update sol

Author: giuseppec

exercises/trees-tex/ex_rnw/sol_research_group_cart.Rnw

@@ -1,19 +1,61 @@
-\begin{enumerate}
-\item[1)] The conclusion is incorrect. Since the tree structure is built recursively, the algorithm does not necessarily identify the optimal tree with lowest empirical risk on the training data.
-This lies in the nature of greedy optimization procedures.
-Empirical risk minimization (ERM) is only performed to identify \textit{the next} splitting rule, and not entire sets of subsequent splitting rules.
-
-\item[2)] CART does automatically select features for splitting nodes if they lead to an expected reduction in empirical risk.
-Irrelevant features are therefore more likely to be picked less often for split rules in model construction.
-(Of course, the subject of assessing feature importance is left for the chapter on random forests.) However, one could gain a rough understanding of a feature's relevance by looking at how often it was picked for splitting a node. However, this kind of "split rule selection frequency" does not necessarily relate to a feature variable's contribution to ERM. 
-
-\item[3)] CART can perform automatic feature selection by remembering surrogate splits in an extra step in model construction. Per default, the \texttt{rpart} package retains up to 5 surrogate splits. For each split rule, a surrogate split rule that leads to sorting observations into child nodes in a similar way is retained. These surrogate splits can then be used to "guide" observations through the tree even if they have some missing feature values. Therefore, CART is generally-speaking well-suited to handle missing observations.
-
-\item[4)] The number of possible split points evaluated per feature variable is equal to the number of different values the respective feature has in the training data minus 1, e.g., a numerical or categorical variable with 4 different values in the training data has 3 potential split points. (Actually, for a continuous feature there is an infinite amount of possible split points, but there are just 3 which lead to different results for the training data.) As each feature variable can be used for the split point, one needs to sum over the feature variables in the data set.
-\begin{align*}
-\text{Number of possible split points} = \sumjp \text{({number of different values in the training data}}_j-1) \le 3 \cdot (n-1)
-\end{align*}
-\end{enumerate}
+\begin{itemize}
+  \item \textbf{(1) Holger's conclusion correct?} \textbf{No.}
+
+\begin{itemize}
+
+\item CART is a \textbf{greedy algorithm}: It minimizes risk $\mathcal{R}(\mathcal{N}_p, j, t)$ only for the \textit{current} split, not the entire tree simultaneously.
+
+\item \textbf{No "Lookahead":} It commits to the best immediate split. It misses cases where a sub-optimal split \textit{now} enables much better splits \textit{later}, potentially resulting in a globally worse tree.
+
+\item \textbf{Computational Constraint:} An exhaustive global search is computationally infeasible (too many combinations). CART accepts this greedy shortcut to remain efficient.
+
+\end{itemize}
+
+  \item \textbf{(2) Automatic feature selection + relevance assessment.}
+  \begin{itemize}
+    \item At each node, CART tests all features $x_j$ and thresholds $t$ and picks the split with largest impurity/risk reduction.
+    \item Features never selected for any split do not appear in the tree $\Rightarrow$ effectively excluded; relevant features tend to be chosen more often (and earlier in the tree) since they yield larger risk reductions.
+    \item Feature relevance proxies after training: (i) used vs.\ unused features (split-frequency, i.e., how often a feature is used, is only a rough proxy and can be misleading), (ii) better: total impurity/risk decrease (often node-size weighted), (iii) permutation importance (validate performance drop when shuffling $x_j$, see random forest later)
+  \end{itemize}
+
+  \item \textbf{(3) Missing values at prediction?} \textbf{Often yes (implementation-dependent).}
+  \begin{itemize}
+    \item Classic CART can use \emph{surrogate splits}: if primary split feature is missing, route via an alternative split that best mimics the primary split on training data.
+    \item Some implementations learn explicit NA-routing (missing $\to$ left/right) or require imputation; surrogate lists are often stored per node (sometimes with a small default maximum number, e.g., rpart uses 5).
+  \end{itemize}
+
+  \item \textbf{(4) Upper bound on \# split points at the root.}
+  \begin{itemize}
+    \item For numeric $x_j$: although thresholds are infinite, only splits between consecutive distinct training values change the partition:
+    \[
+      \#\text{candidates for }x_j = \#\text{unique}(x_j)-1 \le n-1.
+    \]
+    \item With $p=3$ numeric features:
+    \[
+      \#\text{root candidates} \le \sum_{j=1}^{3}(\#\text{unique}(x_j)-1)\le 3(n-1).
+    \]
+    \item (If later considering \emph{unordered categorical} predictors with $k$ levels: in principle up to $2^{k-1}-1$ binary partitions, though implementations may restrict this.)
+  \end{itemize}
+\end{itemize}
+
+% 
+% 
+% \begin{enumerate}
+% \item[1)] The conclusion is incorrect. Since the tree structure is built recursively, the algorithm does not necessarily identify the optimal tree with lowest empirical risk on the training data.
+% This lies in the nature of greedy optimization procedures.
+% Empirical risk minimization (ERM) is only performed to identify \textit{the next} splitting rule, and not entire sets of subsequent splitting rules.
+% 
+% \item[2)] CART does automatically select features for splitting nodes if they lead to an expected reduction in empirical risk.
+% Irrelevant features are therefore more likely to be picked less often for split rules in model construction.
+% (Of course, the subject of assessing feature importance is left for the chapter on random forests.) However, one could gain a rough understanding of a feature's relevance by looking at how often it was picked for splitting a node. However, this kind of "split rule selection frequency" does not necessarily relate to a feature variable's contribution to ERM. 
+% 
+% \item[3)] CART can perform automatic feature selection by remembering surrogate splits in an extra step in model construction. Per default, the \texttt{rpart} package retains up to 5 surrogate splits. For each split rule, a surrogate split rule that leads to sorting observations into child nodes in a similar way is retained. These surrogate splits can then be used to "guide" observations through the tree even if they have some missing feature values. Therefore, CART is generally-speaking well-suited to handle missing observations.
+% 
+% \item[4)] The number of possible split points evaluated per feature variable is equal to the number of different values the respective feature has in the training data minus 1, e.g., a numerical or categorical variable with 4 different values in the training data has 3 potential split points. (Actually, for a continuous feature there is an infinite amount of possible split points, but there are just 3 which lead to different results for the training data.) As each feature variable can be used for the split point, one needs to sum over the feature variables in the data set.
+% \begin{align*}
+% \text{Number of possible split points} = \sumjp \text{({number of different values in the training data}}_j-1) \le 3 \cdot (n-1)
+% \end{align*}
+% \end{enumerate}