Ensembles¶

Rotation Forest: decision trees on PCA-rotated feature subsets.

Bagging and random forests make trees diverse by resampling rows and features, which costs each tree some accuracy. Rotation Forest instead gives every tree the full training set in a different coordinate system: the features are split at random into groups of n_features_per_subset; for each group, principal components are computed on a bootstrap sample of sample_fraction of the rows of a random non-empty subset of the classes; and all components of all groups together form a block-diagonal rotation of the feature space. Axis-parallel splits in rotated coordinates are oblique splits in the original ones, so the trees stay accurate yet differ. Rodriguez et al. found it significantly more accurate than bagging, AdaBoost and random forests on 33 UCI data sets, and Bagnall et al. (2018) confirmed it as the strongest tree ensemble on 121 UCI data sets with continuous features.

Features are standardized before rotation. Prefer Rotation Forest over a random forest for continuous features; it is slower to train, and with mainly categorical features its advantage shrinks.

Parameters:

Attributes:

LogitBoost: additive logistic regression by Newton steps.

Friedman, Hastie and Tibshirani showed that AdaBoost fits an additive model by stagewise minimization of the exponential loss. LogitBoost fits the same kind of additive model to the multinomial log-likelihood instead, by quasi-Newton steps: at every round and for every class it fits a regression tree by weighted least squares to the working response z = (y* - p) / (p (1 - p)) with weights p (1 - p), and adds the centred fits to the class scores. The log-likelihood grows only linearly in the margin of a badly misclassified point, where the exponential loss grows exponentially, so LogitBoost is less sensitive to label noise; and the class scores are calibrated log-odds. The paper found it at least as accurate as AdaBoost on UCI benchmarks, and better for multiclass problems.

Prefer LogitBoost over AdaBoost when labels may be noisy or you need class probabilities. Working responses are capped at z_max for stability, as the paper recommends.

Parameters:

Attributes:

MultiBoost: AdaBoost committees combined by wagging.

AdaBoost mainly reduces bias; bagging and its continuous variant, wagging (random instance weights), mainly reduce variance. MultiBoost gets both: it runs AdaBoost, but about every sqrt(n_estimators) rounds it ends the current sub-committee and restarts from fresh random weights drawn from a continuous Poisson distribution. Restarts also happen when a member is perfect (error 0) or no better than chance, where AdaBoost would have to stop. All members vote with their AdaBoost weights. Webb found MultiBoost more accurate than both AdaBoost and bagging on most of 36 UCI data sets, at the same cost as AdaBoost and with sub-committees that can be trained in parallel.

The weights follow the multiclass SAMME rule (Zhu et al., 2009), so with a single committee and no restarts MultiBoost is :class:~sklearn.ensemble.AdaBoostClassifier.

Parameters:

Attributes:

decision_function(X)

Random forest weights with a local ridge regression at each query.

A random forest prediction is a weighted average of training responses, where the weight of training point i for query x is how often, across trees, i shares a leaf with x (divided by the leaf size). Averaging cannot follow a slope: forests are biased near the edges of the data and wherever the signal is smooth but not flat. Local linear forests keep the forest's adaptive weights but fit, at each query, a weighted linear regression with a ridge penalty on the slopes,

min  sum_i alpha_i(x) (y_i - mu - (x_i - x)' theta)^2 + lambda ||theta||^2,

and predict mu. Friedberg et al. proved asymptotic normality and found lower error than random forests, BART and gradient boosting on smooth signals and a range of real data sets.

The forest is scikit-learn's random forest with leaves of at least min_samples_leaf points; every training point, not only the in-bag ones, contributes to the weights. Inputs are standardized for the ridge penalty, and the weights sum to one, so ll_lambda is on the scale of a local variance. Prediction costs one small weighted regression per query.

Parameters:

Attributes: