Skip to content

Trees

Decision trees with models in the leaves.

M5ModelTreeRegressor 

M5ModelTreeRegressor(min_samples_leaf=4, max_depth=None, smoothing=15.0, prune=True, extrapolation=0.1)

M5 model tree: a regression tree with linear models in the leaves.

A regression tree predicts a constant in each leaf, so it approximates a smooth or linear trend by a staircase and needs many leaves to do so. M5 grows a tree that minimizes the variance of the response, as CART does, and then fits a linear regression at every node on the features tested below it. Each linear model is simplified by greedily dropping features while the pessimistic error estimate, (n + 2 v) / (n - v) times the root mean squared residual with v parameters (the M5' form used by Weka), does not grow. Working upward, a subtree is replaced by its node's linear model when the model's estimated error is no larger than the subtree's. Predictions are smoothed along the path from the leaf to the root: at each step up, p <- (n p + k q) / (n + k) with q the node's own model and n the training size of the node below, which removes the jumps between neighbouring leaves. Quinlan and Wang & Witten found model trees far more accurate than regression trees and comparable to neural networks, with much smaller trees; this is the M5' variant of Weka's M5P.

A leaf's linear model is fitted on few rows and can have very large coefficients, so a new point slightly off those rows could get a wild prediction. As in Quinlan's Cubist, predictions are therefore bounded to the range of the training targets, widened by extrapolation times that range.

Prefer it over a regression tree whenever the response varies smoothly with the features; it stays interpretable and extrapolates linearly within each leaf, up to those bounds.

Parameters:

Name Type Description Default
min_samples_leaf int

Smallest leaf of the grown tree (M5P's minNumInstances).

 4
max_depth int or None

Depth limit of the grown tree, before pruning.

 None
smoothing float

Smoothing constant k; 0 disables smoothing.

 15.0
prune bool
True
extrapolation float or None

How far predictions may go beyond the range of the training targets, as a fraction of that range. None removes the bounds (plain M5).

 0.1

Attributes:

Name Type Description
tree_ Tree

The grown (unpruned) tree structure.
is_leaf_ ndarray of bool, shape (n_nodes,)

Nodes that are leaves after pruning.
n_leaves_ int
bounds_ tuple of float

Lower and upper bound on the predictions.
References

J. R. Quinlan. Learning with Continuous Classes. In Proceedings of the 5th Australian Joint Conference on Artificial Intelligence, pages 343-348, 1992.

Y. Wang and I. H. Witten. Inducing Model Trees for Continuous Classes. In Poster Papers of the 9th European Conference on Machine Learning (ECML), 1997.