Nearest neighbours¶

Nearest-neighbour methods that adapt the neighbourhood or the local fit.

DANNClassifier ¶

DANNClassifier(n_neighbors=5, n_neighbors_metric=None, epsilon=1.0, n_iter=1)

Discriminant adaptive nearest-neighbour classification (DANN).

k-nearest-neighbour classification uses the same round neighbourhood everywhere. Near a class boundary that wastes neighbours on directions in which the class does not change. DANN adapts the metric to each query point: it takes the n_neighbors_metric nearest training points, computes their pooled within-class covariance W and between-class covariance B, and measures distance with

Sigma = W^-1/2 (W^-1/2 B W^-1/2 + epsilon I) W^-1/2,

which stretches the neighbourhood along the local decision boundary and shrinks it across. The query is then classified by a majority vote of its n_neighbors nearest points in that metric. This is a local LDA: epsilon rounds the neighbourhood so it never becomes an infinite strip. Hastie and Tibshirani found it beat k-NN, CART and LDA on most simulated and real problems they tried.

Inputs are standardized before distances are taken. Prefer DANN over k-NN when only some directions matter locally; it costs one small eigendecomposition and a pass over the training data per query.

Parameters:

Name	Type	Description	Default
`n_neighbors`	`int`	Neighbours that vote, in the adapted metric.	`5`
`n_neighbors_metric`	`int or None`	Neighbourhood used to estimate the local metric; None uses `max(n_samples // 5, 50)`, the paper's choice, capped at n.	`None`
`epsilon`	`float`	Softening of the metric; the paper recommends 1.	`1.0`
`n_iter`	`int`	Times the metric is re-estimated in the neighbourhood it defines.	`1`

Attributes:

Name	Type	Description
`classes_`	`ndarray of shape (n_classes,)`
`n_neighbors_metric_`	`int`

References

T. Hastie and R. Tibshirani. Discriminant Adaptive Nearest Neighbor Classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(6):607-616, 1996.

LoessRegressor ¶

LoessRegressor(span=0.3, degree=1, min_neighbors=None, ridge=1e-06)

Locally weighted linear or quadratic regression (LOESS).

k-nearest-neighbour regression predicts the plain average of the k closest training responses: it is biased wherever the regression function has a slope, above all near the edges of the data, and its prediction surface is a step function. LOESS fits, at every query point, a weighted least-squares line (degree=1) or quadratic (degree=2) to the nearest span fraction of the training data, with tricube weights (1 - (d / d_max) ** 3) ** 3 that fall smoothly to zero at the farthest neighbour, and predicts its value at the query. The local fit removes the first-order bias of local averaging (Fan, 1993), and the prediction surface is continuous.

Inputs are standardized before distances are taken, as Cleveland & Devlin recommend for predictors on different scales. A tiny ridge term keeps each local fit solvable when a neighbourhood is degenerate. The robustness iterations of Cleveland (1979) are not included; this is R's loess(family = "gaussian").

Prefer LOESS over k-NN regression for smooth signals in a few dimensions; tune span like k. Prediction costs one small weighted regression per query point.

Parameters:

Name	Type	Description	Default
`span`	`float in (0, 1]`	Fraction of the training data in each neighbourhood. At least `min_neighbors` points are always used.	`0.3`
`degree`	`(1, 2)`	Degree of the local polynomial.	`1`
`min_neighbors`	`int or None`	Smallest neighbourhood; None uses twice the local model's number of coefficients.	`None`
`ridge`	`float`	Ridge penalty of each local fit, relative to its weighted scale.	`1e-6`

Attributes:

Name	Type	Description
`n_neighbors_`	`int`	Neighbourhood size used.

References

W. S. Cleveland and S. J. Devlin. Locally Weighted Regression: An Approach to Regression Analysis by Local Fitting. Journal of the American Statistical Association, 83(403):596-610, 1988.

W. S. Cleveland. Robust Locally Weighted Regression and Smoothing Scatterplots. Journal of the American Statistical Association, 74(368):829-836, 1979.