Discriminant analysis¶

Variants of linear and quadratic discriminant analysis.

RegularizedDiscriminantAnalysis ¶

RegularizedDiscriminantAnalysis(lam=0.5, gamma=0.0, priors=None, tol=1e-10)

Regularized Discriminant Analysis.

Each class covariance is shrunk toward the pooled covariance (lam) and the result toward a scaled identity (gamma). lam=0, gamma=0 is QDA, lam=1, gamma=0 is LDA, and values in between often beat both when the sample is small relative to the number of features.

Following Friedman (1989), with scatter matrices S_k and S = sum S_k::

Sigma_k(lam)        = ((1 - lam) S_k + lam S) / ((1 - lam) n_k + lam n)
Sigma_k(lam, gamma) = (1 - gamma) Sigma_k(lam) + gamma tr(Sigma_k(lam)) / p I

Parameters:

Name	Type	Description	Default
`lam`	`float in [0, 1]`	Shrinkage of the class covariances toward the pooled covariance.	`0.5`
`gamma`	`float in [0, 1]`	Shrinkage toward a multiple of the identity.	`0.0`
`priors`	`array-like of shape (n_classes,)`	Class priors. Estimated from the training data when None.	`None`
`tol`	`float`	Eigenvalues below `tol` times the largest eigenvalue are clipped, which keeps the model defined when a covariance is singular.	`1e-10`

Attributes:

Name	Type	Description
`classes_`	`ndarray of shape (n_classes,)`
`priors_`	`ndarray of shape (n_classes,)`
`means_`	`ndarray of shape (n_classes, n_features)`
`covariances_`	`ndarray of shape (n_classes, n_features, n_features)`	Regularized covariance of each class.

References

J. H. Friedman. Regularized Discriminant Analysis. Journal of the American Statistical Association, 84(405):165-175, 1989.

MixtureDiscriminantAnalysis ¶

MixtureDiscriminantAnalysis(n_subclasses=3, max_iter=100, tol=1e-05, reg_covar=1e-06, random_state=None)

Mixture Discriminant Analysis.

LDA models each class as one Gaussian, all sharing a covariance matrix, so its decision boundaries are linear and a class made of several clusters is summarized by a single centroid. MDA models each class as a mixture of n_subclasses Gaussians, still with one covariance matrix shared by every subclass of every class, and fits the mixtures jointly by EM. Boundaries become piecewise smooth while the number of covariance parameters stays that of LDA, so MDA does not overfit like QDA does on small samples. n_subclasses=1 is LDA.

Prefer MDA over LDA when a class is multimodal or non-convex, for example a class that is the union of several groups. Tune n_subclasses by cross-validation; Hastie & Tibshirani used 2 to 5.

EM starts from k-means within each class (the paper's initialization) and stops when the log-likelihood improves by less than tol relative to its size.

Parameters:

Name	Type	Description	Default
`n_subclasses`	`int`	Mixture components per class, capped at the class size.	`3`
`max_iter`	`int`	Maximum EM iterations.	`100`
`tol`	`float`	Relative log-likelihood tolerance for stopping EM.	`1e-5`
`reg_covar`	`float`	Added to the diagonal of the shared covariance, relative to its mean variance, to keep it invertible.	`1e-6`
`random_state`	`int, RandomState instance or None`	Seeds the k-means initialization.	`None`

Attributes:

Name	Type	Description
`classes_`	`ndarray of shape (n_classes,)`
`means_`	`list of ndarray of shape (n_subclasses_k, n_features)`	Subclass centroids of each class.
`weights_`	`list of ndarray of shape (n_subclasses_k,)`	Mixing proportions within each class.
`covariance_`	`ndarray of shape (n_features, n_features)`	The shared covariance matrix.
`priors_`	`ndarray of shape (n_classes,)`
`n_iter_`	`int`

References

T. Hastie and R. Tibshirani. Discriminant Analysis by Gaussian Mixtures. Journal of the Royal Statistical Society, Series B, 58(1):155-176, 1996.