Linear models¶

Binary logistic regression with Firth's bias-reducing penalty.

Firth's penalty (a Jeffreys prior) removes the first-order small-sample bias of the maximum likelihood estimate and gives finite coefficients even under complete separation, where ordinary logistic regression diverges.

Firth's estimate pulls predicted probabilities toward 0.5, which hurts calibration when events are rare. Puhr et al. (2017) proposed two fixes:

• correction="flic" keeps Firth's slopes and re-estimates the intercept by maximum likelihood, so the average predicted probability equals the observed event rate.
• correction="flac" refits by maximum likelihood on the data augmented with pseudo-observations weighted by the hat values, plus an indicator for the pseudo-data.

In aspective's benchmark (bench/RESULTS.md, 23 PMLB binary datasets) Firth is non-inferior to maximum likelihood in Brier score. With only 100 training rows neither Firth nor FLIC/FLAC was shown to be better or worse than maximum likelihood, so the corrections are offered for calibration under rare events, as studied by Puhr et al., not as a general improvement.

Parameters:

Attributes:

Binary logistic regression with the Gelman et al. default prior.

Gelman, Jakulin, Pittau and Su proposed a default prior for logistic regression that every analysis can use unchanged: put inputs on a common scale, then give each coefficient an independent Cauchy prior with centre 0 and scale 2.5 (the intercept gets scale 10). The prior says a change of one unit on the common scale rarely moves the probability from 0.01 to 0.5, so it is weak, but its heavy tails leave large effects alone. Estimates are the posterior mode, which is always finite, also under separation, where maximum likelihood diverges and ridge shrinks every coefficient by the same rule.

With scale_inputs=True (the paper's recommendation) inputs with more than two distinct values are centred and scaled to standard deviation 0.5, and binary inputs are centred, so that coefficients of both kinds are comparable. coef_ is reported on the original scale. Setting prior_df to another value gives the Student-t family the paper also discusses; prior_df=np.inf is a Gaussian prior.

Prefer it over maximum-likelihood or ridge logistic regression for small or sparse data, separation, and when you want a default with no penalty to tune. :class:FirthLogisticRegression is the alternative that targets bias of the coefficients rather than a prior belief.

Parameters:

Attributes:

Relaxed Lasso with fixed penalties.

The lasso uses one penalty both to select variables and to shrink them, so a penalty strong enough to drop noise variables also over-shrinks the true ones. The relaxed lasso separates the two: Lasso(alpha) selects the support, then the selected variables are refitted with the weaker penalty phi * alpha. phi=1 is the lasso and phi=0 is least squares on the lasso support.

Use :class:RelaxedLassoCV to choose alpha and phi by cross-validation.

Parameters:

Attributes:

Relaxed Lasso with alpha and phi chosen by cross-validation.

Parameters:

Attributes:

Adaptive Lasso with a fixed penalty.

The lasso penalizes every coefficient equally, so to drop the noise variables it must also shrink the large, true effects, and it selects the right variables only under a restrictive irrepresentable condition. The adaptive lasso uses a weighted penalty alpha * sum_j w_j |beta_j| with w_j = 1 / |beta_init_j| ** gamma from an initial root-n consistent fit. Variables that look large are barely penalized, small ones heavily. Zou proved that it has the oracle property: it selects the true variables with probability tending to one and estimates them as efficiently as if the true model were known. It is solved as a lasso on rescaled inputs.

Use :class:AdaptiveLassoCV to choose alpha and gamma.

Parameters:

Attributes:

Adaptive Lasso with alpha and gamma chosen by cross-validation.

For every gamma the weights come from the initial fit on the full training data, as in Zou (2006); then the lasso path on the rescaled inputs is cross-validated with the same folds for every gamma, and the pair with the lowest error is refitted on all the data.

Parameters:

Attributes:

Bootstrap-intersected Lasso with a least-squares refit.

The lasso only selects the true variables consistently under the strict irrepresentable condition. Bach (2008) showed that running the lasso on bootstrap replicates and keeping the variables selected in all of them gives consistent selection under much weaker conditions. The selected set is then refitted by ordinary least squares.

threshold=1 is the strict intersection of the paper; a value like 0.9 is the "soft" Bolasso-S variant, which is more robust in practice.

Parameters:

Attributes:

Componentwise linear L2 boosting with the number of steps chosen by CV.

Gradient boosting with squared loss where each base learner is a simple linear regression on one feature: at every step the feature that best fits the current residuals is chosen and its coefficient moves a small step learning_rate toward the least-squares fit. Stopping early shrinks and selects coefficients much like the lasso, while the coefficient paths are smoother and correlated features enter more gradually. Buehlmann (2006) proved consistency for high-dimensional sparse linear models with many more features than samples. The number of steps is the only tuning parameter and is chosen here by K-fold cross-validation over the whole path, which is computed incrementally in O(n_features) per step after one Gram matrix.

This is the default linear model of the R package mboost (Buehlmann & Hothorn, 2007). Prefer it over the lasso when features are strongly correlated, when you want a sparse model whose coefficients are not over-shrunk at a single penalty, or as a fast, interpretable baseline.

Parameters:

Attributes:

Multivariate adaptive regression splines (MARS).

A linear model in a basis of hinge functions max(0, x_j - t) and max(0, t - x_j) (and, with max_degree > 1, their products) that are chosen from the data. A forward pass adds, at each step, the reflected pair of hinges (on a variable, at a knot, multiplying an existing term) that most reduces the residual sum of squares; a backward pass then removes terms one by one and keeps the subset with the best generalized cross-validation score

GCV = RSS / n / (1 - C(M) / n)^2,   C(M) = M + penalty (M - 1) / 2.

MARS keeps what linear regression and regression trees each do well: it is continuous and piecewise linear, picks variables and interactions automatically, and extrapolates linearly instead of by a constant. Friedman found it more accurate than recursive partitioning on smooth functions in moderate dimension; with max_degree=1 it fits an additive model, the default of the R package earth.

Candidate knots are n_knots interior quantiles of each feature (Friedman's minspan/endspan rules play the same role: they keep knots away from the extremes and from each other). The forward search updates the projections of every candidate onto the current basis incrementally, so a step costs one pass over the candidate hinges.

Parameters:

Attributes:

Additive model with penalized B-splines (P-splines), smoothness chosen by GCV.

Linear regression fits one slope per feature. A generalized additive model (Hastie & Tibshirani, 1986) replaces each slope by a smooth function, y = b0 + f_1(x_1) + ... + f_p(x_p), which keeps the model additive and readable (one curve per feature) while capturing nonlinear effects. Following Eilers and Marx, each f_j is a cubic B-spline on n_knots equally spaced knots with a second-order difference penalty on neighbouring coefficients. The penalty shrinks every curve toward a straight line, so the strongest smoothing gives back ordinary linear regression within the training range. The penalty weight is chosen by generalized cross-validation from a log-spaced grid, all at the cost of one eigendecomposition.

Features with fewer than min_unique distinct values enter linearly. Outside the training range each curve stays at its value at the boundary: a linearly extrapolated spline can reach values far outside anything seen in training on a single unusual point. One smoothing parameter is shared by all features (on standardized curves), as in the mgcv "one smoothing parameter" setting; use it for moderate numbers of features.

Parameters:

Attributes: