Naive Bayes¶

Averaged One-Dependence Estimators.

Naive Bayes assumes all features are independent given the class. AODE instead averages over all models in which one feature (the super-parent) is a parent of every other feature::

P(y, x) ~ sum_i  P(y, x_i) prod_{j != i} P(x_j | y, x_i)

Only super-parents whose value was seen at least min_count times in training take part. When none qualifies the prediction falls back to naive Bayes. There is no model selection and training is a single pass over the data.

AODE works on discrete features. Set n_bins to discretize continuous features into quantile bins; with n_bins=None every distinct value of a feature is treated as a category. Values not seen in training are mapped to an extra, empty category.

Parameters:

Attributes:

Averaged two-dependence estimators (A2DE).

AODE averages over the models in which one feature is a parent of all the others. A2DE takes the next step of the AnDE family: it averages over all models in which a pair of features are parents of all the others,

P(y, x) ~ sum_{i < j}  P(y, x_i, x_j) prod_{k != i, j} P(x_k | y, x_i, x_j),

using only parent pairs whose value combination was seen at least min_count times. Like AODE it needs no structure search and no model selection, and its tables are plain frequency counts. Webb et al. showed that A2DE has lower bias than AODE and, given enough data, lower error: it beat AODE, naive Bayes, TAN and KDB in zero-one loss on most of 62 UCI data sets, above all the larger ones.

Prefer A2DE over :class:AODEClassifier on data sets with thousands of rows; on small ones its larger tables gain less. Prediction costs one pass over the training data per parent pair, so it grows with the square of the number of features.

A2DE works on discrete features. Set n_bins to discretize continuous features into quantile bins; with n_bins=None every distinct value is a category. Values not seen in training map to an extra, empty category. Rows with no usable parent pair fall back to naive Bayes, and with fewer than two features A2DE is naive Bayes.

Parameters:

Attributes:

Tree-Augmented Naive Bayes.

Naive Bayes treats all features as independent given the class. TAN lets every feature depend on the class and on at most one other feature, and learns which one: it builds the maximum-weight spanning tree over the features, weighting each pair by its conditional mutual information I(X_i; X_j | Y) (Chow & Liu, 1968), and roots the tree at the first feature. This is the best tree-shaped Bayesian network in likelihood among all networks where the class is a parent of every feature, and it is found in time quadratic in the number of features with no search.

Prefer TAN over naive Bayes when features are correlated within classes. :class:AODEClassifier averages over all one-parent structures instead of learning one; TAN is cheaper to predict with and gives one readable dependency tree (parents_).

TAN works on discrete features. Set n_bins to discretize continuous features into quantile bins; with n_bins=None every distinct value is a category. Values not seen in training map to an extra, empty category.

Parameters:

Attributes:

k-dependence Bayesian classifier (KDB).

Naive Bayes lets every feature depend on the class only. KDB lets each feature depend on the class and on up to k other features, which gives a dial between naive Bayes (k=0) and the full Bayesian network. The structure is found greedily without search: features are ordered by their mutual information with the class, I(X_i; Y), and each feature takes as parents the k features earlier in that order with the highest conditional mutual information I(X_i; X_j | Y). Sahami showed that going from k=0 to k=1 or 2 reduces the error of naive Bayes on most UCI data sets, and Webb et al. (2012) found KDB with k=1 or 2 among the most accurate Bayesian network classifiers on large data.

Prefer KDB over naive Bayes and :class:TANClassifier when there is enough data to estimate the larger tables: with k=2 every feature has a table over the class and two parents. :class:AODEClassifier is the safer choice on small data sets.

KDB works on discrete features. Set n_bins to discretize continuous features into quantile bins; with n_bins=None every distinct value is a category. Values not seen in training map to an extra, empty category.

Parameters:

Attributes:

Hidden naive Bayes (HNB).

Naive Bayes treats the features as independent given the class. HNB gives every feature a hidden parent that blends the influence of all other features: the conditional probability of feature i is a mixture of one-dependence estimates,

P(x_i | hp_i, y) = sum_{j != i} w_ij P(x_i | x_j, y),

with weights proportional to the conditional mutual information I(X_i; X_j | Y), so strongly related features count most. There is no structure search, every table has only two dimensions besides the class, and training is a single pass plus the mutual-information matrix. Zhang, Jiang and Su found HNB significantly more accurate than naive Bayes, TAN and AODE on 36 UCI data sets.

Prefer HNB over naive Bayes when features are correlated within classes and you want one model, not an average, with the same table sizes as :class:AODEClassifier. With one feature it is naive Bayes.

HNB works on discrete features. Set n_bins to discretize continuous features into quantile bins; with n_bins=None every distinct value is a category. Values not seen in training map to an extra, empty category.

Parameters:

Attributes:

Naive Bayes with a kernel density estimate per feature and class.

Gaussian naive Bayes models each feature within each class by a single normal distribution, which fails for skewed, multimodal or discrete features. Flexible naive Bayes keeps the naive independence assumption but estimates each class-conditional density nonparametrically, as an average of Gaussian kernels centred on the training values. John and Langley proved this is consistent for any continuous density and found it more accurate than Gaussian naive Bayes on most of the UCI data sets they tried, and never much worse.

Features are standardized with the training mean and standard deviation, and the kernel width of class c is 1 / sqrt(n_c) on that scale as in the paper (bandwidth="john-langley"), or Silverman's rule per feature and class (bandwidth="silverman").

Prefer it over :class:~sklearn.naive_bayes.GaussianNB when features are far from normal. Prediction costs one pass over the training data per query, like a nearest-neighbour method.

Parameters:

Attributes: