Random naive Bayes
Encyclopedia
Random naive Bayes extends the Naive Bayes classifier
by adopting the random forest
principles: random input selection, bagging (i.e. bootstrap aggregating
), and random feature selection .
is a probabilistic classifier simplifying Bayes' theorem
by naively assuming class conditional independence. Although this assumption leads to biased posterior probabilities, the ordered probabilities of Naive Bayes result in a classification performance comparable to that of classification trees and neural networks. Notwithstanding Naive Bayes' popularity due to its simplicity combined with high accuracy and speed, its conditional independence assumption rarely holds. There are mainly two approaches to alleviate this naivety:
, Random Naive Bayes builds an ensemble, but unlike AODE
, the ensemble combines zero-dependence classifiers.
, the importance of each feature is estimated on the out-of-bag (oob) data.
Naive Bayes classifier
A naive Bayes classifier is a simple probabilistic classifier based on applying Bayes' theorem with strong independence assumptions...
by adopting the random forest
Random forest
Random forest is an ensemble classifier that consists of many decision trees and outputs the class that is the mode of the class's output by individual trees. The algorithm for inducing a random forest was developed by Leo Breiman and Adele Cutler, and "Random Forests" is their trademark...
principles: random input selection, bagging (i.e. bootstrap aggregating
Bootstrap aggregating
Bootstrap aggregating is a machine learning ensemble meta-algorithm to improve machine learning of statistical classification and regression models in terms of stability and classification accuracy. It also reduces variance and helps to avoid overfitting. Although it is usually applied to decision...
), and random feature selection .
Naive Bayes classifier
The naive Bayes classifierNaive Bayes classifier
A naive Bayes classifier is a simple probabilistic classifier based on applying Bayes' theorem with strong independence assumptions...
is a probabilistic classifier simplifying Bayes' theorem
Bayes' theorem
In probability theory and applications, Bayes' theorem relates the conditional probabilities P and P. It is commonly used in science and engineering. The theorem is named for Thomas Bayes ....
by naively assuming class conditional independence. Although this assumption leads to biased posterior probabilities, the ordered probabilities of Naive Bayes result in a classification performance comparable to that of classification trees and neural networks. Notwithstanding Naive Bayes' popularity due to its simplicity combined with high accuracy and speed, its conditional independence assumption rarely holds. There are mainly two approaches to alleviate this naivety:
- Selecting attribute subsets in which attributes are conditionally independent (cf. Selective Bayesian Classifier ).
- Extending the structure of Naive Bayes to represent attribute dependencies (cf. Averaged One-Dependence Estimators (AODE)AODEAveraged one-dependence estimators is a probabilistic classification learning technique. It was developed to address the attribute-independence problem of the popular naive Bayes classifier...
).
Random naive Bayes' alleviation of the class conditional independence assumption
Random Naive Bayes adopts the first approach by randomly selecting a subset of attributes in which attributes are assumed to be conditionally independent. Naive Bayes' performance might benefit from this random feature selection. Analogous to AODEAODE
Averaged one-dependence estimators is a probabilistic classification learning technique. It was developed to address the attribute-independence problem of the popular naive Bayes classifier...
, Random Naive Bayes builds an ensemble, but unlike AODE
AODE
Averaged one-dependence estimators is a probabilistic classification learning technique. It was developed to address the attribute-independence problem of the popular naive Bayes classifier...
, the ensemble combines zero-dependence classifiers.
Random naive Bayes and random forest
Generalizing Random Forest to Naive Bayes, Random Naive Bayes (Random NB), is a bagged classifier combining a forest of B Naive Bayes. Each bth Naive Bayes is estimated on a bootstrap sample Sb with m randomly selected features. To classify an observation put the input vector down the B Naive Bayes in the forest. Each Naive Bayes generates posterior class probabilities. Unlike Random Forest, the predicted class of the ensemble is assessed by adjusted majority voting rather than majority voting, as each bth Naive Bayes delivers continuous posterior probabilities. Similar to Random ForestsRandom forest
Random forest is an ensemble classifier that consists of many decision trees and outputs the class that is the mode of the class's output by individual trees. The algorithm for inducing a random forest was developed by Leo Breiman and Adele Cutler, and "Random Forests" is their trademark...
, the importance of each feature is estimated on the out-of-bag (oob) data.
See also
- AODEAODEAveraged one-dependence estimators is a probabilistic classification learning technique. It was developed to address the attribute-independence problem of the popular naive Bayes classifier...
- Naive Bayes classifierNaive Bayes classifierA naive Bayes classifier is a simple probabilistic classifier based on applying Bayes' theorem with strong independence assumptions...
- Random forestRandom forestRandom forest is an ensemble classifier that consists of many decision trees and outputs the class that is the mode of the class's output by individual trees. The algorithm for inducing a random forest was developed by Leo Breiman and Adele Cutler, and "Random Forests" is their trademark...
- Random multinomial logitRandom multinomial logitIn statistics and machine learning, random multinomial logit is a technique for statistical classification using repeated multinomial logit analyses via Leo Breiman's random forests.-Rationale for the new method:...