Main Content

Classification Ensembles

Boosting, random forest, bagging, random subspace, and ECOC ensembles for multiclass learning

A classification ensemble is a predictive model composed of a weighted combination of multiple classification models. In general, combining multiple classification models increases predictive performance.

To explore classification ensembles interactively, use the Classification Learner app. For greater flexibility, use fitcensemble in the command-line interface to boost or bag classification trees, or to grow a random forest [12]. For details on all supported ensembles, see Ensemble Algorithms. To reduce a multiclass problem into an ensemble of binary classification problems, train an error-correcting output codes (ECOC) model. For details, see fitcecoc.

To boost regression trees using LSBoost, or to grow a random forest of regression trees[12], see Regression Ensembles.

Apps

Classification LearnerTrain models to classify data using supervised machine learning

Blocks

ClassificationEnsemble PredictClassify observations using ensemble of decision trees

Functions

expand all

templateDiscriminantDiscriminant analysis classifier template
templateECOCError-correcting output codes learner template
templateEnsembleEnsemble learning template
templateKNNk-nearest neighbor classifier template
templateLinearLinear classification learner template
templateNaiveBayesNaive Bayes classifier template
templateSVMSupport vector machine template
templateTreeCreate decision tree template

Create Classification Ensemble

fitcensembleFit ensemble of learners for classification
compactCompact classification ensemble

Modify Classification Ensemble

resumeResume training ensemble
removeLearnersRemove members of compact classification ensemble

Interpret Classification Ensemble

limeLocal interpretable model-agnostic explanations (LIME)
partialDependenceCompute partial dependence
plotPartialDependenceCreate partial dependence plot (PDP) and individual conditional expectation (ICE) plots
predictorImportanceEstimates of predictor importance for classification ensemble of decision trees
shapleyShapley values

Cross-Validate Classification Ensemble

crossvalCross-validate ensemble
kfoldEdgeClassification edge for cross-validated classification model
kfoldLossClassification loss for cross-validated classification model
kfoldMarginClassification margins for cross-validated classification model
kfoldPredictClassify observations in cross-validated classification model
kfoldfunCross-validate function for classification

Measure Performance

lossClassification error
resubLossClassification error by resubstitution
compareHoldoutCompare accuracies of two classification models using new data
edgeClassification edge
marginClassification margins
resubEdgeClassification edge by resubstitution
resubMarginClassification margins by resubstitution
testckfoldCompare accuracies of two classification models by repeated cross-validation

Classify Observations

predictClassify observations using ensemble of classification models
resubPredictClassify observations in ensemble of classification models
oobPredictPredict out-of-bag response of ensemble

Gather Properties of Classification Ensemble

gatherGather properties of Statistics and Machine Learning Toolbox object from GPU
TreeBaggerCreate bag of decision trees
fitcensembleFit ensemble of learners for classification
predictPredict responses using ensemble of bagged decision trees
oobPredictEnsemble predictions for out-of-bag observations

Create ECOC

fitcecocFit multiclass models for support vector machines or other classifiers
compactReduce size of multiclass error-correcting output codes (ECOC) model

Modify ECOC

discardSupportVectorsDiscard support vectors of linear SVM binary learners in ECOC model

Interpret ECOC

limeLocal interpretable model-agnostic explanations (LIME)
partialDependenceCompute partial dependence
plotPartialDependenceCreate partial dependence plot (PDP) and individual conditional expectation (ICE) plots
shapleyShapley values

Cross-Validate ECOC

crossvalCross-validate multiclass error-correcting output codes (ECOC) model
kfoldEdgeClassification edge for cross-validated ECOC model
kfoldLossClassification loss for cross-validated ECOC model
kfoldMarginClassification margins for cross-validated ECOC model
kfoldPredictClassify observations in cross-validated ECOC model
kfoldfunCross-validate function using cross-validated ECOC model

Measure Performance

lossClassification loss for multiclass error-correcting output codes (ECOC) model
resubLossResubstitution classification loss for multiclass error-correcting output codes (ECOC) model
compareHoldoutCompare accuracies of two classification models using new data
edgeClassification edge for multiclass error-correcting output codes (ECOC) model
marginClassification margins for multiclass error-correcting output codes (ECOC) model
resubEdgeResubstitution classification edge for multiclass error-correcting output codes (ECOC) model
resubMarginResubstitution classification margins for multiclass error-correcting output codes (ECOC) model
testckfoldCompare accuracies of two classification models by repeated cross-validation

Classify Observations

predictClassify observations using multiclass error-correcting output codes (ECOC) model
resubPredictClassify observations in multiclass error-correcting output codes (ECOC) model

Gather Properties of ECOC

gatherGather properties of Statistics and Machine Learning Toolbox object from GPU

Classes

expand all

ClassificationEnsembleEnsemble classifier
CompactClassificationEnsembleCompact classification ensemble class
ClassificationPartitionedEnsembleCross-validated classification ensemble
TreeBaggerBag of decision trees
CompactTreeBaggerCompact ensemble of decision trees grown by bootstrap aggregation
ClassificationBaggedEnsembleClassification ensemble grown by resampling
ClassificationECOCMulticlass model for support vector machines (SVMs) and other classifiers
CompactClassificationECOCCompact multiclass model for support vector machines (SVMs) and other classifiers
ClassificationPartitionedECOCCross-validated multiclass ECOC model for support vector machines (SVMs) and other classifiers

Topics

Train Ensemble Classifiers Using Classification Learner App

Create and compare ensemble classifiers, and export trained models to make predictions for new data.

Framework for Ensemble Learning

Obtain highly accurate predictions by using many weak learners.

Ensemble Algorithms

Learn about different algorithms for ensemble learning.

Train Classification Ensemble

Train a simple classification ensemble.

Test Ensemble Quality

Learn methods to evaluate the predictive quality of an ensemble.

Handle Imbalanced Data or Unequal Misclassification Costs in Classification Ensembles

Learn how to set prior class probabilities and misclassification costs.

Classification with Imbalanced Data

Use the RUSBoost algorithm for classification when one or more classes are over-represented in your data.

LPBoost and TotalBoost for Small Ensembles

Create small ensembles by using the LPBoost and TotalBoost algorithms. (LPBoost and TotalBoost require Optimization Toolbox™.)

Tune RobustBoost

Tune RobustBoost parameters for better predictive accuracy. (RobustBoost requires Optimization Toolbox.)

Surrogate Splits

Gain better predictions when you have missing data by using surrogate splits.

Train Classification Ensemble in Parallel

Train a bagged ensemble in parallel reproducibly.

Bootstrap Aggregation (Bagging) of Classification Trees Using TreeBagger

Create a TreeBagger ensemble for classification.

Credit Rating by Bagging Decision Trees

This example shows how to build an automated credit rating tool.

Random Subspace Classification

Increase the accuracy of classification by using a random subspace ensemble.

Predict Class Labels Using ClassificationEnsemble Predict Block

Train a classification ensemble model with optimal hyperparameters, and then use the ClassificationEnsemble Predict block for label prediction.