Main Content

average

Compute performance metrics for average receiver operating characteristic (ROC) curve in multiclass problem

Since R2022a

    Description

    [FPR,TPR,Thresholds,AUC] = average(rocObj,type) computes the averages of performance metrics stored in the rocmetrics object rocObj for a multiclass classification problem using the averaging method specified in type. The function returns the average false positive rate (FPR) and the average true positive rate (TPR) for each threshold value in Thresholds. The function also returns AUC, the area under the ROC curve composed of FPR and TPR.

    example

    [avg1,avg2,Thresholds,AUC] = average(rocObj,type,metric1,metric2) computes the performance metrics and returns avg1 (the average of metric1) and avg2 (the average of metric2) in addition to Thresholds, the corresponding threshold for each of the average values, and AUC, the AUC of the curve generated by metric1 and metric2. (since R2024a)

    average supports the AUC output only when metric1 and metric2 are TPR and FPR, or instead are precision and recall:

    • TPR and FPR — Specify TPR using "TruePositiveRate", "tpr", or "recall", and specify FPR using "FalsePositiveRate" or "fpr". These choices specify that AUC is a ROC curve.

    • Precision and recall — Specify precision using "PositivePredictiveValue", "ppv", "prec", or "precision", and specify recall using "TruePositiveRate", "tpr", or "recall". These choices specify that AUC is the area under a precision-recall curve.

    example

    Examples

    collapse all

    Compute the performance metrics for a multiclass classification problem by creating a rocmetrics object, and then compute the average values for the metrics by using the average function. Plot the average ROC curve using the outputs of average.

    Load the fisheriris data set. The matrix meas contains flower measurements for 150 different flowers. The vector species lists the species for each flower. species contains three distinct flower names.

    load fisheriris

    Train a classification tree that classifies observations into one of the three labels. Cross-validate the model using 10-fold cross-validation.

    rng("default") % For reproducibility
    Mdl = fitctree(meas,species,Crossval="on");

    Compute the classification scores for validation-fold observations.

    [~,Scores] = kfoldPredict(Mdl);
    size(Scores)
    ans = 1×2
    
       150     3
    
    

    The output Scores is a matrix of size 150-by-3. The column order of Scores follows the class order in Mdl, stored in Mdl.ClassNames.

    Create a rocmetrics object by using the true labels in species and the classification scores in Scores. Specify the column order of Scores using Mdl.ClassNames.

    rocObj = rocmetrics(species,Scores,Mdl.ClassNames);

    rocmetrics computes the FPR and TPR at different thresholds and finds the AUC value for each class.

    Compute the average performance metric values, including the FPR and TPR at different thresholds using the macro-averaging method.

    [FPR,TPR,Thresholds,AUC] = average(rocObj,"macro");

    Plot the average ROC curve and display the average AUC value..

    plot(rocObj,AverageCurveType="macro",ClassNames=[])

    Figure contains an axes object. The axes object with title ROC Curve, xlabel False Positive Rate, ylabel True Positive Rate contains 3 objects of type roccurve, scatter, line. These objects represent Macro-average (AUC = 0.9788), Macro-average Model Operating Point.

    To display all the ROC curves and AUC values, do not set the ClassNames argument to [].

    plot(rocObj,AverageCurveType="macro")

    Figure contains an axes object. The axes object with title ROC Curve, xlabel False Positive Rate, ylabel True Positive Rate contains 9 objects of type roccurve, scatter, line. These objects represent setosa (AUC = 1), setosa Model Operating Point, versicolor (AUC = 0.9636), versicolor Model Operating Point, virginica (AUC = 0.9636), virginica Model Operating Point, Macro-average (AUC = 0.9788), Macro-average Model Operating Point.

    Load the fisheriris data set. The matrix meas contains flower measurements for 150 different flowers. The vector species lists the species for each flower. species contains three distinct flower names.

    Train a classification tree that classifies observations into one of the three labels.

    load fisheriris
    mdl = fitctree(meas,species);

    Create a rocmetrics object from the classification tree model.

    roc = rocmetrics(mdl,meas,species); % Input data meas and response species required

    Obtain the average macro recall and precision statistics in addition to the threshold and AUC statistics.

    [avgRecall,avgPrec,thresh,AUC] = average(roc,"macro","recall","precision")
    avgRecall = 9×1
    
             0
        0.6533
        0.9533
        0.9800
        0.9933
        0.9933
        1.0000
        1.0000
        1.0000
    
    
    avgPrec = 9×1
    
           NaN
        1.0000
        0.9929
        0.9811
        0.9560
        0.9203
        0.7804
        0.6462
        0.3333
    
    
    thresh = 9×1
    
        1.0000
        1.0000
        0.9565
        0.3333
       -0.3333
       -0.6667
       -0.9565
       -0.9783
       -1.0000
    
    
    AUC = 
    0.9972
    

    Plot the ROC curve for the recall and precision metrics.

    plot(roc,AverageCurveType="macro",XAxisMetric="recall",YAxisMetric="precision")

    Figure contains an axes object. The axes object with title Precision-Recall Curve, xlabel Recall (True Positive Rate), ylabel Precision (Positive Predictive Value) contains 4 objects of type roccurve. These objects represent setosa (PR-AUC = 1), versicolor (PR-AUC = 0.9928), virginica (PR-AUC = 0.9873), Macro-average (PR-AUC = 0.9972).

    Input Arguments

    collapse all

    Object evaluating classification performance, specified as a rocmetrics object.

    Averaging method, specified as "micro", "macro", or "weighted".

    • "micro" (micro-averaging) — average finds the average performance metrics by treating all one-versus-all binary classification problems as one binary classification problem. The function computes the confusion matrix components for the combined binary classification problem, and then computes the average metrics (as specified by the XAxisMetric and YAxisMetric name-value arguments) using the values of the confusion matrix.

    • "macro" (macro-averaging) — average computes the average values for the metrics by averaging the values of all one-versus-all binary classification problems.

    • "weighted" (weighted macro-averaging) — average computes the weighted average values for the metrics using the macro-averaging method and using the prior class probabilities (the Prior property of rocObj) as weights.

    The algorithm type determines the length of the vectors for the output arguments (FPR, TPR, and Thresholds). For more details, see Average of Performance Metrics.

    Data Types: char | string

    Since R2024b

    Name of a metric to average, specified as a name in rocObj.Metrics or as the name of a built-in metric listed in this table.

    NameDescription
    "TruePositives" or "tp"Number of true positives (TP)
    "FalseNegatives" or "fn"Number of false negatives (FN)
    "FalsePositives" or "fp"Number of false positives (FP)
    "TrueNegatives" or "tn"Number of true negatives (TN)
    "SumOfTrueAndFalsePositives" or "tp+fp"Sum of TP and FP
    "RateOfPositivePredictions" or "rpp"Rate of positive predictions (RPP), (TP+FP)/(TP+FN+FP+TN)
    "RateOfNegativePredictions" or "rnp"Rate of negative predictions (RNP), (TN+FN)/(TP+FN+FP+TN)
    "Accuracy" or "accu"Accuracy, (TP+TN)/(TP+FN+FP+TN)
    "TruePositiveRate", "tpr", or "recall"True positive rate (TPR), also known as recall or sensitivity, TP/(TP+FN)
    "FalseNegativeRate", "fnr", or "miss"False negative rate (FNR), or miss rate, FN/(TP+FN)
    "FalsePositiveRate" or "fpr"False positive rate (FPR), also known as fallout or 1-specificity, FP/(TN+FP)
    "TrueNegativeRate", "tnr", or "spec"True negative rate (TNR), or specificity, TN/(TN+FP)
    "PositivePredictiveValue", "ppv", "prec", or "precision"Positive predictive value (PPV), or precision, TP/(TP+FP)
    "NegativePredictiveValue" or "npv"Negative predictive value (NPV), TN/(TN+FN)
    "f1score"F1 score, 2*TP/(2*TP+FP+FN)
    "ExpectedCost" or "ecost"

    Expected cost, (TP*cost(P|P)+FN*cost(N|P)+FP*cost(P|N)+TN*cost(N|N))/(TP+FN+FP+TN), where cost is a 2-by-2 misclassification cost matrix containing [0,cost(N|P);cost(P|N),0]. cost(N|P) is the cost of misclassifying a positive class (P) as a negative class (N), and cost(P|N) is the cost of misclassifying a negative class as a positive class.

    The software converts the K-by-K matrix specified by the Cost name-value argument of rocmetrics to a 2-by-2 matrix for each one-versus-all binary problem. For details, see Misclassification Cost Matrix.

    Data Types: char | string

    Since R2024b

    Name of a metric to average, specified as a name in rocObj.Metrics or as the name of a built-in metric listed in this table.

    NameDescription
    "TruePositives" or "tp"Number of true positives (TP)
    "FalseNegatives" or "fn"Number of false negatives (FN)
    "FalsePositives" or "fp"Number of false positives (FP)
    "TrueNegatives" or "tn"Number of true negatives (TN)
    "SumOfTrueAndFalsePositives" or "tp+fp"Sum of TP and FP
    "RateOfPositivePredictions" or "rpp"Rate of positive predictions (RPP), (TP+FP)/(TP+FN+FP+TN)
    "RateOfNegativePredictions" or "rnp"Rate of negative predictions (RNP), (TN+FN)/(TP+FN+FP+TN)
    "Accuracy" or "accu"Accuracy, (TP+TN)/(TP+FN+FP+TN)
    "TruePositiveRate", "tpr", or "recall"True positive rate (TPR), also known as recall or sensitivity, TP/(TP+FN)
    "FalseNegativeRate", "fnr", or "miss"False negative rate (FNR), or miss rate, FN/(TP+FN)
    "FalsePositiveRate" or "fpr"False positive rate (FPR), also known as fallout or 1-specificity, FP/(TN+FP)
    "TrueNegativeRate", "tnr", or "spec"True negative rate (TNR), or specificity, TN/(TN+FP)
    "PositivePredictiveValue", "ppv", "prec", or "precision"Positive predictive value (PPV), or precision, TP/(TP+FP)
    "NegativePredictiveValue" or "npv"Negative predictive value (NPV), TN/(TN+FN)
    "f1score"F1 score, 2*TP/(2*TP+FP+FN)
    "ExpectedCost" or "ecost"

    Expected cost, (TP*cost(P|P)+FN*cost(N|P)+FP*cost(P|N)+TN*cost(N|N))/(TP+FN+FP+TN), where cost is a 2-by-2 misclassification cost matrix containing [0,cost(N|P);cost(P|N),0]. cost(N|P) is the cost of misclassifying a positive class (P) as a negative class (N), and cost(P|N) is the cost of misclassifying a negative class as a positive class.

    The software converts the K-by-K matrix specified by the Cost name-value argument of rocmetrics to a 2-by-2 matrix for each one-versus-all binary problem. For details, see Misclassification Cost Matrix.

    Data Types: char | string

    Output Arguments

    collapse all

    Average false positive rates, returned as a numeric vector.

    Average true positive rates, returned as a numeric vector.

    Thresholds on classification scores at which the function finds each of the average performance metric values (FPR and TPR), returned as a vector.

    Area under the average ROC curve composed of FPR and TPR, returned as a numeric scalar.

    Since R2024b

    Average of metric1, returned as a double or single vector, depending on the data.

    Since R2024b

    Average of metric2, returned as a double or single vector, depending on the data.

    More About

    collapse all

    Algorithms

    collapse all

    Alternative Functionality

    • You can use the plot function to create the average ROC curve. The function returns a ROCCurve object containing the XData, YData, Thresholds, and AUC properties, which correspond to the output arguments FPR, TPR, Thresholds, and AUC of the average function, respectively. For an example, see Plot Average ROC Curve for Multiclass Classifier.

    References

    [1] Sebastiani, Fabrizio. "Machine Learning in Automated Text Categorization." ACM Computing Surveys 34, no. 1 (March 2002): 1–47.

    Version History

    Introduced in R2022a

    expand all