evaluateSemanticSegmentation
Evaluate semantic segmentation data set against ground truth
Syntax
Description
computes various metrics to evaluate the quality of the semantic segmentation
results from confusion matrices, ssm
= evaluateSemanticSegmentation(imageSetConfusion
,classNames
)imageSetConfusion
, with
segmentation classes classNames
.
[
computes various metrics to evaluate the quality of the block-based semantic
segmentation results from confusion matrices, ssm
,blockMetrics
] = evaluateSemanticSegmentation(blockSetConfusion
,classNames
)blockSetConfusion
with classes classNames
.
[___] = evaluateSemanticSegmentation(___,
computes semantic segmentation metrics using one or more
Name,Value
)Name,Value
pair arguments to control the evaluation.
Examples
Evaluate Semantic Segmentation Results
The triangleImages
data set has 100 test images with ground truth labels. Define the location of the data set.
dataSetDir = fullfile(toolboxdir("vision"),"visiondata","triangleImages");
Define the location of the test images and ground truth labels.
testImagesDir = fullfile(dataSetDir,"testImages"); testLabelsDir = fullfile(dataSetDir,"testLabels");
Create an imageDatastore
holding the test images.
imds = imageDatastore(testImagesDir);
Define the class names and their associated label IDs.
classNames = ["triangle" "background"]; labelIDs = [255 0];
Create a pixelLabelDatastore
holding the ground truth pixel labels for the test images.
pxdsTruth = pixelLabelDatastore(testLabelsDir,classNames,labelIDs);
Load a semantic segmentation network that has been trained on the training images of triangleImages
.
net = load("triangleSegmentationNetwork");
net = net.net;
Run the network on the test images. Predicted labels are written to disk in a temporary directory and returned as a pixelLabelDatastore
.
pxdsResults = semanticseg(imds,net,Classes=classNames,WriteLocation=tempdir);
Running semantic segmentation network ------------------------------------- * Processed 100 images.
Evaluate the prediction results against the ground truth.
metrics = evaluateSemanticSegmentation(pxdsResults,pxdsTruth);
Evaluating semantic segmentation results ---------------------------------------- * Selected metrics: global accuracy, class accuracy, IoU, weighted IoU, BF score. * Processed 100 images. * Finalizing... Done. * Data set metrics: GlobalAccuracy MeanAccuracy MeanIoU WeightedIoU MeanBFScore ______________ ____________ _______ ___________ ___________ 0.99074 0.99183 0.91118 0.98299 0.80563
Display the properties of the semanticSegmentationMetrics
object.
metrics
metrics = semanticSegmentationMetrics with properties: ConfusionMatrix: [2x2 table] NormalizedConfusionMatrix: [2x2 table] DataSetMetrics: [1x5 table] ClassMetrics: [2x3 table] ImageMetrics: [100x5 table]
Display the classification accuracy, the intersection over union, and the boundary F-1 score for each class. These values are stored in the ClassMetrics
property.
metrics.ClassMetrics
ans=2×3 table
Accuracy IoU MeanBFScore
________ _______ ___________
triangle 0.99302 0.83206 0.67208
background 0.99063 0.9903 0.93918
Display the normalized confusion matrix that is stored in the NormalizedConfusionMatrix
property.
metrics.ConfusionMatrix
ans=2×2 table
triangle background
________ __________
triangle 4697 33
background 915 96755
Input Arguments
dsResults
— Predicted pixel labels
datastore | cell array of datastore objects
Predicted pixel labels resulting from semantic segmentation, specified as
a datastore or a cell array of datastore objects.
dsResults
can be any datastore that returns
categorical images, such as PixelLabelDatastore
. The read
(dsResults
) must return a
categorical array, a cell array, or a table. If the
read
function returns a multicolumn cell array or
table, the second column must contain categorical arrays.
dsTruth
— Ground truth pixel labels
datastore | cell array of datastore objects
Ground truth pixel labels, specified as a datastore or a cell array of
datastore objects. dsTruth
can be any datastore that
returns categorical images, such as PixelLabelDatastore
. Using read
(dsTruth
) must return a categorical
array, a cell array, or a table. If the read
function
returns a multicolumn cell array or table, the second column must contain
categorical arrays.
imageSetConfusion
— Confusion matrices for segmented images
table | cell array
Confusion matrix for the classes in the segmented images, specified as one of the following, where F is the number of images in the data set.
Table with F rows and one variable with the name
ConfusionMatrix
. Each row in the table contains a cell array with the confusion matrix for the corresponding image.F-by-one cell array. Each element of the cell array contains the confusion matrix for the corresponding image.
blockSetConfusion
— Confusion matrices for segmented blocks
table
Confusion matrices for segmented blocks, specified as a table with
B rows and three columns, where B
is the total number of blocks in all images in the data set. The three
columns are the variables ImageNumber
,
ConfusionMatrix
, and BlockInfo
.
You can obtain a table of the correct format by using the segmentationConfusionMatrix
function within a call to the
block-based apply
function. For an example, see Calculate Segmentation Metrics in Block-Based Workflow.
classNames
— Class names
array of strings | cell array of character vectors
Class names, specified as an array of strings or a cell array of character vectors.
Example: ["sky" "grass" "building"
"sidewalk"]
Name-Value Arguments
Specify optional pairs of arguments as
Name1=Value1,...,NameN=ValueN
, where Name
is
the argument name and Value
is the corresponding value.
Name-value arguments must appear after other arguments, but the order of the
pairs does not matter.
Example: metrics =
evaluateSemanticSegmentation(pxdsResults,pxdsTruth,Metrics="bfscore")
computes only the mean BF score of each class, each image, and the entire data
set.
Before R2021a, use commas to separate each name and value, and enclose
Name
in quotes.
Example: metrics =
evaluateSemanticSegmentation(pxdsResults,pxdsTruth,"Metrics","bfscore")
computes only the mean BF score of each class, each image, and the entire data
set.
Metrics
— Segmentation metrics
"all"
(default) | vector of strings
Segmentation metrics in semanticSegmentationMetrics
to compute, specified as a
vector of strings. This argument specifies which variables in the
DataSetMetrics
, ClassMetrics
,
and ImageMetrics
tables to compute.
ConfusionMatrix
and
NormalizedConfusionMatrix
are computed regardless
of the value of Metrics
.
Value | Description | Aggregate Data Set Metric | Image Metric | Class Metric |
---|---|---|---|---|
"all" | Evaluate all semantic segmentation metrics. The function excludes
| All aggregate data set metrics | All image metrics | All class metrics |
"accuracy" | Accuracy indicates the percentage of correctly identified pixels for each class. Use the accuracy metric if you want to know how well each class correctly identifies pixels.
The class accuracy is a simple metric analogous to global accuracy, but it can be misleading. For example, labeling all pixels "car" gives a perfect score for the "car" class (although not for the other classes). Use class accuracy in conjunction with IoU for a more complete evaluation of segmentation results. | MeanAccuracy | MeanAccuracy | Accuracy |
"bfscore" | The boundary F1 (BF) contour matching score indicates how well the predicted boundary of each class aligns with the true boundary. Use the BF score if you want a metric that tends to correlate better with human qualitative assessment than the IoU metric.
For more information, see This metric is
not available when you specify a confusion matrix
( | MeanBFScore | MeanBFScore | MeanBFScore |
"global-accuracy" |
| GlobalAccuracy | GlobalAccuracy | none |
"iou" |
Intersection over union (IoU), also known as the Jaccard similarity coefficient, is the most commonly used metric. Use the IoU metric if you want a statistical accuracy measurement that penalizes false positives.
For more information, see | MeanIoU | MeanIoU | IoU |
"weighted-iou" | Average IoU of each class, weighted by the number of pixels in that class. Use this metric if images have disproportionally sized classes, to reduce the impact of errors in the small classes on the aggregate quality score. | WeightedIoU | WeightedIoU | none |
Example: ["global-accuracy","iou"]
calculates the
global accuracy and IoU metrics across the data set, images, and
classes.
Data Types: string
Verbose
— Flag to display evaluation progress
1
(default) | 0
Flag to display evaluation progress information in the command window,
specified as 1
(true
) or
0
(false
).
The displayed information includes a progress bar, elapsed time,
estimated time remaining, and data set metrics. When
Verbose
is 0
(false
), the
evaluateSemanticSegmentation
function
calculates segmentation metrics without displaying progress
information.
Data Types: logical
Output Arguments
ssm
— Semantic segmentation metrics
semanticSegmentationMetrics
object
Semantic segmentation metrics, returned as a semanticSegmentationMetrics
object.
blockMetrics
— Block-based semantic segmentation metrics
F-by-one cell array
Block-based semantic segmentation metrics, returned as an F-by-one cell array, where F is the number of images in the data set. Each element in the cell array contains information about all of the metrics calculated for all blocks in the corresponding image, formatted as a table.
Each table has K(f) rows, where K(f) is the number of blocks in the fth image in the data set. The table has up to five variables:
The table always includes the
BlockInfo
variable. This table data in this variable are structs that provide spatial information about the block. The four fields of the struct areBlockStartWorld
,BlockEndWorld
,DataStartWorld
, andDataEndWorld
. For more information about these fields, see theIncludeBlockInfo
name-value pair argument of theapply
function.The table includes the metrics in the
DataSetMetrics
property of thessm
output argument. By default, the metrics areGlobalAccuracy
,MeanAccuracy
,MeanIoU
, andWeightedIoU
. However, if you create thessm
and specify a subset of the metrics to calculate by using theMetrics
name-value argument, then the table includes only the specified metrics.
Tips
A value of
NaN
in the dataset, class, or image metrics, indicates that one or more classes were missing during the computation of the metrics when using theevaluateSemanticSegmentation
function. In this case, the software was unable to accurately compute the metrics.The missing classes can be found by looking at the
ClassMetrics
property, which provides the metrics for each class. To more accurately evaluate your network, augment your ground truth with more data that includes the missing classes.
References
[1] Csurka, G., D. Larlus, and F. Perronnin. "What is a good evaluation measure for semantic segmentation?" Proceedings of the British Machine Vision Conference, 2013, pp. 32.1–32.11.
Extended Capabilities
Automatic Parallel Support
Accelerate code by automatically running computation in parallel using Parallel Computing Toolbox™.
To run in parallel, set
'UseParallel'
totrue
or enable this by default using the Computer Vision Toolbox™ preferences.For more information, see Parallel Computing Toolbox Support.
Parallel processing is only supported when the input datastores are
pixelLabelDatastore
objects.
Version History
Introduced in R2017b
See Also
Functions
semanticseg
|plotconfusion
(Deep Learning Toolbox) |jaccard
|bfscore
|segmentationConfusionMatrix
Objects
MATLAB Command
You clicked a link that corresponds to this MATLAB command:
Run the command by entering it in the MATLAB Command Window. Web browsers do not support MATLAB commands.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list:
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)