How to calculate the precision and recall ( determine FP and FN) for image detection based on image segmentation?
Show older comments
IF i used jaccard index and I got value of jaccrad =0.9 if I assume my threshold is 0.5 so if jaccard value >0.5 will be TP and if less will be FP
what about FN?
Answers (1)
Jaccard index is related to how much a "ground truth" binary image overlaps your "test" image. According to the help "similarity = jaccard(BW1,BW2) computes the intersection of binary images BW1 and BW2 divided by the union of BW1 and BW2, also known as the Jaccard index."
So using your definition, you're saying that if they overlap by more than 50%, it detected it correctly -- it's a True Positive. And you're saying that if they overlap by less than 50% it's a False Positive because even though they might overlap some, it's not enough to be considered correct.
A false negative would be if you said there was nothing there (gave a Negative) when there actually was something there.
Look at the different possible cases:
bw1 = true(2,2); % Something there.
bw2 = false(2,2); % Nothing there.
bwHalf = logical([1,1;0,0]);
similarityIndex = jaccard(bw1, bw1)
similarityIndex = jaccard(bw1, bw2)
similarityIndex = jaccard(bw2, bw1)
similarityIndex = jaccard(bw2, bw2)
% Using half overlap
similarityIndex = jaccard(bw1, bwHalf)
similarityIndex = jaccard(bw2, bwHalf)
Case 2 or 3, where the ground truth had something while the test/prediction said there was nothing, both give a Jaccard index of 0. That would be a False Negative -- it said it was negative but it was false because it should have said it was positive.
12 Comments
yasmin ismail
on 20 Nov 2023
Image Analyst
on 20 Nov 2023
I didn't say that, you did. You said that if the overlap is over 50% it's a true positive: "I assume my threshold is 0.5 so if jaccard value >0.5 will be TP" Why you picked that amount is basically a judgment call that you decide on.
yasmin ismail
on 20 Nov 2023
Edited: yasmin ismail
on 20 Nov 2023
Image Analyst
on 21 Nov 2023
Sorry for the delay but there is just so much you don't understand and it would take like an hour or two to explain it to you in person. It looks like you're trying to make an ROC curve. So start with understanding what that is:
Now each individual point on an ROC curve (True Positive Rate vs False Positive Rate) corresponds to running your algorithm on the whole set of images. Each point corresponds to you changing something in your algorithm, such as the threshold, filter window width, or whatever it may be. So you set one value, run all your images through and then computer TP and FP and make that one point on the curve. Then you change the parameter to a new value and computer TP and FN for that new parameter. Now you have a second point. So then you repeat for as many parameter values as you want and you'll start building up the curve. Then you can examine the curve and decide what is the best parameter based on your situations. Obviously for some situations you'd want a really low FN rate (e.g. determination of cancer), while for other situations you might accept a lower FN or FP rate.
I still think that using Jaccard index itself to identify the best performing algorithm is sufficient. I don't see the rationale to threshold it and classify it into FP, FN, TP, and TN. You can just make a plot of Jaccard index vs your parameter value(s) and determine the best parameter(s) to use from that.
yasmin ismail
on 22 Nov 2023
Edited: yasmin ismail
on 22 Nov 2023
Image Analyst
on 22 Nov 2023
By your definitions, TP is the number of images where the jaccard index is more than 0.5. And FN is the number of images where there is actually a blob present but your algorithm says the jaccard index is less than 0.5, because it's saying there is no crack there but there actually is. I believe these are the definitions:
TP = says it's there (jaccard index > 0.5), and it is.
FP = says it's there (jaccard index > 0.5), but it's not there.
TN = says it's not there (jaccard index < 0.5) and it's not
FN = says it's not there (jaccard index < 0.5) but it is.
yasmin ismail
on 22 Nov 2023
yasmin ismail
on 22 Nov 2023
yasmin ismail
on 22 Nov 2023
Image Analyst
on 22 Nov 2023
Edited: Image Analyst
on 22 Nov 2023
I don't know that you can use the Jaccard index for a false negative. If the ground truth is not there, but your algorithm falsely says there is something there then you might just use this
FN = sum(binaryImage, 'all') > 0 && max(groundTruthBinaryImage) == 0;
Let's say you had 100 images. To build up the ROC curve, you'd have to compute the TP and FP rates (meaning the average over all 100 images) and then plot one point on the curve.

Then you'd change something in your algorithm/model such as the threshold but it could be anything, like a filter window width or whatever. Then you'd run all 100 images through the algorithm again and compute new rates. Now you have a second point on the ROC curve. Now do that as many times as you want to get as many points on the curve as you want.
See roc
yasmin ismail
on 23 Nov 2023
yasmin ismail
on 23 Nov 2023
Edited: yasmin ismail
on 23 Nov 2023
Categories
Find more on Image Processing and Computer Vision in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!