How to find true positive and true negative of a segmented image?

Hello everyone. I am working on "Retinal Blood Vessel Segmentation". I want to draw ROC using matlab. I am confused about how to find the TP and FP of a segmented image. After that how to plot ROC? I am clear about the definitions of all three. But how actually for a segmented image I can calculate true positives and true negatives?

Answer 1

[Walter Roberson]

In order to calculate true positive and false positive and the like, you need to have a reference telling you what each pixel really is. True positive would count the places that you predict that a region is part of a segment and the reference tells you that Yes, the region really is part of a segment. False positive counts the places that you predict that a pixel is part of a segment but the reference tells you that it is not part of the segment. True negative counts the places that you predict that a pixel is not part of a segment, and the reference tells you that indeed the pixel is not part of a segment.

Any kind of segmentation like that almost always has a parameter whose value can be moved through a range of values, and the segmentation regenerated, and the positive/negative counts recalculated. You can generate a data point on a curve for each of the four different combinations for each distinct parameter value you try out, so you can plot those with changes in value.

But you need a reference of what each pixel really is -- you need a "ground truth" to calculate this.

You might wonder why you would bother doing any of this if you already have a reference image. The point is that you would have a number of images with known results that you would use to "train" the routine to get the best overall results. Once you have that, you can apply it to images for which the results are not known, and you would have an idea of how good you should expect the outcome to be based upon how well it did on the known images.

[Sidra Aleem]

Thank you @Walter Roberson for detailed and clear answer. From your answer I am clear about things from theoretical purpose. Below attached image has ground truth and segmentation result of an image.

You can see there are a lot of false negatives. Can you please tell me how to calculate them. Like apparently I know what are false negatives in segmentation result, but how to actually come up with a mathematical number for these false negatives. Thank you.

[Walter Roberson]

false_negatives = nnz(~segmented & ground_truth)

Segmentation says it is not part of the segmentation but ground truth says it is, would be a false negative.

false_positives = nnz(segmented & ~ground_truth)

[ruchika lalit]

ca you please give the similar formulation for true positive and true negative or way to find relation between TP TN FP FN

[Walter Roberson]

true_positives = nnz(segmented & ground_truth);
true_negative = nnz(~segmented & ~ground_truth);

[Image Analyst]

The perfcurve() function, in the Statistics and Machine Learning Toolbox, might also be useful as you vary the parameter(s) to tweak your segmentation.

[ruchika lalit]

thanks walter and image anayst for quick response. I have applied the formula suggested by walter and got the results for an image, but how can I do it for the video sequence having about 1500 frames. actually I need to find out recall and precision using its formula of TP, FN,FP. please suggest.

[Walter Roberson]

for frame_idx = 1 : 1500
  this_frame = readFrame(vidobj);
  segmented = do_segmentation(this_frame);
  ground_truth = fetch_ground_truth_for_frame(frame_idx);
  true_positives(frame_idx) = nnz(segmented & ground_truth);
  true_negative(frame_idx) = nnz(~segmented & ~ground_truth);
  ...
end

where fetch_ground_truth_for_frame is a routine you would write that returns the ground truth for a frame given its index. Remember that ground truth cannot be calculated (except for simulated data), only loaded according to whatever expert analysis has been done.

It would be not be at all common for someone to have done a ground truth analysis for 1500 frames of a video sequence -- humans get too bored for doing 1500 frames.

Though, it is possible that you could have a video for which the ground truth was the same for every frame. That would be uncommon, but I guess if you were studying the effect of adding noise to an image it would be possible.

... But when you are talking recall and precision, it is somewhat unlikely that the code I showed for calculating true/false positives/negatives applies to your situation.

[ruchika lalit]

I am sending you the paper which I am trying to implement please check and please tell will the code you gave will work or not

[Walter Roberson]

The only place that paper mentions ground truth is in the captions for figure 4 and figure 5. I think we can say with high confidence that the above code is not applicable to your situation.

[ruchika lalit]

yes you are right as it is giving similar values for most of the frames. will you please suggest some other way to calculate them. and one more thing in paper they have used ABL and GMM for comparison should we apply these two before calculating LBP. Please suggest.

[Walter Roberson]

I do not know why that paper mentioned ground truth in the two figures. Perhaps they took two frames and did specific ground truth analysis for testing the algorithm.

The paper deals with an analysis situation in which the ground truth is not likely to be known, since it deals with changing backgrounds. For example on a road if some trash gets thrown out of a car then the trash becomes part of the background. If someone comes by and removes a construction cone then the area behind it becomes part of the background. If there is a wind then there could be something that is moved around in bursts, and you get into a definitional problem: how much movement is needed to consider something background or not? A piece of paper blowing around is probably not "background", might be moving faster than a pedestrian, but a tree whose branches are moving in the wind is probably background.

[ruchika lalit]

can you please suggest some other way to find them in my case

[Image Analyst]

You have to define what a true positive is. No one or no algorithm is going to know what "truth" is unless you tell it. In your case that might be you hand painting the image to identify what pixels you consider to be "true" vessels. If you paint an image and say that's a vessel, then that's a vessel. Who's to argue with you? I mean some trusted expert (you or your ophthalmologist) has to be the one with expert knowledge as to what is a true vessel pixel or not.