SOLOV2: Multiple conflicting labels

I am using a SoloV2 network to segment the iris and sclera, the "colored" and "white" parts of the eyes, respectively. I am confused about the segmentation results. The trained network gets the mask shapes absolutely correct but labels them incorrectly. Moreover, it sometimes creates identical masks and assigns conflicting labels. I am a novice and unsure if this is because I need to continue training or if something else is amiss with the model.

I attached a sample drawn from the training set and a figure showing segmentation on a series of test images.

Training set sample:

Segmentation results:

Thanks.