Review Labeled Images and Send Feedback

Step 4 of 5 in Get Started with Team-Based Labeling

As a review task owner, you are responsible for completing the assigned review task, sending the correctly labeled images to the project owner, and sending feedback on the incomplete images to the label task owner. For more information on the tasks performed by each role, see the Using This Example section of Get Started with the Image Labeler.

The Image Labeler app enables you to review a set of images as a review task owner. The project owner assigns the review task to you as part of a distributed labeling project with multiple team members. For more information about how to set up a team labeling project as a project owner, see Create Project for a Labeling Team and Publish Tasks and Create Review Tasks for Labeled Images.

This example details the steps a review task owner must follow to:

Open a project shared by the project owner.
Load the assigned review task.
Review images.
Send the review feedback.

Open Labeling Project and Load Assigned Review Task

A review task owner with access to a full MATLAB^® license, can open a labeling project and review images following these instructions.

Open the Image Labeler app using one of these options:

MATLAB^® Toolstrip: On the Apps tab, under Image Processing and Computer Vision, click the app icon.
In the MATLAB command prompt, enter
```
imageLabeler
```

Note

To install the Image Labeler app and access the shared labeling project, a task owner without access to a full MATLAB license must use a compiled executable file shared by the project owner. For more information on the executable file created by the project owner, see Create Executable for Review Task Owners Without Full MATLAB License.

Open Project

The label tasks, along with labeling instructions have been predefined. These are part of the team project created and shared with you by the project owner.

To open an existing team project, in the Image Labeler app interface,click Open Project.
In the file explorer, navigate to the project folder location shared with you. Select the .prj file, and click Open.

Load Assigned Review Task

After you select and open the team labeling project, the app provides the option to open the project in Project Owner or Task Owner mode.

To access the review task assigned to you as a review task owner, follow these steps:

Select the Task Owner mode. The app then displays an interface with a Task Owner Name list and a Task List that contains the Task Name and Task Type for each task assigned to the selected task owner.
From the Task Owner Name list, select Team Member E. The Task List shows Task 5 in the Task Name column and Review in the Task Type column.
Select Task 5 and click Open. The Image Labeler app opens the review Task 5 in the task owner mode.

Image Labeler Layout in Task Owner mode

The ROI Label Definitions, Scene Label Definitions, and Task instructions panes contain the label definitions for all the images in the task that you must review for the current project.

Each ROI label definition is associated with a specific color listed in the ROI Label Definitions pane. The colors of labels on the images correspond to these label definitions.

The Review Labels, Sublabels and Attributes pane lists options for reviewing the labels. You can accept or reject individual labels, and provide feedback, in this pane.

The Image Browser pane displays thumbnails of the images in the review task. In this example, the Image Browser shows the 50 images assigned to Team Member E for the review Task 5.

Review Labeled Images

To complete the review process for the task, you must accept or reject the label definitions for all images in the task. You can also provide a few lines of feedback for each reviewed image. When you reject the labels for an image, this enables you to specify concrete actions for the labelers to correct the labels.

Accept Correctly Labeled Images

Consider the following image for review. Note that the ROI label definition VideoCamera is labeled correctly on the image. The scene label definition BustPresent is correctly marked as No, as there is no bust present in the picture. There is no lamp in the picture, and thus the ROI label definition Lamp is correctly absent in the image. Thus, you can accept the labels on this image. You can do so in the Review Labels, Sublabels, and Attributes pane using these steps:

Because all the labels are marked correctly, you can click Accept All.
Alternatively, you can accept the labels individually by selecting Accept next to the each label.

Review task owner interface for accepting labels.

Reject Incorrectly Labeled Images

Consider the following image for review. Note that the labeler has used the incorrect ROI label definition of Lamp to label the video camera in the image. Furthermore, the label does not fully cover the video camera in the image. The scene label definition BustPresent is correctly marked as No, as there is no bust present in the picture. Thus, you can use the Review Labels, Sublabels, and Attributes pane to accept the scene label and reject the ROI label, with feedback for correction, using these steps:

Accept the scene label by selecting Accept next to the BustPresent label.
Reject the ROI label by selecting Reject next to the Lamp label.

Alternatively, instead of rejecting an incorrect ROI label, you can delete or modify it and accept it as a reviewer. You can also view or hide each label and sublabel in a labeled image by clicking the icon next to the name of that label or sublabel in the ROI Label Definitions pane.

For pixel labels, you can only send acceptance or rejection for each image and provide feedback for the label task owner.

Review task owner interface for rejecting labels.

Send Review Feedback

After reviewing a set of images, you can send the review feedback to the project owner and label task owner. Once you send the feedback, the images are locked for review until the project owner and the respective label task owner take next steps. Images with at least one rejected label go back to the original label task owner to re-label correctly. Images with all labels accepted are marked ready for export and the project owner can export them.

To send your review feedback, from the toolstrip select Send Review Results > Send reviewed images. While you are working on the review task, you can save the project at any stage. This enables the project owner and label task owners to access the feedback for the images that you have completed reviewing so far.

In the Image Browser pane the symbol on the corner of each image thumbnail indicates the status of the image. You can access descriptions for the thumbnail badges by selecting Image Status Legend from the toolstrip. This table shows the review progress for each image based on the displayed badge.