Detect Position and Orientation of a Rectangular Object Using OpenCV

This topic explains the algorithm to detect position and orientation of a rectangular object using OpenCV.

This topic is part of the overall workflow described in Object Detection and Motion Planning Application with Onboard Deployment.

The figure shows the original image captured by the vision module of a Kinova Gen3 Robot. The image consists of a rectangular object lying on a uniform background. The image also shows a small portion of the gripper.

To find a position and orientation of a rectangular object, convert the image to grayscale image for efficient object detection. You can also apply a median blur in the image to smoothen any salt-and-pepper type of noise. The below figure shows the gray image derived from the original image.
Detect edges of the rectangular object. For this, you can use canny edge detection. The output image of canny edge detector is a binary black and white image. The threshold of canny edge detector can be adjusted such that only the periphery of the object is visible in the binary image.
Once you have the binary image, detect contours and approximate these contours by shapes with less number of vertices.
Because you have the prior information about the shape of the object, you can further compute a rectangle with minimum area, which completely encapsulate the less complex shape (which we derived by approximating original contours). The minimum area rectangle contains information of the object’s location and rotation with respect to the horizontal axis.
To avoid any false detection, you can check the area of the rectangle. If the area of the rectangle falls inside the predefined range, the detected rectangle can be accepted as the approximation of original object, and the position and orientation information can be passed to the motion planning algorithm.
In the below figure, you can see the minimum area rectangle with red boundaries, superimposed on the original image. The rectangle has a close resemblance with the rectangular object.

The key OpenCV functions used in this workflow are the following:

Verify Object Detection with NVIDIA Jetson Board

After you create the algorithm for object detection, you need to verify it using the NVIDIA^® Jetson™ board:

Perform the setup and configuration of the Jetson compute board as mentioned in Configure NVIDIA Jetson Board for Object Detection and Motion Planning.
The intended motion of the robot is described in Detect and Pick Object Using KINOVA Gen3 Robot Arm with Joint Angle Control and Trajectory Control example. The robot first moves to a scan position where the vision module is looking straight down. At this point, algorithm waits for the position and orientation data of the object from the object detection node. Hence the node should be able to detect the object when the robot is at scan position.
Manually move the robot near the scan position using joystick or webapp. The joint configuration for the scan position is [0 0 180 266 0 272 90] degrees.
Open new terminal window at the same folder location where you downloaded the ROS packages from KINOVA, and execute ‘source devel/setup.bash’. Launch the kinova_vision node by executing the following command:
```
$ roslaunch kinova_vision kinova_vision_color_only.launch device:=<ipaddress>
```
Replace the <ipaddress> field in the command with an actual IP address from the robot.
Open new terminal window at the same folder location, and execute ‘source devel/setup.bash’. Launch the detect_object node by executing the following command:
```
$ rosrun opencv_node detect_object
```

This launches two image windows. One image is the output of canny edge detection and the other one is the original image with an overlay of a red rectangle. In an ideal setting, the red rectangle must match the boundary of the object.

Troubleshooting Object Detection

If you find any issues with the object detection, try the following steps:

If the red rectangle is not matching the object, adjust the slider on canny edge detection image such that only the object boundary is visible.
If the red square is not visible at all or it is not on the actual object, modify the minimum and maximum area threshold. Navigate to /opencv_node/src folder and open the detect_object.cpp file. Modify AreaThreshMin and AreaThreshMax values accordingly. Rebuild and relaunch the node to validate the object detection.
If you are not sure about threshold values, uncomment the line which prints the area of all detected contours, rebuild the node, and then launch the detect_object node again. This prints the area values for all detected contours on the terminal. You can note the area for the desired object and adjust area threshold values accordingly. Rebuild and relaunch the node to validate the object detection.