Main Content

Deployment and Classification of Webcam Images on NVIDIA Jetson TX2 Platform

This example uses:

Open Live Script

This example shows how to generate CUDA® code from a DAGNetwork object and deploy the generated code onto the NVIDIA® Jetson® TX2 board using the GPU Coder™ Support Package for NVIDIA GPUs. This example uses the resnet50 deep learning network to classify images from a USB webcam video stream.

Prerequisites

Target Board Requirements

  • NVIDIA Jetson Tegra TX2 embedded platform.

  • Ethernet crossover cable to connect the target board and host PC (if the target board cannot be connected to a local network).

  • USB camera to connect to the TX2.

  • NVIDIA CUDA toolkit installed on the target board.

  • NVIDIA cuDNN library on the target board.

  • OpenCV library on the target for reading and displaying images/video.

  • Environment variables on the target for the compilers and libraries. For information on the supported versions of the compilers and libraries and their setup, see Install and Setup Prerequisites for NVIDIA Boards for NVIDIA boards.

Development Host Requirements

Verify NVIDIA Support Package Installation on Host

Use the checkHardwareSupportPackageInstall function to verify that the host system is compatible to run this example.

checkHardwareSupportPackageInstall();

Connect to the NVIDIA Hardware

The GPU Coder Support Package for NVIDIA GPUs uses an SSH connection over TCP/IP to execute commands while building and running the generated CUDA code on the Jetson platform. You must therefore connect the target platform to the same network as the host computer or use an Ethernet crossover cable to connect the board directly to the host computer. Refer to the NVIDIA documentation on how to set up and configure your board.

To communicate with the NVIDIA hardware, you must create a live hardware connection object by using the jetson function. You must know the host name or IP address, username, and password of the target board to create a live hardware connection object. For example, when connecting to the target board for the first time, create a live object for Jetson hardware by using the command:

hwobj= jetson('host-name','username','password');

The jetson object reuses these settings from the most recent successful connection to the Jetson hardware. This example establishes an SSH connection to the Jetson hardware using the settings stored in memory.

hwobj = jetson;
Checking for CUDA availability on the Target...
Checking for 'nvcc' in the target system path...
Checking for cuDNN library availability on the Target...
Checking for TensorRT library availability on the Target...
Checking for prerequisite libraries is complete.
Gathering hardware details...
Checking for third-party library availability on the Target...
Gathering hardware details is complete.
 Board name              : NVIDIA Clara AGX Developer Kit
 CUDA Version            : 11.6
 cuDNN Version           : 8.3
 TensorRT Version        : 8.2
 GStreamer Version       : 1.16.3
 V4L2 Version            : 1.18.0-2build1
 SDL Version             : 1.2
 OpenCV Version          : 4.2.0
 Available Webcams       : Microsoft LifeCam Cinema(TM):
 Available GPUs          : Quadro RTX 6000
 Available Digital Pins  :

In case of a connection failure, a diagnostics error message is reported on the MATLAB command line. If the connection has failed, the most likely cause is incorrect IP address or hostname.

When there are multiple live connection objects for different targets, the code generator performs remote build on the target for which a recent live object was created. To choose a hardware board for performing remote build, use the setupCodegenContext() method of the respective live hardware object. If only one live connection object was created, it is not necessary to call this method.

hwobj.setupCodegenContext;

Verify GPU Environment on the Target

Use the coder.checkGpuInstall function to verify that the compilers and libraries necessary for running this example are set up correctly.

envCfg = coder.gpuEnvConfig('jetson');
envCfg.DeepLibTarget = 'cudnn';
envCfg.DeepCodegen = 1;
envCfg.Quiet = 1;
envCfg.HardwareObject = hwobj;
coder.checkGpuInstall(envCfg);

ResNet-50 Entry-Point Function

The resnet50_wrapper.m entry-point function uses a pre-trained ResNet-50 Network to classify images. ResNet-50 is a DAG Network trained on more than a million images from the ImageNet database. The output contains the categorical scores of each class the image belongs to.

type resnet50_wrapper
function out = resnet50_wrapper(im,ocvFlag) %#codegen
% Wrapper function to call ResNet50 predict function.

%   Copyright 2019-2021 The MathWorks, Inc.

% This example uses OpenCV for reading frames from a web camera and
% displaying output image. Update buildinfo to link with OpenCV library
% available on target.
if ocvFlag
    opencv_link_flags = '`pkg-config --libs opencv4`';
    opencv_compile_flags = '`pkg-config --cflags  opencv4`';
else
    opencv_link_flags = '`pkg-config --libs opencv`';
    opencv_compile_flags = '`pkg-config --cflags --libs opencv`';
end

coder.updateBuildInfo('addLinkFlags',opencv_link_flags);
coder.updateBuildInfo('addCompileFlags',opencv_compile_flags);

% To avoid multiple loads of the network for each run, we use persistent
% rnet
persistent rnet;
if isempty(rnet)
    rnet = resnet50();
end
out = rnet.predict(im);

end

Get OpenCV Version on the Target

Use the pkg-config helper tool to query if OpenCV 4.x is installed on the target board. This example uses the information to update build information to link with the appropriate OpenCV library available on target.

isOpenCV4 = 1;
ocvVersion = hwobj.OpenCVVersion();
if (str2double(ocvVersion(1)) <= 3)
   isOpenCV4 = 0;
end

Generate and Deploy CUDA Code on the Target

To generate a CUDA executable that can be deployed on to an NVIDIA target, create a GPU coder configuration object for generating an executable.

cfg = coder.gpuConfig('exe');

Use the coder.hardware function to create a configuration object for the Jetson platform and assign it to the Hardware property of the GPU code configuration object cfg.

cfg.Hardware = coder.hardware('NVIDIA Jetson');

Set the deep learning configuration to cudnn or tensorrt.

cfg.DeepLearningConfig = coder.DeepLearningConfig('cudnn');

Read an image to use as a sample input for code generation. The deployed executable will take the webcam stream as input.

im = single(imread('peppers.png'));
im = imresize(im,[224,224]);

The custom main file is coded to take video as input and classifies each frame in the video sequence. The custom main_resnet50.cu file is a wrapper that calls the predict function in the generated code. Post processing steps such as displaying output on the input frame are added in the main file using OpenCV interfaces.

cfg.CustomSource = fullfile('main_resnet50.h');
cfg.CustomSource = fullfile('main_resnet50.cu');

To generate CUDA code and deploy it onto target, use the codegen function and pass the GPU code configuration object. After the code generation takes place on the host, the generated files are copied over and built on the target in the workspace directory.

codegen -config cfg -args {im,coder.Constant(isOpenCV4)} resnet50_wrapper -report
Code generation successful: View report

Run the Application on the Target

Copy the synsetWords_resnet50 text file from host computer to the target device by using the putFile command.

putFile(hwobj,'synsetWords_resnet50.txt',hwobj.workspaceDir);

Use the runApplication method of the hardware object to launch the application on the target hardware. The application will be located in the workspace directory.

runApplication(hwobj,'resnet50_wrapper');

If the webcam window is not visible on the target board, it may have been directed to the incorrect display. Use the setDisplayEnvironment function to set the display environment used for redirecting the display on the target. The value must be the same as the $DISPLAY environment value set on the board.

Resnet Classification Output on Jetson TX2

Kill the Application

Use the killApplication method of the hardware object to kill the running application on the target.

killApplication(hwobj,'resnet50_wrapper');

See Also

Objects

Related Topics