Deploy Semantic Segmentation Network Using Dilated Convolutions on FPGA

This example uses:

Computer Vision Toolbox Computer Vision Toolbox
Deep Learning HDL Toolbox Deep Learning HDL Toolbox
Deep Learning HDL Toolbox Support Package for Xilinx FPGA and SoC Devices Deep Learning HDL Toolbox Support Package for Xilinx FPGA and SoC Devices
Deep Learning Toolbox Model Compression Library Deep Learning Toolbox Model Compression Library
GPU Coder Interface for Deep Learning GPU Coder Interface for Deep Learning
MATLAB Coder Interface for Deep Learning MATLAB Coder Interface for Deep Learning

This example shows how to deploy a trained semantic segmentation network that uses dilated convolutions to a Xilinx® Zynq® Ultrascale+™ ZCU102 SoC development kit. Semantic segmentation networks like DeepLab [1] make extensive use of dilated convolutions, also known as atrous convolutions because they can increase the receptive field of the layer without increasing the number of parameters or computations.

A semantic segmentation network classifies every pixel in an image,which results in an image that is segmented by class. Applications for semantic segmentation include road segmentation for autonomous driving and cancer cell segmentation for medical diagnosis.

The network attached to this example was created in the example Get Started with Semantic Segmentation Using Deep Learning (Computer Vision Toolbox).

To obtain an improved frames per second (FPS) performance, you can quantize the network. This example shows how to calibrate and deploy a quantized network and then compare the performance between the quantized network and a single data type network.

Load the Pretrained Network

To load the pretrained semantic segmentation network, enter:

load("trainedSemanticSegmentationNet.mat")

Use the analyzeNetwork function to view a graphical representation of the network and detailed parameter settings for the layers in the network.

analyzeNetwork(net)

Define FPGA Board Interface

Define the target FPGA board programming interface by using the dlhdl.Target object. Specify that the interface is for a Xilinx board with an Ethernet interface.

hTarget = dlhdl.Target('Xilinx','Interface','Ethernet');

To use the JTAG interface, install Xilinx Vivado® Design Suite 2022.1. To set the Xilinx Vivado tool path, enter:

hdlsetuptoolpath('ToolName', 'Xilinx Vivado', 'ToolPath', 'C:\Xilinx\Vivado\2022.1\bin\vivado.bat');

Prepare Network for Deployment

Prepare the network for deployment by creating a dlhdl.Workflow object and specifying the network and bitstream name. Ensure that the bitstream name matches the data type and FPGA board. In this example, the target FPGA board is the Xilinx ZCU102 SOC board and the bitstream uses a single data type.

wfObj = dlhdl.Workflow('network', net, 'Bitstream', 'zcu102_single','Target',hTarget);

To run the example on a Xilinx ZC706 board, enter:

hW = dlhdl.Workflow('Network', snet, 'Bitstream', 'zc706_single','Target',hTarget);

Compile Network

Run the compile method of the dlhdl.Workflow object to compile the network and generate the instructions, weights, and biases for deployment. Because the total number of frames exceeds the default value of 30, set the InputFrameNumberLimit to 64 to run predictions in chunks of 64 frames to prevent timeouts.

dn = compile(wfObj,'InputFrameNumberLimit',64)

### Compiling network for Deep Learning FPGA prototyping ...
### Targeting FPGA bitstream zcu102_single.
### An output layer called 'Output1_softmax' of type 'nnet.cnn.layer.RegressionOutputLayer' has been added to the provided network. This layer performs no operation during prediction and thus does not affect the output of the network.
### Optimizing network: Fused 'nnet.cnn.layer.BatchNormalizationLayer' into 'nnet.cnn.layer.Convolution2DLayer'
### Notice: The layer 'imageinput' of type 'ImageInputLayer' is split into an image input layer 'imageinput' and an addition layer 'imageinput_norm' for normalization on hardware.
### The network includes the following layers:

### Notice: The layer 'softmax' with type 'nnet.cnn.layer.SoftmaxLayer' is implemented in software.
### Notice: The layer 'Output1_softmax' with type 'nnet.cnn.layer.RegressionOutputLayer' is implemented in software.
### Compiling layer group: conv_1>>conv_4 ...
### Compiling layer group: conv_1>>conv_4 ... complete.

### Allocating external memory buffers:

          offset_name          offset_address     allocated_space  
    _______________________    ______________    __________________

    "InputDataOffset"           "0x00000000"     "1024.0 kB"       
    "OutputResultOffset"        "0x00100000"     "1024.0 kB"       
    "SchedulerDataOffset"       "0x00200000"     "80.0 kB"         
    "SystemBufferOffset"        "0x00214000"     "288.0 kB"        
    "InstructionDataOffset"     "0x0025c000"     "92.0 kB"         
    "ConvWeightDataOffset"      "0x00273000"     "84.0 kB"         
    "EndOffset"                 "0x00288000"     "Total: 2592.0 kB"

### Network compilation complete.

dn = struct with fields:
             weights: [1×1 struct]
        instructions: [1×1 struct]
           registers: [1×1 struct]
    syncInstructions: [1×1 struct]
        constantData: {{}  [1×4096 single]}
             ddrInfo: [1×1 struct]
       resourceTable: [6×2 table]

Program Bitstream onto FPGA and Download Network Weights

To deploy the network on the Xilinx ZCU102 SoC hardware, run the deploy method of the dlhdl.Workflow object. This function uses the output of the compile function to program the FPGA board and download the network weights and biases. The deploy function programs the FPGA device and displays progress messages, and the required time to deploy the network.

deploy(wfObj)

### Programming FPGA Bitstream using Ethernet...
### Attempting to connect to the hardware board at 172.21.88.150...
### Connection successful
### Programming FPGA device on Xilinx SoC hardware board at 172.21.88.150...
### Attempting to connect to the hardware board at 172.21.88.150...
### Connection successful
### Copying FPGA programming files to SD card...
### Setting FPGA bitstream and devicetree for boot...
# Copying Bitstream zcu102_single.bit to /mnt/hdlcoder_rd
# Set Bitstream to hdlcoder_rd/zcu102_single.bit
# Copying Devicetree devicetree_dlhdl.dtb to /mnt/hdlcoder_rd
# Set Devicetree to hdlcoder_rd/devicetree_dlhdl.dtb
# Set up boot for Reference Design: 'AXI-Stream DDR Memory Access : 3-AXIM'
### Programming done. The system will now reboot for persistent changes to take effect.
### Rebooting Xilinx SoC at 172.21.88.150...
### Reboot may take several seconds...
### Attempting to connect to the hardware board at 172.21.88.150...
### Connection successful
### Programming the FPGA bitstream has been completed successfully.
### Loading weights to Conv Processor.
### Conv Weights loaded. Current time is 28-Aug-2024 15:57:01

Load Test Image

Read the example image.

imgTest = imread('triangleTest.jpg');
figure
imshow(imgTest)

Run Prediction for Test Image

Segment the test image using semanticseg_FPGA and display the results using labeloverlay.

networkInputSize = net.Layers(1).InputSize(1:2);
imgTestSize = size(imgTest);

assert(all(mod(imgTestSize(1:2), networkInputSize)) == 0, 'The 2D image input size should be a multiple of network input size');

numberOfBlocks = imgTestSize./networkInputSize;
totalBlocks = prod(numberOfBlocks);

splitImage = mat2cell(imgTest, networkInputSize(1)*ones(1, numberOfBlocks(1)), networkInputSize(1)*ones(1, numberOfBlocks(2)));
multiFrameInput = zeros([networkInputSize 1 totalBlocks]);
for i=1:totalBlocks
    multiFrameInput(:,:,:,i) = splitImage{i};
end

multiFrameInput = dlarray(multiFrameInput,'SSCB');
result = semanticseg_FPGA(multiFrameInput, wfObj.Network, wfObj);

### Finished writing input activations.
### Running in multi-frame mode with 64 inputs.


              Deep Learning Processor Profiler Performance Results

                   LastFrameLatency(cycles)   LastFrameLatency(seconds)       FramesNum      Total Latency     Frames/s
                         -------------             -------------              ---------        ---------       ---------
Network                     253840                  0.00115                      64           16252946            866.3
    imageinput_norm           7511                  0.00003 
    conv_1                   25842                  0.00012 
    conv_2                   96988                  0.00044 
    conv_3                   97253                  0.00044 
    conv_4                   26227                  0.00012 
 * The clock frequency of the DL processor is: 220MHz

The performance of the single data type network is 866.3 frames per second. Concatenate the images to highlight the triangles identified in the input image.

concatenatedResult = [];
for i=1:numberOfBlocks(2)
    subset = result(:,:,numberOfBlocks(1)*(i-1)+1:i*numberOfBlocks(1));
    verticalConcatenation = [];
    for j=1:numberOfBlocks(1)
        verticalConcatenation = [verticalConcatenation; subset(:,:,j)];
    end
    concatenatedResult = [concatenatedResult verticalConcatenation];
end

croppedFinal = labeloverlay(imgTest, concatenatedResult);
figure
imshow(croppedFinal)

Prepare the Quantized Network for Deployment

Load the data. The data set includes 32-by-32 triangle images.

dataFolder = fullfile(toolboxdir('vision'),'visiondata','triangleImages');
imageFolderTrain = fullfile(dataFolder,'trainingImages');

Create an image datastore for the images by using an imageDatastore object.

imdsTrain = imageDatastore(imageFolderTrain);

Create a quantized network by using dlquantizer. Set the target execution environment to FPGA.

dlQuantObj = dlquantizer(net,'ExecutionEnvironment','FPGA');

Use the calibrate function to exercise the network and collect range information for the learnable parameters in the network layers.

dlQuantObj.calibrate(imdsTrain);

Prepare the network for deployment by creating a dlhdl.Workflow object. Specify the network and bitstream name. Ensure that the bitstream name matches the data type and FPGA board. In this example the target FPGA board is the Xilinx ZCU102 SOC board. The bitstream uses an int8 data type.

wfObj_int8 = dlhdl.Workflow('Network', dlQuantObj, 'Bitstream', 'zcu102_int8', 'Target', hTarget);

Compile and Deploy the Quantized Network

dn = compile(wfObj_int8,'InputFrameNumberLimit',64)

### Compiling network for Deep Learning FPGA prototyping ...
### Targeting FPGA bitstream zcu102_int8.
### An output layer called 'Output1_softmax' of type 'nnet.cnn.layer.RegressionOutputLayer' has been added to the provided network. This layer performs no operation during prediction and thus does not affect the output of the network.
### Optimizing network: Fused 'nnet.cnn.layer.BatchNormalizationLayer' into 'nnet.cnn.layer.Convolution2DLayer'
### The network includes the following layers:

### Notice: The layer 'imageinput' with type 'nnet.cnn.layer.ImageInputLayer' is implemented in software.
### Notice: The layer 'softmax' with type 'nnet.cnn.layer.SoftmaxLayer' is implemented in software.
### Notice: The layer 'Output1_softmax' with type 'nnet.cnn.layer.RegressionOutputLayer' is implemented in software.
### Compiling layer group: conv_1>>conv_4 ...
### Compiling layer group: conv_1>>conv_4 ... complete.

### Allocating external memory buffers:

          offset_name          offset_address     allocated_space  
    _______________________    ______________    __________________

    "InputDataOffset"           "0x00000000"     "512.0 kB"        
    "OutputResultOffset"        "0x00080000"     "512.0 kB"        
    "SchedulerDataOffset"       "0x00100000"     "0.0 kB"          
    "SystemBufferOffset"        "0x00100000"     "88.0 kB"         
    "InstructionDataOffset"     "0x00116000"     "52.0 kB"         
    "ConvWeightDataOffset"      "0x00123000"     "28.0 kB"         
    "EndOffset"                 "0x0012a000"     "Total: 1192.0 kB"

### Network compilation complete.

dn = struct with fields:
             weights: [1×1 struct]
        instructions: [1×1 struct]
           registers: [1×1 struct]
    syncInstructions: [1×1 struct]
        constantData: {}
             ddrInfo: [1×1 struct]
       resourceTable: [6×2 table]

deploy(wfObj_int8)

### Programming FPGA Bitstream using Ethernet...
### Attempting to connect to the hardware board at 172.21.88.150...
### Connection successful
### Programming FPGA device on Xilinx SoC hardware board at 172.21.88.150...
### Attempting to connect to the hardware board at 172.21.88.150...
### Connection successful
### Copying FPGA programming files to SD card...
### Setting FPGA bitstream and devicetree for boot...
# Copying Bitstream zcu102_int8.bit to /mnt/hdlcoder_rd
# Set Bitstream to hdlcoder_rd/zcu102_int8.bit
# Copying Devicetree devicetree_dlhdl.dtb to /mnt/hdlcoder_rd
# Set Devicetree to hdlcoder_rd/devicetree_dlhdl.dtb
# Set up boot for Reference Design: 'AXI-Stream DDR Memory Access : 3-AXIM'
### Programming done. The system will now reboot for persistent changes to take effect.
### Rebooting Xilinx SoC at 172.21.88.150...
### Reboot may take several seconds...
### Attempting to connect to the hardware board at 172.21.88.150...
### Connection successful
### Programming the FPGA bitstream has been completed successfully.
### Loading weights to Conv Processor.
### Conv Weights loaded. Current time is 28-Aug-2024 15:59:49

Run Prediction