Generate HDL Code from Frame-Based Models by Using Neighborhood Modeling Methods

If you have a frame-based model, you can model an optical flow algorithm by using multiple neighborhood processing methods. In this example, you use the HDL Coder™ frame-to-sample conversion optimization to generate synthesizable pixel-based HDL code from two frame-based models that demonstrate different modeling patterns.

This example shows two different modeling patterns for a neighborhood processing algorithm for optical flow. One of the patterns uses a MATLAB Function block with the neighborhood processing function hdl.npufun. The other pattern uses the Neighborhood Processing Subsystem block.

Model Algorithm with MATLAB Function Block

In the hdlFrame_OpticalFlow_LK_MLFB model, there is a single MATLAB Function block inside the device under test (DUT) that contains neighborhood processing operations needed for the Lucas Kanade method for optical flow. To generate synthesizable HDL code from the frame-based model, HDL Coder uses the frame-to-sample conversion supported function, hdl.npufun to create a streaming sample-based neighborhood processing algorithm. Using this function in a frame-based model enables you to contain the entire image processing algorithm in a single MATLAB Function block.

Open the hdlFrame_OpticalFlow_LK_MLFB/DUT_LK/MATLAB Function LK subsystem to see the optical flow algorithm.

load_system("hdlFrame_OpticalFlow_LK_MLFB");
open_system("hdlFrame_OpticalFlow_LK_MLFB/DUT_LK/MATLAB Function LK");

function flowVector  = opticalFlowForHDL_lk(I, Idelay)
% Implements the Lucas Kanade method for optical flow

%   Copyright 2021-2023 The MathWorks, Inc.

Ix = hdl.npufun(@imfilter_ComputIx, [1 5], I);
Iy = hdl.npufun(@imfilter_ComputIy, [5 1], I);
It = I - Idelay;

WIxx = hdl.npufun(@imfilter_kernel, [5 5], Ix.*Ix);
WIxy = hdl.npufun(@imfilter_kernel, [5 5], Ix.*Iy);
WIyy = hdl.npufun(@imfilter_kernel, [5 5], Iy.*Iy);
WIxt = hdl.npufun(@imfilter_kernel, [5 5], Ix.*It);
WIyt = hdl.npufun(@imfilter_kernel, [5 5], Iy.*It);

[Vx, Vy] = hdl.npufun(@calc_roots_pixel, [1 1], WIxx, WIxy, WIyy, WIxt, WIyt);
flowVector = complex(Vx,Vy);

end

function WI_1x1 = imfilter_ComputIx(I_1x5)

coder.inline('always');

d5 = [-1 8 0 -8 1]/12;
d5 = fliplr(d5(:)');
[WI_1x1] = sum(d5 .* I_1x5);

end

function WI_1x1 = imfilter_ComputIy(I_5x1)

coder.inline('always');

d5 = [-1 8 0 -8 1]/12;
d5 = flipud(d5(:));
[WI_1x1] = sum(d5(:) .* I_5x1);

end

function WI_1x1 = imfilter_kernel(I_5x5)

coder.inline('always');

p5 = [1 4 6 4 1]/16;
W = p5(:)*p5(:)';
[WI_1x1] = sum(W(:) .* I_5x5(:));

end

function [VxPix, VyPix] = calc_roots_pixel(aPix, bPix, cPix, ...
    txPix, tyPix)

coder.inline('always');

r = aPix + cPix;
s = sqrt(single(4 * bPix * bPix + (aPix - cPix) * (aPix - cPix)));
eig1 = (single(r) + s);
eig2 = (single(r) - s);

vx = single(0);
vy = single(0);
thresh = 0.0039;
thresh2 = 0;

if (eig1 > thresh && eig2 > thresh)
    d = bPix * bPix - aPix * cPix;
    iDelta = 1/d;
    vx =  single( -(tyPix * bPix - txPix * cPix)*iDelta);
    vy =  single( -(txPix * bPix - aPix * tyPix)*iDelta);
elseif (eig1 > thresh && eig2 < thresh)
    rr = bPix+aPix;
    cc = cPix+bPix;
    norm = cc*cc + rr*rr;

    if (norm > thresh2)
        invnorm = 1/norm;
        tmp = -(txPix + tyPix) * invnorm;
        vx =  single(rr * tmp);
        vy =  single(cc * tmp);
    end
end

VxPix = vx;
VyPix = vy;

end

Run the Model

The input port of the DUT, Frame, connects to a 2-D matrix that contains the data for the current frame. This input signal is a frame composed of 360x640 pixels. Within the DUT, a persistent variable is used to store the value of the input and read it on the next cycle. This delayed data is also used in the optical flow algorithm. Simulate the model to see the frame size and simulation results of the optical flow output.

sim("hdlFrame_OpticalFlow_LK_MLFB");

Generate HDL Code

Generate synthesizable HDL code by using the frame-to-sample conversion. Set the HDL block property ConvertToSamples on the Inport block of the DUT to convert the input signal from frame-based to sample-based input.

hdlset_param('hdlFrame_OpticalFlow_LK_MLFB/DUT_LK/Frame','ConvertToSamples','on');

For the MATLAB Function block that contains the optical flow algorithm, set the HDL block property Architecture to MATLAB Datapath. Enable the frame-to-sample conversion optimization and generate HDL code using the makehdl command. For more information on the frame-to-sample conversion optimization, see HDL Code Generation from Frame-Based Algorithms.

hdlset_param('hdlFrame_OpticalFlow_LK_MLFB/DUT_LK/MATLAB Function LK','Architecture','MATLAB Datapath');
hdlset_param('hdlFrame_OpticalFlow_LK_MLFB','FrameToSampleConversion','on');

makehdl('hdlFrame_OpticalFlow_LK_MLFB/DUT_LK');

The frame-to-sample conversion separates the frame-based inputs into sample, valid, and ready signals for a sample-based hardware-targeted interface.

Model Algorithm with the Neighborhood Processing Subsystem Block

In the hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem model, the DUT contains multiple Neighborhood Processing Subsystem blocks that contain the neighborhood processing operations needed for the Lucas Kanade optical flow algorithm. Separating the optical flow algorithm into various Simulink blocks enables you to visualize and model the different aspects of the algorithm in a more modular way than with a single MATLAB Function block.

Open the hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem/DUT subsystem to see the optical flow algorithm.

load_system("hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem");
open_system("hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem/DUT");

Run the Model

The input port of the DUT, Frame, connects to a 2-D matrix that contains the data for the current frame. This input signal is a frame composed of 360x640 pixels. A Delay block is used to store a full frame of data so that the previous frame's values can also be used in the optical flow algorithm. Simulate the model to see the frame size and simulation results of the optical flow output.

sim("hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem");

Although the hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem and hdlFrame_OpticalFlow_LK_MLFB models differ in design, the output is the same. Both models can also generate synthesizable HDL code for sample-based hardware.

Generate HDL Code

Generate synthesizable HDL code by using the frame-to-sample conversion. Set the HDL block property ConvertToSamples on the Inport block of the DUT to convert the input signal from frame-based to sample-based input.

hdlset_param('hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem/DUT/Frame','ConvertToSamples','on');

Enable the frame-to-sample conversion optimization and generate HDL code using the makehdl command.

hdlset_param('hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem','FrameToSampleConversion','on');

makehdl('hdlFrame_OpticalFlow_LK_NeighborProcessingSubsystem/DUT');

The frame-to-sample conversion separates the frame-based inputs into sample, valid, and ready signals for a sample-based hardware-targeted interface.