Code Generation and Deployment for Acoustic-Based Machine Fault Detection Using Deep Learning on ARM Cortex-M Hardware

Since R2024b

This example uses:

This example demonstrates code generation for acoustics-based machine fault recognition using a long-short-term-memory (LSTM) network and spectral descriptors. This example uses Embedded Coder™ with deep learning support to generate a "processor-in-the-loop" (PIL) executable function that leverages C code on ARM® Cortex-M hardware. The input data consists of acoustic time-series recordings from a faulty or healthy air compressors, and the output is the state of the mechanical machine predicted by the LSTM network. For details on audio preprocessing and network training, see the Acoustics-Based Machine Fault Recognition (Audio Toolbox).

Prepare Input Data Set

Specify a sample rate fs of 16 kHz and a windowLength of 512 samples, as defined in the Acoustics-Based Machine Fault Recognition (Audio Toolbox). Set numFrames to 100.

fs = 16000;
windowLength = 512;
numFrames = 100;

To run the example on a test signal, generate a pink noise signal. To test the system on a real data, download the air compressor data set [1].

downloadDataset = true;;

if ~downloadDataset
    pinkNoiseSignal = pinknoise(windowLength*numFrames,"single");
else
    % Download AirCompressorDataset.zip 
    component = "audio";
    filename = "AirCompressorDataset/AirCompressorDataset.zip";
    localfile = matlab.internal.examples.downloadSupportFile(component,filename);

    % Unzip the downloaded zip file to the download folder
    downloadFolder = fileparts(localfile);
    if ~exist(fullfile(downloadFolder,"AirCompressorDataset"),"dir")
        unzip(localfile,downloadFolder)
    end
    
    % Create an audioDatastore object to manage the data.
    dataStore = audioDatastore(downloadFolder,IncludeSubfolders=true,LabelSource="foldernames",OutputDataType="single");

    % Use countEachLabel to get the number of samples of each category in the data set.
    countEachLabel(dataStore)
end

ans=8×2 table
      Label      Count
    _________    _____

    Bearing       225 
    Flywheel      225 
    Healthy       225 
    LIV           225 
    LOV           225 
    NRV           225 
    Piston        225 
    Riderbelt     225

Initialize Test Signal and Load the Network

Create one dsp.AsyncBuffer (DSP System Toolbox) object to read audio in streaming mode and another dsp.AsyncBuffer (DSP System Toolbox) object to accumulate scores for MATLAB® and PIL methods.

audioSource = dsp.AsyncBuffer;
scoreBufferMATLAB = dsp.AsyncBuffer;
scoreBufferPIL = dsp.AsyncBuffer;

Get the labels corresponding to the network outputs.

model = load("AirCompressorFaultRecognitionModel.mat");
labels = string(model.labels);

Set signalToBeTested to pinkNoiseSignal or set signalToBeTested to test the file of your choice from the drop-down list.

if ~downloadDataset
    signalToBeTested = pinkNoiseSignal;
else
    [allFiles,~] = splitEachLabel(dataStore,1);
    allData = readall(allFiles);
    signalToBeTested = allData(3);
    signalToBeTested = cell2mat(signalToBeTested);
end

Detect Machine Fault in MATLAB

To run the streaming classifier in MATLAB®, use the pretrained network in AirCompressorFaultRecognition.mat and wrapper file recognizeAirCompressorFault.m developed in Acoustics-Based Machine Fault Recognition (Audio Toolbox).

Stream one audio frame at a time to simulate deploying the system in a real-time embedded system. Use the recognizeAirCompressorFault developed in Acoustics-Based Machine Fault Recognition (Audio Toolbox) to compute audio features and perform deep learning classification.

write(audioSource,signalToBeTested);
resetNetworkState = true;

while audioSource.NumUnreadSamples >= windowLength

    % Get a frame of audio data
    x = read(audioSource,windowLength);

    % Apply the streaming classifier function
    score = recognizeAirCompressorFault(x,resetNetworkState);
    
    % Store the score for analysis
    write(scoreBufferMATLAB,extractdata(score)');
    
    resetNetworkState = false;
end

Compute the recognized fault from the scores, and display it.

scoresMATLAB = read(scoreBufferMATLAB);
[~,labelIndex] = max(scoresMATLAB(end,:),[],2);
detectedFault = labels(labelIndex)

detectedFault = 
"Healthy"

Plot the scores of each label for each frame.

plot(scoresMATLAB)
legend("" + labels,Location="northwest") 
xlabel("Time Step")
ylabel("Score")
str = sprintf("Predicted Scores Over Time Steps.\nPredicted Class: %s",detectedFault);
title(str)

Detect Machine Fault on Hardware using PIL

Hardware Requirements:

Cortex-M™ hardware such as STM32F769I-Discovery board.
In this case, the network requires a flash memory of 570 KB. The board used in this example comes equipped with a 2 MB flash memory capacity, which exceeds the network's flash memory requirements. The memory requirement for the network is determined by adding together the inputs and learnables of the network. This can be examined using the analyzeNetwork function.
Please ensure that the board's flash size is more than the network's requirements to avoid flash overflow error during code generation.

Generate PIL Executable for Cortex-M Hardware

To generate a PIL MEX function, create a coder.config object for a static library and set the verification mode to "PIL". Set the target language to C.

cfg = coder.config("lib","ecoder",true);
cfg.VerificationMode = "PIL";
cfg.TargetLang = "C";

Create a coder.hardware object for the STM32F769I-Discovery board. Set the hardware property of the coder.config object cfg to coder.hardware object hw. In the following code, If using hardware other than the STM32F769I-Discovery board, replace the "STM32F769I-Discovery" with the connected hardware and replace "COM5" with the port number to which you have connected the Cortex-M hardware.

hardware = "STM32F769I-Discovery";
hw = coder.hardware(hardware);
cfg.Hardware = hw;
cfg.Hardware.PILCOMPort = "COM5";
cfg.HardwareImplementation.TargetHWDeviceType = "ARM Compatible->ARM Cortex-M";

Create a deep learning configuration object and configure the deep learning config target library to none which generates C Code. Set the code replacement library to "ARM Cortex-M" for more optimizations.

cfg.DeepLearningConfig = coder.DeepLearningConfig("TargetLibrary","none");
cfg.CodeReplacementLibrary = "ARM Cortex-M";

Enable the code execution profile flag to obtain timing details.

cfg.CodeExecutionProfiling = true;

Create an audio data frame of length windowLength.

audioFrame = ones(windowLength,1,"single");

Call the codegen function to generate C code for the recognizeAirCompressorFault function. Specify the configuration object and prototype arguments. Check your current folder to see the generated PIL file recognizeAirCompressorFault_pil.

codegen -config cfg recognizeAirCompressorFault -args {audioFrame,resetNetworkState} -report

### Connectivity configuration for function 'recognizeAirCompressorFault': 'STM32 Microcontroller'
### COM port: COM5
### Baud rate: 115200
Code generation successful: View report

Detect Machine Fault on ARM Cortex-M Hardware

Stream one audio frame at a time to to simulate deploying the system in a real-time embedded system. Use the generated recognizeAirCompressorFault_pil file to compute audio features and perform deep learning classification.

release(audioSource);
write(audioSource,signalToBeTested);
resetNetworkState = true;
while audioSource.NumUnreadSamples >= windowLength

    % Get a frame of audio data
    x = read(audioSource,windowLength);

    % Apply the streaming classifier function
    score = recognizeAirCompressorFault_pil(x,resetNetworkState);
    
    % Store the score for analysis
    write(scoreBufferPIL,extractdata(score)');

    resetNetworkState = false;
end

### Connectivity configuration for function 'recognizeAirCompressorFault': 'STM32 Microcontroller'
### COM port: COM5
### Baud rate: 115200
### Starting application: 'codegen\lib\recognizeAirCompressorFault\pil\recognizeAirCompressorFault.elf'
    To terminate execution: clear recognizeAirCompressorFault_pil
### Downloading executable to the hardware on Drive: G:
C:\Users\bhpatray\OneDrive - MathWorks\Documents\MATLAB\ExampleManager\bhpatray.latestBdoc\deeplearning_shared-ex42361478\codegen\lib\recognizeAirCompressorFault\pil\recognizeAirCompressorFault.bin 
1 File(s) copied 
    Execution profiling data is available for viewing. Open Simulation Data Inspector.
    Execution profiling report will be available after termination.

Compute the recognized fault from the scores and display it.

scoresPIL = read(scoreBufferPIL);
[~,labelIndex] = max(scoresPIL(end,:),[],2);
detectedFault = labels(labelIndex)

detectedFault = 
"Healthy"

Plot the scores of each label for each frame.

plot(scoresPIL)
legend(labels,Location="northwest")
xlabel("Time Step")
ylabel("Score")
str = sprintf("Predicted Scores Over Time Steps.\nPredicted Class: %s",detectedFault);
title(str)

Evaluate PIL Performance

Obtain the execution profiling report after terminating the PIL.

clear recognizeAirCompressorFault_pil;

    Execution profiling report: coder.profile.show(getCoderExecutionProfile('recognizeAirCompressorFault'))

profileInfo = getCoderExecutionProfile('recognizeAirCompressorFault');

Running the PIL for a single instance takes 1,153,450 ticks, and with the board's clock speed set at 216 MHz, this translates to an approximate time of 5.3 milliseconds.

clockSpeed = profileInfo.TimerTicksPerSecond;
disp(["ClockSpeed :" clockSpeed]);

    "ClockSpeed :"    "216000000"

[initiationTicks, executionTicks, terminationTicks] =  profileInfo.Sections.MaximumExecutionTimeInTicks;
timeTakenForInstance = seconds(single(executionTicks)/single(clockSpeed));
disp(timeTakenForInstance);

   0.0053463 sec

Compare PIL and MATLAB Machine Fault Detection Scores

Compare the outputs of MATLAB and the PIL and check if they are within the tolerance value of one another.

checkWithinTol = ismembertol(scoresMATLAB,scoresPIL,1e-5);
isVerificationPassed = isequal(checkWithinTol,ones(size(scoresMATLAB)))

isVerificationPassed = logical
   1

Further Exploration

You can change the signalToBeTested label to 'bearing' or 'piston' and measure the accuracy of other labels.
You can also generate the signal using pinkNoise and set the downloadDataset to false to test the example with random signal.
The compatible hardware can be set to hardware in "Generate PIL Executable for Cortex-M Hardware" section. Ensure hardware flash memory is more than the memory requirement of network. The memory requirement for the network is determined by adding together the inputs and learnables of the network. This can be examined using the analyzeNetwork function.
To attain real time requirements, change the architecture of the network and train the network by referring to Acoustics-Based Machine Fault Recognition (Audio Toolbox) (or) use hardware which can meet network requirements.

References

[1] Verma, Nishchal K., Rahul Kumar Sevakula, Sonal Dixit, and Al Salour. "Intelligent Condition Based Monitoring Using Acoustic Signals for Air Compressors." IEEE Transactions on Reliability 65, no. 1 (March 2016): 291–309. https://doi.org/10.1109/TR.2015.2459684.

How useful was this information?

Unrated 1 star 2 stars 3 stars 4 stars 5 stars