MATLAB Answers

How to check the robustness of the Neural network model?

25 views (last 30 days)
Le Truong An
Le Truong An on 13 Jul 2019
Commented: Le Truong An on 14 Jul 2019
Hi everyone,
I need your help for my project.
  • I used Neural Pattern Recognition app, and have already built an NN model for classification with 4 labels. The NN model worked very well. Accuracy classification reaches more than 90%.
  • However, when I want to check this model with new data (new data = the original data through an AWGN channel having a 10 dB signal-to-noise ratio (SNR). The classification result is always less than 30% accuracy.
  • How to create an augmented dataset and train it?
  • How to check the robustness of the Neural network model?
Please help me!!!
Code to create a neural network:
% Solve a Pattern Recognition Problem with a Neural Network
% Script generated by Neural Pattern Recognition app
%% Load data
% X - input data.
% Y - target data.
load('mydata.mat')
x = X;
t = Y;
%% Choose a Training Function
trainFcn = 'trainscg'; % Scaled conjugate gradient backpropagation.
%% Create a Pattern Recognition Network
hiddenLayerSize = 50;
net = patternnet(hiddenLayerSize, trainFcn);
% Choose Input and Output Pre/Post-Processing Functions
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};
% Setup Division of Data for Training, Validation, Testing
net.divideFcn = 'dividerand'; % Divide data randomly
net.divideMode = 'sample'; % Divide up every sample
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;
% Choose a Performance Function
net.performFcn = 'crossentropy'; % Cross-Entropy
% Choose Plot Functions
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
'plotconfusion', 'plotroc'};
%% Train the Network
[net,tr] = train(net,x,t);
%% Test the Network
y = net(x);
e = gsubtract(t,y);
performance = perform(net,t,y)
tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);
%% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y)
valPerformance = perform(net,valTargets,y)
testPerformance = perform(net,testTargets,y)
%% View the Network and plots confusion matrix
view(net)
figure, plotconfusion(t,y)
save ('mynet');
Code to check neural network with data added white noise:
clc;
clear all;
%load
load('mynet');
%% add Gaussian white noise SNR=10dB
load('mydata.mat')
X10dB = awgn(X,10,'measured');
x = X10dB; % train data
t = Y; % target
%% Test the Network
y = net(x);
e = gsubtract(t,y);
performance = perform(net,t,y)
tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);
% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y)
valPerformance = perform(net,valTargets,y)
testPerformance = perform(net,testTargets,y)
%% Plots
figure, plotconfusion(t,y)

  0 Comments

Sign in to comment.

Accepted Answer

Greg Heath
Greg Heath on 14 Jul 2019
Edited: Greg Heath on 14 Jul 2019
If you are going to test with white noise, include white noise in your design (i.e., training + validation)
Then, given a fixed input level of white noise for design (i.e., design + noise1) you can obtain individual performance measures of training, validation and test as a function of added noise level.
Hope this helps,
Thank you for formally accepting my answer
Greg

  1 Comment

Le Truong An
Le Truong An on 14 Jul 2019
Thank for your help!
I tried to follow your help.
Step 1: I built a Neural Network model with new dataset = original data add white noise (SNR = 10dB). And then I saved NN model.
Step 2: Test that NN model as follows
And I got better results than my previous results.
But I'm not sure. I did the right thing according to your instructions.
Could you please explain it clearly?
Thanks!
testNNmodel.png

Sign in to comment.

More Answers (0)