MATLAB Answers

0

How to check the robustness of the Neural network model?

Asked by Le Truong An on 13 Jul 2019
Latest activity Commented on by Le Truong An on 14 Jul 2019
Hi everyone,
I need your help for my project.
  • I used Neural Pattern Recognition app, and have already built an NN model for classification with 4 labels. The NN model worked very well. Accuracy classification reaches more than 90%.
  • However, when I want to check this model with new data (new data = the original data through an AWGN channel having a 10 dB signal-to-noise ratio (SNR). The classification result is always less than 30% accuracy.
  • How to create an augmented dataset and train it?
  • How to check the robustness of the Neural network model?
Please help me!!!
Code to create a neural network:
% Solve a Pattern Recognition Problem with a Neural Network
% Script generated by Neural Pattern Recognition app
%% Load data
% X - input data.
% Y - target data.
load('mydata.mat')
x = X;
t = Y;
%% Choose a Training Function
trainFcn = 'trainscg'; % Scaled conjugate gradient backpropagation.
%% Create a Pattern Recognition Network
hiddenLayerSize = 50;
net = patternnet(hiddenLayerSize, trainFcn);
% Choose Input and Output Pre/Post-Processing Functions
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};
% Setup Division of Data for Training, Validation, Testing
net.divideFcn = 'dividerand'; % Divide data randomly
net.divideMode = 'sample'; % Divide up every sample
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;
% Choose a Performance Function
net.performFcn = 'crossentropy'; % Cross-Entropy
% Choose Plot Functions
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
'plotconfusion', 'plotroc'};
%% Train the Network
[net,tr] = train(net,x,t);
%% Test the Network
y = net(x);
e = gsubtract(t,y);
performance = perform(net,t,y)
tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);
%% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y)
valPerformance = perform(net,valTargets,y)
testPerformance = perform(net,testTargets,y)
%% View the Network and plots confusion matrix
view(net)
figure, plotconfusion(t,y)
save ('mynet');
Code to check neural network with data added white noise:
clc;
clear all;
%load
load('mynet');
%% add Gaussian white noise SNR=10dB
load('mydata.mat')
X10dB = awgn(X,10,'measured');
x = X10dB; % train data
t = Y; % target
%% Test the Network
y = net(x);
e = gsubtract(t,y);
performance = perform(net,t,y)
tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);
% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y)
valPerformance = perform(net,valTargets,y)
testPerformance = perform(net,testTargets,y)
%% Plots
figure, plotconfusion(t,y)

  0 Comments

Sign in to comment.

1 Answer

Answer by Greg Heath
on 14 Jul 2019
Edited by Greg Heath
on 14 Jul 2019
 Accepted Answer

If you are going to test with white noise, include white noise in your design (i.e., training + validation)
Then, given a fixed input level of white noise for design (i.e., design + noise1) you can obtain individual performance measures of training, validation and test as a function of added noise level.
Hope this helps,
Thank you for formally accepting my answer
Greg

  1 Comment

Thank for your help!
I tried to follow your help.
Step 1: I built a Neural Network model with new dataset = original data add white noise (SNR = 10dB). And then I saved NN model.
Step 2: Test that NN model as follows
And I got better results than my previous results.
But I'm not sure. I did the right thing according to your instructions.
Could you please explain it clearly?
Thanks!
testNNmodel.png

Sign in to comment.