NN accuracy on test set low

I have implemented a neural network in Matlab R2013a for character recognition. I have used trainbr function for nn training. 80% samples were used for training and the rest for testing. When i plot the confusion matrix, i get 100% accuracy on the training set. But for the test set the accuracy is very low(around 60%). What could be possibly wrong?

Answer 1

[Greg Heath]

See my example at

http://www.mathworks.com/matlabcentral/newsreader/view_thread/334932#919949

Hope this helps.

Greg

Answer 2

[Greg Heath]

Insufficient info:

How many characters?

How many examples for each character?

What are the dimensions of the input and target matrices?

Are the summary statistics of the training and test subsets sufficiently similar?

How many input, hidden and output nodes?

What values of hidden nodes did you try ?

How many random weight initializations for each value ?

Although trainbr should mitigate the effect of using more hidden nodes than are needed, you still need many trials to establish sufficient confidence intervals.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Anitha]

I am using 13 characters. There are a total of 234 characters in the database. 18 for each class.

The dimension of input matrix is 18 x 234.

The dimension of target matrix is 13 x 234

Number of input nodes=18 Number of hidden nodes=30( I have tried 10,15,20 also.But without crossvalidation got better result with 30 hidden nodes) Number of output nodes=13

[Greg Heath]

Sorry, that does not make sense to me. Consider the following

 13 characters of the alphabet A-to-M
 234 examples, 18 for each character
 All characters are columnized 8x5 images

 size(input) = [ 40 234]
 size(target) = [ 13 234] % columns of eye(13)

Where did I go wrong?

[Anitha]

I have extracted 18 features for each of the characters. So

size(input)=[18 234] and since there are 13 characters,

size(target)=[13 234]

[Greg Heath]

 [ I N ] = [ 18 234]
 [ O N ] = [ 13 234]

The default trn/val/tst split for trainbr is 0.8/0.0/0.2. The resulting number of training equations is

 Ntrn   = N - round(0.2*N) % 187
 Ntrneq = Ntrn*O           % 2431

With H=30 hidden nodes, the number of unknown weights is

 Nw = (I+1)*H+(H+1)*O    % 973

The ratio is

r = Ntrneq/Nw % ~2.5

Which should be ok for trainbr.

I suggest making multiple designs (20?) in a loop with different mixes of training examples, testing examples and initial weights. For examples of multiple designs in a loop, search using

 greg Ntrials

Post your code if you still have problems.

[Anitha]

I have done the code as you said with multiple designs.Now i am getting good results. After running 10 trials, i get best test set performance in trial 8. Now i want to train my final neural network with the parameters of this trial 8. But the problem is that, I have saved the rng state but when I use the same state for traing my net again, I get different results. This is my code.

clear all, close all, clc, plt=0;
 tic
load inputdata
load targetdata
 x=input;
 t=target;
hiddenLayerSize = 30;
net = patternnet(hiddenLayerSize);
net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};
net.outputs{2}.processFcns = {'removeconstantrows','mapminmax'};
net.trainFcn='trainbr';
net.trainParam.epochs=100;
 rng(4151941)     
 Ntrials=10
 for i = 1:Ntrials
   s{i} = rng;            
   [ net tr y e] = train(net,x,t);
    mseval(i) = tr.best_tperf;  % Best mseval over all epochs of ith run
     tstind = tr.testInd;
         ytst = y(:,tstind);
         ttst = t(:,tstind);
         plt=plt+1,figure(plt)
         plotconfusion(ttst,ytst)
          title([ ' TEST SET CONFUSION MATRIX. TRIAL =  ', num2str(i)] )
         hold off
 end
 [ minmseval ibest ] = min(mseval);
 rng     = s{ibest};            % For repeating the best design
  bestnet = configure(net,x,t);
  bestIW0 = bestnet.IW
  bestb0  = bestnet.b
  bestLW0 = bestnet.LW
 [ bestnet tr y e] = train(net,x,t) 
         tstind = tr.testInd;
         ytst = y(:,tstind);
         ttst = t(:,tstind);
         plt=plt+1,figure(plt)
         plotconfusion(ttst,ytst)
         title([ ' OVERALL TEST SET CONFUSION MATRIX=  '] )
         hold off
         save net
         view(bestnet)

[Greg Heath]

Two mistakes

1. No configure statement in the loop

2. Used net instead of bestnet in the last train statement

Answer 3

[Greg Heath]

 1. Not necessary to specify default process functions.
 2. How did you know my birthdate is 4151941 ??
 3. You are reusing the same net for each trial without using CONFIGURE. 
    Therefore, the initial weights of each trial are the final weights of the last trial. 
    I suspect that if the design results are not monotonically better it is because TRAIN is
    using a new trn/tst division. 
 4. Use configure after the RNG initialization.
 5. An alternate approach is to CONTINUALLY save one or all of
      a. the best current RNG state
      a. the best current net
      b. the best current Wb = getwb(net)
 6. I think you should do all three at the same time and compare results

Hope this helps.

Thank you for formally accepting my answer

Greg

[Anitha]

Hi Greg,

I used an example from your previous post. That is why i set the rng as 4151941. I have modified the code as you said with configure after initialization. I got the best result in trial 4. But when i try to train the final network with these parameters, i still get different results. The weights and bias are all different as well. The confusion matrix is also different. What shall i do?

clear all, close all, clc, plt=0;
 tic
load inputdata
load targetdata
 x=input;
 t=target;
hiddenLayerSize = 30;
net = patternnet(hiddenLayerSize);
net.trainFcn='trainbr';
net.trainParam.epochs=100;
rng(4151941);     
 Ntrials=15
 for i = 1:Ntrials
   s{i} = rng;
   net  = configure(net,x,t);
   netIW0{i} = net.IW
   netb0{i}  = net.b
   netLW0{i} = net.LW
   [ net tr y e] = train(net,x,t);
      % Best mseval over all epochs of ith run
         tstind = tr.testInd;
         ytst = y(:,tstind);
         ttst = t(:,tstind);
         %mseval(i) = mse(net,ttst,ytst)
         mseval(i)=tr.best_tperf;
         plt=plt+1,figure(plt)
         plotconfusion(ttst,ytst)
          title([ ' TEST SET CONFUSION MATRIX. TRIAL =  ', num2str(i)] )
         hold off
 end
 [ minmseval ibest ] = min(mseval);
 rng=s{ibest};            % For repeating the best design
  bestnet = configure(net,x,t);
  bestIW0 = bestnet.IW
  bestb0  = bestnet.b
  bestLW0 = bestnet.LW
 [ bestnet tr y e] = train(net,x,t);
         msetst=tr.best_tperf;
         tstind = tr.testInd;
         ytst = y(:,tstind);
         ttst = t(:,tstind);
         fWb=getwb(bestnet);
         %msetst= mse(bestnet,ttst,ytst)
         plt=plt+1,figure(plt)
         plotconfusion(ttst,ytst)
         title([ ' OVERALL TEST SET CONFUSION MATRIX=  '] )
         hold off
         save bestnet
         view(bestnet)

[Greg Heath]

The second train statement contains net instead of bestnet

Answer 4

[Anitha]

Hi greg, Thanks a lot..Got it now..

[Greg Heath]

See my second post in the MATLAB Central thread.

http://www.mathworks.com/matlabcentral/newsreader/view_thread/334932#919949