Deep learning using CNN - question about training.

I have a question about a CNN I'm training. during the training everything goes well, with high accuracy results on training and validation sets. however once I stop training the model, the final model has much much inferior accuracy returned to me (see the image below).

Now I have two questions: 1) obviously, why does it happen? I'm expecting to get a model with on par performance on both training and validation sets, with that of the plot. 2) After I manually stop the training process to finish the training at a certain point, it takes quite a long time for the model to be returned (anywhere between 5 to 50 mins w.r.t the size of the data and other parameters such as depth of the network and etc.). Why does this happen?

Answer 1

[Johannes Bergstrom]

Does you network use batch normalization layers?

After training finishes, trainNetwork loops thorugh the whole data set to calculate the batch normalization statistics required to create a network ready for prediction. This answers your question 2) - why it takes so long.

But also, you train for much less than one epoch, which means that the data seen so far by the network might not be representative of the training nor validation sets.

Try making sure that you

1. Shuffle the training data (see 'Shuffle')

2. Train for multiple epochs

Finally, looking at the large 'bump' in the training loss for the very first 10 iterations, it seems like your learning rate is too high.