Mini-Batch loss and accuracy trends

Hi,

I'm training a Fast RCNN detector for the first time. I've got a data set of 3000 images with about 3 label per image.

The training gives me the following results:

I would ask why the Mini-batch loss and the Mini-batch accuracy have trands that go up and down sharply and can't settle around fix values.

Below my training options:

opts = trainingOptions('adam',...
    'InitialLearnRate', 0.000001, ...
    'LearnRateSchedule', 'piecewise', ...
    'LearnRateDropFactor', 0.1, ...
    'LearnRateDropPeriod', 8, ...
    'MaxEpochs', 10, ...
    'MiniBatchSize', 32, ...
    'Verbose', true);

Could it be the initial learning rate too high or the number of epoch too low?

Thank you!

Answer 1

[Kritika Bansal]

Hi,

As you said you’re using Fast R-CNN for the first time, the example given on https://www.mathworks.com/help/vision/ref/trainfastrcnnobjectdetector.html#bvkk00v-6 page can help you understand the role of various inputs going into trainingOptions function. There can be many probable reasons behind your network not getting converged.

Since you are training your network using ‘adam’ solver, consider using the other training options associated with it like GradientDecayFactor and SquaredGradientDecayFactor. Also, consider training your network using other methods available in MATLAB R2016a and above, like sgdm and rmsprop. You can learn more about the solver options available to you in the link below:

https://www.mathworks.com/help/deeplearning/ref/trainingoptions.html#bu80qkw-3_head

Also, check to see if the images in your training data do not contain excessive NAN values, as too many NaNs may lead to propagation of NaNs through the network and hence the training will fail to converge.

As far as your initial learning rate in considered, 10^-6 seems to be very small, and with every 10 epochs it is further decreasing. An initial learning rate of 10^-3 may be of help in your case.