Clear Filters
Clear Filters

SSD Object Detector training results in NaN loss and RMSE

9 views (last 30 days)
I've create an SSD with mobilenetv2 with the example from "Create SSD Object Detection Network". But changed the class count to just 1.
For training I've used the sample from "Object Detection Using SSD Deep Learning".
| Epoch | Iteration | Time Elapsed | Mini-batch | Mini-batch | Mini-batch | Base Learning |
| | | (hh:mm:ss) | Loss | Accuracy | RMSE | Rate |
| 1 | 1 | 00:00:01 | 3.1220 | 50.98% | 3.39 | 1.0000e-05 |
| 4 | 50 | 00:00:27 | NaN | 0.00% | NaN | 1.0000e-05 |
| 8 | 100 | 00:00:53 | NaN | 0.00% | NaN | 1.0000e-05 |
| 11 | 150 | 00:01:19 | NaN | 0.00% | NaN | 1.0000e-05 |
Is there something I'm missing? Is the SSD model created in the first sample not an actual working model?
Best regards
Link Sample 1:
Edit: I've tried decreasing the learning rate with no success.
Sai Bhargav Avula
Sai Bhargav Avula on 12 May 2020
can you provide more details like what other changes you made to the code ?
can you share the code that you are working on?
Daniel Jampen
Daniel Jampen on 12 May 2020
The only thing I've changed is the line in "Create SSD Object detection network"
numClasses = 5;
numClasses = 1;
I've done this because the "Object Detection using SSD Deep Learning" uses just one class for training. (Vehicle)
When using more anchor boxes with bigger sizes it trains longer without resulting in NaN. But at somepoint NaN will still show up.
To my knowledge tho, anchorbox size and count should only impact accuracy and not resulting in NaN values?

Sign in to comment.

Answers (1)

Ryan Comeau
Ryan Comeau on 10 May 2020
I do not know the exact thing which may be causing this, but if I had to bed on it, I would check all of the bounding boxes in your data set and make sure they are correctly labelling the objects. If you removed 1 class but left the bounding boxes there it could be finding NaN value in this way. Superimpose the bounding boxed on the image and ensure you have the correct labels and locations of these objects.
Second, the lowest recommended learning rate i've seen in literature(i don't have a specific paper to link her unfortunately) is about 1e-6. Low learning rates like this can cause your network to not converge at all since the weights will never be updated enough. What I recommend is use the learn rate drop schedule that is provided here. Here is a sample of what i've used to achive some satisfactory results.
options = trainingOptions('sgdm',...
'LearnRateSchedule','piecewise', ...
'LearnRateDropFactor',0.7, ...
'LearnRateDropPeriod',1, ...
Third, to further gain performance, tune your strides and sizes of convolution kernels, you'll need to adjust this to your specific task i cannot help here.
Hope this helps,
  1 Comment
Daniel Jampen
Daniel Jampen on 12 May 2020
Thank you for your insights.
Since the Sample in "Link Sample 2:" does work when training with
lgraph = ssdLayers(inputSize, numClasses, 'resnet50');
I assume the training data set is correct.
I will make some test with your other points.
Thank you very much.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!