Training loss considerably lower than validation loss in regression network
3 views (last 30 days)
Show older comments
So I succesfully transfer learnied Alexnet to classify an image on the precense of manhole covers or not. After that I used the convolutinoal layers from this network to train a regression network that determines the position of the manhole in the image. However during training the RMSE of the validation and training set are prety much the same but it is the validation loss that is considerable lower than the training loss. I used a custom loss function in my regression layer and training and validation are split 70/30.
I started looking on the internet what could cause this as you would expect the validation loss to be higher or similar as the training loss. On the one hand, somebody mentioned that this could be caused by the validation set being too easy compared to the training set. But even after reshuffeling the training and validation sets from a different random seed point the same thing happend. On the other hand, regularisation in the form of drop-out layers could cause this as they are activeated during training but not during validation. But this is also not the case as my modified alexnet network only consists of the convolutional, max pooling layers and a fully connected layer at the end.
Additionally I manually calculated the loss of the validation and training dataset after training. The validation loss was the same as the final validation loss during training. However, the training loss all of a sudden was lower than the last training loss during training and also lower than the validation loss. This is as expected but why is the loss not correctly computed during training. I don't think this is a problem with my custom loss function as previously I trained with the default loss function of matlab and saw the same thing.
Training loss during training = 0.4002
Training loss after training (manually computed) = 0.0421
Validation loss during training = 0.0720
Validation loss after training (manually computed) = 0.0720
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/273943/image.png)
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/273944/image.png)
3 Comments
Roland Kruse
on 18 Jan 2021
Strangely, I have just the opposite problem: while the validation losses agree the training (mini-batch) loss printed during training is much lower than the one I manually calculate afterwards.
Answers (0)
See Also
Categories
Find more on Image Data Workflows in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!