Save intermediate model in matlab while training a deep learning model and resume training from that stage later

5 views (last 30 days)
Hi,
I am training a 3 pipeline deep learning model in matlab which takes a lot of time to train. I need to store intermediate variable values while training, stop the training process and then resume training at a later point from the stage at which the training was stopped previously. Does Matlab have any options to do this? Any help in this regard would be highly appreciated.
Thanks in Advance

Accepted Answer

Mahesh Taparia
Mahesh Taparia on 15 Oct 2021
Hi
You can set the checkpoint path in trainingOptions as suggested in the above answer. The trained weights will be saved into the specified path as a mat file. To resume the training process, you can load those weights in the net variable and start the training process. For example, you can refer this documentation.
Hope it will help!
  1 Comment
Jeet Agrawal
Jeet Agrawal on 13 Apr 2023
Can you help me understanding load model more.
Do i need to create another file and load model and train from their?
suppose i am saving model after 50 iteration using
checkpointfrequency =50 and checkpointunit = iteration
lastly it has saved checkpoint at 750 now how to run again from 751?
As per link i can reduce my epoch but how can i reduce iteration?
suppose my data set has 20000 files and minibatchsize is 40 for one epoch it will take 500 iteration. I am not able to wait for 500 iteration to complete because it will take around 5 hours. so, want to save at 50 epochs only.
could you please share snipet?

Sign in to comment.

More Answers (1)

yanqi liu
yanqi liu on 15 Oct 2021
sir,may be use CheckpointPath,such as
options = trainingOptions('sgdm', ...
'MaxEpochs', 5, ...
'MiniBatchSize', 1, ...
'InitialLearnRate', 1e-3, ...
'CheckpointPath', tempdir);

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!