How to manage RAM usage when running trainACFObjectDetector()

In a project I am working on, I am training an object detector using the Oxford Hand Dataset in order to detect a hand in an image.

I have put the training data together into a Ground Truth data source using the groundTruth() function.

All of this works, but when I train the dataset using the trainACFObjectDetector() function, the system just continues to use RAM up until MATLAB just crashes.

Is there a way that I can tell MATLAB to manage the RAM that is used while training? I am nowhere near the number of images I want to train with (at least 1000) and also am nowhere near the number of iterative stages.

The system I am using has about 90 GB of ram. I would imagine that this should be plenty...

I am thinking there has to be a way to have the training algorithm throw out RAM it doesn't need, instead of just eating up more and more throughout the training process.

Thank you in advance to anyone who offers their help!

Answer 1

[Ryan Comeau]

Hello,

I have also had hilarious experiences with MATLAB RAM usage. I can however give a brief explanation of what is causing the RAM to be filled a lot. This is based on my personal experience and I don't have objective measures for you.

1. I see this particular computer vision uses a table to read the data. It is possible that this table is being read all at once and the images are being loaded. What you need is the Image Datastore(imds)https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.imagedatastore.html and the box label datastore(blds) https://www.mathworks.com/help/vision/ref/boxlabeldatastore.html. These will load in only the images which are currently being used.
2. You must take a look at your network architecture and consider the convolution kernels that you're using and how much intermediate information is being used by you convolutions. If you have a 2000x2000 image with a 2x2 pixel convolution filter with a stride of 1, a lot of intermediate information is being used. If you make a 7x7 filter with a 5 pixel stride, the intermediate information will be less
3. I'm not familiar with this particular object detector, but the batch size has a large effect on RAM usage. If you make the batch size smaller, it will reduce the RAM being used and once and will hopefully prevent a crash.

Hope this helps

RC