A problem when using "multi-gpu" as "ExecutionEnvironment" for training a CNN

Hello,

I am experiencing weird problems when I use the “multi-gpu” as the “ExecutionEnvironment” in the training option for training a CNN.

I am using the simple CNN example shown in the MATLAB’s website: https://www.mathworks.com/help/deeplearning/ug/create-simple-deep-learning-network-for-classification.html

I only added the “multi-gpu” as the “ExecutionEnvironment” and changed the “MaxEpochs” to 50 in the training option.

I am using two of NVIDA RTX 2080 Ti 11GB from the same manufacturer (Galax). The two graphic cards are the same models from the same manufacturer.

In MATLAB R2020a, the CNN example works fine with faster training speeds when I use the “multi-gpu” as the “ExecutionEnvironment” in the training option. The training progress plot is attached below.

But in MATLAB R2022a, the training progress works really weird when I run the same code. The validation data curve goes high while the training data curve fluctuates at the middle and never goes up. The screenshot showing this is attached below.

And the validation accuracy in the command window after the training is also bad (63.92%) while the validation accuracy only in the training progress plot is good.

This issue also happens in other MATLAB versions later than R2020a. When I tested with a single GPU separately and individually, the training works fine.

Can anyone please help me with this issue? I need to use my two GPUs for the "multi-gpu" training in MATLAB R2022a for my work so I first need to fix this weird problem in the example.

I would appreciate any help. Thank you very much.

Answer 1

[Joss Knight]

Most likely this is this issue, which is fixed in the latest update to R2022a. You can also try downgrading your GPU drivers.

[Hyung Joon]

Thank you very much for your support. I uninstalled my Geforce driver (a 500 level version, probably quite recent one) and installed an older version (one of the 400 level versions). Then the "multi-gpu" now works!! I installed the MATLAB R2022a on July 21st. I think it was not the latest update. Thank you very much again for your help!!

[Joss Knight]

Glad to hear it!