trainNetwork stops after one iteration

Hi all, i run into trouble using trainNetwork. Sometimes training stops after one iteration without any error message and the GPU utilization goes down to 0%. The process itself runs with 100% CPU utilization.

We use 4D-Imagearrays for training. The trainingOptions are:

training_options = trainingOptions('sgdm','MaxEpochs',200,...
    'InitialLearnRate', 0.01,...
    'ExecutionEnvironment', 'gpu',...
    'Shuffle', 'every-epoch',...
    'ValidationData', {validation_images, validation_labels},...
    'ValidationPatience', 8,...
    'MiniBatchSize', 256);

Our systems have 1080Ti's and 12G memory. For some reason this error does not occur on K80 GPUs. For me it seems that this is related to the architecture of the GPUs.

Does someone run into the same issue? Thanks for suggestions to solve this.

Edit: I tested all nodes. The training runs normal on the ones with K80 GPUs. On all machines with 1080TIs the training aborts after one iteration with high CPU utilization.

[Joss Knight]

The training stops after a single iteration with no message at all? Or do you mean it stops after displaying only a single line of verbose output?

Set 'VerboseFrequency' to 1 and 'ValidationPatience' to Inf and run again, just to check.

[Niklas]

Hi Joss,

yes the training stops without any further messages. I submitted a service request and we weren't able to track this error.

We now switched to 2018a and everything seems fine so far.