GPU arrays support only fundamental numeric or logical data types

I tried to use Faster R-CNN with the pretrained model of ResNet18 where I froze some of the layers to detect 5 different objects inside image. I've successfully train the model using only the training data (in table). However when I include the validation data, the MATLAB asked me to change the validation and training data to be in datastore format (basically I followed the steps here https://www.mathworks.com/help/vision/ug/object-detection-using-faster-r-cnn-deep-learning.html). I've tried to train the model, the training can go on, but there is a pop-up warning for every iteration(?) saying the GPU arrays support only fundamental numeric or logical data types. Also, the training plot for loss function can't be shown.

load data.mat
lgraph=resnet18_12_freeze
%trainingData
imds = imageDatastore(trainingData.imageFilename);
blds = boxLabelDatastore(trainingData(:,2:end));
trainingData=combine(imds,blds);

%validationData
imds = imageDatastore(validationData.imageFilename);
blds = boxLabelDatastore(validationData(:,2:end));
validationData=combine(imds,blds);

inputSize = [224 224 3];
trainingData = transform(trainingData, @(data)preprocessData(data,inputSize));
augmentedTrainingData = transform(trainingData,@augmentData);
augmentedData = cell(4,1);

trainingData = transform(augmentedTrainingData,@(data)preprocessData(data,inputSize));
validationData = transform(validationData,@(data)preprocessData(data,inputSize));

options = trainingOptions('sgdm', ...
'MiniBatchSize', 2, ...
'InitialLearnRate', 1e-3, ...
'MaxEpochs', 50, ...
'VerboseFrequency', 50, ...
'ValidationData', validationData, ...
'ValidationFrequency',50, ...
'ExecutionEnvironment', 'gpu', ...
'CheckpointPath', tempdir, ...
'Plots','training-progress');
  
  
      
detector = trainFasterRCNNObjectDetector(trainingData, lgraph, options, ...
        'NegativeOverlapRange',[0.1 0.5], ...
        'PositiveOverlapRange',[0.5 1]);

Warning message

Warning: GPU arrays support only fundamental numeric or logical data types. 
> In nnet.internal.cnn.util/TrainingPlotReporter/cleanUpAfterPlotError (line 109)
In nnet.internal.cnn.util/TrainingPlotReporter/reportIteration (line 58)
In nnet.internal.cnn.util/VectorReporter/computeAndReport (line 64)
In nnet.internal.cnn.util/VectorReporter/reportIteration (line 20)
In nnet.internal.cnn/Trainer/train (line 142)
In vision.internal.cnn.trainNetwork (line 102)
In trainFasterRCNNObjectDetector>iTrainEndToEnd (line 901)
In trainFasterRCNNObjectDetector (line 428)
In trainFasterRCNN (line 35) 

Answer 1

[Joss Knight]

It looks distinctly like one of your custom functions, preprocessData or augmentData, is doing something illegal with gpuArray objects. Try calling dbstop if all error so that MATLAB will break at the problematic line of code.

[Ricky Hartono]

Thanks for the reply.

I've tried to disable the preprocessData and the augmentData functions.

Then I've tried to call the dbstop if all error and this is the result

[Joss Knight]

It's difficult to tell whether this is the error or an earlier error. I don't think it is because it is not throwing the error about gpuArray support. Keep running the code (F5) to try to find the error that ends up being thrown as a warning in that cleanupAfterPlotError function. I'm afraid this isn't really something we can do over this interface so you'll have to do a bit of debugging and investigating on your own.

[Ricky Hartono]

Ok, thank you for the answer

[Joss Knight]

Here's an idea: wrap the code in your custom functions preprocessData and augmentData with a try...catch block. Then you can put a breakpoint inside the catch clause, allowing you to determine the original source of the error - assuming it was indeed your custom code.

try
    % Put your original code here
catch ME
    % Put a breakpoint here so you can inspect the error
    getReport(ME) % This displays the error and call stack
    rethrow(ME);
end

Clearly we need to do better here with reporting the correct call stack so you can see the source of the error, but hopefully this should help.