I get error when I try to test an agent trained on a PC( with GPU) on a second computer which has no GPU (Reinforcement Learning)

Hello,

I trained a DDPG agent on PC where I placed the critic and actor network on a GPU. After training, I am able to run and test the agent performance on that PC. However, when I copy the trained agent to another PC, I am not able to run it. I get a warning related to GPU. This second PC has no GPU, so I am using the CPU as the device. The first PC uses MATLAB 2023a and the second PC is 2022b, which I donot think should be a problem. Please see image of the error attached. Thanks

[Joss Knight]

This is a MATLAB forum, not Tensorflow.

[Rik]

Since TensorFlow is a Python tool, I don't see the relevance on this forum.

Can you explain why you posted this comment?

[Bay Jay]

@Ahmad Taher: Joss Knight and Rik are right. My problem is specific to MATLAB reinforcement learning (with deep learning). Your recommendation is appreciated but would not be helpful for my case. Thanks

@Emmanouil Tzorakoleftherakis

[Emmanouil Tzorakoleftherakis]

Can you please share the code snippet you used to use GPUs for training?

[Bay Jay]

gcpu = gpuDeviceCount("available");

if gcpu == 0

device = 'cpu';

else

device = 'gpu';

end

statePath = [

featureInputLayer(numObservations,'Normalization','none','Name','observation')

fullyConnectedLayer(numOfNeurons,'Name','fc1')

reluLayer('Name','relu1')

fullyConnectedLayer(numOfNeurons,'Name','fc2')

additionLayer(2,'Name','add') % ADDITION layer: Would be used to merge the observation and action

reluLayer('Name','relu2')

fullyConnectedLayer(numOfNeurons,'Name','fc3')

reluLayer('Name','relu3')

fullyConnectedLayer(1,'Name','fc4')];

actionPath = [

featureInputLayer(numActions,'Normalization','none','Name','action') %enter numActions = 3

fullyConnectedLayer(numOfNeurons,'Name','fc5')];

% Create the critic network = neural network+the action path

criticNetwork = layerGraph(statePath);

criticNetwork = addLayers(criticNetwork,actionPath);

criticNetwork = connectLayers(criticNetwork,'fc5','add/in2'); %( Meaning:Connect fc5 to the secondinput of th addition layer)

criticNetwork = dlnetwork(criticNetwork);

criticOptions = rlOptimizerOptions(...

'LearnRate',1e-3,'GradientThreshold',1,'L2RegularizationFactor');

critic = rlQValueFunction(criticNetwork,observationInfo,actionInfo,...

'ObservationInputNames','observation','ActionInputNames','action');

critic.UseDevice = device; % 'cpu' or 'gpu';

% 2. ACTOR NETWORK

actorNetwork = [

featureInputLayer(numObservations,'Normalization','none','Name','observation')

fullyConnectedLayer(numOfNeurons,'Name','fc1')

reluLayer('Name','relu1')

fullyConnectedLayer(numOfNeurons,'Name','fc2')

reluLayer('Name','relu2')

fullyConnectedLayer(numActions,'Name','fc4')

tanhLayer('Name','tanh1')

% softmaxLayer('Name','actionProb')

scalingLayer('Name','ActorScaling1','Scale',[])];

actorNetwork = layerGraph(actorNetwork);

actorNetwork = dlnetwork(actorNetwork);

actorOptions = rlOptimizerOptions('LearnRate',1e-4,'GradientThreshold',1,'L2RegularizationFactor',1e-4, 'Algorithm', "adam");

actor = rlContinuousDeterministicActor(actorNetwork,observationInfo,actionInfo);

actor.UseDevice = device; % 'cpu' or 'gpu';

trainOpts = rlTrainingOptions(...

MaxEpisodes=5000, ...

MaxStepsPerEpisode=ceil(Tf/Ts), ...

ScoreAveragingWindowLength=10,...

Verbose=false, ...

Plots="training-progress",...

StopTrainingCriteria = "AverageReward",...

StopTrainingValue=5000,...

[Takeshi Takahashi]

Can you re-save the agent on the PC with GPU?

Before saving the agent, please follow these steps:

actor = getActor(agent);
actor.UseDevice = "cpu";
setActor(agent, actor);
critic = getCritic(agent);
critic.UseDevice = "cpu";
setCritic(agent, critic);

Answer 1

[Walter Roberson]

Your Training_Script1 is trying to load() a .mat file that has a gpuArray() object stored in it, but you are trying to do that on a machine that has no attached GPU. MATLAB is trying to create the gpuArray() object, but is failing to do so.

If you need to load something with an embedded gpuArray object on a system without GPU, then that is not going to be possible.

Sometimes this problem can show up if you have multiple variables saved in a .mat file, some of which have gpuArray objects stored in them, but for you work on the non-GPU system, you do not happen to need the variables that have gpuArray objects. In such a case, the trick is to be selective about which variables you load, such as

datastruct = load('MySavedFile.mat', 'trainedobject');  %only load the items you need
trainedobject = datastruct.trainedobject;

instead of

load('MySavedFile.mat');      %loads everything in the file

[Emmanouil Tzorakoleftherakis]

@Bay Jay this answer explains why you see the error. Please give us some time to investigate if there is a workaround.