I want to print out multiple actions in reinforcement learning

I want to print out multiple actions in reinforcement learning

this is agent code

function agent = createAgent(observationInfo,actionInfo,Ts)

L = 16; % number of neurons
statePath = [
    featureInputLayer(observationInfo.Dimension(1),'Normalization','none','Name','observation')
    %fullyConnectedLayer(L,'Name','fc1')
    %reluLayer('Name','relu1')
    fullyConnectedLayer(L,'Name','fc2')
    additionLayer(2,'Name','add')
    reluLayer('Name','relu2')
    %fullyConnectedLayer(L,'Name','fc3')
    %reluLayer('Name','relu3')
    fullyConnectedLayer(1,'Name','fc4')];

actionPath = [
    featureInputLayer(actionInfo.Dimension(1),'Normalization','none','Name','action')
    fullyConnectedLayer(L, 'Name', 'fc5')];

criticNetwork = layerGraph(statePath);
criticNetwork = addLayers(criticNetwork, actionPath);
criticNetwork = connectLayers(criticNetwork,'fc5','add/in2');
criticNetwork = dlnetwork(criticNetwork);

 
criticOptions = rlOptimizerOptions('LearnRate',1e-3,'GradientThreshold',1);


critic = rlQValueFunction(criticNetwork,observationInfo,actionInfo,...
    'ObservationInputNames','observation','ActionInputNames','action');


actorNetwork = [
    featureInputLayer(observationInfo.Dimension(1),'Normalization','none','Name','observation')
    fullyConnectedLayer(L,'Name','fc1')
    reluLayer('Name','relu1')
    %fullyConnectedLayer(L,'Name','fc2')
    %reluLayer('Name','relu2')
    %fullyConnectedLayer(L,'Name','fc3')
    %reluLayer('Name','relu3')
    fullyConnectedLayer(1,'Name','fc4')
    tanhLayer('Name','tanh1')
    scalingLayer('Name','ActorScaling1','Scale',2.5,'Bias',-0.5)];
actorNetwork = dlnetwork(actorNetwork);

actorOptions = rlOptimizerOptions('LearnRate',1e-3,'GradientThreshold', 1);
actor = rlContinuousDeterministicActor(actorNetwork,observationInfo,actionInfo);


agentOptions = rlDDPGAgentOptions(...
    'SampleTime',Ts,...
    'CriticOptimizerOptions',criticOptions,...
    'ActorOptimizerOptions',actorOptions,...
    'ExperienceBufferLength',1e6);
agentOptions.NoiseOptions.Variance = 0.6;
agentOptions.NoiseOptions.VarianceDecayRate = 1e-5;


agent = rlDDPGAgent(actor,critic,agentOptions);

This is run code

%%
clear;
clc;

%% Create Environment

% load environment parameters
parameters

mdl = 'my_CAV_slx';
% open_system(mdl);

obsInfo = rlNumericSpec([10  1]);
actInfo = rlNumericSpec([1 1], 'LowerLimit', accMin, 'UpperLimit', accMax);

blks = mdl + "/RL Agent1";

env = rlSimulinkEnv(mdl, blks, obsInfo, actInfo);

% set reset function
env.ResetFcn = @(in)ResetParamFcn(in);

%% Create Agents

% fix seed
rng(0)

% longitudinal control
agent1 = createAgent(obsInfo, actInfo, Ts);

%% Train Agents

trainingOpts = rlTrainingOptions('MaxEpisodes',maxEpisodes,'MaxStepsPerEpisode',maxSteps,'Verbose',false,'Plots','training-progress','StopTrainingCriteria','AverageReward');

doTraining = true;
if doTraining
    % Train the agent.
    trainingStats = train(agent1, env, trainingOpts);
    save('my_CAV.mat', 'agent1')
else
    % Load pretrained agents for the example.
    load('my_CAV.mat')
end

%% Simulate Agents

simOptions = rlSimulationOptions('MaxSteps',maxSteps);
experience = sim(env, agent1, simOptions);

[Emmanouil Tzorakoleftherakis]

I am not sure what you mean by print out. Do you want your agent to output multiple actions? Or do you want to know how to do inference with the agent you created above and see what the output action is?

[SEONGJAE SHIN]

Thank you for comment. I want to output multiple actions.

Answer 1

[Emmanouil Tzorakoleftherakis]

Hi,

If you want to create an agent that outputs multiple actions, you need to make sure the actor network is set up accordingly. I would recommend using the 'default agent' feature as shown here. In summary, this feature creates a default network architecture for your actor and critic based on the observation and action spaces you provide. If your action space indicates that you have multiple actions, the actor and critic that will be created will reflect that. You can use this shortcut as a way to get an initial design which you can then tweak as needed.

The link I shared above also shows you how to perform inference on the actor/agent you already created using 'getAction'.

Hope that helps