RL Toolbox: DQN epsilon greedy exploration with epsilon=1 does not act random

Setup:

• Costum Simulink Environment
• DQN Agent

To get a baseline of the environment I started training with a DQN Agent with:

opt.EpsilonGreedyExploration.Epsilon=1;
opt.EpsilonGreedyExploration.EpsilonDecay=0.0;
opt.EpsilonGreedyExploration.EpsilonMin=1;

This means that the Agent should not exploit the greedy action at all.

Sstated by the documentation (https://de.mathworks.com/help/reinforcement-learning/ug/dqn-agents.html):

During each control interval, the agent either selects a random action with probability ϵ or selects an action greedily with respect to the value function with probability 1-ϵ.

--> Epsilon=1 means probability of zero to have the greedy agent. It is not clearly stated how the random action is sampled, but it should be uniform.

Now with the above setting, the DQN Agent should never exploit the greedy policy during training. However, when starting the Simulation and watching the output of the episodes, it is clear that the Agent does in fact exploit the policy and does not act random.

• What is going on here? Why does the agent not act random during training?
• Is the sampling of the actions uniform? (Not related to the epsilon=1 behavior)
• When exactly is the decay executed? I think i read somewhere in the doc that it is every training step, i.e., for DQN every time step of the simulation with the SampleTime of rlDQNAgentOptions? Would be handy to just have this information clearly stated in the part of the doc that expxlains epsilon greedy

I quite like the toolbox so far, there are just some implementation details that are a bit hard to grasp,i.e., its not 100 % clear to me how it is done by MATLAB.

Answer 1

[Emmanouil Tzorakoleftherakis]

Hello,

Maybe I misread the question, but you are saying "when starting the Simulation and watching the output of the episodes...". Just to clarify, if you hit the "play" button in Simulink or if you use the "sim" command, exploration is out of the picture - Simulink will only do inference on the agent. Exploration is used only when you call "train".

To your other question, sampling in DQN is indeed uniform for exploration

[Tobias Schindler]

While training, i.e. after this command:

agent = rlDQNAgent(critic,agentOpts);

[Emmanouil Tzorakoleftherakis]

This command only creates the agent, it does not train the agent. If you run the Simulink model after this command there will be no exploration. Exploration will come into the picture when you call train (agent, env)

[Tobias Schindler]

Sorry, my fault. Was the wrong line. This one trains

trainingStats = train(agent,env,trainOpts);

[Emmanouil Tzorakoleftherakis]

Thanks. How many actions do you have in this problem? I am wondering if this behavior persists after the first few episodes?

[Tobias Schindler]

I have 8 actions. I think that it persits over multiple episodes, but I am note sure. Would have to check.

Are you able to reproduce my problem with one of the examples?

[Emmanouil Tzorakoleftherakis]

I tried in R2020b with this example and I am not able to reproduce the behavior. For one, using the settings you have I got an error saying that 'EpsilonDecay' should be greater than zero. That does not change anything if the minimum Epsilon value is 1 so I left the decay at the default value and I was seeing random actions sampled uniformly.

I am assuming you are using an earlier release? Can you try with the default 'EpsilonDecay'?

[Tobias Schindler]

Thanks for checking! I'll try to reproduce it with the example as well and check for differences between my original model and the example.

Will be reporting back!

[Tobias Schindler]

Forgot about this question and I did not encounter this problem anymore in other models / setups, not sure what the problem was.