Incorrect tanhLayer output in RL agent
3 views (last 30 days)
Show older comments
Mohammad Ashraful Islam
on 5 Apr 2020
Commented: H. M.
on 20 Oct 2022
Last layer in my actor network is set to tanhLayer. However, I am seeing output that goes above 1 or below -1 from the RL agent block. Is this normal behavior of RL agent?
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/282102/image.png)
4 Comments
Asvin Kumar
on 11 Apr 2020
I am unable to reproduce the error. Here's what I got:
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/283579/image.png)
Each view corresponds to a leg of the bipedal robot. The three signals are the normalized torques applied to the ankle, knee and hip.
Mind sharing your model to have a look?
Accepted Answer
Asvin Kumar
on 13 Apr 2020
I’ve tried this. I still don’t see the values going beyond [–1, 1]. However, I might be able to answer your question. If you have a look at the helper functions createTD3Agent.m and createDDPGAgent.m, you will notice the ‘agentoptions’ object. The parameters called ‘ExplorationModel’ or ‘NoiseModel’ specify details about the kind of noise added to the predicted action. This can either be an ‘OrnsteinUhlenbeckActionNoise’ object or a ‘GaussianActionNoise’ object each with their own set of parameters. Have a more detailed look at the Noise Options here: rlDDPGAgentoptions and rlTD3AgentOptions. This noise is added to encourage the agent to explore the environment.
The output action from the tanhLayer in the ‘actorNetwork’ will still be in the range of [–1, 1]. Once the noise is added, the new action values will be saturated to the limits specified in the ‘ActorInfo’. These limits will be [-Inf, Inf] by default and won’t saturate your action values when not mentioned.
3 Comments
Abdul Basith Ashraf
on 5 Apr 2021
Edited: Abdul Basith Ashraf
on 5 Apr 2021
If only I knew when that noise was added, I could have saved a ton of my time. Finally I know it. Noise is added to the output of the actor. I had used a default value for variance of 0.6, but my input was in the range -1e-4 to +1e-4 And because of that my output was always saturated
H. M.
on 20 Oct 2022
Is that means, we can only use action limits,
actInfo = rlNumericSpec([numAct 1],'LowerLimit',-1,'UpperLimit',1);
without using tanhlayer and this will guarantee the action to be in the desired range.
More Answers (0)
See Also
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!