Do we need a delay block in reinforcement learning simulink?

11 views (last 30 days)
Hi
I have started using sumulink recently to use reinforcement learning toolbox. In Matlab examples, there is not any delay unit but in my simulation I got loop error and it will be fixed after I use delay block. The training can be excuted now but the learning is not succesful and I am wondering whether this delay is the reason or not because it is not used in matlab examples. I have screenshot of before and after using delay block. Errors were shown when there is no delay.
Caused by:
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Error due to multiple causes.
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input data dependency violation due to action subsystems. See Subsystem Examples in the
Simulink library for valid and invalid examples of action subsystems
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Calculate Reward/Abs' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1, 2) of 'MODEL2D/Calculate Reward/If1' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Calculate Reward/Abs1' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (Action) of 'MODEL2D/Calculate Reward/If Action Subsystem1' are involved in
the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1, 2, 3) of 'MODEL2D/Calculate Reward/Merge' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (Action) of 'MODEL2D/Calculate Reward/If Action Subsystem2' are involved in
the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (Action) of 'MODEL2D/Calculate Reward/If Action Subsystem3' are involved in
the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/UAVEnv/Add' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Calculate Reward/Subtract1' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Calculate Reward/Gain1' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/UAVEnv/Add1' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1, 2) of 'MODEL2D/Calculate Reward/If' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (Action) of 'MODEL2D/Calculate Reward/If Action Subsystem' are involved in
the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1, 2) of 'MODEL2D/Calculate Reward/Merge1' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (Action) of 'MODEL2D/Calculate Reward/If Action Subsystem4' are involved in
the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Check if Done/Abs1' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Check if Done/Compare To Constant2/Compare' are involved in
the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Check if Done/Subtract1' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Check if Done/Compare To Constant/Compare' are involved in
the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Check if Done/Abs' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Check if Done/Compare To Constant1/Compare' are involved in
the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1, 2) of 'MODEL2D/Check if Done/Logical Operator' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Check if Done/Subtract' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Check if Done/Compare To Constant3/Compare' are involved in
the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1, 2, 3) of 'MODEL2D/Check if Done/Logical Operator1' are involved in the
loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1, 2, 3) of 'MODEL2D/RL Agent/AgentWrapper' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/UAVEnv/Sin' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1, 2) of 'MODEL2D/UAVEnv/Product' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1, 2, 3, 4) of 'MODEL2D/Calculate Reward/Add2' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/UAVEnv/Cos' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1, 2) of 'MODEL2D/UAVEnv/Product1' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Calculate Reward/Subtract' are involved in the loop
Error using rl.env.SimulinkEnvWithAgent>localHandleSimoutErrors (line 817)
Input ports (1) of 'MODEL2D/Calculate Reward/Gain' are involved in the loop

Answers (1)

Emmanouil Tzorakoleftherakis
A delay block may be necessary to break algebraic loops or to rely on observations from previous time steps (see e.g. this example). I would try to identify which specific observation causes the issue - no need to add a delay to all of them.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!