RL Training Error for PMSM Control: Reward Block Sample Time Issue in Triggered Subsystem AND Simulation Step Size Question

After completing the "Train TD3 Agent for PMSM Control" tutorial, I’m attempting to integrate reinforcement learning into my custom PMSM Simulink model—specifically, replacing the current loop PI controller with an RL agent, just like in the tutorial.

However, after modifying the Simulink model, I encountered the following error when starting training:

Error using rl.internal.train.OffPolicyTrainer/run_internal_/nestedRunEpisode (line 304)
An error occurred while running the simulation for model 'rl_Basic_Model' with the following RL agent blocks:
 	rl_Basic_Model/Current_Regulator/rlCurrentController/Reinforcement Learning/RL Agent

Error in rl.internal.train.OffPolicyTrainer/run_internal_ (line 371)
                            out = nestedRunEpisode(policy);
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in rl.internal.train.OffPolicyTrainer/run_ (line 39)
            result = run_internal_(this);
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in rl.internal.train.Trainer/run (line 8)
            result = run_(this);
            ^^^^^^^^^^^^^^^^^^^^
Error in rl.internal.trainmgr.OnlineTrainingManager/run_ (line 135)
            trainResult = run(trainer);
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in rl.internal.trainmgr.TrainingManager/run (line 4)
            result = run_(this);
            ^^^^^^^^^^^^^^^^^^^^
Error in rl.agent.AbstractAgent/train (line 86)
    trainingResult = run(tm);
    ^^^^^^^^^^^^^^^^^^^^^^^^^
Caused by:
    Error using rl.env.internal.reportSimulinkSimError (line 29)
    'rl_Basic_Model/Current_Regulator/rlCurrentController/Reinforcement Learning/RL Agent/reward' has sample time 5e-05. Sample time of block must be inherited (-1) because blocks in a triggered subsystem must run only when the triggered subsystem runs.

understand this error is caused by the triggered subsystem in my model. Let me explain why I use triggers: In my SVPWM block, I output two triggers:

• A pwm trigger to trigger current loop calculations.
• A 1ms_trigger for speed loop or MTPA calculations.Finally, the block outputs pulses connected to the Universal Bridge block.(My PWM frequency is 20kHz (Ts=5e-5s), and the simulation step size is set to 3e-7s.)

My Questions:

1. How can I resolve this sample time error while retaining my existing trigger logic? Is it possible to proceed with RL training without removing the triggers?
2. When the RL Toolbox calls my Simulink model, does the environment update (e.g., changes in the motor, inverter, etc.) still follow my configured simulation step size (3e-7s as mentioned above)?
3. The relationship between agent-sample time, simulation step.

Any guidance or suggestions would be greatly appreciated! Thank you.

Answer 1

[Anmol]

Hi Haoming,

I understand that you are trying to integrate reinforcement learning into a custom PMSM Simulink Model. You are trying to replace the current loop PI controller with an RL agent. After modifying the Simulink model, you are getting an error. The trigger logic of the code should remain the same after the modifications.

1. This error is originating from using a fixed sample time 5e-05 for the reward model. This model is inside a triggered subsystem. For debugging the error, you can set the sample time of reward model as inherited (-1). This allows the agent to run only when its parent triggered subsystem is activated by your PWM trigger.
2. All blocks in a triggered subsystem must have Sample time set to inherited (-1) or constant (inf). This requirement allows the blocks in a triggered subsystem to run only when the triggered subsystem itself runs. This requirement also means that a triggered subsystem cannot contain continuous blocks, such as an Integrator block as per the documentation.
3. Yes, when the RL Toolbox calls the Simulink model, the environment updates still follow the configured simulation step size.
4. The agent-sample time is the rate at which the RL agent interacts with the environment. It defines the discrete moments in time when the agent observes the current state of the system, chooses an action and applies that action to the system. Simulation step size is the fixed time interval at which the solver calculates the state of all continuous blocks.

You can refer to the following links for more information about these:

• https://www.mathworks.com/matlabcentral/answers/418803-can-step-size-be-changed-during-a-simulink-simulation
• https://www.mathworks.com/help/simulink/ug/triggered-subsystems.html
• https://www.mathworks.com/help/simulink/gui/fixedstepsizefundamentalsampletime.html
• https://www.mathworks.com/matlabcentral/answers/555799-reinforcement-learning-sample-time?s_tid=answers_rc1-2_p2_MLT

Hope, it helps.