rlSimulinkEnv

Create environment object from a Simulink model already containing at least one agent block

Syntax

env = rlSimulinkEnv(mdl,agentBlocks)

env = rlSimulinkEnv(mdl,agentBlocks,observationInfo,actionInfo)

env = rlSimulinkEnv(___,'UseFastRestart',fastRestartToggle)

Description

The rlSimulinkEnv function creates an environment object from a Simulink^® model that already includes at least one agent block. The environment object acts as an interface so that when you call sim or train, these functions in turn call the (compiled) Simulink model to generate experiences for the agents. This environment supports training multiple agents at the same time.

To create an environment object from a Simulink model that does not include an agent block, use the createIntegratedEnv function instead. For more information on reinforcement learning environments, see Create Custom Simulink Environments.

env = rlSimulinkEnv(mdl,agentBlocks) creates the reinforcement learning environment object env for the Simulink model mdl. agentBlocks contains the paths to one or more reinforcement learning agent blocks in mdl. If you use this syntax, each agent block must reference an agent object already in the MATLAB^® workspace.

env = rlSimulinkEnv(mdl,agentBlocks,observationInfo,actionInfo) creates the reinforcement learning environment object env for the model mdl. The two cell arrays observationInfo and actionInfo must contain the observation and action specifications for each agent block in mdl, in the same order as they appear in agentBlocks.

env = rlSimulinkEnv(___,'UseFastRestart',fastRestartToggle) creates a reinforcement learning environment object env and additionally enables fast restart. Use this syntax after any of the input arguments in the previous syntaxes.

Examples

Create Simulink Environment Using Agent in Workspace

Create a Simulink environment using the trained agent and corresponding Simulink model from the Control Water Level in a Tank Using a DDPG Agent example.

Load the agent in the MATLAB® workspace.

load WaterTankDDPG

Create an environment for the rlwatertank model, which contains an RL Agent block. Because the agent used by the block is already in the workspace, you do not need to pass the observation and action specifications to create the environment.

env = rlSimulinkEnv("rlwatertank","rlwatertank/RL Agent")

env = 
SimulinkEnvWithAgent with properties:

           Model : rlwatertank
      AgentBlock : rlwatertank/RL Agent
        ResetFcn : []
  UseFastRestart : on

Validate the environment by performing a short simulation for two sample times.

validateEnvironment(env)

You can now train and simulate the agent within the environment by using train and sim, respectively.

Create Reinforcement Learning Environment from Simulink Model

For this example, you use rlNumericSpec to define an observation space consisting of a single channel carrying a three-element vector. You then use rlFiniteSetSpec to define an action space consisting of a single channel carrying only one of three possible values. You then use these observation and action specifications to create a custom Simulink® environment that relies on the rlSimplePendulumModel Simulink model.

The model represents a simple frictionless pendulum that initially hangs in a downward position. Open the model.

mdl = "rlSimplePendulumModel";
open_system(mdl)

An rlNumericSpec object specifies an environment channel that carries signals (actions or observations) that belong to a continuous set. By contrast, an rlFiniteSetSpec object specifies a channel that carries signals that belong to a finite set (a set containing only a finite number of elements).

If you have an existing environment, you can extract its action or observation specifications (which in general are vectors of rlNumericSpec and rlFiniteSetSpec objects) using the getActionInfo or getObservationInfo functions.

In this example, instead, you need to create a new custom environment. To do so you must first define the environment action and observation channels.

To define the channel that represents the observation space, use rlNumericSpec. The channel carries a vector containing three signals (the sine, cosine, and time derivative of the angle).

obsInfo = rlNumericSpec([3 1]) 

obsInfo = 
  rlNumericSpec with properties:

     LowerLimit: -Inf
     UpperLimit: Inf
           Name: [0×0 string]
    Description: [0×0 string]
      Dimension: [3 1]
       DataType: "double"

To define the channel that represents the action space, use rlFiniteSetSpec. The channel carries a scalar expressing the torque and can be one of three possible values, -2 Nm, 0 Nm and 2 Nm.

actInfo = rlFiniteSetSpec([-2 0 2])

actInfo = 
  rlFiniteSetSpec with properties:

       Elements: [3×1 double]
           Name: [0×0 string]
    Description: [0×0 string]
      Dimension: [1 1]
       DataType: "double"

You can use dot notation to assign property values for the rlNumericSpec and rlFiniteSetSpec objects.

obsInfo.Name = "observations";
actInfo.Name = "torque";

You can now use these specifications to create both a new custom environment and an agent object that works within your environment.

To create your custom Simulink environment, use rlSimulinkEnv. Specify the Simulink model as first argument, the path of the agent block as a second argument, and the observation and action specifications that you have created in the previous step. For more information on custom Simulink environments, see Create Custom Simulink Environments.

agentBlk = mdl + "/RL Agent";
env = rlSimulinkEnv(mdl,agentBlk,obsInfo,actInfo)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : []
  UseFastRestart : on

Specify a reset function using dot notation. For this example, randomly initialize theta0 in the model workspace using the setVariable (Simulink) function.

env.ResetFcn = @(in) setVariable(in,"theta0",randn,"Workspace",mdl)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : @(in)setVariable(in,"theta0",randn,"Workspace",mdl)
  UseFastRestart : on

Here, in is a Simulink.SimulationInput (Simulink) object, and the values of theta0 that you specify overrides the existing theta0 value in the model workspace for the duration of the simulation or training. The value of theta0 is then reverted to the original when the simulation or training completes. For more information on reset functions, see Reset Function for Simulink Environments.

You can now use env (together with an agent object) as argument for the built-in functions train and sim, which train and simulate the agent within the environment.

Create Simulink Environment for Multiple Agents

Create an environment for the Simulink model from the example Train Multiple Agents to Perform Collaborative Task.

Load the file containing the agents. For this example, load the agents that have been already trained using decentralized learning.

load decentralizedAgents.mat

Create an environment for the rlCollaborativeTask model, which has two agent blocks. Because the agents used by the two blocks (agentA and agentB) are already in the workspace, you do not need to pass their observation and action specifications to create the environment.

env = rlSimulinkEnv( ...
    "rlCollaborativeTask", ...
    ["rlCollaborativeTask/Agent A","rlCollaborativeTask/Agent B"])

env = 
SimulinkEnvWithAgent with properties:

           Model : rlCollaborativeTask
      AgentBlock : [
                     rlCollaborativeTask/Agent A
                     rlCollaborativeTask/Agent B
                   ]
        ResetFcn : []
  UseFastRestart : on

It is good practice to specify a reset function for the environment such that agents start from random initial positions at the beginning of each episode. For an example, see the resetRobots function defined in Train Multiple Agents to Perform Collaborative Task.

You can now simulate or train the agents within the environment using the sim or train functions, respectively.

Input Arguments

mdl — Simulink model name

string | character vector

Simulink model name, specified as a string or character vector. The model must contain at least one RL Agent block.

Example: "myModel"

agentBlocks — Agent block paths

string | character vector | string array

Agent block paths, specified as a string, character vector, or string array.

If mdl contains a single RL Agent block, specify agentBlocks as a string or character vector containing the block path.

If mdl contains multiple RL Agent blocks, specify agentBlocks as a string array, where each element contains the path of one agent block.

mdl can contain RL Agent blocks whose path is not included in agentBlocks. Such agent blocks behave as part of the environment, selecting actions based on their current policies. When you call sim or train, the experiences of these agents are not returned and their policies are not updated.

Multiagent simulation is not supported for MATLAB environments.

The agent blocks can be inside of a model reference. For more information on configuring an agent block for reinforcement learning, see RL Agent.

Example: "myModel/RL Agent"

observationInfo — Observation specifications

rlNumericSpec object | rlFiniteSetSpec object | array of rlNumericSpec objects | array of rlFiniteSetSpec objects | cell array

Observation information, specified as a specification object, an array of specification objects, or a cell array.

If mdl contains a single agent block, specify observationInfo as an rlNumericSpec object, an rlFiniteSetSpec object, or an array containing a mix of such objects.

If mdl contains multiple agent blocks, specify observationInfo as a cell array, where each cell contains a specification object or array of specification objects for the corresponding block in agentBlocks.

For more information, see getObservationInfo.

[rlNumericSpec([2 1]) rlFiniteSetSpec([3,5,7])]

actionInfo — Action specifications

rlNumericSpec object | rlFiniteSetSpec object | cell array

Action information, specified as a specification object or a cell array.

If mdl contains a single agent block, specify actionInfo as an rlNumericSpec or rlFiniteSetSpec object.

If mdl contains multiple agent blocks, specify actionInfo as a cell array, where each cell contains a specification object for the corresponding block in agentBlocks.

For more information, see getActionInfo.

rlNumericSpec([1 1])

fastRestartToggle — Option to toggle fast restart

"on" (default) | "off"

Option to toggle fast restart, specified as either "on" or "off". Fast restart allows you to perform iterative simulations without compiling a model or terminating the simulation each time.

For more information on fast restart, see How Fast Restart Improves Iterative Simulations (Simulink).

Example: "off"

Output Arguments

env — Reinforcement learning environment

SimulinkEnvWithAgent object

Reinforcement learning environment, returned as a SimulinkEnvWithAgent object.

Note

Before training or simulating an agent within a Simulink environment, to make sure that the RL Agent block runs at the intended sample time, set the SampleTime property of your agent object appropriately.

For more information on reinforcement learning environments, see Create Custom Simulink Environments.

Version History

Introduced in R2019a