How to save further simulation data (e.g. the states) when simulating an RL-Agent on an environment.

Hello everyone,

I created a custom environment with the function rlCreateEnvTemplate("MyEnvironment") and trained an AC- and DDPG-agent for this. Afterwards I ran the function experience = sim(env,agents) and plot the results. Is there a possibility to save not only the actions, observations,... in the structure 'experience' but also e.g. the states since they contain information i want to plot as well?

Alternatively is there a possibility to add this information i want to plot to the observations but only use 3 out of 24 entries from the observation vector to train the network of actor and critic?

Thanks for your help.

Answer 1

[Shubham]

Hi Rebecca,

In MATLAB, when you are working with custom reinforcement learning environments and agents such as Actor-Critic (AC) and Deep Deterministic Policy Gradient (DDPG), the experience structure typically contains the data from simulation, including observations, actions, rewards, and other signals that the agents use for learning.

To address your questions:

1. The experience structure does not by default save the internal states of the environment unless they are part of the observation. If you need to save the full states for post-simulation analysis, you can modify your environment to include the full state as part of the observation or as additional information in the info output of the step function. However, this additional state information should not be used by the agent for learning if it is not part of the actual observation space.
2. If you want to include additional information in the observations but only use a subset of them for training the agent's networks, you would need to customize the neural network architectures for the actor and critic. In MATLAB, you can define custom layers or networks that only process the relevant parts of the observation vector. You can achieve this by creating a custom observation preprocessing function that selects the relevant features from the observation space before they are fed into the networks. This preprocessing step can be included as part of the agent's policy or value function representation.