How to deal with both discrete and continuous action in the same environment?

Suppose in a custom environment there are two actions- one is discrete and the other continuous. How can I build such an environment in MATLAB? All examples are only of one type -either discrete or continuous.

Thanks.

[Maha Mosalam]

yes, I had the same issue...If any proposed solutions for that

Answer 1

[Ronit]

Hi,

To create a custom environment class, “rl.env.MATLABEnvironment” function in MATLAB can be used.

But it appears that an action space combined of both discrete and continuous actions cannot be created. Here’s a workaround for handling the mixed action space within the step function of your environment by manually processing the action input based on your specific requirements.

Following is a sample custom environment with this workaround:

classdef MyCustomEnv < rl.env.MATLABEnvironment
    properties
        % Define properties for your environment
        State = [];
    end
    
    methods
        function this = MyCustomEnv()
            % Initialize Observation settings
            ObservationInfo = rlNumericSpec([2 1]);
            ObservationInfo.Name = 'My Observations';
            ObservationInfo.Description = 'Description of the observation';
            
            % Initialize Action settings
            % Here, you might define either a continuous or discrete action space
            % and handle the mixed nature internally.
            ActionInfo = rlNumericSpec([2 1], 'LowerLimit', [-1; -10], 'UpperLimit', [1; 10]);
            ActionInfo.Name = 'My Actions';
            
            % The constructor must call the step function with these info
            this = [e-mail](ObservationInfo, ActionInfo);
            
            % Initialize environment state
            this.State = [0; 0];
        end
        
        % Reset function
        function InitialObservation = reset(this)
            this.State = [0; 0];
            InitialObservation = this.State;
        end
        
        % Step function
        function [Observation, Reward, IsDone, LoggedSignals] = step(this, Action)
            LoggedSignals = [];
            
            % Here, you would manually interpret the Action vector
            % For example, assuming the first element is discrete (after scaling/rounding)
            % and the second is continuous.
            DiscreteAction = round(Action(1)); % Simplify for example
            ContinuousAction = Action(2);
            
            % Implement your environment dynamics here using the interpreted actions
            
            % Example update of the state
            this.State = this.State + [DiscreteAction; ContinuousAction];
            
            Observation = this.State;
            Reward = -sum(this.State.^2);
            IsDone = norm(this.State) > 10;
        end
    end
end

This approach requires careful handling and proper description, as it does not follow the standard approach of defining and using the action space. You'll need to ensure that any agents interacting with this environment are correctly configured and generated to match the environment's expected format.

Refer to the documentation of "rl.env.MATLABEnvironment" for more information on how to create and customize your own environment: https://www.mathworks.com/help/reinforcement-learning/ug/create-custom-environment-from-class-template.html

I hope this helps!