How to build an Actor-Critic model with shared layers?

2 views (last 30 days)
Heesu Kim
Heesu Kim on 8 Mar 2021
Edited: Heesu Kim on 8 Mar 2021
Hi.
I'm trying to build an Actor-Critic model uisng Reinforcement Learning Toolbox.
What I'm currently intending is to share low level features over Actor and Critic, like this.
-----------------------------
inputImage
|
backboneNet
| |
actorNet criticNet
-------------------------------
So, I created three LayerGraphs for the backbone, as well as actor and critic, and connected them.
And then, the next steps are the same as the guideline of generating AC agent, like this.
%% Backbone Network
% ...(Some codes to define the backbone layers)...
backboneNet = createLgraphUsingConnections(all_layers, all_connections) ;
%% Critic Network
% ...(Some codes to define the critic layers)...
criticNet = layerGraph(fullyConnectedLayer(1, 'Name', 'critic_FC')) ;
all_layers = [...
backbone.Layers
criticNet.Layers] ;
all_connections = [...
backbone.Connections
criticNet.Connections] ;
connection = subgraphConnections(backbone.Layers(end), criticNet.Layers(1));
all_connections = [all_connections; connection];
criticNet = createLgraphUsingConnections(all_layers, all_connections) ;
critic = rlValueRepresentation(...
criticNet, obsInfo,...
'Observation', {'inputImage'},...
criticOpts);
%% Actor Network
% ...(Some codes to define the actor layers)...
actorNet_concatOut = layerGraph(concatenationLayer(1, 2, 'Name', 'mean&sdev'));
all_layers = [...
backbone.Layers
actorNet_meanPath.Layers
actorNet_sdevPath.Layers
actorNet_concatOut.Layers] ;
all_connections = [...
backbone.Connections
actorNet_meanPath.Connections
actorNet_sdevPath.Connections
actorNet_concatOut.Connections] ;
connection1 = subgraphConnections(...
backbone.Layers(end),...
[actorNet_meanPath.Layers(1) actorNet_sdevPath.Layers(1)]);
connection2 = subgraphConnections(...
[actorNet_meanPath.Layers(end) actorNet_sdevPath.Layers(end)],...
actorNet_concatOut.Layers(1));
all_connections = [all_connections; connection1; connection2];
actorNet = createLgraphUsingConnections(all_layers, all_connections) ;
actor = rlStochasticActorRepresentation(...
actorNet, obsInfo, actInfo,...
'Observation', {'inputImage'},...
actorOpts);
%% Agent
agent = rlACAgent(actor,critic,agentOpts);
But If I put in the backboneNet as above, the actorNet gradients appear as zero so the actorNet doen't learn anything.
When I was not using the backboneNet, everything was fine.
I guess I'm doing something wrong or misunderstand how to define shared layers.
Is there anybody know how to define shared layers over the actor and critic?
Thanks

Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!