How to use DAG network - multi-input network (Static + Temporal inputs using LSTM and Fully connected layers)

4 views (last 30 days)
Yildirim Kocoglu
Yildirim Kocoglu on 28 Nov 2020
Commented: Ankita Jain on 15 Mar 2022
Hello there,
I'm trying to combine static features with temporal data to affect the outcome of the sequence output with no success yet.
I tried using both R2020a and R2020b to see if there is any difference but, I get the same error mentioned below in both versions.
The problem is a regression/prediction problem (not classification) and both input and output are sequences of unique objects (each row) with different sequence lengths (except the static data). The static data I'm trying to add is at time t = 0 and they are not pictures (each row corresponds to the unique object in each row of XTrain and YTrain).
If anyone can give suggestions or guide me in the right direction, I'll highly appreciate it.
I got the main idea from the paper called "Early Predictions for Medical Crowdfunding: A Deep Learning Approach Using Diverse Inputs" by Wang et al. (2019) which should be publicly available but, unlike them, I'm not trying to initialize the hidden state at time t=0 yet but, trying to use the older approach that they mention at the beginning of the paper. Architecture of one of the older approaches that they mention is below:
Attached is the simplified data saved as excel that is required to run the code.
Everything in the code works until the training part and then I get the following error:
Error using trainNetwork (line 170)
Invalid network.
Error in Static_temporal_NN_Test (line 95)
net = trainNetwork({XTrain(1:3,:);Static_input(1:3,:)'},YTrain(1:3,:),lgraph,options);
Caused by:
Network: Incompatible layer types. The network contains layer types not supported with recurrent layers.
Detected recurrent layers:
layer 'bilstm1' (BiLSTM)
Detected incompatible layers:
layer 'static_input' (Image Input)
Network: Incompatible layer types. The network contains layer types not supported with image input layers.
Detected image input layers:
layer 'static_input' (Image Input)
Detected incompatible layers:
layer 'flattenstatic1' (Flatten)
Network: Invalid input layers. If the network has a sequence input layer, then it must not have any other input layers.
Layer 'concat1': Input size mismatch. Size of input to this layer is different from the expected input size.
Inputs to this layer:
from layer 'flattenstatic1' (output size 100)
from layer 'bilstm1' (output size 100)
Full code below:
clear;
clc;
close all;
%% Import Data
Input = xlsread('Static_temporal_data.xlsx','XTrain','B2:CG4');
Output = xlsread('Static_temporal_data.xlsx','YTrain','B2:CG4');
Static_input = xlsread('Static_temporal_data.xlsx','Static_Features','B2:H4');
XTrain = cell(size(Input,1),1);
YTrain = cell(size(Output,1),1);
for i = 1:size(Input,1)
XTrain{i,1} = Input(i,~isnan(Input(i,:)));
YTrain{i,1} = Output(i,~isnan(Output(i,:)));
end
%% Training Options
miniBatchSize = 1;
maxEpochs = 2;
options = trainingOptions('adam', ...
'MaxEpochs',maxEpochs,'MiniBatchSize',miniBatchSize,'Shuffle','never', ...
'GradientThreshold',1, ...
'InitialLearnRate',0.01, ...
'LearnRateSchedule','piecewise', ...
'LearnRateDropPeriod',125, ...
'LearnRateDropFactor',1, ...
'Verbose',0, ...
'Plots','training-progress');
%% BiLSTM Layer(s)
numFeatures = 1;
numHiddenUnits = 50;
% Structure of the LSTM network
layers = [ ...
sequenceInputLayer(numFeatures, 'Name', 'temporal_input')
bilstmLayer(numHiddenUnits,'BiasInitializer','ones','OutputMode','sequence', 'Name', 'bilstm1')
];
%% Static Layers
numfeatures_static = [7 1];
numResponses_static = 100;
layers_static = [ ...
imageInputLayer(numfeatures_static,'Name', 'static_input')
fullyConnectedLayer(numResponses_static,'BiasInitializer','ones', 'Name', 'fullyconnect1_static')
reluLayer('Name', 'relustatic_1')
flattenLayer('Name', 'flattenstatic1')
];
%% Concatenation Layer + Fully connected Layer + Regression Layer
numResponses = 1;
conclayers = [...
concatenationLayer(3,2,'Name','concat1')
fullyConnectedLayer(numResponses,'BiasInitializer','ones', 'Name', 'fullyconnect1_temporal')
regressionLayer('Name', 'regressionlayer')];
%% Multiple Input DAG NN + Layer Graph plot
lgraph = layerGraph(layers);
plot(lgraph);
axis off;
lgraph = addLayers(lgraph, layers_static);
lgraph = addLayers(lgraph, conclayers);
plot(lgraph);
axis off;
lgraph = connectLayers(lgraph, 'flattenstatic1', 'concat1/in1');
lgraph = connectLayers(lgraph, 'bilstm1', 'concat1/in2');
plot(lgraph);
axis off;
%% Train Network
net = trainNetwork({XTrain(1:3,:);Static_input(1:3,:)'},YTrain(1:3,:),lgraph,options);
  3 Comments
Ankita Jain
Ankita Jain on 15 Mar 2022
Have you find out any solution? I am trying the same. And I guess custom layer option provided my matlab will help to create two input layer.

Sign in to comment.

Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!