Convolutional LSTM (C-LSTM) in MATLAB

118 views (last 30 days)
Jake
Jake on 9 Oct 2018
Edited: Dieter Mayer on 29 Aug 2022
I'd like to train a convolutional neural network with an LSTM layer on the end of it. Similar to what was done in:
  1. https://arxiv.org/pdf/1710.03804.pdf
  2. https://arxiv.org/pdf/1612.01079.pdf
Is this possible?

Answers (5)

Shounak Mitra
Shounak Mitra on 9 Oct 2018
Hi Jake,
Unfortunately, we do not directly support C-LSTM. We are working on it and it should be available soon.
-- Shounak
  7 Comments
David Willingham
David Willingham on 26 Aug 2022
Hi Dieter,
Apologies for not updating this answers post sooner. This workflow is now supported. the following code will illustrated this:
% Load data
[XTrain,YTrain] = japaneseVowelsTrainData;
% Define layers
layers = [ sequenceInputLayer(12,'Normalization','none', 'MinLength', 9);
convolution1dLayer(3, 16)
batchNormalizationLayer()
reluLayer()
maxPooling1dLayer(2)
convolution1dLayer(5, 32)
batchNormalizationLayer()
reluLayer()
averagePooling1dLayer(2)
lstmLayer(100, 'OutputMode', 'last')
fullyConnectedLayer(9)
softmaxLayer()
classificationLayer()];
options = trainingOptions('adam', ...
'MaxEpochs',10, ...
'MiniBatchSize',27, ...
'SequenceLength','longest');
% Train network
net = trainNetwork(XTrain,YTrain,layers,options);
Dieter Mayer
Dieter Mayer on 29 Aug 2022
Edited: Dieter Mayer on 29 Aug 2022
Hi David,
Thanks for your reply! Is this workflow shows a real convolution LSTM (LSTM carries out convolutional operations instead of matrix multiplication) and is not only implied to a input matrix, which is a result of a convolution net work applied before?
Sorry for asking that, I have to learn the syntax of using the deep learning toolbox, I am a beginner. The background is, that I will use such a Conv-LSTM to make precipitation forecasts for grids bases on precipitation radar inputs from several timesteps of the last minutes / hours as discussed in this paper publication

Sign in to comment.


Yi Wei
Yi Wei on 17 Dec 2019
Hi, can matlab support C-LSTM now?
  5 Comments
ytzhak goussha
ytzhak goussha on 23 Feb 2021
Hey,
Sorry I didn't follow this thread and didn't see the questions.
Here is a simplified C-LSTM network.
The input it a 4D image (height x width x channgle x time)
The input type is sqeuntial.
When you need to put CNN segments, you simply unfold->CNN->Fold->flatten and feed to LSTM layer.
Ioana Cretu
Ioana Cretu on 18 May 2021
Hi! When I try to train the model I have this error:
Error using trainNetwork (line 170)
Invalid network.
Caused by:
Layer 'fold': Unconnected output. Each layer output must be connected to the input of another layer.
Detected unconnected outputs:
output 'miniBatchSize'
Layer 'unfold': Unconnected input. Each layer input must be connected to the output of another layer.
I connected the layers using this:
lgraph = layerGraph(Layers);
lgraph = connectLayers(lgraph,'fold/miniBatchSize','unfold/miniBatchSize');
What do you think the cause is?

Sign in to comment.


Chen
Chen on 25 Aug 2021
Please refer to this excellent example in:
It is possible to train the hybrid together.

Jonathan
Jonathan on 4 Aug 2022
inputSize = [28 28 1];
filterSize = 5;
numFilters = 20;
numHiddenUnits = 200;
numClasses = 10;
layers = [ ...
sequenceInputLayer(inputSize,'Name','input')
sequenceFoldingLayer('Name','fold')
convolution2dLayer(filterSize,numFilters,'Name','conv')
batchNormalizationLayer('Name','bn')
reluLayer('Name','relu')
sequenceUnfoldingLayer('Name','unfold')
flattenLayer('Name','flatten')
lstmLayer(numHiddenUnits,'OutputMode','last','Name','lstm')
fullyConnectedLayer(numClasses, 'Name','fc')
softmaxLayer('Name','softmax')
classificationLayer('Name','classification')];
lgraph = layerGraph(layers);
lgraph = connectLayers(lgraph,'fold/miniBatchSize','unfold/miniBatchSize');

David Willingham
David Willingham on 26 Aug 2022
Updating this answer. This workflow has been supported since R2021. The following example illustrates how to combin CNN's with LSTM layers:
% Load data
[XTrain,YTrain] = japaneseVowelsTrainData;
% Define layers
layers = [ sequenceInputLayer(12,'Normalization','none', 'MinLength', 9);
convolution1dLayer(3, 16)
batchNormalizationLayer()
reluLayer()
maxPooling1dLayer(2)
convolution1dLayer(5, 32)
batchNormalizationLayer()
reluLayer()
averagePooling1dLayer(2)
lstmLayer(100, 'OutputMode', 'last')
fullyConnectedLayer(9)
softmaxLayer()
classificationLayer()];
options = trainingOptions('adam', ...
'MaxEpochs',10, ...
'MiniBatchSize',27, ...
'SequenceLength','longest');
% Train network
net = trainNetwork(XTrain,YTrain,layers,options);

Products


Release

R2018a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!