Clear Filters
Clear Filters

selfAttentionLayer can't process sequence-to-label problem?

2 views (last 30 days)
selfAttentionLayer why can't handle the following simple sequence classification problem, already through the flattenLayer into one-dimensional data, on the contrary, lstm specify "outputMode" as "last" will pass.
% Here use simple data, for demonstration purposes only
XTrain = rand(3,200,1000); % dims "CTB"
TTrain = categorical(randi(4,1000,1));
% define my layers
numClasses = numel(categories(TTrain));
layers = [inputLayer(size(XTrain),"CTB");
flattenLayer;
selfAttentionLayer(6,48);
% lstmLayer(20,OutputMode="last"); % use lstmLayer is ok!
layerNormalizationLayer;
fullyConnectedLayer(numClasses);
softmaxLayer];
net = dlnetwork(layers);
% train network
lossFcn = "crossentropy";
options = trainingOptions("adam", ...
MaxEpochs=1, ...
InitialLearnRate=0.01,...
Shuffle="every-epoch", ...
GradientThreshold=1, ...
Verbose=true);
netTrained = trainnet(XTrain,TTrain,net,lossFcn,options);
Error using trainnet
Number of observations in predictors (1000) and targets (1) must match. Check that the data and network are consistent.

Accepted Answer

cui,xingxing
cui,xingxing on 7 Jan 2024
Edited: cui,xingxing on 27 Apr 2024
In terms of the output feature map dimensions, there is a time "T" dimension that has to be eliminated in order to match the output dimensions, which can usually be done by indexing1dLayer. So the layers array is added before the fullyConnectedLayer.
% Here use simple data, for demonstration purposes only
XTrain = rand(3,200,1000); % dims "CTB"
TTrain = categorical(randi(4,1000,1));
% define my layers
numClasses = numel(categories(TTrain));
layers = [inputLayer(size(XTrain),"CTB");
flattenLayer;
selfAttentionLayer(6,48);
% lstmLayer(20,OutputMode="last"); % use lstmLayer is ok!
layerNormalizationLayer;
indexing1dLayer; % Add this!!!
fullyConnectedLayer(numClasses);
softmaxLayer];
net = dlnetwork(layers);
% train network
lossFcn = "crossentropy";
options = trainingOptions("adam", ...
MaxEpochs=1, ...
InitialLearnRate=0.01,...
Shuffle="every-epoch", ...
GradientThreshold=1, ...
Verbose=true);
netTrained = trainnet(XTrain,TTrain,net,lossFcn,options);
Iteration Epoch TimeElapsed LearnRate TrainingLoss _________ _____ ___________ _________ ____________ 1 1 00:00:02 0.01 1.5374 7 1 00:00:06 0.01 1.5272 Training stopped: Max epochs completed
-------------------------Off-topic interlude-------------------------------
I am currently looking for a job in the field of CV algorithm development, based in Shenzhen, Guangdong, China. I would be very grateful if anyone is willing to offer me a job or make a recommendation. My preliminary resume can be found at: https://cuixing158.github.io/about/ . Thank you!
Email: cuixingxing150@gmail.com
  5 Comments

Sign in to comment.

More Answers (0)

Categories

Find more on Image Data Workflows in Help Center and File Exchange

Products


Release

R2023b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!