selfAttentionLayer can't process sequence-to-label problem?

71 views (last 30 days)
selfAttentionLayer why can't handle the following simple sequence classification problem, already through the flattenLayer into one-dimensional data, on the contrary, lstm specify "outputMode" as "last" will pass.
% Here use simple data, for demonstration purposes only
XTrain = rand(3,200,1000); % dims "CTB"
TTrain = categorical(randi(4,1000,1));
% define my layers
numClasses = numel(categories(TTrain));
layers = [inputLayer(size(XTrain),"CTB");
flattenLayer;
selfAttentionLayer(6,48);
% lstmLayer(20,OutputMode="last"); % use lstmLayer is ok!
layerNormalizationLayer;
fullyConnectedLayer(numClasses);
softmaxLayer];
net = dlnetwork(layers);
% train network
lossFcn = "crossentropy";
options = trainingOptions("adam", ...
MaxEpochs=1, ...
InitialLearnRate=0.01,...
Shuffle="every-epoch", ...
GradientThreshold=1, ...
Verbose=true);
netTrained = trainnet(XTrain,TTrain,net,lossFcn,options);
Error using trainnet
Number of observations in predictors (1000) and targets (1) must match. Check that the data and network are consistent.

Accepted Answer

cui,xingxing
cui,xingxing on 7 Jan 2024
In terms of the output feature map dimensions, there is a time "T" dimension that has to be eliminated in order to match the output dimensions, which can usually be done by indexing1dLayer. So the layers array is added before the fullyConnectedLayer.
% Here use simple data, for demonstration purposes only
XTrain = rand(3,200,1000); % dims "CTB"
TTrain = categorical(randi(4,1000,1));
% define my layers
numClasses = numel(categories(TTrain));
layers = [inputLayer(size(XTrain),"CTB");
flattenLayer;
selfAttentionLayer(6,48);
% lstmLayer(20,OutputMode="last"); % use lstmLayer is ok!
layerNormalizationLayer;
indexing1dLayer; % Add this!!!
fullyConnectedLayer(numClasses);
softmaxLayer];
net = dlnetwork(layers);
% train network
lossFcn = "crossentropy";
options = trainingOptions("adam", ...
MaxEpochs=1, ...
InitialLearnRate=0.01,...
Shuffle="every-epoch", ...
GradientThreshold=1, ...
Verbose=true);
netTrained = trainnet(XTrain,TTrain,net,lossFcn,options);
Iteration Epoch TimeElapsed LearnRate TrainingLoss _________ _____ ___________ _________ ____________ 1 1 00:00:02 0.01 1.5374 7 1 00:00:06 0.01 1.5272 Training stopped: Max epochs completed
  5 Comments

Sign in to comment.

More Answers (0)

Categories

Find more on Sequence and Numeric Feature Data Workflows in Help Center and File Exchange

Products


Release

R2023b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!