Transfer Learning on Unet: Image Input Size not matching for layers

Hello,

I am trying to retrain a couple layers of the U-net architecture with new data. However, some of the layers have a different input size and therefore, are giving me an error. How do I change the input to the layers without messing up the U-net architecture?

load SimNet1.mat
Net2 = SimNet1; %renaming U-Net
analyzeNetwork(Net2)
plot(Net2)
layers = Net2.Layers;
lgraph = layerGraph(layers)
lgraph = connectLayers(lgraph,'Encoder-Stage-1-ReLU-2','Decoder-Stage-4-DepthConcatenation/in2')
lgraph = connectLayers(lgraph,'Encoder-Stage-2-ReLU-2','Decoder-Stage-3-DepthConcatenation/in2')
lgraph = connectLayers(lgraph,'Encoder-Stage-3-ReLU-2','Decoder-Stage-2-DepthConcatenation/in2')
lgraph = connectLayers(lgraph, 'Encoder-Stage-4-ReLU-2','Decoder-Stage-1-DepthConcatenation/in2')
figure;
plot(lgraph)

%Real US Directories - Transfer Learning Images (260)
segDir = fullfile(%Fire Location); % Segmentations 
USDir = fullfile(%File Location); % Labels = US Images

imds = imageDatastore(USDir); %DataStore of input training images - ultrasound images
classNames = ["bone","background"]; %labels
labelIDs   = [1 0];
pxds = pixelLabelDatastore(segDir,classNames,labelIDs); 

larray = [convolution2dLayer([1 1], 2,'NumChannels',64,'NumFilters',2,'Name','NewFinalConvLayer')];
lgraph = replaceLayer(lgraph,'Final-ConvolutionLayer',larray);

larray2 = pixelClassificationLayer('Name','NewPixelClassificationLayer','Classes',["bone" "background"]);
lgraph = replaceLayer(lgraph,'Segmentation-Layer',larray2);

larray3 = softmaxLayer('Name','NewSoftMaxLayer');
lgraph = replaceLayer(lgraph,'Softmax-Layer',larray3);

%Error is occuring for the next two layers
larray4 = [convolution2dLayer([3 3], 2,'NumChannels',128,'NumFilters',64,'Name','NewDecoderStage41layer')];
lgraph = replaceLayer(lgraph,'Decoder-Stage-4-Conv-1',larray4);

larray5 = [convolution2dLayer([3 3], 2,'NumChannels',64,'NumFilters',64,'Name','NewDecoderStage42Layer')];
lgraph = replaceLayer(lgraph,'Decoder-Stage-4-Conv-2',larray5);

figure;
plot(lgraph)

options = trainingOptions('adam','InitialLearnRate', 3e-4, ...
    'MaxEpochs',100,'MiniBatchSize',15, ...
    'Plots','training-progress','Shuffle','every-epoch'); 

ds = pixelLabelImageDatastore(imds,pxds) %returns a datastore based on input image data(imds - US images) 
%and pxds (required network output - segmentations)

TLNet7 = trainNetwork(ds,lgraph,options)
save TLNet7

[Srivardhan Gadila]

@Hridayi can you provide the values of the imageSize, numClasses & 'EncoderDepth' of your above U-Net network and also can you attach the SimNet1.mat file?

[Hridayi]

@Srivardhan Gadila

These are the parameters used to train SimNet1:

imageSize = [256 256 1]; %Size of images being used
numClasses = 2;  %Binary images so 2
encoderDepth = 4; %How many layers will there be in Unet apart from input and output

Unfortunately, I can't attach the file because it exceeds 5 MB even when I put it in a zipped folder.

I've also tried another way by freezing the weights of previous layers but that method does not give me good segmentations. It also won't allow me to change the PixelClassificationLayer or Softmax layer as they don't have any weights to change. Below is my code for that method.

I'm not sure which of these methods is better but I'm only getting results from my original method. However, by directly replacing the layers, I can't change other layers of the network which take different input sizes since it's a U-net which changes size at different encoder and decoders.

load SimNet3.mat %Same parameters as SimNet1 only difference is this network has more images
trainedNet = SimNet3; %renaming U-Net

layers = trainedNet.Layers;
connections = trainedNet.Connections;

layers(1:55) = FreezeWeights(layers(1:55)); %freeze up to the replaced layers
lgraph = CreateLgraphUsingConnections(layers,connections);

%FinalConvLayer
a = lgraph.Layers(56,1); %copy the layer
am = mean(reshape(a.Weights,1,[]));%find the mean of the weights
a.Weights = (2*am)*rand(size(a.Weights))- am; %replace layer with random matrix centered around the mean of the pretrained layer
am = mean(reshape(a.Bias,1,[])); % find mean of the bias
a.Bias = (2*am)*rand(size(a.Bias))- am;%replace layer with random matrix centered around the mean of the pretrained laye

lgraph = replaceLayer(lgraph,a.Name,a);%replace FinalConv
clear a am;

%Real US Directories - Transfer Learning Images (260 images)
segDir = fullfile('Directory'); % Segmentations 
USDir = fullfile('Directory'); % Labels = US Images

imds = imageDatastore(USDir); %DataStore of input training images - ultrasound images
classNames = ["bone","background"]; %labels
labelIDs   = [1 0];
pxds = pixelLabelDatastore(segDir,classNames,labelIDs); %Has Ground Truth pixel data on training images-Seg images are pixel labeled (only have 255/0

options = trainingOptions('adam','InitialLearnRate', 5e-4, ...
    'MaxEpochs',100,'MiniBatchSize',15, ...
    'Plots','training-progress','Shuffle','every-epoch'); 

ds = pixelLabelImageDatastore(imds,pxds); %returns a datastore based on input image data(imds - US images) 
%and pxds (required network output - segmentations)

TLNet13 = trainNetwork(ds,lgraph,options)
save TLNet13

Answer 1

[Srivardhan Gadila]

Based on the information on imageSize, numClasses & 'EncoderDepth', for the Net2 in your question add the value 'same' for the 'padding' Name-Value pair argument as follows for the NewFinalConvLayer, NewDecoderStage41layer & NewDecoderStage42Layer:

larray = [convolution2dLayer([1 1], 2,'NumChannels',64,'NumFilters',2,'Name','NewFinalConvLayer','Padding','same')];
lgraph = replaceLayer(lgraph,'Final-ConvolutionLayer',larray);

larray2 = pixelClassificationLayer('Name','NewPixelClassificationLayer','Classes',["bone" "background"]);
lgraph = replaceLayer(lgraph,'Segmentation-Layer',larray2);

larray3 = softmaxLayer('Name','NewSoftMaxLayer');
lgraph = replaceLayer(lgraph,'Softmax-Layer',larray3);

%Error is occuring for the next two layers
larray4 = [convolution2dLayer([3 3], 2,'NumChannels',128,'NumFilters',64,'Name','NewDecoderStage41layer','Padding','same')];
lgraph = replaceLayer(lgraph,'Decoder-Stage-4-Conv-1',larray4);

larray5 = [convolution2dLayer([3 3], 2,'NumChannels',64,'NumFilters',64,'Name','NewDecoderStage42Layer','Padding','same')];
lgraph = replaceLayer(lgraph,'Decoder-Stage-4-Conv-2',larray5);
analyzeNetwork(lgraph)

This would maintain the same U-Net structure but with the replaced layers and should no longer cause the mentioned error.

Answer 2

[Samia OUKIL]

How to segment color image(Skin lesion) with Unet and transfert learning?

Hello, I am a beginner in deep learning! So,I have a medical image database (Skin lesion) to segment with U-net and transfer learning! For that, I downloaded " u-net-release-2015-10-02.tar.gz"(https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/) but I don't know how to segment my database!

Please, if you can help and guide me to segment with Unet and how to use transfer learning?

[Hridayi]

Hi,

I suggest checking out these links:

Segmentation in MATLAB: https://www.mathworks.com/help/vision/ug/getting-started-with-semantic-segmentation-using-deep-learning.html

U-net Layers: https://www.mathworks.com/help/vision/ref/unetlayers.html

For transfer learning, you can use the freezing certain layers approach that I had commented above or you can try this approach: https://blogs.mathworks.com/pick/2017/02/24/deep-learning-transfer-learning-in-10-lines-of-matlab-code/

[Samia OUKIL]

Thank you so much!

Good luck

[sab kara]

slt if you find eny thing about this theme please give me a code i need it