How to use transfer learning on U-net created in Matlab?

Hi,

I'm interested in using a U-net that I have trained in Matlab on one set of data and then applying transfer learning on the last couple layers with another set of data (as shown in https://www.mathworks.com/help/deeplearning/gs/get-started-with-transfer-learning.html )

How would this process work with a network created using U-net layers in Matlab? Also, is the U-net layer in Matlab already pre-trained using some other set of data when it's used in the script?

Thank you

Edit:

I'm attempting to change the final convolutional layer using this code:

load SimNet1.mat
Net2 = SimNet1; %renaming U-Net
layers = Net2.Layers
%layers(56)
layers(56) = convolution2dLayer([1 1], 2,'NumChannels',64); %Should I specify weights and bias?
layers

% Directories of new training images (not used in pre-trained network)
segDir = fullfile('images location'); 
USDir = fullfile('images location'); % Labels 

imds = imageDatastore(USDir); %DataStore of input training images - ultrasound images
classNames = ["bone","background"]; %labels
labelIDs   = [1 0];
pxds = pixelLabelDatastore(segDir,classNames,labelIDs); %Has Ground Truth pixel data on training images  - Seg images are pixel labeled (only have 255/0)

options = trainingOptions('adam','InitialLearnRate', 3e-4, ...
    'MaxEpochs',100,'MiniBatchSize',15, ...
    'Plots','training-progress','Shuffle','every-epoch'); 

ds = pixelLabelImageDatastore(imds,pxds) %returns a datastore based on input image data(imds - US images) 
%and pxds (required network output - segmentations)

TLNet1 = trainNetwork(ds,layers,options)
save TLNet1

But I am recieving an error that says:

Shouldn't these layers already be connected from the pre-trained network?

[sab kara]

find you the soluston? i need it please

Answer 1

[Daniel Vieira]

what you made works only for sequential models, where each layer is necessarily connected to the next in the same order that they are stacked in "Layers". The uNet is not a sequential model, some layers have multiple outputs, some have multiple inputs, a "early" layer may be connedted to a "late" layer. When you just pick the layers and pass it to trainNetwork, you are ignoring the connections.

Instead of passing the layers, you should pass the lgraph object. And before passing, you need to change it with the methods addLayers, removeLayers, connectLayers, disconnectLayers. You mustn't leave any layer input or output unassigned in the Connections property of lgraph.

[Hridayi]

If I use plot(Net2), Matlab outputs an plot of how the layers should be connected in the Unet. Can that plot be used to train the last couple of layers with new data? If so, what commands are required?

When I plot the layers using lgraph, the network loses all it's structure.

[Hridayi]

I updated the layers to match the U-net structure. However, when I went to replace the last couple layers, the PixelClassificationLayer gave an error:

load SimNet1.mat
Net2 = SimNet1; %renaming U-Net
analyzeNetwork(Net2)
plot(Net2)
layers = Net2.Layers;
lgraph = layerGraph(layers)
lgraph = connectLayers(lgraph,'Encoder-Stage-1-ReLU-2','Decoder-Stage-4-DepthConcatenation/in2')
lgraph = connectLayers(lgraph,'Encoder-Stage-2-ReLU-2','Decoder-Stage-3-DepthConcatenation/in2')
lgraph = connectLayers(lgraph,'Encoder-Stage-3-ReLU-2','Decoder-Stage-2-DepthConcatenation/in2')
lgraph = connectLayers(lgraph, 'Encoder-Stage-4-ReLU-2','Decoder-Stage-1-DepthConcatenation/in2')
figure;
plot(lgraph)

larray = [convolution2dLayer([1 1], 2,'NumChannels',64,'Name','NewFinalConvLayer')];
lgraph = replaceLayer(lgraph,'Final-ConvolutionLayer',larray);

larray2 = [pixelClassificationLayer,'Name','NewPixelClassificationLayer'];
lgraph = replaceLayer(lgraph,'Segmentation-Layer',larray2);

larray3 = [softmaxLayer,'Name','NewSoftmaxLayer'];
lgraph = replaceLayer(lgraph,'Softmax-Layer',larray3);
figure;
plot(lgraph)

I am not sure what that error means.

[Daniel Vieira]

this error is because you made a syntax mistake. The right form would be:

larray2=pixelClassificationLayer('Name','NewPixelClassificationLayer');

Instead you are concatenating the output of pixelClassificationLayer with 2 char arrays, it doen't make sense. Make the same change to the other lines where you did this.

[Hridayi]

Hello, I had a follow up question on this.

If I wanted to use transfer learning on the U-net itself, do I need to only replace the last 3 layers(Final convolution, pixelclassification, softmax)? What are the correct layers to replace? I also had difficulties with replacing other layers in the network because of input size not matching for the U-net (Asked on MATLAB Answers)

Thanks for your help.