How to apply Ensemble Learning using two Trained Deep Learning Models

Hello, I hope you are doing well. i have the two trained model one is Resnet50 and other is Resnet18.

I want to apply Ensemble learning or Weighted average or Majority vote. I am going through the link below. I want to implment the same metholodogy.

https://blogs.mathworks.com/deep-learning/2019/06/03/ensemble-learning/

I have write the following code but it takes the model seprately. Not using Ensemble Learning

Can anybody help me with that

net1 = load('Resnet50.mat')
net2 = load('Resnet18.mat')
rxTestPred = classify(net1.resnet.trainedNet,rxTestFrames);
testAccuracy = mean(rxTestPred == rxTestLabels);
disp("Resnet50 Test accuracy: " + testAccuracy*100 + "%")
rxTestPred = classify(net2.resnet.trainedNet,rxTestFrames);
testAccuracy = mean(rxTestPred == rxTestLabels);
disp("Resnet18 Test accuracy: " + testAccuracy*100 + "%")

[SH]

@Johanna Pingel @yanqi liu @Image Analyst @Walter Roberson

[Walter Roberson]

Sorry, I have done very little Deep Learning.

[SH]

@Walter Roberson Thanks for your comment.

[SH]

@David Willingham

Answer 1

[Drew]

The short answer is that for ensemble averaging, use the scores from each classifier, not just the class label predictions from each classifier. So, start by returning the scores from the "classify" method. Read on for more details.

The three ensemble techniques listed on the blog that you mentioned (https://blogs.mathworks.com/deep-learning/2019/06/03/ensemble-learning/) are:

1. Averaging: a simple average over all the predictions (output of the softmax layer) from the different networks
2. Weighted average: the weights are proportional to an individual model's performance. For example, the predictions for the best model could be weighted by 2, while the rest of the models have no weight;
3. Majority voting: for each test observation, the prediction is the most frequent class in all predictions

Majority voting requires at least 3 classifiers. You have just two classifers, so you can try averaging or weighted average. Start with averaging. You just need to average the scores for each output class, and use those average scores to determine the new predicted output class for the ensemble of two classifiers.

So, in your code, you need to return the score vector, then take the average of the scores, then find the max score, choose that label for each observation, then calculate the accuracy of the ensemble classifier.

net1 = load('Resnet50.mat')
net2 = load('Resnet18.mat')

% Resnet50
[rxTestPred1,scores1] = classify(net1.resnet.trainedNet,rxTestFrames);
testAccuracy1 = mean(rxTestPred1 == rxTestLabels);
disp("Resnet50 Test accuracy: " + testAccuracy1*100 + "%")

% Resnet18
[rxTestPred2,scores2] = classify(net2.resnet.trainedNet,rxTestFrames);
testAccuracy2 = mean(rxTestPred2 == rxTestLabels);
disp("Resnet18 Test accuracy: " + testAccuracy2*100 + "%")

% "Average Ensemble" of the two nets:
% Create a new score vector which averages the scores from the two nets
% average of each column will average the two scores for each class
scoreAvg=mean([scores1;scores2]); 
% Next, find the index of the max score in scoreAvg, find the label, and do the
% scoring as desired. 
% Here is one way to extract the class labels. Of course the two networks
% must have the same output classes in the same order for this score averaging to work.
classNames1 = net1.Layers(end).Classes; 
classNames2 = net2.Layers(end).Classes;
% continue with finding the max score, and the accuracy ...

If this answer helps you, please remember to accept the answer.