How to use Levenberg-Marquardt backprop with GPU?

Levenberg-Marquardt backprop train my shallow neural net very efficienetly and gives a very good result. However, it doesn't seem to support GPU training. Is there a way to implement GPU support for Levenberg-Marquardt backprop?

Thanks

Answer 1

[Joss Knight]

This isn't supported out of the box yet. You could convert your network to use dlarray and train it with a custom training loop. Then you could write your own LevMarq solver.

[psimeson]

Do you guys have plan to add useGPU option for trainlm?

[Joss Knight]

No, there are no plans to further develop trainlm.

[psimeson]

I am new to matlab. I have been using it for last couple of weeks.

Here's my nn:

net = feedforwardnet([10 20 20 10]);
net.layers{1}.transferFcn = 'logsig';
net.layers{2}.transferFcn = 'radbas';
net.layers{3}.transferFcn = 'logsig';
net.layers{4}.transferFcn = 'purelin';
net = train(net,input.',output.');

By default, MatLab uses trainlm to train this net. To speed it up I use parallel pool, which uses 12 CPUs. However, it is not fast enough. For this process can I use more than 12 CPUs? If yes then how do I go about doing so on HPC platforms?

If the GPU implementation of LM is not that hard to code then please guide me and I will try to code it up.

[Joss Knight]

This looks trivial to translate into a deep learning version. I'll ask someone to help you do so.

You can create a parallel pool with more than 12 workers by creating a new local profile without the upper limit. But it rarely helps. If your computer has 12 CPUs, to run more processes just means queuing more things up.

[psimeson]

Sounds good.

I can run my code on high preformace computing(HPC), so I can access lots of CPUs. Could you please suggest me a way to create local profile on a HPC? I want to give it a try.

[Joss Knight]

https://www.google.com/search?rlz=1C1GCEB_enGB814GB815&sxsrf=ACYBGNR0__V12oULUwGb7__4q4XaTRfrkQ%3A1578068556979&ei=TGoPXqi6O9KM8gLK6JL4Cg&q=MATLAB+help+local+cluster+profile&oq=MATLAB+help+local+cluster+profile&gs_l=psy-ab.3..33i21.16536.17319..17504...0.2..0.65.451.8......0....1..gws-wiz.......0i71j35i304i39j33i13i21.WvH4-5OW8AY&ved=0ahUKEwiokJS-6-fmAhVShlwKHUq0BK8Q4dUDCAs&uact=5

[psimeson]

Thanks Joss for the link.

Did you ask someone to help me with my problem?

[Joss Knight]

I have asked. Unfortunately I don't know enough about shallow nets to help you directly.

[psimeson]

Please keep me in the loop. Thanks a lot.

[Amanjit Dulai]

Here is how you might translate the example above to use some of our newer functionality. In the example below, we train the network you described earlier on a simple regression problem. See the function 'iNetworkForward' for the definition of the network:

% Load the data
[X,T] = simplefit_dataset;
inputSize = size(X,1);
outputSize = size(T,1);

% Split the data into test and training data
rng('default');
testFraction = 0.15;
[XTrain, TTrain, XTest, TTest] = ...
    iSplitIntoTrainAndTestSets(X, T, testFraction);

% Initialize the weights for the network
layerSizes = [10 20 20];
params = iInitializeWeights(inputSize, layerSizes, outputSize);

% Specify the training options
executionEnvironment = "cpu";
velocity = [];
miniBatchSize = 20;
numEpochs = 1000;
numObservations = size(XTrain,2);
numIterationsPerEpoch = floor(numObservations./miniBatchSize);

% Cast the data to dlarray
XTrain = dlarray(XTrain, 'CB');
TTrain = dlarray(TTrain, 'CB');
if executionEnvironment == "gpu"
    XTrain = gpuArray(XTrain);
    TTrain = gpuArray(TTrain);
end

% Train the model
for epoch = 1:numEpochs
    for iteration = 1:numIterationsPerEpoch
        % Get a batch of data.
        indices = (iteration-1)*miniBatchSize+1:iteration*miniBatchSize;
        XBatch = XTrain(:,indices);
        TBatch = TTrain(:,indices);
        
        % Get the loss and gradients
        [loss, gradients] = dlfeval( ...
            @iNetworkForwardWithLoss, params, XBatch, TBatch );
        
        % Update the network
        [params, velocity] = sgdmupdate(params, gradients, velocity);
        
        % Report the loss
        fprintf('Loss: %f\n', extractdata(loss));
    end
end

% Run the network on test data
XTest = dlarray(XTest, 'CB');
YTest = iNetworkForward(params, XTest);
YTest = extractdata(YTest);

% Plot the ground truth and predicted data
plot([TTest' YTest']);

%% Helper functions
function [XTrain,TTrain,XTest,TTest] = iSplitIntoTrainAndTestSets( ...
    X, T, testFraction )
numObservations = size(X,2);
idx = randperm(numObservations);
splitIndex = floor(numObservations*testFraction);
testIdx = idx(1:splitIndex);
trainIdx = idx((splitIndex+1):end);
XTrain = X(:,trainIdx);
TTrain = T(:,trainIdx);
XTest = X(:,testIdx);
TTest = T(:,testIdx);
end

function params = iInitializeWeights(inputSize, layerSizes, outputSize)
params = struct;
params.W1 = dlarray( iGlorot(inputSize, layerSizes(1)) );
params.b1 = dlarray( zeros([layerSizes(1) 1]) );
params.W2 = dlarray( iGlorot(layerSizes(1), layerSizes(2)) );
params.b2 = dlarray( zeros([layerSizes(2) 1]) );
params.W3 = dlarray( iGlorot(layerSizes(2), layerSizes(3)) );
params.b3 = dlarray( zeros([layerSizes(3) 1]) );
params.W4 = dlarray( iGlorot(layerSizes(3), outputSize) );
params.b4 = dlarray( zeros([outputSize 1]) );
end

function weights = iGlorot(fanIn, fanOut)
weights = (2*rand([fanOut fanIn])-1) * sqrt(6/(fanIn+fanOut));
end

function Y = iNetworkForward(params, X)
Z1 = fullyconnect(X, params.W1, params.b1);     % 1st fully connected layer
Z1 = sigmoid(Z1);                               % Logistic sigmoid
Z2 = fullyconnect(Z1, params.W2, params.b2);    % 2nd fully connected layer
Z2 = exp(-Z2.^2);                               % Radial basis function
Z3 = fullyconnect(Z2, params.W3, params.b3);    % 3rd fully connected layer
Z3 = sigmoid(Z3);                               % Logistic sigmoid
Y = fullyconnect(Z3, params.W4, params.b4);     % 4th fully connected layer
end

function [loss, dLossdW] = iNetworkForwardWithLoss(weights, X, T)
Y = iNetworkForward(weights, X);
loss = mse(Y, T)/size(T,1);
dLossdW = dlgradient(loss, weights);
end

There are a few differences from doing things this way compared to using 'feedforwardnet':

• As you mentioned, 'feedforwardnet' is trained with Levenberg Marquardt by default. But the example above is using stochastic gradient descent with momentum, which is simpler.
• The example above uses the 'Glorot' weight initializer, which is a more modern technique associated with Deep Learning. 'feedforwardnet' uses the Nguyen-Widrow method.
• The example above does not perform any scaling on the data. 'feedforwardnet' will by default rescale the input and target data to the range -1 to 1. Sometimes this can help training.