This is machine translation

Translated by Microsoft
Mouseover text to see original. Click the button below to return to the English version of the page.

Note: This page has been translated by MathWorks. Click here to see
To view all translated materials including this page, select Country from the country navigator on the bottom of this page.

Optimize Neural Network Training Speed and Memory

Memory Reduction

Depending on the particular neural network, simulation and gradient calculations can occur in MATLAB® or MEX. MEX is more memory efficient, but MATLAB can be made more memory efficient in exchange for time.

To determine whether MATLAB or MEX is being used, use the 'showResources' option, as shown in this general form of the syntax:

net2 = train(net1,x,t,'showResources','yes')

If MATLAB is being used and memory limitations are a problem, the amount of temporary storage needed can be reduced by a factor of N, in exchange for performing the computations N times sequentially on each of N subsets of the data.

net2 = train(net1,x,t,'reduction',N);

This is called memory reduction.

Fast Elliot Sigmoid

Some simple computing hardware might not support the exponential function directly, and software implementations can be slow. The Elliot sigmoid elliotsig function performs the same role as the symmetric sigmoid tansig function, but avoids the exponential function.

Here is a plot of the Elliot sigmoid:

n = -10:0.01:10;
a = elliotsig(n);

Next, elliotsig is compared with tansig.

a2 = tansig(n);
h = plot(n,a,n,a2);

To train a neural network using elliotsig instead of tansig, transform the network’s transfer functions:

[x,t] = bodyfat_dataset;
net = feedforwardnet;
net.layers{1}.transferFcn = 'elliotsig';
net = train(net,x,t);
y = net(x)

Here, the times to execute elliotsig and tansig are compared. elliotsig is approximately four times faster on the test system.

n = rand(5000,5000);
tic,for i=1:100,a=tansig(n); end, tansigTime = toc;
tic,for i=1:100,a=elliotsig(n); end, elliotTime = toc;
speedup = tansigTime / elliotTime

speedup =


However, while simulation is faster with elliotsig, training is not guaranteed to be faster, due to the different shapes of the two transfer functions. Here, 10 networks are each trained for tansig and elliotsig, but training times vary significantly even on the same problem with the same network.

[x,t] = bodyfat_dataset;
tansigNet = feedforwardnet;
tansigNet.trainParam.showWindow = false;
elliotNet = tansigNet;
elliotNet.layers{1}.transferFcn = 'elliotsig';
for i=1:10, tic, net = train(tansigNet,x,t); tansigTime = toc, end
for i=1:10, tic, net = train(elliotNet,x,t), elliotTime = toc, end