polynomialLearnRate

Polynomial learning rate schedule

Since R2024b

Description

A polynomial learning rate schedule drops the learning rate using a power law.

Tip

For most tasks, setting the NumSteps argument to the number of training steps can yield better results. To easily use a polynomial learning rate schedule that drops the learning rate over the length of the training process, specify the LearnRateSchedule argument of the trainingOptions function as "polynomial".

Creation

Description

schedule = polynomialLearnRate creates a polynomialLearnRate object.

schedule = polynomialLearnRate(Name=Value) specifies optional properties using one or more name-value arguments. For example, Power=2 specifies an exponent of 2 in the power law.

Properties

InitialFactor — Initial scaling factor

1 (default) | positive scalar

Initial scaling factor, specified as a positive scalar.

The software starts by scaling the base learning rate by InitialFactor and transitions toward scaling the learning rate by FinalFactor.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

FinalFactor — Final scaling factor

1e-5 (default) | positive scalar

Final scaling factor, specified as a positive scalar.

The software starts by scaling the base learning rate by InitialFactor and transitions toward scaling the learning rate by FinalFactor.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Power — Exponent of power law

1 (default) | positive scalar

Exponent of the power law, specified as a positive scalar.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

NumSteps — Number of steps to drop learning rate

30 (default) | finite positive integer

Number of steps to drop the learning rate, specified as a finite positive integer.

If FrequencyUnit is "iteration", then the software updates the learning rate each iteration for NumSteps iterations. If FrequencyUnit is "epoch", then the software updates the learning rate each epoch for NumSteps epochs.

Tip

For most tasks, setting the NumSteps argument to the number of training steps can yield better results. To easily use a polynomial learning rate schedule that drops the learning rate over the length of the training process, specify the LearnRateSchedule argument of the trainingOptions function as "polynomial".

polynomialLearnRate objects are finite learning rate schedules, so NumSteps is a finite positive integer.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

FrequencyUnit — Frequency unit

"epoch" (default) | "iteration"

Frequency unit, specified as "epoch" or "iteration".

If FrequencyUnit is "iteration", then the software updates the learning rate each iteration for NumSteps iterations. If FrequencyUnit is "epoch", then the software updates the learning rate each epoch for NumSteps epochs.

Examples

Create Polynomial Learning Rate Schedule

Create a polynomial learning rate schedule with the default settings.

schedule = polynomialLearnRate

schedule = 
  polynomialLearnRate with properties:

    InitialFactor: 1
      FinalFactor: 1.0000e-05
            Power: 1
    FrequencyUnit: "epoch"
         NumSteps: 30

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

For most tasks, setting the NumSteps argument to the number of training steps can yield better results. As an alternative to creating a polynomial learning rate object, to easily use a polynomial learning rate schedule that drops the learning rate over the length of the training process, specify the LearnRateSchedule argument of the trainingOptions function as "polynomial".

options = trainingOptions("adam",LearnRateSchedule="polynomial");

Customize Polynomial Learning Rate Schedule

Create a polynomial learning rate schedule that drops the learning rate for 100 iterations, starting with an initial scaling factor of 0.5.

schedule = polynomialLearnRate( ...
    NumSteps=100, ...
    InitialFactor=0.5)

schedule = 
  polynomialLearnRate with properties:

    InitialFactor: 0.5000
      FinalFactor: 1.0000e-05
            Power: 1
    FrequencyUnit: "epoch"
         NumSteps: 100

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

Algorithms

Polynomial Learning Rate Schedule

A polynomial learning rate schedule drops the learning rate using a power law.

The software uses this formula to calculate the learning rate:

$$\alpha =\{\begin{array}{cc}\left({\gamma }_N+{\left(1-\frac{k-1}{N}\right)}^{\lambda }\left({\gamma }_0-{\gamma }_N\right)\right){\alpha }_0& \text{if }k\le N\\ {\gamma }_N{\alpha }_0& \text{if }k>N\end{array},$$

where:

• α₀ is the base learning rate, specified by the InitialLearnRate option of the trainingOptions function.

• γ₀ is the initial scaling factor, specified by the InitialFactor argument.

• γ_N is the final scaling factor, specified by the FinalFactor argument.

• λ is the power, specified by the Power argument.

• N is the number of steps, specified by the NumSteps argument.

If FrequencyUnit is "iteration", then the variable k denotes the schedule iteration number. The value of k can be different from the training iteration number. For example, if you train using {"warmup",schedule}, where schedule is a polynomialLearnRate object with FrequencyUnit set to "iteration", then at training iteration 15, the polynomialLearnRate object uses k with a value of 10 because the warm-up learning rate schedule runs for 5 training iterations.

If FrequencyUnit is "epoch", then the variable k denotes the schedule epoch number. In this case, the schedule uses the same learning rate for each iteration of the epoch. The value of k can be different from the training epoch number. For example, if you train using {warmupLearnRate(FrequencyUnit="epoch"),schedule}, where schedule is a polynomialLearnRate object with FrequencyUnit set to "epoch", then at training epoch 15, the polynomialLearnRate object uses k with a value of 10 because the warm-up learning rate schedule processes the first 5 epochs.

This plot shows an overview of the polynomial learning rate schedule with the NumSteps argument set to the length of the training process.

Version History

Introduced in R2024b