piecewiseLearnRate

Piecewise learning rate schedule

Since R2024b

Description

A piecewise learning rate schedule object drops the learning rate periodically by multiplying it by a specified factor.

Tip

To easily use a piecewise learning rate schedule with the default options, specify the LearnRateSchedule argument of the trainingOptions function as "piecewise".

Creation

Description

schedule = piecewiseLearnRate creates a piecewiseLearnRate object.

This syntax is equivalent to setting the LearnRateSchedule argument of the trainingOptions function to "piecewise".

schedule = piecewiseLearnRate(Name=Value) specifies optional properties using one or more name-value arguments. For example, DropFactor=0.5 specifies halving the learning rate.

Properties

DropFactor — Scaling factor used to drop the learning rate

0.1 (default) | scalar in the range (0,1)

Scaling factor used to drop the learning rate, specified as a scalar in the range (0,1).

If FrequencyUnit is "epoch", then the software updates the learning rate every Period epochs. If FrequencyUnit is "iteration", then the software updates the learning rate every Period iterations.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Period — Number of steps before updating learning rate

1 (default) | positive integer

Number of steps before updating the learning rate, specified as a positive integer.

If FrequencyUnit is "epoch", then the software updates the learning rate every Period epochs. If FrequencyUnit is "iteration", then the software updates the learning rate every Period iterations.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

FrequencyUnit — Frequency unit

"epoch" (default) | "iteration"

Frequency unit, specified as "epoch" or "iteration".

If FrequencyUnit is "epoch", then the software updates the learning rate every Period epochs. If FrequencyUnit is "iteration", then the software updates the learning rate every Period iterations.

NumSteps — Number of steps to drop learning rate

Inf (default)

This property is read-only.

Number of steps to drop learning rate, specified as Inf.

piecewiseLearnRate objects are infinite learning rate schedules, so NumSteps is Inf.

Data Types: double

Examples

Create Piecewise Learning Rate Schedule

Create a piecewise learning rate schedule with the default settings.

schedule = piecewiseLearnRate

schedule = 
  piecewiseLearnRate with properties:

       DropFactor: 0.1000
           Period: 10
    FrequencyUnit: "epoch"
         NumSteps: Inf

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

As an alternative to creating a piecewise learning rate schedule object, to easily use a piecewise learning rate schedule with the default options, specify the LearnRateSchedule argument of the trainingOptions function as "piecewise".

options = trainingOptions("adam",LearnRateSchedule="piecewise");

Customize Piecewise Learning Rate Schedule

Create a piecewise learning rate schedule that halves the learning rate every 100 iterations.

schedule = piecewiseLearnRate( ...
    DropFactor=0.5, ...
    Period=100, ...
    FrequencyUnit="iteration")

schedule = 
  piecewiseLearnRate with properties:

       DropFactor: 0.5000
           Period: 100
    FrequencyUnit: "iteration"
         NumSteps: Inf

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

Algorithms

Piecewise Learning Rate Schedule

A piecewise learning rate schedule object drops the learning rate periodically by multiplying it by a specified factor.

The software uses this formula to calculate the learning rate:

$$\alpha ={\alpha }_0{\gamma }^{\lfloor k/P\rfloor },$$

where:

• α₀ is the base learning rate, specified by the InitialLearnRate option of the trainingOptions function.

• γ is the scaling factor, specified by the DropFactor argument.

• P is the period, specified by the Period argument.

If FrequencyUnit is "iteration", then the variable k denotes the schedule iteration number. The value of k can be different from the training iteration number. For example, if you train using {"warmup",schedule}, where schedule is a piecewiseLearnRate object with FrequencyUnit set to "iteration", then at training iteration 15, the piecewiseLearnRate object uses k with a value of 10 because the warm-up learning rate schedule runs for 5 training iterations.

If FrequencyUnit is "epoch", then the variable k denotes the schedule epoch number. In this case, the schedule uses the same learning rate for each iteration of the epoch. The value of k can be different from the training epoch number. For example, if you train using {warmupLearnRate(FrequencyUnit="epoch"),schedule}, where schedule is a piecewiseLearnRate object with FrequencyUnit set to "epoch", then at training epoch 15, the piecewiseLearnRate object uses k with a value of 10 because the warm-up learning rate schedule processes the first 5 epochs.

This plot shows an overview of the piecewise learning rate schedule with the default settings.

Version History

Introduced in R2024b