Learning Rate Calculator

Warmup + cosine LR schedule with batch scaling.

FreeNo accountOn your device

Runs in your browser. How to verify

Loading tool…

About this tool

Enter a base learning rate, total steps and warmup steps, then pick a step to read the learning rate there. The schedule warms up linearly and then decays on a cosine curve to 10% of the base. You can also scale the base by batch size.

How to use

1
Step 1
Enter base LR, total steps, warmup steps and the step to evaluate.
2
Step 2
Set batch size and reference batch for scaling (equal values mean no change).
3
Step 3
Press Calculate.

How it works

For steps before warmup, lr = base × step / warmup. After warmup, lr = base × (0.1 + 0.9 × ½(1 + cos(π × progress))) where progress runs from 0 to 1 over the remaining steps. The base is first multiplied by batch ÷ reference batch (linear scaling). The table also shows the rate at 0, warmup, halfway and the final step.

warmup: lr = base · t / warmup      then: lr = base · (0.1 + 0.9 · ½(1 + cos(π · progress)))

Worked example

A 10k-step run

Input
Base 3e-4; total 10,000; warmup 300; evaluate at step 500; batch 256; reference 256
Output
LR at 500 = 0.000299717; at 300 = 0.0003; at 5,000 = 0.000171556; at 10,000 = 3e-05
Walkthrough
At step 150, halfway through warmup, the rate is 0.00015. With batch 512 against a reference of 256, the base becomes 0.0006.

Limitations

  • The final learning rate is fixed at 10% of the base; it cannot be changed here.
  • Cosine decay only; no linear, constant or inverse-square-root schedules.
  • No chart of the curve, only sample points.

Questions

How do I calculate a learning rate with warmup and cosine decay?
Enter the base rate, total steps, warmup steps and an evaluation step.
How many warmup steps should I use?
A few percent of total steps is common; this tool's default is 300 for 10,000.
What is linear batch-size scaling?
Multiply the learning rate by batch size over a reference batch size. Doubling the batch doubles the rate.
Does it decay to zero?
No. It decays to 10% of the base.

Features

  • Warmup + cosine
  • Batch scaling
  • Sample points

Related resources

Guides and videos from around the web—opens on YouTube or the publisher’s site.

  • Mean, Variance and Standard Deviation

    StatQuest with Josh Starmer

    Stats foundations behind many ML metrics tools.

    Watch on YouTube
  • Finding mean, median, and mode

    Khan Academy

    Baseline stats for accuracy and evaluation helpers.

    Watch on YouTube