Learning Rate Calculator
Warmup + cosine LR schedule with batch scaling.
About this tool
Enter a base learning rate, total steps and warmup steps, then pick a step to read the learning rate there. The schedule warms up linearly and then decays on a cosine curve to 10% of the base. You can also scale the base by batch size.
How to use
How it works
For steps before warmup, lr = base × step / warmup. After warmup, lr = base × (0.1 + 0.9 × ½(1 + cos(π × progress))) where progress runs from 0 to 1 over the remaining steps. The base is first multiplied by batch ÷ reference batch (linear scaling). The table also shows the rate at 0, warmup, halfway and the final step.
warmup: lr = base · t / warmup then: lr = base · (0.1 + 0.9 · ½(1 + cos(π · progress)))
Worked example
A 10k-step run
- Input
- Base 3e-4; total 10,000; warmup 300; evaluate at step 500; batch 256; reference 256
- Output
- LR at 500 = 0.000299717; at 300 = 0.0003; at 5,000 = 0.000171556; at 10,000 = 3e-05
- Walkthrough
- At step 150, halfway through warmup, the rate is 0.00015. With batch 512 against a reference of 256, the base becomes 0.0006.
Limitations
- The final learning rate is fixed at 10% of the base; it cannot be changed here.
- Cosine decay only; no linear, constant or inverse-square-root schedules.
- No chart of the curve, only sample points.
Questions
How do I calculate a learning rate with warmup and cosine decay?
How many warmup steps should I use?
What is linear batch-size scaling?
Does it decay to zero?
Features
- Warmup + cosine
- Batch scaling
- Sample points
Related resources
Guides and videos from around the web—opens on YouTube or the publisher’s site.
Mean, Variance and Standard Deviation
StatQuest with Josh Starmer
Stats foundations behind many ML metrics tools.
Watch on YouTubeFinding mean, median, and mode
Khan Academy
Baseline stats for accuracy and evaluation helpers.
Watch on YouTube
- Machine Learning Crash Course
Google Developers
Official ML crash course pairing with metric calculators.
- Data Index
Math is Fun
Stats primers when reading ML evaluation outputs.