Model Parameter Calculator
Rough width×depth parameter heuristic for neural nets.
About this tool
A one-line estimate of how big a network is from its width, depth and vocabulary. Parameters are about multiplier × depth × width² plus vocab × width, with a multiplier of 12 for standard transformers. With the defaults it gives 123.5 million.
How to use
How it works
A transformer layer holds about 12·d² weights (4d² in attention plus 8d² in a feed-forward block with d_ff = 4d). Multiply by depth and add the embedding table, vocab × width. This is the fast back-of-envelope used in scaling-law discussions.
N ≈ multiplier · depth · width² + vocab · width
Worked example
Default 768-wide, 12-layer model
- Input
- Width 768; depth 12; vocab 50,257; multiplier 12
- Output
- 123.532M (body 84.935M + embedding 38.597M)
- Walkthrough
- 12 × 12 × 768² = 84,934,656 for the layers, and 50,257 × 768 = 38,597,376 for the embeddings.
Limitations
- Assumes d_ff = 4·width. Other widths need the transformer parameter calculator.
- Ignores biases, position embeddings and untied output heads.
- Not valid for convolutional or mixture-of-experts models.
Questions
How many parameters does a model with width 768 and 12 layers have?
Why is the multiplier 12?
How is it different from the transformer parameter calculator?
Is it accurate for billion-parameter models?
Features
- One-line heuristic
- Body and embedding split
- Runs in your browser
Related resources
Guides and videos from around the web—opens on YouTube or the publisher’s site.
Mean, Variance and Standard Deviation
StatQuest with Josh Starmer
Stats foundations behind many ML metrics tools.
Watch on YouTubeVectors | Essence of linear algebra
3Blue1Brown
Embeddings and vectors start with linear algebra intuition.
Watch on YouTube
- Metrics and scoring
scikit-learn
Precision, recall, F1 definitions used by AI metric tools.
- Hugging Face docs
Hugging Face
Model hub docs next to tokenizer and embedding helpers.