Transformer Parameter Calculator
Estimate GPT-style transformer parameter counts.
About this tool
Enter the vocabulary size, model width (d_model), number of layers, heads and feed-forward width and get an estimate of total parameters, split into embeddings, attention, feed-forward and layer norm. With GPT-2 small settings it gives about 123.6 million.
How to use
How it works
Embedding = vocab × d_model. Per layer, attention is about 4·d² (Q, K, V and output projections) and the feed-forward network is 2·d·d_ff. Layer-norm scale and shift add 2·d per norm. The LM head is assumed to share weights with the embedding (tied), so it adds nothing. Head dim is d_model / heads.
N ≈ V·d + L·(4d² + 2·d·d_ff) + layer norms
Worked example
GPT-2 small
- Input
- Vocab 50,257; d_model 768; layers 12; heads 12; d_ff 3,072
- Output
- 123.569M total (123,568,896): embedding 38.597M, attention 28.312M, FFN 56.623M, LayerNorm 36.864K
- Walkthrough
- Attention is 12 × 4 × 768² = 28.3M and FFN is 12 × 2 × 768 × 3072 = 56.6M. The model is usually quoted as 124M because that includes position embeddings and biases.
Limitations
- An estimate: biases, learned position embeddings and the final layer norm are not counted, so real models differ by under 1% for GPT-2-size models.
- Assumes tied input and output embeddings.
- Models with grouped-query attention, gated feed-forward (SwiGLU) or mixture-of-experts layers need different formulas.
Questions
How do I calculate the number of parameters in a transformer?
Why is GPT-2 small 124M and not 123.6M?
Does it work for LLaMA-style models?
What is d_ff?
Features
- Component breakdown
- GPT-2 defaults
- Runs in your browser
Related resources
Guides and videos from around the web—opens on YouTube or the publisher’s site.
Machine Learning for Everybody
freeCodeCamp.org
Accessible ML survey for AI calculator and metric tools.
Watch on YouTubeVectors | Essence of linear algebra
3Blue1Brown
Embeddings and vectors start with linear algebra intuition.
Watch on YouTube
- Data Index
Math is Fun
Stats primers when reading ML evaluation outputs.
- OpenAI Research
OpenAI
Research notes for people exploring AI tooling trends.