Transformer Parameter Calculator

Estimate GPT-style transformer parameter counts.

FreeNo accountOn your device

Runs in your browser. How to verify

Loading tool…

About this tool

Enter the vocabulary size, model width (d_model), number of layers, heads and feed-forward width and get an estimate of total parameters, split into embeddings, attention, feed-forward and layer norm. With GPT-2 small settings it gives about 123.6 million.

How to use

1
Step 1
Enter vocab size, d_model, layers, heads and d_ff.
2
Step 2
Press Calculate.
3
Step 3
Read the total and the breakdown.

How it works

Embedding = vocab × d_model. Per layer, attention is about 4·d² (Q, K, V and output projections) and the feed-forward network is 2·d·d_ff. Layer-norm scale and shift add 2·d per norm. The LM head is assumed to share weights with the embedding (tied), so it adds nothing. Head dim is d_model / heads.

N ≈ V·d + L·(4d² + 2·d·d_ff) + layer norms

Worked example

GPT-2 small

Input
Vocab 50,257; d_model 768; layers 12; heads 12; d_ff 3,072
Output
123.569M total (123,568,896): embedding 38.597M, attention 28.312M, FFN 56.623M, LayerNorm 36.864K
Walkthrough
Attention is 12 × 4 × 768² = 28.3M and FFN is 12 × 2 × 768 × 3072 = 56.6M. The model is usually quoted as 124M because that includes position embeddings and biases.

Limitations

  • An estimate: biases, learned position embeddings and the final layer norm are not counted, so real models differ by under 1% for GPT-2-size models.
  • Assumes tied input and output embeddings.
  • Models with grouped-query attention, gated feed-forward (SwiGLU) or mixture-of-experts layers need different formulas.

Questions

How do I calculate the number of parameters in a transformer?
Add the embedding (vocab × width) to layers × (4·width² + 2·width·d_ff). The tool does it from your inputs.
Why is GPT-2 small 124M and not 123.6M?
The official count includes positional embeddings and bias terms, which this estimate leaves out.
Does it work for LLaMA-style models?
Only roughly. Gated feed-forward layers and grouped-query attention change the counts.
What is d_ff?
The hidden width of the feed-forward block, usually 4 times d_model.

Features

  • Component breakdown
  • GPT-2 defaults
  • Runs in your browser

Related resources

Guides and videos from around the web—opens on YouTube or the publisher’s site.

  • Machine Learning for Everybody

    freeCodeCamp.org

    Accessible ML survey for AI calculator and metric tools.

    Watch on YouTube
  • Vectors | Essence of linear algebra

    3Blue1Brown

    Embeddings and vectors start with linear algebra intuition.

    Watch on YouTube
  • Data Index

    Math is Fun

    Stats primers when reading ML evaluation outputs.

  • OpenAI Research

    OpenAI

    Research notes for people exploring AI tooling trends.