Model Parameter Calculator

Rough width×depth parameter heuristic for neural nets.

FreeNo accountOn your device

Runs in your browser. How to verify

Loading tool…

About this tool

A one-line estimate of how big a network is from its width, depth and vocabulary. Parameters are about multiplier × depth × width² plus vocab × width, with a multiplier of 12 for standard transformers. With the defaults it gives 123.5 million.

How to use

1
Step 1
Enter width, depth and vocabulary size.
2
Step 2
Leave the multiplier at 12 for a transformer.
3
Step 3
Press Calculate.

How it works

A transformer layer holds about 12·d² weights (4d² in attention plus 8d² in a feed-forward block with d_ff = 4d). Multiply by depth and add the embedding table, vocab × width. This is the fast back-of-envelope used in scaling-law discussions.

N ≈ multiplier · depth · width² + vocab · width

Worked example

Default 768-wide, 12-layer model

Input
Width 768; depth 12; vocab 50,257; multiplier 12
Output
123.532M (body 84.935M + embedding 38.597M)
Walkthrough
12 × 12 × 768² = 84,934,656 for the layers, and 50,257 × 768 = 38,597,376 for the embeddings.

Limitations

  • Assumes d_ff = 4·width. Other widths need the transformer parameter calculator.
  • Ignores biases, position embeddings and untied output heads.
  • Not valid for convolutional or mixture-of-experts models.

Questions

How many parameters does a model with width 768 and 12 layers have?
About 124 million including a 50k-token embedding table.
Why is the multiplier 12?
4d² for attention plus 8d² for a feed-forward block with 4× expansion.
How is it different from the transformer parameter calculator?
That one uses separate d_ff, heads and layer norm; this is a quick two-term heuristic.
Is it accurate for billion-parameter models?
Roughly; for large models the embedding term is small and the 12·L·d² term dominates.

Features

  • One-line heuristic
  • Body and embedding split
  • Runs in your browser

Related resources

Guides and videos from around the web—opens on YouTube or the publisher’s site.

  • Mean, Variance and Standard Deviation

    StatQuest with Josh Starmer

    Stats foundations behind many ML metrics tools.

    Watch on YouTube
  • Vectors | Essence of linear algebra

    3Blue1Brown

    Embeddings and vectors start with linear algebra intuition.

    Watch on YouTube