Text Chunker

Split text into token-sized chunks with overlap.

FreeNo accountOn your device

Runs in your browser. How to verify

Loading tool…

About this tool

Paste long text and get it split into numbered chunks of up to 500 tokens, each overlapping the previous by 50 tokens. This is the basic fixed-size chunking used to prepare documents for embeddings and retrieval. The sizes are fixed in this tool.

How to use

1
Step 1
Paste the text to split.
2
Step 2
Press Run.
3
Step 3
Copy the chunks, each marked --- Chunk n/total ---.

How it works

The text is converted to tokens with o200k_base, cut into windows of 500 tokens, and each new window starts 50 tokens before the previous one ended, so context carries across the boundary. The tool shows how many chunks were produced. Windows are cut by token count, so they can end mid-sentence or mid-word.

step = maxTokens − overlapTokens; windows advance by step across the token ID array

Worked example

A 1,201-token document

Input
"word " repeated 1,200 times (1,201 tokens)
Output
--- Chunk 1/3 --- (500 tokens) --- Chunk 2/3 --- (500 tokens) --- Chunk 3/3 --- (301 tokens)
Walkthrough
Chunk 2 starts at token 450, because of the 50-token overlap, and chunk 3 starts at token 900. The last chunk holds what is left.

Limitations

  • Chunk size (500) and overlap (50) cannot be changed in this tool.
  • No sentence, paragraph or Markdown-aware splitting; boundaries can fall in the middle of a sentence.
  • A boundary inside a multi-byte character (such as an emoji or some CJK text) may produce a replacement character.
  • Tokens are counted with o200k_base, so chunk sizes for other models are approximate.

Questions

How do I chunk text for RAG?
Paste the text and press Run. You get overlapping 500-token chunks ready for embedding.
What chunk size is best for embeddings?
It depends on the model and content; 256 to 512 tokens is a common start. This tool uses 500.
Why overlap chunks?
So a sentence cut at a boundary still appears whole in one of the neighbouring chunks.
Can I change the chunk size?
Not here. The chunk size calculator helps you estimate how many chunks other settings produce.

Features

  • 500-token chunks
  • 50-token overlap
  • Numbered output
  • Runs in your browser

Related resources

Guides and videos from around the web—opens on YouTube or the publisher’s site.

  • Machine Learning for Everybody

    freeCodeCamp.org

    Accessible ML survey for AI calculator and metric tools.

    Watch on YouTube
  • TensorFlow 2.0 Complete Course

    freeCodeCamp.org

    Practical ML stack context next to AI utilities.

    Watch on YouTube