Text Chunker
Split text into token-sized chunks with overlap.
About this tool
Paste long text and get it split into numbered chunks of up to 500 tokens, each overlapping the previous by 50 tokens. This is the basic fixed-size chunking used to prepare documents for embeddings and retrieval. The sizes are fixed in this tool.
How to use
How it works
The text is converted to tokens with o200k_base, cut into windows of 500 tokens, and each new window starts 50 tokens before the previous one ended, so context carries across the boundary. The tool shows how many chunks were produced. Windows are cut by token count, so they can end mid-sentence or mid-word.
step = maxTokens − overlapTokens; windows advance by step across the token ID array
Worked example
A 1,201-token document
- Input
- "word " repeated 1,200 times (1,201 tokens)
- Output
- --- Chunk 1/3 --- (500 tokens) --- Chunk 2/3 --- (500 tokens) --- Chunk 3/3 --- (301 tokens)
- Walkthrough
- Chunk 2 starts at token 450, because of the 50-token overlap, and chunk 3 starts at token 900. The last chunk holds what is left.
Limitations
- Chunk size (500) and overlap (50) cannot be changed in this tool.
- No sentence, paragraph or Markdown-aware splitting; boundaries can fall in the middle of a sentence.
- A boundary inside a multi-byte character (such as an emoji or some CJK text) may produce a replacement character.
- Tokens are counted with o200k_base, so chunk sizes for other models are approximate.
Questions
How do I chunk text for RAG?
What chunk size is best for embeddings?
Why overlap chunks?
Can I change the chunk size?
Features
- 500-token chunks
- 50-token overlap
- Numbered output
- Runs in your browser
Related resources
Guides and videos from around the web—opens on YouTube or the publisher’s site.
Machine Learning for Everybody
freeCodeCamp.org
Accessible ML survey for AI calculator and metric tools.
Watch on YouTubeTensorFlow 2.0 Complete Course
freeCodeCamp.org
Practical ML stack context next to AI utilities.
Watch on YouTube
- OpenAI Research
OpenAI
Research notes for people exploring AI tooling trends.
- Hugging Face docs
Hugging Face
Model hub docs next to tokenizer and embedding helpers.