Knowledge base size estimator
Estimate words, tokens and chunks for the documents behind an AI assistant.
FreeRuns in your browserNo signupUpdated 2026-11-05
Your documents
Everything is estimated in your browser.
Tokens
167,650
126,000 words × 1.33
Chunks
610
500-token chunks, 50 overlap
Raw text
738.3 KB
About 6 characters per word
Vector storage
3.57 MB
Before database index overhead
Could it go straight into the prompt?
84% of the context windowIt fits, but leaves little room for the question, instructions and answer. Use retrieval (RAG).
| Source | Words | Tokens | Chunks |
|---|---|---|---|
| Website pages | 21,600 | 28,720 | 80 |
| PDF pages (manuals, brochures) | 43,200 | 57,480 | 120 |
| Help centre articles | 27,000 | 35,940 | 120 |
| Word documents and policies | 27,000 | 35,910 | 90 |
| FAQ entries or spreadsheet rows | 7,200 | 9,600 | 200 |
Tokens are estimated at 1.33 per word, the usual figure for English. To use your own ratio, paste a typical document into a tokenizer, divide its token count by its word count, and enter the result above. Arabic, German and code usually need more tokens per word.
About this tool
How it works and how to read the result.
Need this done for you?
AI agents & chatbots
AI assistants that answer, act and hand over when needed.
See the serviceWhat it does
This knowledge base size calculator estimates how much text sits behind an AI assistant: words, tokens, chunks, raw text size and vector storage. Enter how many website pages, PDF pages, help articles, documents, FAQ entries and past support messages you have, with an average length for each. It removes a share for menus, footers and duplicates, converts words to tokens, splits each item into chunks with overlap and compares the total with a model's context window. All assumptions are editable, and nothing leaves your browser.
How to use it
- Enter the number of items in each source and their average length in words.
- Set how much text you expect to remove in clean-up.
- Keep 1.33 tokens per word for English, or enter a ratio you have measured.
- Set the chunk size, overlap and embedding dimensions to match your setup.
- Enter the context window of the model you plan to use, and read the result.
How to read the result
The default example has 40 website pages, 120 PDF pages, 60 help articles, 15 longer documents and 200 FAQ entries. After removing 10% in clean-up, that is 126,000 words, or about 167,650 tokens at 1.33 tokens per word.
Split into 500-token chunks with 50 tokens of overlap, each item is chunked on its own: a 2,000-word policy, 1,800 words after clean-up, becomes 6 chunks, while each FAQ entry is a single chunk. That gives 610 chunks. At 1,536 dimensions stored as 4-byte floats, the vectors take about 3.6 MB, and the raw text about 740 KB.
Against a 200,000-token context window, the documents fill 84%. They would technically fit in one prompt, but leave little room for instructions, the question and the answer, so the panel recommends retrieval.
The words-to-tokens ratio is the main assumption. To adjust it, paste a typical document into a tokenizer, divide the token count by the word count and enter the result. Arabic or German text will usually give a higher ratio, which raises every figure.
We build AI assistants over company documents, including clean-up and chunking, as part of AI agents. Measure your own ratio with the token counter, and work out embedding costs with the RAG chunk size calculator.
Questions
Frequently asked.
How many tokens are in a word?
For English, about 1.33 tokens per word with current OpenAI and Anthropic tokenizers, or roughly 0.75 words per token. It varies with the text: technical terms, code, numbers and other languages such as Arabic or German usually need more. Measure a typical document with a tokenizer and enter your own ratio for a closer estimate.
Should I put my whole knowledge base in the prompt?
Only if it is small. Sending everything with each question costs more per question and leaves less room for instructions and the answer. Retrieval (RAG) sends only the relevant chunks. As a rule of thumb, if your documents fill more than about half the context window, use retrieval.
What chunk size should I use?
Between about 300 and 800 tokens suits most business documents, with an overlap of around 10%. Short FAQ entries work best as one chunk each. Long manuals benefit from chunks that follow headings. Test retrieval on real questions and adjust from there.
More in AI & automation
Related tools.
Your next move
Like what you see? Tell us about it.
A process that takes too long, a website that undersells you, or an idea ready to take shape. Tell us where you are and we will suggest a useful next step.
India · Working with clients worldwide