What is Tokenization?
Tokenization — The process of breaking down text into smaller units (tokens) that AI models can process and understand.
Tokens are not always whole words. Common words like ‘the’ are single tokens, while uncommon words may be split into multiple tokens. A rough rule: 1 token ≈ 4 characters in English. Token counts directly affect API costs and context window limits.
Frequently Asked Questions
How do tokens affect AI costs?
Most AI APIs charge per token processed. Longer inputs and outputs cost more. Understanding tokenization helps you optimize prompts and reduce expenses.
Are tokens the same across all models?
No. Each model family uses its own tokenizer. The same sentence may produce different token counts on GPT-4 vs. Claude vs. Llama.
What is a context window?
The context window is the maximum number of tokens a model can process in a single request. Larger windows allow more input data but increase cost and latency.