Understanding tokens is essential for managing context windows and costs.
Key Points
- Models price by tokens
- Context limits in tokens
- Different tokenizers vary
- Non-English often uses more tokens
Typical Ratios
- 1 token ≈ 4 characters
- 1 token ≈ 0.75 words
- 100 tokens ≈ 75 words