tiktoken Cheatsheet

Q: Which encoding should I use?

Use `tiktoken.encoding_for_model('gpt-4o')` to get the correct encoding for any supported model. Common encodings: cl100k_base (GPT-4, GPT-3.5), o200k_base (GPT-4o), p50k_base (text-davinci). Use cl100k_base for general purposes.

Q: How do I estimate API costs?

Count tokens with `len(enc.encode(text))`, then multiply by the model's per-token pricing. Note that chat messages have overhead tokens (~3 per message plus roles). tiktoken doesn't calculate costs directly — use it for counts, then apply your pricing formula.

tiktoken is OpenAI's fast BPE tokenizer for encoding text into LLM-understandable token IDs. Supports GPT-4o, GPT-4, GPT-3.5, DeepSeek, and other major models. Accurate token counting is essential for cost control and context window management.

Updated: 2026-07-20·8 commands

Quick Start

``bash pip install tiktoken

python -c " import tiktoken enc = tiktoken.get_encoding('cl100k_base') tokens = enc.encode('Hello world') print(f'Tokens: {len(tokens)}') print(f'IDs: {tokens}') print(f'Decoded: {enc.decode(tokens)}') "``

Startup & Modes(8)

Command	Level	Example
`pip install tiktoken` Install tiktoken	Basic	`pip install tiktoken`
`tiktoken encode` Encode text into token IDs	Basic	`python -c 'import tiktoken; enc = tiktoken.get_encoding("cl100k_base"); tokens = enc.encode("Hello world"); print(tokens)'`
`tiktoken decode` Decode token IDs into text	Basic	`python -c 'import tiktoken; enc = tiktoken.get_encoding("cl100k_base"); print(enc.decode([9906, 1917]))'`
`tiktoken count` Count tokens in text	Basic	`python -c 'import tiktoken; enc = tiktoken.get_encoding("cl100k_base"); print(len(enc.encode("Hello world, how are you?")))'`
`tiktoken encoding for model` Get encoding for a specific model	Basic	`python -c 'import tiktoken; enc = tiktoken.encoding_for_model("gpt-4o"); print(enc.name)'`
`tiktoken batch count` Batch count tokens for multiple texts	Intermediate	`python -c 'import tiktoken; texts=["Hello","World"]; enc=tiktoken.get_encoding("cl100k_base"); print([len(enc.encode(t)) for t in texts])'`
`tiktoken cost estimate` Estimate token count and cost for chat messages	Intermediate	`python -c 'import tiktoken; enc=tiktoken.get_encoding("cl100k_base"); msgs=[{"role":"user","content":"Hello"}]; tpm=3; print(sum(len(enc.encode(m["content"])) for m in msgs) + tpm*len(msgs))'`
`tiktoken special tokens` Encode with special tokens	Expert	`python -c 'import tiktoken; enc=tiktoken.get_encoding("cl100k_base"); tokens=enc.encode("Hello", allowed_special={"<\|endoftext\|>"}); print(tokens)'`

FAQ

Which encoding should I use?

How do I estimate API costs?

This cheatsheet is compiled from official tool documentation. Last updated: 2026-07-20.