Token

A token is the basic unit of text that a language model processes, produced by a tokenizer splitting raw text into subword units, words, or characters. Common English words are often single tokens, while rare words are split into multiple subwords.

Token count determines compute cost and context window usage — the OpenAI GPT tokenizer averages approximately 4 characters or 0.75 words per token.