Token (AI)

In the AI ✕ Artificial intelligence is a computer system’s ability to perform tasks we associate with human intelligence, such as understanding language, recognizing images or making decisions. More in the glossary → context, a token is the smallest unit of text into which a language model splits what it reads and writes. It does not always match a word: it can be a whole short word, a chunk of a longer one, or a punctuation mark; as a rough rule, in English one token is about four characters. Models do not see letters or words as such, but these pieces turned into numbers. The concept matters in practice for two reasons: the cost of AI tools is usually calculated per token processed, and each model has a maximum number of tokens it can handle at once (its context window ✕ The context window is how much text a model can hold "in mind" at once. Exceeding it makes it forget what was said at the start. More in the glossary → ). That is why very long texts consume more and may not fit whole. Understanding tokens helps you estimate costs and avoid exceeding limits.