Tokenization is the process of splitting text or other data into units that a model can process.
For language models, tokens may represent words, parts of words, punctuation, or other symbols, and tokenization affects context length, cost, and output behavior.