Tokenization

Language models operate on tokens rather than raw sentences. A token may represent a whole word, part of a word, punctuation, or another symbol, and token counts influence context limits, processing cost, latency, and how much information can be provided to a model.