A tokenizer converts text into token identifiers from a fixed vocabulary before the model sees it. Token counts determine context usage and often affect latency and API billing.
A token is a unit of input or output processed by a language model, such as part of a word, punctuation mark, or byte sequence.
A tokenizer converts text into token identifiers from a fixed vocabulary before the model sees it. Token counts determine context usage and often affect latency and API billing.