All glossary terms
AI Fundamentals & AI for Proposals
Token
An individual unit of text, such as a word or piece of punctuation, that an AI language model processes; tokenization is the act of breaking text into these units.
An individual unit of text, such as a word or piece of punctuation, that an AI language model processes; tokenization is the act of breaking text into these units.