Token
The basic unit of text processing for language models, typically representing a word, subword, or character depending on the tokenizer used.
Definition
The basic unit of text processing for language models, typically representing a word, subword, or character depending on the tokenizer used.
Why It Matters for AI Governance
Token counts directly affect AI costs and governance. Each token in a prompt that contains personal data represents a potential compliance exposure. Cost tracking and PII detection both operate at the token level.
How CrewCheck Handles This
CrewCheck's LLM gateway applies token-related controls at the request boundary. Every AI call passes through detection, policy evaluation, and audit logging — ensuring that token is addressed consistently across all teams and providers.
The governance dashboard provides real-time visibility into token events, with drill-down capabilities for compliance officers and exportable evidence for auditors.