#Tokenization

Articles about Tokenization — exploring patterns, best practices, and real-world implementations in production systems.

1 post tagged with tokenization. ← All posts

#A2A (5)#ADK (8)#AG-UI (6)#AI (9)#AI Agents (212)#AI Engineering (39)#AI Governance (16)#AI Red Teaming (8)#AI Safety (6)#AI Security (16)#AML (3)#API Design (9)#API Security (8)#APIs (37)#AWS (9)#Accounting (9)#Agent Skills (3)#Agentic AI (24)#Agentic Commerce (12)#Agents (4)#Amazon Bedrock (8)#Architecture (36)#Audit (3)#Automation (6)#Azure (11)#Azure AI Foundry (9)#Backend Engineering (220)#BigQuery (6)#CI/CD (7)#Capital Markets (14)#Card Payments (12)#Cards (13)#Career (7)#Checkpointing (4)#Claude Code (8)#Cloud Architecture (3)#Code Review (8)#Collaboration (4)#Communication (4)#Compliance (42)#Concurrency (23)#Consulting (3)#Cost Optimisation (5)#Credit (14)#Credit Risk (14)#Crypto (12)#Cryptocurrency (12)#Custody (9)#Data (4)#Data Structures (3)#Databases (14)#Deployment (3)#DevSecOps (10)#Developer Experience (3)#Distributed Systems (64)#Documentation (3)#Embeddings (11)#Engineering (7)#Engineering Culture (3)#Engineering Practices (16)#Evaluation (31)#FREE-AI (8)#FX (5)#FinOps (5)#FinTech (6)#Financial AI (14)#Financial Systems (122)#Fintech (116)#Foreign Exchange (5)#Forward Deployed Engineer (8)#Forward Deployed Engineering (8)#Fraud (10)#Function Tools (5)#GCP (5)#Go (220)#Google ADK (36)#Governance (46)#Granite (6)#GraphQL (3)#Guardrails (21)#HIPAA (3)#HTTP (3)#Harness Engineering (8)#Human-in-the-Loop (8)#IBM watsonx (8)#Integration (3)#Interfaces (3)#KYC (11)#KYC and AML (12)#Kubernetes (8)#LLM (5)#LLMs (45)#LangGraph (11)#Ledger (12)#Lending (14)#MCP (10)#MLOps (3)#Machine Learning (12)#Markets (4)#Memory (8)#Microservices (3)#Microsoft Agent Framework (150)#Middleware (6)#Migration (9)#Monitoring (3)#Multi-Agent (10)#Multi-Agent AI (14)#Multi-Agent Systems (49)#Multimodal (3)#NIM (5)#NVIDIA (8)#OWASP (7)#Observability (25)#Open Source (6)#OpenTelemetry (4)#Opinion (6)#Orchestration (10)#Payment Rails (16)#Payments (51)#Performance (6)#Privacy Engineering (3)#Product (4)#Production (6)#Programming (10)#Prompt Engineering (26)#Prompt Injection (7)#Providers (4)#Python (89)#Quality (3)#RAG (23)#RBI (3)#REST (5)#Rails (16)#Reconciliation (3)#Regulation (9)#Reliability (27)#Resilience (3)#Responsible AI (4)#Retrieval (3)#Risk (13)#SRE (4)#Scalability (3)#Security (63)#Sessions (3)#Settlement (9)#Software Architecture (9)#Software Engineering (94)#Spanner (4)#Streaming (21)#Structured Output (4)#System Design (8)#Testing (30)#Tokenization (1)#Tool Use (9)#Tools (3)#Trading (8)#Treasury (6)#Type System (3)#Type Systems (3)#Vector Databases (8)#Vector Search (3)#Workflows (14)#gRPC (4)
Pratik Dhanave · ·12 min read

Tokens and Tokenization

The unit a language model actually reads is neither a word nor a character — it is a token, and once you see the world the way the model does, half of its strange behavior stops being strange.

The unit a language model actually reads is neither a word nor a character but a token. How byte-pair encoding builds a vocabulary, why tokenization explains half of an LLM's strange behavior, and how to count tokens exactly in Go.

#AI Engineering#LLMs#Tokenization#Go#Backend Engineering

All posts on this site are written by Pratik Dhanave, an Agentic AI Architect with 7+ years building production distributed systems, multi-agent AI platforms, and cloud-native infrastructure. About the author → Each article includes working code, architecture diagrams, and references to the specific frameworks and standards discussed. Browse all posts or explore related topics using the tag cloud above.