Article explains how Large Language Models work internally, from tokenization through inference

LLMs (AI systems that understand and generate text) break input text into tokens—subword units like whole words, word fragments, punctuation, or spaces—using an algorithm called Byte Pair Encoding (BPE) that iteratively merges the most frequent adjacent character pairs into a vocabulary of ~50,000–100,000 tokens.

Each token is converted into an embedding, a high-dimensional vector that captures what the token represents; words appearing in similar contexts (like 'king' and 'queen') end up with vectors pointing in similar directions, while unrelated words (like 'king' and 'banana') point in different directions.

The model processes tokens in parallel but adds positional encoding—additional vectors that encode each token's position in the sequence—so the model can distinguish between word order (e.g., 'dog bites man' vs. 'man bites dog').