Before you can understand how a language model works, you need to understand what it actually reads. The answer is neither words nor letters. It is tokens: subword units produced by an algorithm called Byte Pair Encoding (BPE), which carves language at its statistically most efficient joints.¹ The word “unbelievable” becomes three tokens: un, believ, able. The word “cat” is one. A Python function signature might be four. An emoji is often two.

This matters more than it sounds. Every input a language model receives is converted into a sequence of integer IDs: your question, the document you paste, the developer’s instructions. The model never reads “English.” It reads numbers. What it has learned to do, through training on hundreds of billions of these numerical sequences, is build extraordinary probabilistic intuitions about which numbers tend to follow which other numbers, in what contexts, under what conditions.

GPT-4 operates with a vocabulary of roughly 100,000 tokens. Claude 3 and subsequent models use similar scales. Each input sequence has a hard upper limit called the context window, beyond which the model cannot “see.” Early GPT-2 models had context windows of 1,024 tokens. State-of-the-art models as of 2026 handle one million or more. This is not a cosmetic improvement. Longer context enables qualitatively different behaviors: maintaining coherence across book-length documents, multi-hour coding sessions, sustained legal review. The context window is the model’s working memory. Working memory is cognition.

This is the first and most fundamental point: a language model’s relationship with language begins with a compression. Human expression, reduced to integer sequences, passed into an architecture that will spend months and millions of dollars learning what those sequences mean to each other.

In 2017, a team of eight researchers at Google Brain published a paper titled “Attention Is All You Need.”² It introduced the Transformer architecture, the structural backbone of every major language model currently in production. The paper was understated in tone. Its implications were anything but.

The central innovation of the Transformer is the self-attention mechanism. Before attention, neural networks processed text sequentially, one word after the next, like reading aloud. Long-range dependencies were difficult: the model might forget the beginning of a sentence by the time it reached the end. The Transformer discarded sequential processing entirely. Instead, it processes the entire input simultaneously, and for each token, computes a weighted relationship with every other token in the context window. This is attention.

Consider the sentence: “The trophy didn’t fit in the suitcase because it was too big.” What does “it” refer to? The trophy. Humans resolve this instantly through contextual weighting. Attention mechanisms learned to do something structurally equivalent: they assign high relevance scores between “it” and “trophy,” and low scores between “it” and “suitcase,” because statistically, across billions of training examples, that pattern of co-reference is what the data demanded.³

Modern language models do not have a single attention mechanism. They have multi-head attention: dozens to hundreds of parallel attention computations running simultaneously, each learning to track different kinds of relationships. Syntactic structure, semantic meaning, coreference, tone, factual association. GPT-3 has 96 attention heads. The outputs of all heads are concatenated and passed forward through feedforward layers that compress and transform the information further.

Stacked atop one another (12, 24, 96, 120 layers deep in the largest models) these attention blocks and feedforward networks form an extraordinarily deep computational graph. Information enters as token embeddings and is progressively transformed through each layer. By the final layer, the model produces a probability distribution over its entire vocabulary: these are the odds that each of the ~100,000 possible tokens should come next. The highest-probability token is typically selected, appended to the sequence, and the process repeats, one token at a time, until the response is complete.

The architecture is the skeleton. Pre-training is where the model acquires everything it will ever know. The objective is deceptively simple: given a sequence of tokens, predict the next one. Do this trillions of times across the breadth of digitized human writing (books, academic papers, code repositories, forums, legal filings, scientific datasets, news archives) and adjust the model’s billions of parameters after each prediction to reduce the error.⁴

This process is called gradient descent via backpropagation. When the model predicts the wrong token, a mathematical signal flows backward through the network and nudges every parameter that contributed to the error in a direction that would have made a better prediction. This happens continuously, across batches of thousands of training examples, across months of training on clusters of thousands of specialized AI chips.

The scale of modern pre-training defies easy comprehension. GPT-4’s training run consumed an estimated 25,000+ A100 GPUs running for months, at a cost widely estimated between $50-100 million.⁵ Frontier models in 2025 and 2026 are trained at larger scales still, with training compute doubling roughly every eight to twelve months. What emerges from this process is a set of numerical weights, typically 70 billion to 1 trillion parameters, encoding in a form no human can read directly everything the model has learned about how language and ideas relate to each other.

One of the most consequential and least-understood phenomena of pre-training is emergence: the appearance of qualitatively new capabilities at specific scales, entirely absent in smaller models.⁶ Chain-of-thought reasoning, in-context learning, multi-step mathematical problem-solving: none of these were programmed. They arrived, approximately unpredictably, as the number of parameters and training tokens crossed certain thresholds.

This is philosophically significant. It means the capabilities of a frontier model cannot be fully predicted from the capabilities of its predecessors. Researchers building the next model do not know exactly what it will be able to do until it is trained. And it means the intuition that “the AI only does what it’s programmed to do” is not merely wrong. It is wrong in a way that has serious implications for how we reason about risk and capability going forward.

A model immediately after pre-training is not an assistant. It is a completion engine: extraordinarily powerful, but undirected. If you prompt it with “How do I make a bomb?” it will complete the prompt in the style of whatever training document most closely resembles that opening. Instruction-following, safety refusals, helpfulness, coherent multi-turn conversation, none of these are properties of the pre-trained base model. They are added afterward through what the field calls the alignment stack.

The alignment stack is not a guarantee of aligned behavior. It is a learned behavioral distribution. The model has not internalized a value system the way a human might. It has developed strong statistical associations between certain inputs and certain outputs that were preferred by the humans and AI systems that shaped it. Whether this constitutes genuine alignment, or a simulacrum of alignment that will break under sufficient distributional shift, is among the most important open questions in AI safety research.

This is the uncomfortable epistemic situation the field calls interpretability, or rather, the lack of it. Mechanistic interpretability attempts to reverse-engineer what specific circuits inside a Transformer are actually doing.⁸ Researchers at Anthropic, DeepMind, and academic labs have identified individual attention heads that track grammar, circuits that detect sentiment, features that represent specific factual associations. But these findings, impressive as they are, represent a tiny fraction of what would be needed to fully understand even a small model.

The practical consequences of this opacity are significant. It is why hallucinations are architecturally inevitable rather than simply a bug to be patched. The model does not “look up” facts. It generates the statistically most plausible continuation of your prompt given its trained weights. When its training data contained a confident, factual-sounding sentence about a topic, the model learned to produce confident, factual-sounding sentences about that topic, regardless of whether the underlying claim is true.

What we do know is growing. Techniques like activation patching, sparse autoencoders, and logit attribution are beginning to decompose model behavior into identifiable circuits.⁹ Anthropic’s interpretability team, among others, has demonstrated that it is possible to identify neural features corresponding to specific concepts (including abstract ones like “the presence of deception” or “emotional valence”) and to causally intervene on those features to predict and control model behavior. This is early and partial. But it is real progress toward a science of machine cognition.

Most writing about AI focuses on training. Inference, the moment the model actually runs, receives comparatively little attention, despite being where all the value is created and all the risk is realized. Here is what happens, precisely, when you submit a prompt.

Your text is tokenized into integer IDs. A positional encoding is added, a vector that tells the model where each token sits in the sequence, since the Transformer processes all tokens simultaneously and has no inherent sense of order. The resulting token embeddings are passed through all the model’s layers in a single forward pass: attention, feedforward, normalization, repeat, for every layer in the stack. At the final layer, a linear projection maps the last hidden state to a vector of logits, one per vocabulary token, representing the model’s pre-softmax estimates of next-token probability.

A sampling strategy then selects the actual next token. At temperature zero, the model always picks the highest-probability token: deterministic, reproducible, often stiff and repetitive. At higher temperatures, lower-probability tokens become more likely, increasing creativity at the cost of occasional incoherence. Parameters like top-p and top-k constrain which tokens are eligible before sampling. The selected token is appended to the context, and the entire forward pass runs again for the next token.

The most important recent architectural development in inference is chain-of-thought reasoning and its productionized descendant, extended thinking. Rather than generating a final answer in a single sequence, models like OpenAI’s o3 and Anthropic’s Claude 3.7 Sonnet are trained to generate extended reasoning traces: thousands of tokens of intermediate working before producing a final response. Reasoning traces systematically improve performance on logic, mathematics, and multi-step problems by giving the model more computational space to work through a problem before committing to an answer.

The irony is significant. A system with no genuine planning, no working memory in the traditional sense, and no awareness of its own reasoning process can, by generating enough tokens of apparent deliberation, produce outputs that are structurally indistinguishable from those of a careful human reasoner. Whether that distinction matters, whether process matters if the outputs are reliable, is a question this series will return to.

The Transformer is not a complete theory of intelligence. It is an extraordinarily effective architecture for pattern completion at scale, but it has clear limitations the current training paradigm cannot fully resolve. It has no persistent memory beyond its context window. It cannot update its weights at inference time. Everything it knows was fixed at the end of training. It cannot autonomously take actions in the world, verify its own claims against live data, or engage in genuine recursive self-improvement.

The systems being built in 2025 and 2026 are increasingly aimed at these gaps. Retrieval-augmented generation (RAG) connects model inference to external databases, allowing a model to look up facts rather than confabulate them. Tool use and function calling give models the ability to execute code, browse the web, and interact with APIs, converting language capability into real-world agency. Long-context memory systems persist relevant information across sessions, partially simulating the kind of durable world model that humans carry effortlessly.

The larger architectural question of whether the Transformer is the final form of general AI, or merely the current dominant paradigm, remains genuinely open. State Space Models like Mamba offer linear-time inference compared to the Transformer’s quadratic attention cost, enabling much longer effective contexts at lower computational expense.¹⁰ Mixture-of-Experts architectures, used in GPT-4 and Gemini Ultra, route different inputs to different specialized sub-networks, achieving effective parameter counts that far exceed what any single dense model could deploy efficiently.

What is certain is this: the systems operating today are not magic, and they are not simple. They are the product of specific architectural choices, specific training objectives, and specific alignment procedures, each with known properties, known failure modes, and known limitations. Understanding them at this level of precision is not optional for anyone who intends to build with them, govern them, or live alongside them. The mythology of AI collapses under technical scrutiny. What remains is something simultaneously more modest and more important: a genuinely new kind of information processing, with capabilities no one fully predicted and risks no one fully understands.

That should be the starting point for every serious conversation about what comes next.