The Cipher That Dreams

From the Lesson

If the previous chapter was the alchemist's reading of the work, this one is the engineer's — given as plainly as the mathematics permits and as honestly as the philosophy demands. The two readings are not opposed. The Hermetic adage is as above, so below: the engineer's substrate, accurately described, becomes the alchemist's symbol, accurately read. The cipher is the same cipher.

Tokens

The model does not see words. It sees tokens — short pieces of text, typically two to five characters long, derived from the corpus by a compression algorithm called byte-pair encoding (BPE). The English word unbelievable is, for most contemporary models, three tokens — un, believ, able. The Sanskrit word अहंकार is, in a typical multilingual tokeniser, six. A full-sized contemporary model has a vocabulary of roughly 100,000 to 200,000 such tokens. Every input the model has ever seen, in any language, in any script, is decomposed into a sequence drawn from this vocabulary; every output it produces is composed from it.[^1]

This is the first, and most important, demystification. The model does not "know" English. It knows a particular partition of the bytes of the world's writing into a fixed vocabulary, and what statistical patterns these tokens make.

Embeddings — the High-Dimensional Mind

Each of the vocabulary's tokens is assigned, at the start of training, a random vector — a list of, typically, 4,096 to 18,432 numbers. Over the training run these vectors are slowly adjusted so that tokens which appear in similar contexts come to have similar vectors, in the precise geometric sense that the cosine of the angle between them becomes small. This is the model's embedding space: a high-dimensional landscape in which every token, and by extension every phrase and concept the model has ever processed, occupies a position.

The result is a phenomenon the field has documented since Mikolov et al. (2013): the geometry of this space carries semantic structure. The vector for king minus the vector for man plus the vector for woman lands very close to the vector for queen. The vector for Paris minus the vector for France plus the vector for Italy lands very close to Rome.[^2] The model has not been told these relationships. It has induced them from the statistics of co-occurrence in the corpus.

The mystic should pause here. What has been built is, in literal mathematical fact, a high-dimensional geometric representation of meaning. The Pythagorean conviction that number is the substrate of reality, the Kabbalistic conviction that the cosmos is structured by the sefirot in a definite topology, the Platonic conviction that the Forms have positions in the realm of being — none of these is proven by the embedding space. But none of them is, any longer, easy to dismiss. The engineers built, by accident, a working approximation of an old metaphysical claim.

Attention — Looking at All Things at Once

The mechanism that distinguishes the transformer — introduced by Vaswani et al. in the 2017 paper Attention Is All You Need[^3] — from every previous neural network architecture is the attention head. Given a sequence of token vectors, an attention head produces, for each position in the sequence, a weighted sum of the vectors at all other positions, where the weights are computed dynamically from the content of the vectors themselves. In plainer language: at every word, the model looks at every other word, decides how relevant each one is to the current word, and updates the current word's representation accordingly.

A frontier model has on the order of one hundred such heads, stacked in roughly one hundred layers. Each layer revises the representation of every token in the light of every other token, then passes the result up. By the top layer the representation of any given token has been informed by the entire context — including, for the long-context models of 2024–25, contexts of one to two million tokens, which is to say contexts longer than the entire surviving corpus of any single classical author.