Embeddings - Giving Tokens a Starting Representation

Remember the problem from tokenization: Token 15339 ("Hello") and Token 47234 ("hello") are just unrelated IDs in this toy tokenizer. Numerical closeness between IDs says nothing about meaning.

The solution: Turn each token into a vector of numbers. Not just one number, but many.

From One Number to Many

A token embedding is a learned list of numbers used as the token's initial representation. The width can range from hundreds to many thousands of values, depending on the model.

The eight values above are invented for explanation. Real embedding widths are model-specific. The important idea is that each token ID selects one high-dimensional starting vector.

Visualizing Embeddings

Imagine a 2D space, even though real embeddings are much wider. The coordinates below are invented only to show the idea of a vector space:

Here's what this looks like in 2D space:

In this illustration, "cat", "dog", and "kitten" cluster together while "run" is farther away. A trained model's actual geometry is more complex and need not form such a clean picture.

Measuring similarity with cosine similarity:

One common analysis tool is cosine similarity. It measures how much two vectors point in the same direction, regardless of their length. It is useful for inspecting vectors, but it is not the operation that defines all transformer behavior.

These three scores are invented, not measurements from a named model. In some embedding analyses, related items have higher cosine similarity, but raw token embeddings do not guarantee a neat semantic map.

The Embedding Table

How does this work? The model has an embedding table: a big lookup table that maps each token ID to its vector.

In this toy example, token 15339 selects row 15339 and returns a 768-value vector. Both the vocabulary and width depend on the actual model.

Embeddings Are Learned

Here's the crucial part: the model doesn't know what these embeddings should be at the start.

At initialization, the embedding table is filled with random numbers. Token "cat" might start as [0.01, -0.23, 0.44, ...]. Token "dog" might be [-0.88, 0.12, -0.03, ...].

During training, the model learns better embeddings. It adjusts the numbers so that:

• Useful geometric relationships can emerge
• Different tokens can support related predictions
• The embeddings help the full network predict what comes next

Why This Matters

Embeddings replace arbitrary IDs with trainable vectors:

The network can learn to process "Hello" and "hello" similarly even when their IDs differ. That similarity may appear partly in token embeddings and partly in the contextual representations created after attention and MLP transformations.

Relationships such as "run" and "running" or "Paris" and "France" are distributed across learned weights and context-dependent activations, not stored only in one embedding-table row.

The Magic of High Dimensions

Why use a wide vector instead of only two or three values? Because language is complex. The exact width is a model-specific design choice.

The word "bank" can mean:

• A financial institution
• The side of a river
• To tilt an airplane

A wide representation can support many overlapping features at once. Concepts are usually distributed across many dimensions rather than assigned cleanly to one dimension each.

The model doesn't explicitly assign dimensions to concepts. It just learns what works. But the result is that embeddings capture rich, multi-faceted meaning.

But embeddings are just the start. The real power comes from what happens next: learning.

Attention - Letting Tokens Share Context

Token embeddings provide starting vectors, but the model still needs to combine information across positions. Attention is one coordination step. Each position can draw from the positions allowed by the model's attention mask. In a causal language model, that normally means the current and earlier tokens, not future tokens.

The Challenge: Ambiguity in Language

Consider the sentence: "The river reached the bank."

The word "bank" is ambiguous. It could mean:

• A financial institution (money bank)
• The side of a river (riverbank)
• To tilt or lean

A human reader looks at "river" and immediately knows this is the riverside meaning. The model needs a similar ability: to look at context and adjust understanding. Attention provides exactly that.

Queries, Keys, and Values

At a given layer, each current hidden representation is projected into Query, Key, and Value vectors. For each attention head:

Positional Information

Content-only dot-product attention does not carry enough information about token order. Architectures introduce position in different places, so positional information is not always added directly to embeddings:

The causal mask also supplies direction by blocking future positions. Exact positional behavior depends on the model's architecture, but some position-sensitive mechanism is needed to represent order and distance well.

The Attention Mechanism Step-by-Step

Let's trace what happens when a causal model processes "The river reached the bank." Here, "river" occurs before "bank," so the final token is allowed to attend to it:

Step 1: Compute Compatibility Scores

For the token "bank", compute a learned compatibility score between its Query and each allowed token's Key using a dot product. The values below are invented to make the flow visible:

A dot product depends on both direction and magnitude. It is not generally cosine similarity. During training, the projections learn scores that are useful for the task. In this invented example, "river" receives the highest display score because the diagram is illustrating the riverside reading of "bank".

Step 2: Stabilize and Normalize

Raw dot products can grow with dimension. Standard scaled dot-product attention divides by √(d_k), where d_k is the per-head key/query width, then applies softmax. Masks and implementation-specific biases may also be added before softmax. This separate numerical example is constructed only to show the conversion:

In this toy distribution, "river" has weight 0.57. These weights are not taken from a trained model and should not be interpreted as a universal explanation of how models resolve the word "bank".

Step 3: Blend Values

Multiply each token's Value vector by its attention weight, then sum:

In the toy calculation, the Value vector at "river" contributes the largest share to this head's output for "bank". A real model distributes features across heads and layers, so one visible attention weight is not a complete explanation of the model's decision.

Multi-Head Attention

Many transformers use multiple heads in parallel, each with its own Query, Key, and Value projections. The labels below are intuition, not fixed or guaranteed head roles:

Each head produces an output vector. The outputs are usually concatenated and projected to form the attention block's result. Individual heads can show mixed, overlapping, or context-dependent behavior rather than one clean specialization.

Putting It Together: The Attention Mechanism

Learning - How Matrix Multiplications and Attention Create Understanding

We have token embeddings now. Their width is model-specific, and at the start of training from scratch their values are typically initialized without useful language structure. The model must learn representations that help it predict.

This is where the transformer architecture comes in. Two main mechanisms work together: attention (which you learned about in the previous tab) and feedforward networks (made of matrix multiplications). They're the tools the model uses to transform embeddings and discover patterns across billions of training examples.

What Is a Matrix Multiplication?

A matrix multiplication is a mathematical operation that transforms vectors. Think of it as a function: you put a vector in, you get a different vector out.

The matrix contains weights (numbers). When you multiply the input vector by the matrix, you get a new vector. The weights control how the transformation happens.

How Transformers Work: Attention + Feedforward

Each layer in a transformer has two parts that work together. Here's what happens:

This cycle repeats through the model's transformer blocks. A simplified decoder block looks like this, although normalization order, activation functions, positional method, and other details vary by architecture:

Key distinction: Attention mixes information across permitted positions. The MLP transforms features at each position separately. Both operations can contribute to many behaviors, so it is too strong to say attention only finds relevance while the MLP alone performs reasoning.

Some analyses find rough depth-related tendencies, but there is no universal rule that assigns local syntax to early layers and reasoning to late layers.

Example: Understanding Context (Attention + Feedforward)

Consider the sentence: "The river reached the bank." The vectors and weights below are invented teaching values:

Together, attention, MLPs, residual paths, and normalization can make the representation at "bank" depend on "river." The example is useful intuition, not a claim that one head or one MLP contains the complete interpretation.

Learning Through Training

At the start of training, the weight matrices are random. The model makes terrible predictions. But here's how it improves:

Training software aggregates prediction errors into a loss, computes gradients with backpropagation, and updates parameters with an optimizer. A simplified training loop is:

Over training, optimization usually lowers prediction loss and produces parameters that generalize to many unseen sequences. Exact data volumes and convergence behavior vary by model and run.

Why Multiple Layers?

LLMs apply a model-specific number of transformer blocks. Many have dozens of blocks; some have fewer or more. The diagram below is a teaching sketch, not the anatomy of every model.

Representations usually change and become more task-useful with depth, but features can be distributed, reused, or carried through residual paths. Heads and MLPs do not follow a fixed ladder from grammar to semantics to reasoning in every model.

Putting It All Together: The Complete Transformer Pipeline

Here's the complete pipeline:

Attention and MLP sublayers repeatedly transform the hidden representations. Attention mixes across permitted positions, while the MLP transforms the features at each position. Their roles overlap and interact, and neither has one fixed interpretation.

The weights in embeddings, attention, MLPs, normalization, and output components are learned. Parameter counts and training-token counts vary widely, so all fixed counts in this article are explicitly toy or model-specific examples.

The Big Picture: Everything Connected

You now understand the full transformer architecture:

• Tokenization: Text becomes numbers (token IDs)
• Token embeddings: Token IDs select context-free learned starting vectors
• Position: Added vectors, Query-Key transformations, or attention biases represent order
• Attention: Scaled Q-K dot products weight Value vectors across allowed positions
• MLPs: Learned nonlinear transformations operate at each position
• Transformer blocks: These mechanisms repeat a model-specific number of times to create contextual representations

Every prompt follows this broad journey, though the exact tokenizer, position method, dimensions, number of heads, number of blocks, and parameter count depend on the model. The useful behavior comes from the complete trained system, not from one component in isolation.

Current Technical References

• Hugging Face Transformers: Tokenizer summary. Current documentation on tokenizer families and tradeoffs. Accessed Jul. 30, 2026.
• PyTorch: torch.nn.attention. Current official attention API documentation. Accessed Jul. 30, 2026.
• PyTorch: scaled_dot_product_attention. Official formula and implementation notes. Last updated May 7, 2026; accessed Jul. 30, 2026.