One source of confusion when learning about LLMs is the relationship between tokenization and embeddings. These are distinct but tightly connected steps in how language models process text. Understanding this relationship is essential to grasp how raw text transforms into semantic meaning that machines can process.

From Text to Token IDs

As discussed in the tokenization article, the first step in any LLM pipeline is to convert raw text into tokens using a tokenizer. But tokens at this stage are just integer IDs. The number 42 means nothing semantically. It's an arbitrary index into a vocabulary.

Think of tokenization as breaking down a sentence into individual puzzle pieces, where each piece is labeled with a number. These numbers are just identifiers—they don't tell you anything about what the piece represents or how it relates to other pieces. That's where embeddings come in.

The Embedding Lookup Table

Once tokenization is complete, each token ID is used to look up a corresponding vector from an embedding matrix. This is essentially a large lookup table where each row represents a learned embedding for a specific token in the vocabulary.

For example, if a model has a vocabulary of 50,000 tokens and uses 768-dimensional embeddings, the embedding matrix is 50,000 × 768. The token ID 42 simply retrieves row 42 from this matrix, which is a 768-dimensional vector.

These initial embeddings are sometimes called token embeddings or input embeddings. They represent each token in isolation, without any awareness of the surrounding context.

From Static to Contextual Embeddings

The initial token embeddings are just the starting point. As the sequence of token vectors flows through the transformer's layers, something crucial happens: the model uses self-attention to allow each token to "look at" and integrate information from all other tokens in the sequence.

This is where the magic of transformers comes in. Each layer refines the representation of each token based on its context. By the time a token's vector reaches the final layer, it no longer represents the token in isolation—it represents the token as it appears in this specific sentence, with this specific meaning.

This transformation from static to contextual embeddings is one of the key innovations that made modern LLMs so powerful. Earlier approaches like Word2Vec or GloVe produced only static embeddings—one vector per word, regardless of context. Transformers dynamically compute embeddings that adapt to the specific usage.

Sentence and Document Embeddings

When we talk about embedding a sentence or a document, we're typically referring to a pooled representation derived from the contextual token embeddings.

Common pooling strategies include:

• Mean pooling: Average all token vectors
• CLS token: Use the vector of a special classification token (common in BERT-style models)
• Max pooling: Take the element-wise maximum across all token vectors
• Weighted pooling: Average tokens with attention-based weights

The result is a single fixed-length vector that represents the entire input. This is what gets used in semantic search, clustering, and similarity tasks.

Different pooling strategies work better for different tasks. Mean pooling tends to work well for general-purpose sentence embeddings, while CLS tokens are often used in classification tasks. The choice depends on how the model was trained and what you're trying to accomplish.

The Overloaded Term "Embedding"

It's important to recognize that "embedding" is used to describe several related but distinct concepts:

• Token embeddings: Initial vectors from the embedding lookup table
• Contextual embeddings: Token vectors after passing through transformer layers
• Sentence embeddings: Pooled representations of entire sequences
• Embedding models: Models specifically trained to produce high-quality sentence/document embeddings

Understanding which kind of embedding is being discussed is crucial. When practitioners talk about "using embeddings for RAG," they almost always mean sentence-level embeddings produced by a model specifically designed for that purpose—not raw token embeddings.

This terminology confusion can lead to misunderstandings. Someone might say "the embedding of this word" and mean the initial token embedding, while someone else uses the same phrase to refer to the contextual embedding after transformer processing. Always clarify which type of embedding is being referenced.

The Complete Pipeline

Putting it all together, here's how text becomes meaning in an LLM:

Each step serves a specific purpose:

• Tokenization breaks text into processable units
• Token IDs provide discrete identifiers
• Token embeddings map IDs to continuous vector space
• Transformer layers inject context through self-attention
• Contextual embeddings capture meaning in context
• Pooling creates fixed-size representations for variable-length inputs

Why This Matters

Understanding the relationship between tokenization and embeddings clarifies several important points:

• Tokenization determines the granularity at which the model sees language
• Token embeddings are learned parameters that evolve during training
• Context is not inherent in the initial embeddings—it's constructed by the transformer
• The same tokenizer and embedding model must be used consistently across a pipeline
• Different types of embeddings serve different purposes

Together, tokenization and embeddings form the bridge between human language and machine computation. One provides the discrete units, the other provides the semantic representation—and the transformer weaves them into contextual understanding.

This pipeline is so fundamental that it appears in virtually every modern NLP system, from chatbots to search engines to content recommendation systems. Master this concept, and you'll have a solid foundation for understanding how LLMs actually work under the hood.

Back to Home