The one-sentence explanation
Encoding converts information from one representation into another. An embedding maps an object to a dense vector, usually learned by a model, that captures useful features or meaning.
An embedding is a kind of encoding, but not every encoding is an embedding.
The shortest distinction is this: encoding answers “How can we represent this?” Embedding goes further and asks “How can the representation express relationships and meaning?”
One word, four representations
The same information can take different forms depending on the task. All four representations below let a computer process “cat,” but only the last deliberately places semantic relationships in the representation space.
| Method | Example output | Main purpose |
|---|---|---|
| UTF-8 | 63 61 74 | Store and transmit text |
| Token ID | 12345 | Locate an item in a model vocabulary |
| One-hot | [0, 0, 1, 0, ...] | Identify a category without ambiguity |
| Embedding | [0.21, -0.73, 0.44, ...] | Express features, meaning, and similarity |
The core differences
| Dimension | Encoding | Embedding |
|---|---|---|
| Goal | Make information storable, transferable, or computable | Make features and relationships computable |
| Output | Bytes, identifiers, sparse vectors, or dense vectors | Usually a fixed-size dense floating-point vector |
| Learned? | Not necessarily | Usually learned during training |
| Meaning of distance | Usually no semantic guarantee | Distance, angle, or dot product often measures similarity |
| Reversible? | Some encodings decode losslessly | Usually cannot reconstruct the exact input |
Why vector distance matters
UTF-8 bytes and token IDs primarily identify objects. Token IDs 12345 and 12346 being adjacent says nothing about whether their tokens have similar meanings.
An embedding is valuable because it creates a space that can be compared. In a well-trained space, “cat” and “dog” will usually be closer than “cat” and “database”:
< distance(embedding("cat"), embedding("database"))
This turns semantic search, recommendations, clustering, and RAG retrieval into vector operations. What “close” means, however, depends on the training data, objective, and distance metric. Different embeddings do not automatically encode the same notion of meaning.
Where do they appear in a Transformer?
Transformer terminology adds another easily confused word: encoder. The tokenizer, embedding layer, and Transformer encoder are separate stages with separate responsibilities.
INPUT
Raw text
“eat an apple”
ENCODING
Token IDs
Tokenizer splits and numbers
EMBEDDING
Initial vectors
Embedding layer lookup
ENCODER
Contextual vectors
Uses the surrounding tokens
A token ID is a discrete identifier. The embedding layer converts that identifier into an initial vector, and the Transformer encoder updates the vector using context.
For example, “Apple announced” and “eat an apple” may begin with the same initial embedding for “apple.” After contextual encoding, the final vectors should lean toward the company and fruit meanings respectively.
How can you recognize an embedding?
- 01Is the output a dense vector?
- 02Was the vector learned from a model or data?
- 03Does distance between vectors express some kind of similarity?
If the answers are mostly yes, the representation is usually an embedding. The final takeaway is simple: encoding makes data computable; embedding makes relationships between objects computable.
Loading comments...