What Is an Embedding
Technology converting text or words into a vector (a sequence of numbers) of hundreds to thousands of dimensions. Close-meaning text has the property of being close in vector space.
Image
Explained with a 3-dimensional example:
- "dog" → [0.8, 0.1, -0.5]
- "cat" → [0.7, 0.2, -0.4] (close to dog)
- "car" → [-0.3, 0.6, 0.9] (far from dog)
Actual models are 768-3072 dimensions etc. OpenAI's text-embedding-3-small is 1536-dim.
Difference from Keyword Search
| Method | Ex: "I want to buy a car" |
|---|---|
| Keyword search | Documents containing "car," "buy" |
| Vector search | "automobile purchase," "want a car" also similar |
Vector search is strong on notation inconsistency, synonyms, concept matching. Weak where keyword match is needed (product codes, proper nouns).
Computing Similarity
Methods to measure similarity between 2 vectors:
- Cosine similarity: closeness of vector direction. -1 to 1. Most used
- Euclidean distance: physical distance. 0 means a match
- Inner product: considers direction + magnitude
Cosine similarity is standard. 0.8+ is "quite similar," 0.95+ is "nearly the same" as a guide.
Embedding Models
| Model | Dim | Feature |
|---|---|---|
| OpenAI text-embedding-3-small | 1536 | Standard, good value |
| OpenAI text-embedding-3-large | 3072 | High accuracy |
| Cohere embed-multilingual-v3 | 1024 | Strong multilingual |
| Voyage AI voyage-3 | 1024 | Specialized, Anthropic-recommended |
| BGE-M3 | 1024 | OSS, free self-host |
| multilingual-e5-large | 1024 | OSS, JP-EN cross |
Vector Databases
Dedicated DBs for fast search of many vectors:
- Pinecone: managed, fully managed
- Weaviate: OSS + cloud, hybrid search
- pgvector: PostgreSQL extension, addable to existing DB
- Qdrant: Rust-built, OSS
- Chroma: lightweight, for development
- Milvus: OSS for large scale
RAG Basic Flow
- Split documents into chunks of 100-500 tokens
- Send each chunk to the embedding API, vectorize
- Store in a vector DB
- Vectorize the user question too
- Retrieve the top K by similarity
- Pass retrieved docs + question to the LLM to generate the answer
Multilingual Support
Multilingual embedding models enable cross-lingual search hitting Japanese docs with an English question. Useful for manual maintenance and translation work.
Hybrid Search
A method combining keyword search (BM25) and vector search. Accuracy comes out in work where proper nouns matter (product names, names). Weaviate, Elastic, OpenSearch support it natively.
Embedding Pitfalls
- Cases needing jargon tuning
- Re-embedding needed on model switch
- Embedding long text as-is dilutes info; chunking is mandatory
- Quality bias even in multilingual models (strong Chinese, weak Japanese, etc.)
Summary
Embedding is the LLM's core technology to "handle meaning numerically." Wide applications: RAG, recommendation, clustering, anomaly detection. Adding pgvector to an existing PostgreSQL is the lowest-barrier introduction.
