Can I mix embedding models for chunks and queries?

No. Vectors from different models live in different spaces and cannot be compared. Re-embed everything if you change the model.

Is embedding search better than keyword search?

Each has strengths. Embeddings handle paraphrases, while keyword search is better for exact names, codes, and rare terms. Combining them (hybrid search) often works best.

RAG Step 2: Embeddings and Vector Search

Introduction

An embedding converts text into a list of numbers so that passages with similar meaning end up close together. Searching then becomes a matter of finding the nearest vectors to the question. This lesson generates embeddings with an open-source model and searches them with plain NumPy.

Prerequisites

Document chunks from the previous lesson
pip install sentence-transformers numpy

Step-by-Step Guide

Step 1: Understand what an embedding is

An embedding model maps text to a fixed-length vector. Sentences about the same idea get vectors pointing in similar directions even when they share few words, which is what allows search by meaning instead of keywords alone.

Step 2: Create embeddings for your chunks

Load an embedding model and encode all chunk texts. Normalizing the vectors to unit length makes cosine similarity equal to a simple dot product. You can use any embedding model or hosted embeddings API, but you must use the same model for chunks and questions.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

texts = [r["text"] for r in records]
vectors = model.encode(texts, normalize_embeddings=True)
print(vectors.shape)   # (number_of_chunks, 384)

Step 3: Search by similarity

Embed the question with the same model, score every chunk with a dot product, and take the highest scores. Scores near 1 mean very similar, and lower scores mean less related.

import numpy as np

def search(question: str, k: int = 3):
    q = model.encode([question], normalize_embeddings=True)[0]
    scores = vectors @ q
    top = np.argsort(scores)[::-1][:k]
    return [(float(scores[i]), records[i]) for i in top]

for score, rec in search("How many vacation days do employees get?"):
    print(round(score, 3), rec["source"], rec["text"][:120])

Step 4: Look at failures

Try questions you know the answer to and check whether the right chunk appears in the top results. When it does not, the cause is usually chunking, a mismatch between how people ask and how documents are written, or an embedding model that suits your domain poorly.

Step 5: Know when to move to a vector database

NumPy search is exact and fine for thousands to tens of thousands of chunks. For millions of chunks, frequent updates, or per-user filtering, use a vector index such as FAISS, a dedicated vector database, or a database extension like pgvector, which use approximate search to stay fast.

FAQs

Can I mix embedding models for chunks and queries?: No. Vectors from different models live in different spaces and cannot be compared. Re-embed everything if you change the model.
Is embedding search better than keyword search?: Each has strengths. Embeddings handle paraphrases, while keyword search is better for exact names, codes, and rare terms. Combining them (hybrid search) often works best.

Summary

Embed chunks and questions with the same model, normalize, and rank by dot product. Check real questions against the results, and graduate to a vector index when scale or update needs demand it.