Use public benchmarks such as MTEB to shortlist embedding models, not to choose the production winner. The final decision should come from your own query-document relevance set, with chunking, preprocessing, index settings and candidate count held constant. Measure first-stage retrieval before reranking, then add latency, vector footprint, language coverage and domain fit to the decision.
That separation matters because an embedding model can look strong on a public leaderboard yet perform poorly on the jargon, document structure or query patterns your users actually produce.
Use public benchmarks to shortlist models, not to choose the winner
MTEB provides tooling and benchmark coverage for evaluating embedding models across multiple task families, including retrieval. Its value is breadth: it can help narrow a large model market to a smaller set of candidates that appear competitive on relevant tasks.
But a leaderboard is not your corpus. Public benchmark data may differ from your production data in language, document length, domain terminology, query style and relevance criteria. Treat benchmark performance as a screening signal. The model that wins your deployment should still prove itself on representative queries and documents from your own use case.
Public benchmarks also need their own trust checks. Benchmark age, exposure and evaluation protocol can affect how confidently a score should be interpreted; AiTechJournal’s guide to benchmark contamination in AI models explains how to separate strong benchmark performance from evidence that a test set may have been exposed during training.
Build an in-domain relevance set
The most useful evaluation asset is a set of real or realistic queries paired with relevance judgments for the documents or chunks that should be retrieved. For customer support, that might include paraphrased questions, product names, error codes and policy wording. For technical search, it may include abbreviations, version numbers, stack traces and internal terminology.
A small but representative labeled set is usually more informative than a large synthetic test that does not resemble production traffic. Include easy queries, ambiguous queries, long-tail terminology, multi-concept questions and cases where several documents are partially relevant.
Do not build the relevance set around one embedding model’s output. Start from the user question and independently decide which documents should count as relevant. Otherwise the evaluation can inherit the retrieval biases of the model you are trying to measure.
Freeze a version of the relevance set for the controlled model comparison and record any later changes. This makes reruns comparable and keeps model improvements separate from changes in the test data itself. Add new query slices separately when you discover a production failure mode instead of silently rewriting the original test.
Measure candidate recall before final answer quality
For first-stage retrieval, Recall@k asks whether the relevant material appears somewhere within the top k candidates. If a relevant document is absent from the candidate set, a later reranker or language model cannot use it.
For example, if a query has one known relevant chunk and that chunk appears within the top 10 results, Recall@10 for that query is successful. If it appears at rank 18, Recall@10 fails even though Recall@20 succeeds. The right cutoff depends on how many candidates your next stage can realistically process.
| Metric | Question it answers | Useful when |
|---|---|---|
| Recall@k | Did relevant material enter the top-k candidate set? | Candidate generation before reranking or answer generation |
| MRR | How early does the first relevant result appear? | There is usually one especially important answer/result |
| nDCG | How well are multiple results ordered when relevance can differ by degree? | Several documents can be relevant, with some better than others |
For RAG, Recall@k is often the first metric to inspect because downstream generation cannot repair missing evidence. End-to-end answer quality should still be measured separately, as covered in AiTechJournal’s broader guide to evaluating RAG systems.
Use ranking metrics when position matters
Mean Reciprocal Rank (MRR) rewards systems that place the first relevant result near the top. A relevant document at rank 1 contributes more than the same document at rank 5. That makes MRR useful when users or downstream logic strongly prefer the first relevant hit.
nDCG is more useful when several retrieved documents can be relevant and relevance is graded rather than binary. It rewards highly relevant items appearing near the top while discounting useful items that appear lower.
Do not collapse every retrieval problem into one score. A support assistant that needs any one authoritative policy page may care more about early first-hit quality. A research search tool returning several useful sources may care more about the ordering of multiple relevant results.
Build a controlled comparison
Change the embedding model, not the whole retrieval system at the same time. Keep the following variables fixed during the core comparison:
- Document and chunk boundaries
- Text normalization and preprocessing
- Query wording
- Vector-index type and search settings
- Similarity function where the model supports the same setup
- Top-k candidate count
- Relevance judgments
- Reranker configuration, if reranking is evaluated in a separate stage
If one model uses different chunks, another uses hybrid search, and a third gets a larger candidate budget, the result is a comparison of pipelines rather than embedding models. Pipeline experiments are valuable, but run them after the controlled model comparison so you can attribute the improvement.
Do not let reranking hide weak candidate retrieval
Reranking can improve precision by rescoring a smaller candidate set with a more expressive model. That makes it useful, but it can also make a weak first-stage retriever look better if you examine only the final ranked list.
Score the system twice:
- Before reranking: measure candidate Recall@k and, where useful, first-stage ranking metrics.
- After reranking: measure the final ordering and the downstream answer quality.
If two embedding models produce similar final results after reranking but one has substantially better candidate recall, the stronger retriever may be safer when query distribution changes or the reranker budget is reduced. Conversely, if both have strong recall and the reranker dominates final precision, a cheaper embedding model may be sufficient.
Measure deployment cost separately from relevance
Retrieval quality is only one axis. Embedding models also differ in vector dimensionality, inference speed, hardware requirements and deployment options.
Higher-dimensional vectors require more vector data to be stored and moved through the retrieval system. That can affect index size, memory or storage requirements, network transfer and search cost. The exact impact depends on the index and database, so measure your own implementation rather than assuming dimensions translate directly into one fixed latency penalty.
Split latency into at least two measurements:
- Query embedding latency: time to convert the user query into a vector.
- Vector-search latency: time for the index to return the top candidates once the vector exists.
Also record throughput or concurrency if the system will serve many simultaneous users. A model that is fast for one query can behave differently under sustained load.
| Dimension | What to record | Why it matters |
|---|---|---|
| Retrieval quality | Recall@k, MRR and/or nDCG | Measures whether relevant material is found and ranked well |
| Query embedding | Median and tail latency under representative load | Affects search responsiveness |
| Vector search | Index latency at the chosen top-k | Separates model inference from database/search cost |
| Vector footprint | Dimensions, index size, memory/storage use | Influences infrastructure cost and scale |
| Operations | Hosting model, batching, rate limits, hardware | Determines whether the model is practical to run |
Test domain and language fit explicitly
Average benchmark quality can hide failure slices that matter more than the mean. Break out evaluation results by language, document type, query length, jargon level and other characteristics that reflect the real corpus.
A multilingual product should not validate only English queries. A medical, legal or engineering search system should include domain terminology rather than relying on general-language paraphrases. If users often paste identifiers, product codes or version strings, include those too.
Also inspect failures manually. Metrics tell you how often retrieval fails; error review tells you why. Common failure patterns include semantically similar but operationally different concepts, overly broad chunks, short ambiguous queries and domain terms that are poorly represented.
A practical embedding-model scorecard
A useful final decision does not require turning every metric into one artificial master score. Keep the dimensions visible and choose according to product priorities.
| Question | Evidence to review |
|---|---|
| Does it retrieve the needed evidence? | Recall@k on the in-domain relevance set |
| Does it rank useful results early? | MRR or nDCG, depending on the relevance structure |
| Does reranking compensate for weaknesses? | Before-versus-after reranking results |
| Does it work on our real language and terminology? | Slice metrics and manual error analysis |
| Can we afford its serving footprint? | Embedding latency, vector-search latency, dimensions and index footprint |
| Is the system operationally sustainable? | Hosting requirements, throughput, quotas, update process and model availability |
The practical workflow is straightforward: use MTEB or similar public benchmarks to create a shortlist, build a representative relevance set, run a controlled comparison, measure first-stage retrieval before reranking, then add operational constraints. The best embedding model is the one that retrieves the right material reliably for your workload at a serving cost you can sustain—not necessarily the model sitting at the top of a generic leaderboard.
