Every RAG demo feels instant. Almost every RAG product in production feels broken. The gap between the two is not model quality or prompt engineering. It is latency, and it accumulates in places most teams never measure.
This post is about where the seconds actually go in a retrieval-augmented generation pipeline, and what to do about each one. It assumes you have already built something that works and are now trying to make it feel fast enough that people keep using it.
1. First, Measure the Right Thing
The single most common mistake is optimising total response time. Users do not experience total response time. They experience time-to-first-token — the gap between hitting Enter and seeing the first word appear.
A response that streams its first token in 400ms and finishes in 6 seconds feels dramatically faster than one that returns nothing for 3 seconds and then dumps a complete answer. Same information, very different perceived quality.
So before you change anything, instrument these five spans separately:
- Query embedding — turning the user's question into a vector
- Vector search — the actual nearest-neighbour lookup
- Reranking — if you have a rerank stage
- Prompt assembly — fetching full documents, formatting, token counting
- Generation — split into time-to-first-token and total streaming time
Do not skip this step and guess. In my experience the bottleneck is almost never where the team assumed it was before they had numbers on a dashboard.
2. The Embedding Call Nobody Counts
Here is the stage that surprises people. Before you can search your vector database, you have to embed the user's query — and if you are calling a hosted embedding API to do it, that is a full network round-trip on the critical path of every single request.
A hosted embedding call typically costs 80 to 300ms depending on your region and the provider's load. If your application server sits in one region and the embedding API in another, you can easily pay 400ms before your vector database has done any work at all.
Three fixes, in order of effort:
- Cache aggressively. Embeddings are deterministic for a given model and input. The same query text always produces the same vector. A simple hash-keyed cache eliminates this cost entirely for repeat questions, and in most real products the head of the query distribution is very repetitive.
- Co-locate. Run your application in the same region as your embedding provider. This is a configuration change that routinely saves 100ms or more.
- Run the embedding model locally. Modern small embedding models are a few hundred megabytes and run comfortably on CPU. Removing the network hop entirely can take this stage under 20ms. The trade is deployment complexity and you must re-embed your entire corpus if you switch models.
3. Vector Search Is Rarely the Problem
Teams tend to blame the vector database because it is the newest and least familiar component. It is usually innocent.
Approximate nearest neighbour indexes such as HNSW have roughly logarithmic search complexity. Going from 100,000 vectors to 10 million typically adds single-digit milliseconds. If your retrieval step is genuinely slow, look for one of these instead:
- Cold index. Serverless vector databases may page your index out after a period of inactivity. The first query after idle can take seconds while the last few are fast. If your latency graph is bimodal, this is almost certainly why.
- Unindexed metadata filters. Filtering by tenant, document type, or date is standard practice, but if that field is not indexed, the database may fall back to scanning. This turns a logarithmic operation linear.
- Over-fetching. Requesting the top 100 results when you only use the top 5 wastes bandwidth and serialisation time on every request.
- Network round-trips. Hosted vector databases are another network hop. The same co-location advice applies.
4. Reranking: Slower Per Step, Faster Overall
This one is counterintuitive, so it is worth stating plainly: adding a reranker often makes the whole pipeline faster.
Without reranking, teams compensate for mediocre retrieval by stuffing more context into the prompt. Retrieve 15 chunks, hope the right one is in there, let the model sort it out. That inflates prompt size, which directly inflates both cost and time-to-first-token, because the model must process every input token before producing output.
With reranking, you retrieve 20 to 30 cheap candidates, score them properly with a cross-encoder, and pass only the best 3 to 5 into the prompt. The rerank stage costs you something — typically 50 to 200ms for a small hosted reranker over 20 candidates — but you claw it back through a much smaller prompt and often improve answer quality at the same time.
The rule of thumb: if your prompts routinely exceed a few thousand tokens of retrieved context, reranking will probably pay for itself in latency alone.
5. Prompt Assembly, the Quiet Offender
Between retrieval and generation sits a stage most people never instrument. It usually involves fetching full document text from your primary database using the IDs returned by the vector search, formatting everything into a template, and counting tokens to ensure you fit the context window.
Two things go wrong here constantly:
- Sequential database calls. Fetching five documents in a loop means five round-trips. Batch them into a single query. This is a one-line change that frequently saves 100ms or more.
- Synchronous token counting. Tokenising several thousand words to check length is real CPU work. If you are doing it on the request path in a single-threaded runtime, you are blocking the event loop. Estimate with a character heuristic first and only tokenise precisely when you are near the limit.
6. Stream Everything
If you take one thing from this post: stream your responses. Not because it reduces total time — it does not — but because it changes what the user is waiting for.
Without streaming, the user stares at a spinner for the full duration and forms an opinion about your product during that silence. With streaming, they start reading after a few hundred milliseconds and the remaining generation time happens while they are occupied.
Streaming also lets you overlap work that would otherwise be sequential. You can begin rendering the answer while citations are still being resolved, or start generating from the first retrieved chunk while the rest are still being fetched. The engineering is more involved, but the perceived improvement is larger than anything else on this list.
7. A Sensible Optimisation Order
If you are starting from a slow pipeline and want the shortest path to something that feels good:
- Add streaming. Biggest perceived improvement, no accuracy cost.
- Cache query embeddings. Cheap to implement, removes an entire network hop for repeat queries.
- Batch your document fetches. Usually a small diff for a real gain.
- Add reranking and shrink your prompt. Improves quality and often latency together.
- Co-locate your services. Configuration work, no code changes.
- Only then consider a smaller generation model, and only if your measurements say generation is genuinely the bottleneck.
The Honest Summary
RAG latency is an accumulation problem, not a single-component problem. Nothing in the pipeline is catastrophically slow on its own. You get an embedding call here, a network hop there, an oversized prompt, a sequential fetch loop, and no streaming — and the total lands somewhere between four and eight seconds, which is exactly the range where users stop trusting the feature.
Instrument the five spans. Fix them in order of perceived impact rather than technical interest. And stream, always.
If you are building something in this space and want a second pair of eyes on the architecture, get in touch — this is the kind of problem I enjoy.