RAG Query Cost and Latency Calculator

Introduction: RAG query pipeline cost and latency estimates

A retrieval-augmented generation (RAG) request typically retrieves relevant documents or chunks from a vector database, adds that context to an LLM prompt, and generates a final answer. This calculator estimates the cost and latency of that RAG serving path from the workload, pricing, retrieval, cache, and index-refresh inputs you provide.

Use the RAG estimates for planning monthly spend, testing the value of a higher answer-cache hit rate, comparing model or vector-store assumptions, and identifying whether tokens, retrieval fan-out, retrieval timing, or model time is driving the result. The latency figures are simplified average-style estimates rather than service-level guarantees.

RAG inputs explained (units and how they’re used)

These inputs describe the request volume, token usage, retrieval work, caching behavior, and index maintenance costs of the RAG pipeline.

• Monthly Queries (Q): total RAG queries served per month.
• Prompt Tokens per Query (P): tokens sent to the model for each RAG request, including instructions, history, and retrieved context. If your provider charges different prompt and completion rates, this calculator uses a blended rate as described in the limitations.
• Completion Tokens per Query (R): tokens generated in the RAG answer.
• Model Price per 1K Tokens ($) (M): price per 1,000 tokens. The calculator treats P + R as billable tokens at this one rate.
• Retrieval Fan-out (documents) (F): documents or chunks fetched for each RAG query. For pricing, each document represents one vector read operation, or its equivalent.
• Cache Hit Rate (0–1) (h): fraction of queries answered from cache. Enter a decimal, so 0.35 means 35%.
• Latency per Retrieval (ms) (L_r): average time for one retrieval. The model applies it linearly as F × L_r.
• Base LLM Latency (ms) (L_llm): average model time excluding retrieval, such as prompt processing and generation under typical load.
• Vector Read Cost per 1K Retrievals ($) (V): charge per 1,000 retrieval operations.
• Embedding Price per 1K Tokens ($) (p): charge per 1,000 tokens embedded during refresh or reindexing.
• Documents in Index (E): total documents or chunks in the RAG index.
• Average Tokens per Document (T): approximate tokens embedded for each document or chunk.
• Monthly Refresh Percentage (%) (D%): share of the index re-embedded each month, converted in the calculation to D = D% / 100.

Formula: RAG query cost model

The RAG monthly cost model combines model inference, vector reads, and recurring embedding refresh. Per-query amounts are calculated by allocating the monthly embedding cost across Q queries where applicable.

1) RAG LLM cost

For each uncached RAG request, billable model tokens are approximated as P + R. With one blended price per 1K tokens (M):

LLM cost per query:

2) RAG vector read cost

Each uncached RAG query triggers F retrievals. If the vector store charges V dollars per 1,000 retrievals:

Vector cost per query:

C_vector = (F × V) / 1000

3) RAG embedding refresh amortization

A RAG index also has a recurring maintenance cost when new or changed content is re-embedded. With E documents, T tokens per document, and a monthly refreshed fraction D:

• Monthly embedding tokens: E × T × D
• Monthly embedding cost: (E × T × D × p) / 1000

To compare index upkeep with serving cost, the calculator spreads this monthly RAG embedding cost across monthly queries Q:

Embedding refresh cost per query:

C_embed = (E × T × D × p) / (1000 × Q)

Cold-cache vs warm-cache RAG cost

For RAG answer caching, this calculator treats a cache hit as avoiding both retrieval and LLM inference because a stored final answer is returned. Under that simplifying assumption:

• Cold-cache (no cache): C_cold = C_llm + C_vector + C_embed
• Warm-cache (hit rate h): C_warm = (1 − h) × (C_llm + C_vector) + C_embed

Embedding refresh remains in both RAG scenarios because maintaining the index is modeled independently of answer-cache performance.

RAG retrieval and generation latency model

This RAG latency estimate adds base LLM time to linear retrieval time, using the requested fan-out and per-retrieval latency.

• Cold latency: L_cold = L_llm + (F × L_r)
• Warm latency: L_warm = (1 − h) × L_cold (cache hits are treated as approximately 0 ms of incremental compute in this simplified view)

Production cached answers still have routing, lookup, and network overhead. To represent part of that overhead in this model, use a lower effective cache hit rate rather than assuming every hit removes the full cold-path time.

How to use: RAG cache cost worked example with the default inputs

This RAG example uses the page defaults: Q=100,000, P=800, R=600, M=$0.003, F=8, h=0.35, L_r=45ms, L_llm=700ms, V=$0.15, p=$0.0001, E=250,000, T=750, and D%=20 (therefore D=0.20).

• LLM cost/query: ((800+600)/1000)×0.003 = 1.4×0.003 = $0.0042
• Vector cost/query: (8×0.15)/1000 = $0.0012
• Embedding monthly cost: tokens = 250,000×750×0.20 = 37,500,000; cost = 37,500,000/1000×0.0001 = $3.75
• Embedding cost/query: $3.75 / 100,000 = $0.0000375

Cold RAG cost/query is approximately 0.0042 + 0.0012 + 0.0000375 = $0.0054375 (~$0.0054), implying monthly cold serving of approximately $543.75.

Warm RAG cost/query at 35% answer-cache hits is approximately (1−0.35)×(0.0042+0.0012) + 0.0000375, or $0.0035475 (~$0.00355). The resulting monthly warm serving estimate is approximately $354.75.

Cold RAG latency is approximately 700 + 8×45 = 1060ms. Under the simplified cache-hit treatment, warm latency is approximately 0.65×1060 = 689ms. In this example, base LLM time is the larger latency component, model tokens are the larger serving-cost component, and embedding refresh is small per query because it is allocated across a high query volume.

How to interpret RAG cost and latency results

Read the RAG results by separating variable request-path expense from the index-refresh cost that is allocated across the month.

• If per-query RAG cost is high, first inspect token counts (P and R). Large retrieved context or long completions commonly dominate model spend.
• If vector retrieval cost is high, examine F, filtering, and reranking so that fan-out stays as low as quality requirements allow.
• If embedding amortization is large, the monthly refresh cost is being distributed over relatively few queries, or the refreshed portion of a large index is substantial.
• If RAG latency is high, compare L_llm with F × L_r to see whether generation or retrieval is the larger modeled contributor.

Scenario comparison: RAG caching and retrieval fan-out levers

The RAG scenario table identifies operational changes that affect the calculator’s cost and latency inputs; the displayed Optimized row applies only the built-in cache and fan-out changes described below.

The calculator’s Optimized scenario raises the entered cache hit rate by 0.20, capped at 0.95, and sets fan-out to 80% of the entered value, with a minimum of one retrieval. It leaves token counts, model price, embedding refresh, base LLM latency, and per-retrieval latency unchanged, so treat it as a narrow cache-and-retrieval what-if rather than a complete architecture recommendation.

Assumptions and limitations of this RAG serving model

This RAG calculator intentionally uses a compact model, so its estimates should be calibrated against observed production traces and invoices.

• Blended token price: Many providers price prompt and completion tokens differently. This calculator applies one rate (M) to both.
• Answer-cache semantics: A hit avoids both retrieval and LLM inference in this model because it returns a stored final answer. Caching retrieval results or prompt prefixes produces different savings.
• Average-style latency: Real RAG services also need p95 and p99 analysis. Queueing, throttling, cold starts, and regional network variation are not modeled.
• Linear retrieval latency: The formula uses F × L_r. Systems that retrieve in parallel may behave more like max(L_r) plus overhead.
• Costs not included: Rerankers, tool or function calls, retries, guardrails, moderation, streaming, vector index builds, writes, storage, observability, and application or server costs are outside this estimate.
• Variable token counts: Tokens per document and per query change with chunking and prompt design. Treat the output as directional until logging validates the assumptions.

RAG pricing and timing sourcing note

For a usable RAG estimate, enter current provider pricing for M and p, vector-database pricing for V, and measured production values for L_r and L_llm. Sampling real request traces and running the calculator separately with p50 and p95 timing inputs provides a more informative planning range than relying on one average.