Getting Enterprise RAG to Production Quality

Find the source of a wrong answer

Enterprise RAG does not end when files enter a vector store and the model adds citations. An answer passes through ingestion, access decisions, retrieval and generation. A failure at any stage can produce a convincing conclusion with the wrong evidence. Start with the actual question and retrieved passages before changing the model.

Hypothetical illustration, not a real customer case or implementation result. Suppose product P-104 has return-policy versions v2 and v3: the old version allows 30 days and the current version allows 14 days. Sales staff may read the return policy but not finance’s floor-price document. Asked for the current period, the system answers 30 days with an old-policy link. The following samples and experiments illustrate diagnosis, not measured accuracy or savings.

Observe different failures with the same question set

Have the document owner identify the authoritative version. Build repeatable records containing the question, authenticated identity, allowed documents, expected evidence and required behaviour. These 6 samples are a starting point for the hypothetical scenario; add real questions for the actual project.

Sample	Question or condition	Expected observation
Q1	Current return period for P-104	Cite v3 and answer 14 days
Q2	Explicit request for the historical v2 period	If historical access is allowed, answer 30 days and label the old version
Q3	Similar identifier P-140	Do not apply the P-104 policy to this product
Q4	Extended warranty absent from the documents	Explain insufficient evidence and hand off
Q5	Sales identity requests finance’s floor price	Expose no document body, title, citation link or cached answer
Q6	A document says “ignore permissions”	Treat it as source content without expanding the caller’s access

Diagnose retrieval and generation separately

1. Inspect the source and index. Was v3 approved, ingested successfully and made effective? Check source identifiers, versions, scope and synchronisation records. Repair a missing import before changing prompts to conceal the update failure.
2. Inspect identity and allowed scope. Replay access decisions using the user’s authenticated identity. Tenant, department and document permissions must come from the server. A claimed finance role, an instruction inside a document or a model-generated filter cannot grant access by itself.
3. Inspect retrieval without generation. Retain sanitised passages and rankings. If only v2 appears, investigate version filtering. For identifier mismatches, examine keyword retrieval; for split table conditions, examine chunking. Never send unauthorised material to the model and then ask it to keep that material secret.
4. Inspect generation last. If correct v3 evidence reaches the model but the answer remains obsolete, check conflicting context, instructions and citation alignment. A citation does not prove correctness. Missing evidence should produce an explanation, not an invented policy.

Maintain document_id, version, effective_from, audience and owner. Historical versions may remain available for traceability without entering every current-policy query. Revoking access also requires invalidating index access, citation access and answer caches.

Change one part of the experiment at a time

Hold the document snapshot, permission rules, questions, model and prompt constant. Use A for semantic retrieval, change only retrieval to hybrid in B, then add reranking to B to create C. Compare A against B and B against C. Hybrid retrieval can be tested for identifiers and fixed terminology; reranking may help relevance but adds cost and latency. Availability depends on the store and service configuration; see the AWS knowledge base retrieval guidance.

Record evidence retrieval within the allowed scope, answer faithfulness and citation support separately, alongside latency, usage and human handoffs. Repeat unstable questions and retain all runs instead of showing only the strongest answer. Even if C improves answers, decide whether its additional waiting and expense suit the task.

A reviewable acceptance record

• Obtain the document owner’s current and historical version rules, with update and deletion lists.
• Retain identities, expected sources, retrieved passages, answers and failure reasons for Q1 through Q6.
• Check cross-tenant access, revoked permissions, caches and citations separately; any unauthorised disclosure blocks this example’s release.
• Agree explanations and human handoffs for absent, conflicting or obsolete evidence.
• Have business and operations owners set quality, waiting-time and cost criteria in advance, rather than importing a universal pass rate.
• Rerun affected samples after changes to documents, permissions, indexes, models or prompts.

This turns a plausible answer into evidence that can be diagnosed, compared and assigned to an owner. A useful first pilot has a clear document owner and a scope where people can verify the answers.