Many teams building retrieval‑augmented generation (RAG) systems for high‑stakes classification route every ambiguous case directly to a large language model (LLM), trusting the retrieved context to resolve ambiguity. That approach can work for demos, but it breaks down under audit, regulatory scrutiny, or when a compliance officer asks why a decision was made months earlier.
Vineet Vijay, a Lead AI and machine learning engineer, has spent the past year building RAG‑based classifiers in regulated enterprise settings. In those environments a wrong answer is not just an awkward chatbot reply — decisions must withstand scrutiny long after the model produced them. This requirement forces a different design philosophy than many AI engineering guides assume.
The hidden costs of an all‑LLM pipeline
Three main problems emerge with an "everything to the model" design:
- Auditability: "The model decided based on retrieved context" is not an acceptable explanation. You need a decision path a human can reconstruct without rerunning inference and hoping for the same output.
- Cost at scale: if a system handles tens of thousands of cases a day and every case triggers an LLM call with multiple retrieved documents, both inference bill and latency scale with volume in a way that rule‑based logic does not.
- Model drift on easy cases: LLMs are excellent at nuanced judgments but can be inconsistent on cases that should have deterministic answers. A structured match against known criteria should not depend on a language model’s fluctuations.
The cascade approach
The fix is to stop treating the LLM as the front line and use it as the escalation path instead. Practically, this is a three‑stage pipeline:
-
Deterministic first stage: exact matches, structured field comparisons, and any clear rules get resolved here with no model call. This stage should clear the majority of cases (often more than half, depending on data quality), and every decision is fully explainable because it’s a lookup rather than an inference.
-
Retrieval‑focused second stage: cases that survive the first stage — i.e., are not clearly resolved — get a retrieval pass that pulls specific evidence for the ambiguity: prior reviewer decisions on similar cases, contextual documents explaining apparent conflicts, or historical precedent clarifying edge cases. Retrieval quality matters more than generation: if you retrieve the wrong context, even the best LLM will produce a confident but incorrect answer.
-
LLM escalation third stage: the LLM only sees the residue that stages one and two couldn’t resolve. In one system the author worked on, routing only the truly ambiguous 10–15% of cases to the LLM cut inference cost by roughly 6× compared to an all‑LLM baseline, while improving consistency on the deterministic majority to effectively perfect.
Prompt design for asymmetric risk
When an instance reaches the LLM stage, teams often use a neutral prompt like “Decide whether this case should be approved or flagged.” That framing is wrong for high‑stakes classification because the costs of the two error types are not symmetric: missing a true issue can cause real harm, while a false flag typically costs reviewer time and a delay.
An asymmetric‑risk prompt makes that tradeoff explicit. Concretely, instruct the model to treat uncertainty as a reason to escalate rather than to clear, provide calibrated examples of both error types and their consequences, and request a confidence score alongside the classification. Use that confidence score as a second cascade point: anything below a threshold goes to a human reviewer regardless of the model’s binary classification.
This may look like a prompt engineering detail, but in practice it separates systems that reduce reviewer workload from those that appear to work while quietly increasing risk.
Evaluating such a system properly
Standard RAG evaluation metrics were not built for this use case, and applying them without adaptation will give a false sense of confidence. Important adjustments include:
- Measure retrieval quality separately from final classification accuracy. A system can have strong retrieval ranking but still make bad final decisions if the generation step misweights evidence.
- Deliberately oversample cases that reach stage three in your evaluation set, because that’s where judgment is actually tested. If your eval mirrors production distribution, it will be dominated by deterministic cases your cascade already handles well, leaving you blind to the failures that matter.
- "LLM as judge" evaluation can work only if the judge prompt encodes the same asymmetric risk framing as production. A judge that treats both error types equally will favor the wrong tradeoff when you tune the system.
- Build a feedback loop from confirmed outcomes back into the retrieval corpus. When a human reviewer overturns a model decision, that case and its correct resolution should become retrievable context for future similar cases; without this, the system never improves on ambiguous cases.
Broader lesson
Reaching for the most capable model for every decision is understandable, but in domains where wrong answers have real consequences, the valuable engineering work is deciding what should never touch the model. Cascade architecture is not merely a workaround for LLM limitations; it is what a mature RAG system looks like once you have had to defend its decisions to someone whose job is to find flaws in your logic.
Before writing a single prompt for an AI system in regulated or high‑stakes domains, ask not “How do I get the model to handle this?” but “Which parts of this decision should never have been the model’s job in the first place?”
Vineet Vijay is a Lead AI and machine learning engineer.
