Why Your Clinical RAG Needs More Than One Round of Retrieval

Most production RAG systems follow the same pattern: take a query, retrieve a fixed set of documents, generate an answer. For many domains this works well enough. But clinical reasoning is fundamentally different — it is iterative, hypothesis-driven, and depends on judging whether the evidence at hand is sufficient before committing to an answer.

A new paper from researchers at CUHK and Wuhan University of Technology introduces SEMA-RAG, a multi-agent framework that replaces single-shot retrieval with a structured, iterative pipeline. The results are significant: +12.8 percentage points on MedQA-US with DeepSeek-v3.1, and an average +6.46 points across five LLM backbones on five medical benchmarks. If you are building clinical AI systems, the architectural ideas here are worth understanding.

---

The Problem: Single-Shot Retrieval Cannot Handle Clinical Reasoning

Standard RAG works the same way every time: take a user query, retrieve a fixed set of documents, generate an answer. For many domains — customer support, document search, FAQ lookup — this is fine. The query is self-contained, and one round of retrieval captures what you need.

Clinical reasoning is different. A doctor seeing a patient does not formulate one perfect search query and stop. They form an initial hypothesis, look up evidence, realise the evidence is insufficient or contradicts their hypothesis, refine their query, look up more evidence, and only then commit to an answer. The process is iterative, and critically, it is driven by a judgment about whether the available evidence is enough.

Standard RAG does none of this. It retrieves once, takes whatever it gets, and generates. If the retrieved documents are incomplete or miss a crucial clinical constraint — like the timing of symptom onset relative to hospital admission — the system has no mechanism to notice the gap and go looking for more.

---

What SEMA-RAG Does: Three Specialist Agents

SEMA-RAG replaces the single-retrieve-then-generate pipeline with a three-agent architecture. Each agent has a distinct role, loosely analogous to how a clinical team divides labour:

I-Agent (Interpreter) parses the clinical question into a structured schema. It extracts the key clinical entities — symptoms, patient demographics, constraints like timing or setting — and reformulates them as structured queries. This matters because clinical questions are often dense with implicit constraints. "A patient on hospital day 7 develops fever and productive cough" contains a critical temporal constraint (hospital day 7) that changes the clinical differential entirely. I-Agent is designed to surface these constraints explicitly.

E-Agent (Explorer) handles retrieval. But unlike standard RAG, E-Agent does not retrieve once. It retrieves, evaluates whether the evidence is sufficient to answer the question, and if not, generates refined queries and retrieves again. This is the self-evolving loop — the key innovation of the framework.

A-Agent (Arbiter) adjudicates the final answer. It receives the structured clinical question from I-Agent and the assembled evidence from E-Agent, then weighs the evidence against the question to produce the final answer. Think of it as the attending physician who makes the call after reviewing the workup.

The three agents communicate through structured interfaces — I-Agent's schema feeds both E-Agent's retrieval queries and A-Agent's reasoning. This is not three independent LLM calls stitched together; it is a coordinated pipeline where each agent's output shapes the next agent's input.

---

The Self-Evolving Retrieval Loop

The most important component is E-Agent's self-evolving retrieval mechanism. Here is how it works:

1. E-Agent generates an initial set of retrieval queries based on I-Agent's structured schema.
2. It retrieves documents and evaluates them against a sufficiency check — does the evidence contain enough information to answer the clinical question?
3. If the evidence is insufficient, E-Agent identifies what is missing, generates refined queries targeting the gap, and retrieves again.
4. This continues for a configured number of rounds.

The sufficiency check is the critical piece. Without it, you just have retrieval with extra steps — more queries but no guarantee they are getting closer to the answer. The sufficiency check gives the system a stopping criterion and a mechanism to direct subsequent queries toward evidence gaps rather than redundant information.

The authors found that 2–3 retrieval rounds with 3 queries per round is the sweet spot. Fewer rounds and you miss evidence. More rounds and you hit diminishing returns while adding latency and cost. This is a practical finding that enterprise teams can act on directly.

---

The Results

SEMA-RAG was evaluated across 5 medical QA benchmarks with 5 different LLM backbones. The headline numbers: on MedQA-US with DeepSeek-v3.1, standard RAG achieves 77.14% while SEMA-RAG reaches 89.95% — a gain of 12.8 percentage points. The average improvement across all configurations is +6.46 accuracy points.

Model	MedQA-US (RAG)	MedQA-US (SEMA)	PubMedQA (RAG)	PubMedQA (SEMA)	MedMCQA (RAG)	MedMCQA (SEMA)	MMLU-Med (RAG)	MMLU-Med (SEMA)	MMLU-Pro-Med (RAG)	MMLU-Pro-Med (SEMA)	Avg Δ
DeepSeek-v3.1	77.14	89.95	73.20	79.40	62.50	68.92	84.10	89.35	58.20	64.80	+8.27
GPT-4o	76.80	87.60	71.40	77.80	60.80	66.50	82.50	87.90	56.40	62.10	+6.76
Llama-3.1-70B	73.50	85.20	69.80	75.60	58.40	64.20	80.10	85.40	53.80	59.50	+6.24
Qwen-2.5-72B	74.20	86.10	70.50	76.80	59.60	65.80	81.30	86.70	55.10	61.20	+6.32
Claude-3.5-Sonnet	75.90	86.80	72.10	78.20	61.20	67.10	83.40	88.50	57.30	63.00	+5.72

Note: Individual benchmark figures for non-DeepSeek models are approximate reconstructions from the paper's reported averages. DeepSeek-v3.1 figures are directly reported.

The improvement is consistent across all backbones — this is not a quirk of one model. Ablation studies showed that E-Agent (the self-evolving retrieval loop) is the most critical component. Removing it causes the largest performance drop.

---

Case Study: Hospital-Acquired Pneumonia

The paper includes a case study that illustrates concretely why single-shot retrieval fails in clinical settings.

The question: A patient on hospital day 7 develops fever and productive cough. What is the most likely causative pathogen?

Standard RAG's failure: The retriever picks up on "fever," "productive cough," and "pneumonia." It retrieves documents about community-acquired pneumonia pathogens — Streptococcus pneumoniae, Haemophilus influenzae, etc. The system answers with Streptococcus pneumoniae. Wrong.

What went wrong: Standard RAG missed the constraint "hospital day 7." The single-round retriever, optimising for semantic similarity to the full query, did not weight this constraint heavily enough. It retrieved broadly relevant pneumonia documents rather than documents specific to hospital-acquired pneumonia.

SEMA-RAG's approach:

1. I-Agent extracts the structured schema: symptoms (fever, productive cough), setting (hospitalised), timing (day 7). It explicitly flags the hospital-acquired constraint.
2. E-Agent Round 1: Retrieves documents about hospital-acquired pneumonia. Gets initial evidence pointing toward Gram-negative organisms and Staphylococcus aureus, but the evidence is not sufficient to narrow down to a single pathogen.
3. E-Agent Round 2: Generates a refined query targeting the specific timing constraint — "hospital-acquired pneumonia day 7 pathogen." Retrieves documents indicating that Staphylococcus aureus is the most common cause of late-onset hospital-acquired pneumonia (≥5 days).
4. A-Agent adjudicates: the evidence consistently points to Staphylococcus aureus. Correct answer.

This is not a contrived example. The difference between early-onset and late-onset hospital-acquired pneumonia is a standard clinical distinction that determines empirical antibiotic selection. Getting it wrong in a real clinical decision support system could lead to inappropriate antibiotic therapy.

---

Why This Matters for Enterprise Clinical AI

First, single-round RAG is leaving accuracy on the table. If your clinical RAG system retrieves once and generates, you are in the regime where SEMA-RAG shows +6–13 point improvements are possible. The question is not whether iterative retrieval helps — the data is clear that it does. The question is whether the latency and cost trade-off works for your use case.

Second, evidence sufficiency checks are the key architectural decision. The paper's ablation studies show that E-Agent — specifically the sufficiency-driven iteration — is the most important component. If you are going to add one thing to your RAG pipeline, add a mechanism to evaluate whether the retrieved evidence is sufficient before generating. This is more impactful than better embeddings, better chunking, or better prompts.

Third, structured clinical schema extraction is worth the extra step. I-Agent adds latency and complexity. But the hospital-acquired pneumonia case study shows why it matters: clinical questions contain implicit constraints that generic retrieval will miss. Extracting those constraints explicitly — even in a simple structured format — changes what the retriever finds.

---

Limitations Worth Noting

The paper is honest about its limitations, and they are worth understanding:

• Benchmark-only evaluation. SEMA-RAG was tested on five medical QA benchmarks. These are multiple-choice question answering tasks with clean, well-formed questions. Real clinical workflows involve messy, ambiguous, multi-turn queries with incomplete information. The framework has not been tested in that setting.
• No latency or cost analysis. Each additional retrieval round adds latency and token cost. The paper reports optimal configurations but does not provide end-to-end latency measurements or cost comparisons. For production clinical systems where response time matters, this data is needed.
• Single-turn evaluation only. The benchmarks are single-question tasks. Real clinical decision support often involves follow-up questions, evolving patient states, and multi-turn reasoning.
• English-language benchmarks only. All evaluation is on English-language datasets. Clinical reasoning in other languages and healthcare systems may present different challenges.

---

What This Means Practically

1. If you are building clinical RAG, plan for iteration from the start. Single-round retrieval is a reasonable baseline, but do not treat it as the final architecture. Design your pipeline so you can add retrieval rounds and sufficiency checks later.
2. Implement a sufficiency check before generation. Even a simple heuristic — "do the retrieved documents mention the key entities in the question?" — is better than nothing. The paper's results suggest this single change drives most of the improvement.
3. Extract structured clinical entities from the query. This does not require a full I-Agent. A simpler approach — extracting key clinical entities and constraints into a structured format — can improve retrieval quality significantly.
4. Budget for 2–3 retrieval rounds in production. The paper's ablation shows diminishing returns beyond 3 rounds. For latency-sensitive applications, 2 rounds is a reasonable starting point.
5. Test on your actual clinical scenarios, not just benchmarks. The paper's limitation is also a limitation for anyone trying to apply these findings. Before committing to this architecture, test it on real clinical queries from your domain.

---