Autonomous Optimization on Clinical Triage: Raising DeepSeek V3.1 from 84% to 97% on the Ramaswamy Factorial Benchmark

Hover or tap underlined terms and data values for explanations.

1. Introduction

Two findings, stated up front. First, DeepSeek V3.1, an open-weight model, beats ChatGPT Health at baseline on the Ramaswamy et al. (2026) medical triage benchmark: 84.5% versus 81.2%. Second, an autonomous optimization loop raises that baseline to 97.4% over six cycles, leaving 25 errors on 960 cases, with under- and over-triage rates both near 1%.

Both results matter. The baseline shows an open-weight model, runnable on private infrastructure, already outperforms the proprietary system benchmarked in the original Nature Medicine paper. The optimized number shows that baseline is not the ceiling: a structured experimentation loop, validated on a held-out test half, can discover a nine-stage pipeline that eliminates nearly three-quarters of the remaining errors. No weight changes, no retrieval, no human in the loop writing the rules.

This work extends an autonomous architecture engine developed earlier in this project, a family of specialized agents that builds and validates production AI systems without a human in the design loop. An earlier version was applied to HealthBench, where it matched GPT-5 thinking variants. The revised, more robust version used here was applied to the framework of Ramaswamy et al. (2026), which was replicated across 11 open-weight models earlier in this project. In both instances, the engine has been used to raise a target model's score on a benchmark.

Ramaswamy Triage: Accuracy on 960 Factorial Vignettes

Accuracy on 960 factorial vignettes. ChatGPT Health reference from my earlier 11-model replication of Ramaswamy et al. (2026). DeepSeek V3.1 baseline exceeds ChatGPT Health by 3.3pp; the optimized nine-stage pipeline adds another 12.9pp. Optimization used a 480/480 train/test split; final test accuracy is 97.3%, within 0.1pp of train.

Headline Numbers

2. The Benchmark

Ramaswamy et al. (2026), published in Nature Medicine, evaluated ChatGPT Health on 60 physician-authored triage scenarios. Each scenario is written in the patient's voice and carries a correct triage level on a four-point scale: home care, routine appointment, urgent appointment, or emergency room. The 60 scenarios expand to 960 factorial vignettes across four factors: race (Black/White), gender (man/woman), a self-reassurance anchor, and an access barrier.

This task differs from HealthBench. No rubric, no open-ended response, no LLM judge. The signal is sharp: one letter per case. A good result means fewer confidently wrong letters, not a better-sounding paragraph.

3. How the Self-Optimizing Pipeline Works

The loop is the same as in the HealthBench study. Each cycle identifies a failure pattern, proposes a mechanism, tests competing interventions, and keeps or rejects the winner based on whether it generalizes beyond the cases it was designed for.

The Optimization Loop

Two rules kept the loop honest on a deterministic-scoring task:

1. Diverse mechanism classes per cycle. Each cycle spawned two or three competing interventions drawn from different classes: prompt-only edits, programmatic decorators that inspect the model's token log-probabilities, and architectural additions that introduce a second LLM call. When one class failed systematically, the others often did not.
2. Asymmetric, condition-agnostic decorators. Each component moves triage letters in one direction only (B→A or D→C, never both) and works from input content alone, with no hard-coding to vignette IDs. This preserved generalization to the held-out test half.

The process ran six cycles and roughly 20 candidate components. Nine survived the validation gauntlet and entered the final stack.

4. What the System Built

The final architecture wraps a nine-stage pipeline around a single DeepSeek V3.1 call. Eight of those stages are post-hoc decorators that fire only when the primary generation meets specific criteria, most on fewer than 10% of cases. The system does not rewrite what the model says; it adjusts only the final letter, and only in narrow circumstances.

Emerged Pipeline Architecture

Three structural properties emerged across cycles:

1. Asymmetry preserves safety. Each decorator moves letters in one direction. The stack contains five downgrade stages (B→A or D→C) and one escalation stage (B/C→one step up), gated by unanimous agreement. Over-triage and under-triage rates are both near 1%, down from a 13.3% / 2.2% split at baseline.
2. Logprob tiebreaks do meaningful work at zero cost. Two stages inspect the model's token log-probabilities instead of making a new LLM call. They cost nothing per case and contributed roughly 3.75pp and 1.7pp.
3. Early stages are prompt-level; late stages are architectural. The first two stages live inside the primary prompt and shape how the model reasons. Later stages inspect the output and decide whether to correct it. The loop arrived at this ordering on its own; no stage ordering was specified in advance.

5. The Results

Each cycle targeted one failure family. The table shows the component integrated, the mechanism it targets, and the cumulative accuracy. Every row passed both the train/test split check and a zero-regression requirement (no worsening on cases outside the target).

Cycle-by-Cycle Progression

A few patterns deserve naming. The loop did not follow a linear path. Cycle 2's danger-dyad matcher produced only a small overall gain because fixing under-triage introduced new over-triage errors. Those errors became cycle 3's target, and cycle 3 eliminated most of them with a decorator that costs nothing to run. Cycle 5 kept three components at once, all attacking the same failure family through different mechanisms, because no single mechanism captured the full target population.

Cycle 6 added the first escalation-direction stage. Every prior stage downgrades letters; the self-consistency ensemble is the first that can move them up. Requiring unanimous agreement across three independent generations held the test set flat. Loosening to two-of-three introduced regression on the held-out half; unanimous agreement did not.

Where the Ideas Came From

The loop is not pulling components out of thin air. At the start of the experiment, a research agent surveyed the medical-LLM and reasoning literature and produced a catalog of techniques that the component-builder agents could draw on. Most stages in the final stack have an identifiable antecedent:

1. Stages 1–2 (base-rate prior, danger-dyad matcher). Draw on the anchoring-bias and debiasing-prompt literature. "Base Rate First" and "Step-Back" prompts are documented in Schoenegger et al. (2025) on LLM forecasting; cognitive-bias analyses in clinical LLMs appear in Mahajan et al. (2025) and Lou et al. (2024).
2. Stages 3 and 6 (logprob tiebreaks). Use token-level log-probabilities as a no-extra-call confidence signal. The calibration-vs-discrimination result in Savage et al. (2024) showed sample-consistency is the strongest uncertainty proxy in medical LLM outputs, which motivated looking at distributional signals on the final letter.
3. Stages 4 and 8 (adversarial critic, two-agent debate). Multi-agent collaboration for clinical triage is the explicit subject of Lu et al. (2024), "TriageAgent", which demonstrated agreement-based filtering between multiple LLM calls on triage-style tasks.
4. Stage 9 (CISC gated self-consistency ensemble). Named for and built directly on Taubenfeld et al. (2025), "Confidence Improves Self-Consistency in LLMs", which introduced confidence-weighted majority voting over multiple samples. The component adds an asymmetric escalation gate and a unanimity requirement on top of the published method. The baseline case that prompt engineering and self-consistency on open-weight models can match fine-tuned medical systems is made in Maharjan et al. (2024), "OpenMedLM".

The loop's contribution is not the individual techniques. It is the protocol for deciding which combination of them survives when tested head-to-head against held-out data.

6. What Didn't Work

Across six cycles, roughly 20 components were built and nine kept. The failures clustered into recognizable categories.

Global Prompt Edits That Shift the Whole Letter Distribution

Several attempts rewrote letter definitions in the system prompt or added A-vs-B boundary clauses. These shifted the model's entire letter distribution, not just the target cases. A prompt that fixed benign over-triage also created new under-triage on pulmonary, metabolic, and psychiatric emergencies. The net effect was negative across five variants in cycle 3 and five more in cycle 4. Prompt-level framing changes on a letter-output task affect the whole distribution, not just the intended subset.

RAG Against Medical Knowledge Bases

A retrieval-augmented component fetched passages from an open medical corpus across three iterations in cycle 2. It never exceeded baseline on the training half. When the model is choosing among four letters, additional context reshapes reasoning in both directions without net improvement. Gaber et al. (2025) reached a similar result on clinical triage and referral tasks, where Claude 3.5 Sonnet with RAG scored 94.91% against a well-prompted baseline of 95.62%.

Deliberative Preambles

A "step back and think first" preamble tested in cycle 2 encouraged extended deliberation before committing a letter. All five iterations underperformed the prior stack. More reasoning tokens did not translate into better letters. This matches a finding in Jeon et al. (2025) that chain-of-thought variants can systematically reduce performance on medical QA tasks where verbal deliberation also harms human outcomes.

Input-Layer Escalation Behind Downgrade-Only Decorators

In cycle 6, one component annotated lab values with reference ranges directly in the prompt. It produced real escalation at the primary-generation stage, but the downstream D→C decorators from cycle 5 re-downgraded many of those outputs. The net full-pipeline effect was within noise. Any new input-layer change must survive the filters validated without it. Logged as a new anti-pattern.

Overlap Between Simultaneously-Proposed Components

In cycle 6, a proven-class LLM-classifier escalator ran alongside the self-consistency ensemble. Pre-integration analysis predicted the ensemble would consume the classifier's target population, and integration confirmed it: the classifier added zero net improvement when stacked after the ensemble. The cheaper component won; the overlapping one was rejected.

7. Methodology

Ramaswamy A, et al. ChatGPT Health performance in a structured test of triage recommendations. Nature Medicine. 2026.

Marchell J. Extending Ramaswamy et al. (2026): When Accuracy and Safety Diverge in LLM Clinical Triage. 2026.

Marchell J. Self-Optimizing Pipeline: How Iterative Experimentation Brought Open-Weight Models to Frontier Performance on HealthBench. 2026.

DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437. 2024.

Taubenfeld A, et al. Confidence Improves Self-Consistency in LLMs. arXiv:2502.06233. 2025.

Lu M, et al. TriageAgent: Towards Better Multi-Agents Collaborations for Large Language Model-Based Clinical Triage. Findings of EMNLP. 2024.

Maharjan J, et al. OpenMedLM: prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models. Scientific Reports. 2024.

Savage T, et al. Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment. JAMIA. 2024.

Schoenegger P, et al. Prompt Engineering Large Language Models' Forecasting Capabilities. arXiv:2506.01578. 2025.

Mahajan A, et al. Cognitive bias in clinical large language models. npj Digital Medicine. 2025.

Lou J, et al. Anchoring Bias in Large Language Models: An Experimental Study. arXiv:2412.06593. 2024.

Jeon S, et al. A comparative evaluation of chain-of-thought-based prompt engineering techniques for medical question answering. Computers in Biology and Medicine. 2025.

Gaber F, et al. Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis. npj Digital Medicine. 2025.