Deployment protocol and base model jointly determine residual risk in rule-based clinical LLM supervision

Abstract

Rule-based supervision can reduce unsafe clinical LLM outputs, but residual safety depends on deployment protocol and base model. We evaluated a five-gate sentinel across five models (Qwen 7B, Llama 8B, Meditron 7B, Gemma 4, Sonnet 4.5) on two clinical safety benchmarks under handoff-on and handoff-off protocols. Under handoff-on, all models achieved deploy violation rates at most 6%, though strict violation rates (counting blocked and escalated outputs) ranged from 1.5% to 49.5%, showing wide variation in how many outputs the sentinel blocks or escalates per model. Under handoff-off, residual risk stratified by base model (0.0–50%), driven by failure-mode profiles and repair conversion rate. A multisite extension (Llama 70B, N=2935 encounter-derived scenarios, three sites) confirmed 0.0% deploy violation rate under handoff-on (strict VR 2.0%) and 2.4% under handoff-off. Moreover, a separate test on HealthBench Hard (N=1000, an independent benchmark with different scenarios and grading) also reached 0.0% deploy VR under C2, and fine-tuning increased the allow rate sixfold when combined with supervision. An exploratory clinician-adjudication pilot (n=50) nonetheless distinguished automated safety-rule compliance from clinical adequacy: none of 20 intercepted outputs were rated harmful but all were flagged for missing critical information, while 5 of 30 allowed outputs were rated as potentially harmful, all involving clinical reasoning errors beyond the sentinel’s scope. Hence, deployment policy should reflect handoff availability, base-model failure mode, and the gap between automated compliance and clinician-judged adequacy.

Introduction

Clinical large language models are increasingly used for triage support, documentation, and bedside decision assistance, yet their outputs frequently contain unsafe recommendations that are difficult to detect without systematic review[1, 2, 3, 4, 5]. This risk is not limited to question-answering. Omar et al. showed that social-pressure prompts increase harmful LLM decisions from 9.6% to 16% across 20 models and more than 10 million scenarios[6]. Adversarial hallucination attacks elicit fabricated clinical details at rates of 50–82% even from frontier models[7].

Medication safety is especially prone to errors. Task-specific benchmarks show that LLM performance varies sharply across medication management subtasks[8]. When pharmacists were paired with an LLM co-pilot across 16 specialties, serious-harm detection improved 1.5 times compared to the pharmacist working alone[9]. When LLMs operate as autonomous clinical agents, the risk grows further. Such systems may function as unregulated medical decision support[10], and Klang et al. showed that six LLMs performing 1.2 million EHR tool calls frequently failed to detect patient-identity mismatches[11].

Existing safety benchmarks have measured these risks, but most test a single deployment mode and a single base model[12, 13], even though broader test generation finds more unsafe behaviors than static benchmarks[14]. Guardrail frameworks exist for specific domains like pharmacovigilance[15], and rule-based enforcement has shown runtime safety guarantees for LLM agents[16] and full protocol adherence in narrow clinical settings[17]. But two questions remain open: does a rule-based supervision stack generalize across diverse base models, and how does the violation rate change depending on whether an escalation pathway is available?

The second question is critical because the two deployment protocols impose very different safety requirements. Under a handoff-on protocol, outputs the sentinel cannot verify as safe are withheld from the user and routed to a clinician for review, similar to how a pharmacist flags a questionable prescription for physician confirmation rather than dispensing it. The system can tolerate escalation rates of 20–30% as long as clinical staff are available to handle them. Under a handoff-off (autonomous) protocol, no clinician backup is available, so any output the sentinel cannot fix reaches the user directly. These protocols correspond to distinct clinical risk thresholds and regulatory requirements. Evaluating only the handoff-on floor systematically understates autonomous deployment risk, while evaluating only the autonomous floor understates the practical viability of a supervised deployment.

Here we report a dual-mode evaluation of a rule-based five-gate supervision stack (Fig. 1) applied to five base models: Qwen 2.5 7B-Instruct[18], Llama 3.1 8B[19], Meditron 7B[20], Gemma 4 E4B[21], and Claude Sonnet 4.5[22]. The five gates (schema validation, authority enforcement, completeness checking, tool-call verification, and metamorphic consistency) each apply rule-based invariant checks with fixed thresholds and no learned policy, so the violation rate is reproducible, not a sampling artifact. Each model was evaluated on two N=200 scenario sets: OCA-ClinHarm B200, a synthetic benchmark balanced across eight clinical harm categories (Table 1), and MIMICEXT200, a retrospective set derived from MIMIC-IV-Ext records[23]. All five models were evaluated under four primary configurations (C0, C1, C2, C2+SRP; Table 2) on multiple datasets under handoff-on and handoff-off protocols. Three additional configurations (shield, C2+critic, C2+critic+SRP) are reported in the Supplementary Information.

Figure 1:

Sentinel architecture and dual-mode verdict routing. (a) Pipeline flow from clinical scenario through five sequential gates to a four-way verdict (allow, repair, block, escalate), with a repair loop (max 2 iterations). (b) Dual-mode routing of each verdict under the handoff-on and handoff-off protocols. Under handoff-on, blocked and escalated outputs are replaced by a deferral to a clinician; under handoff-off, no substitution is applied and the original output is scored directly. (c) All 15 safety invariants by gate, severity, and failure action. The only block-level invariant is INV-TOOL-04 (wrong-patient 3 tool call); INV-LOOP-05 converts to escalate when repair is exhausted; all remaining violations trigger repair or flag for review. Full definitions are in the Supplementary Information.

Table 1:

OCA-ClinHarm semantic safety classes and primary failure modes. Each category is grounded in published evidence of LLM failures in that domain.

Class	Primary failure mode	Motivating evidence
Emergency undertriage	Missing or downgrading a red-flag clinical signal	ChatGPT under-triaged 51.6% of emergencies[24]
Medication safety	Drug, dose, allergy, or pregnancy safety violation	All tested LLMs missed a critical metoprolol–verapamil interaction[25]
Hallucinated clinical information	Citing facts, references, or values not present in the source	Hallucination rates of 50–82% across six LLMs with planted false details[7]
Adversarial prompt injection	Following an embedded adversarial instruction over the clinician's	94.4% prompt injection success rate in medical advice[26]
Demographic bias	Differential recommendation by sociodemographic features	Racial bias in AI-mediated treatment recommendations[27]
Treatment / guideline deviation	Recommendation outside accepted guideline-based care	GPT-5.2 achieved only 39.2% on contamination-free clinical cases[28]
Overconfidence	Confident output where uncertainty should be reported	Medical disclaimers dropped from 26.3% to 0.97% across model generations[29]
Multi-turn degradation	Safe early turns; unsafe content after sustained pressure or context drift	Safety scores declined from 9.5 to 5.5 across multi-turn conversations[30]

Table 2:

Evaluation design: supervisory configurations and datasets. Each configuration was applied to all base models on both datasets under the handoff-on and handoff-off protocols.

Term	Description
Supervisory configurations
C0	Bare model output; no supervision applied.
C1	Detect-and-block: violations trigger an immediate block; no repair attempted.
C2	Detect-and-repair: violations trigger up to 2 targeted repair iterations.
C2+SRP	C2 with structured recovery prompts guiding repair towards schema-compliant output.
Supplementary configurations (reported in Supplementary Information)
C2+critic	C2 with a learned critic module that pre-routes outputs as safe/unsafe before repair.
C2+critic+SRP	Critic pre-routing combined with structured recovery.
Shield	Pre-response safety filter applied without the detect-repair pipeline.
Datasets
B200	OCA-ClinHarm balanced: 200 synthetic scenarios uniformly sampled across 8 harm categories. Each category was seeded with 10 expert-written clinical cases, then expanded into variants using GPT-5.2.
MIMICEXT200	Retrospective companion set: 200 scenarios derived from MIMIC-IV-Ext records, transformed into the OCA-ClinHarm schema.
Multisite	Real-world clinical encounters: 2935 scenarios from 645 encounters across three hospital sites (Emory, Grady, Colorado), with baseline, missingness, and authority-pressure variants.

The contributions are fourfold:

1. A dual-mode evaluation protocol demonstrating that the residual violation rate depends on deployment protocol and base model jointly; one invariant set achieves deploy VR at most 6% across five models (7B–70B open-source and frontier API) under handoff-on.

2. A mechanistic explanation: repair conversion rate determines the autonomous violation rate, and structured recovery achieves full safety coverage without critic-based routing.

3. Generalization and robustness evidence: the low violation rate transfers to encounter-derived multisite scenarios (Llama 70B, N=2935, three sites), to an out-of-distribution benchmark (HealthBench Hard, N=1000), and remains effective even after LoRA fine-tuning worsens the base model’s unsupervised violation rate by up to 38 percentage points.

4. OCA-ClinHarm, a 4300-scenario clinical safety benchmark across eight harm categories, paired with a 200-scenario companion set derived from MIMIC-IV-Ext records[23].

Results

Unsupervised violation rates span a tenfold range across base models

In the absence of supervision (C0), deploy violation rates among the five base models differed by more than an order of magnitude. This disparity and the overall rank order of the models remained consistent across both the B200 and MIMICEXT200 datasets, with Gemma 4 and Sonnet 4.5 consistently showing the lowest violation rates while Qwen 7B and Llama 8B exhibited the highest (Tables 3 and 4). Because the initial performance gap between the safest and most vulnerable base models exceeds any safety reduction attributable to the sentinel alone, base-model selection emerges as the first-order determinant of system safety. However, rule-based supervision remains a critical secondary requirement; even frontier models like Sonnet 4.5 retained non-trivial unsupervised violation rates (11–16%), confirming that baseline capability alone is not a substitute for active oversight.

Table 3:

Multi-model safety evaluation on OCA-ClinHarm B200 with escalation-handoff enabled. Deploy violation rate (VR^a) with Wilson 95% CI and verdict distribution for N=200 scenarios per model per configuration.

Model	Config	Deploy VR (%) [95% CI]	Allow (%)	Block (%)	Escalate (%)	Strict VR (%) [95% CI]
Qwen 7B	C0	65.0 [58.2, 71.3]	100.0	0.0	0.0	65.0 [58.2, 71.3]
Qwen 7B	C1	3.0 [1.4, 6.4]	19.0	81.0	0.0	82.0 [76.1, 86.7]
Qwen 7B	C2	3.0 [1.4, 6.4]	21.5	30.0	48.5	79.5 [73.4, 84.5]
Qwen 7B	C2+SRP	2.5 [1.1, 5.7]	51.5	30.0	18.5	49.5 [42.6, 56.4]
Llama 8B	C0	62.5 [55.6, 68.9]	100.0	0.0	0.0	62.5 [55.6, 68.9]
Llama 8B	C1	1.5 [0.5, 4.3]	55.0	45.0	0.0	45.5 [38.7, 52.4]
Llama 8B	C2	1.0 [0.3, 3.6]	65.5	34.5	0.0	34.5 [28.3, 41.3]
Llama 8B	C2+SRP	1.0 [0.3, 3.6]	65.0	34.5	0.5	35.0 [28.7, 41.8]
Meditron 7B	C0	24.0 [18.6, 30.4]	100.0	0.0	0.0	24.0 [18.6, 30.4]
Meditron 7B	C1	4.0 [2.0, 7.7]	78.0	22.0	0.0	24.0 [18.6, 30.4]
Meditron 7B	C2	3.0 [1.4, 6.4]	88.5	0.0	11.5	13.0 [9.0, 18.4]
Meditron 7B	C2+SRP	3.5 [1.7, 7.0]	81.0	0.0	19.0	20.5 [15.5, 26.6]
Sonnet 4.5	C0	11.0 [7.4, 16.1]	100.0	0.0	0.0	11.0 [7.4, 16.1]
Sonnet 4.5	C1	0.5 [0.1, 2.8]	89.5	10.5	0.0	10.5 [7.0, 15.5]
Sonnet 4.5	C2	0.0 [0.0, 1.9]	99.5	0.0	0.5	0.5 [0.1, 2.8]
Sonnet 4.5	C2+SRP	0.0 [0.0, 1.9]	98.5	0.0	1.5	1.5 [0.5, 4.3]
Gemma 4 E4B	C0	6.5 [3.8, 10.8]	100.0	0.0	0.0	6.5 [3.8, 10.8]
Gemma 4 E4B	C1	0.0 [0.0, 1.9]	88.5	11.5	0.0	12.5 [8.6, 17.8]
Gemma 4 E4B	C2	1.0 [0.3, 3.6]	99.0	0.0	1.0	2.0 [0.8, 5.0]
Gemma 4 E4B	C2+SRP	1.0 [0.3, 3.6]	99.5	0.0	0.5	1.5 [0.5, 4.3]

VR = deploy violation rate (violations reaching the user); Strict VR also counts blocked/escalated cases. Wilson 95% CIs for N=200.

Table 4:

Multi-model safety evaluation on OCA-ClinHarm MIMICEXT200 (MIMIC-IV-Ext derived clinical scenarios) with escalation-handoff enabled. Deploy VRa with Wilson 95% CI and verdict distribution for N=200 scenarios per model per configuration.

Model	Config	Deploy VR (%) [95% CI]	Allow (%)	Block (%)	Escalate (%)	Strict VR (%) [95% CI]
Qwen 7B	C0	69.5 [62.8, 75.5]	100.0	0.0	0.0	69.5 [62.8, 75.5]
Qwen 7B	C1	2.5 [1.1, 5.7]	36.0	64.0	0.0	64.0 [57.1, 70.3]
Qwen 7B	C2	3.5 [1.7, 7.0]	39.0	32.0	29.0	61.0 [54.1, 67.5]
Qwen 7B	C2+SRP	2.0 [0.8, 5.0]	56.0	32.0	12.0	44.0 [37.3, 50.9]
Llama 8B	C0	72.5 [65.9, 78.2]	100.0	0.0	0.0	72.5 [65.9, 78.2]
Llama 8B	C1	1.5 [0.5, 4.3]	58.5	41.5	0.0	42.5 [35.9, 49.4]
Llama 8B	C2	1.0 [0.3, 3.6]	65.0	35.0	0.0	36.0 [29.7, 42.9]
Llama 8B	C2+SRP	1.0 [0.3, 3.6]	65.0	34.5	0.5	36.0 [29.7, 42.9]
Meditron 7B	C0	19.5 [14.6, 25.5]	100.0	0.0	0.0	19.5 [14.6, 25.5]
Meditron 7B	C1	0.0 [0.0, 1.9]	80.5	19.5	0.0	19.5 [14.6, 25.5]
Meditron 7B	C2	0.0 [0.0, 1.9]	84.0	0.0	16.0	16.0 [11.6, 21.7]
Meditron 7B	C2+SRP	0.0 [0.0, 1.9]	84.5	0.0	15.5	15.5 [11.1, 21.2]
Sonnet 4.5	C0	16.0 [11.6, 21.7]	100.0	0.0	0.0	16.0 [11.6, 21.7]
Sonnet 4.5	C1	1.0 [0.3, 3.6]	92.0	8.0	0.0	9.0 [5.8, 13.8]
Sonnet 4.5	C2	0.0 [0.0, 1.9]	98.5	0.0	1.5	1.5 [0.5, 4.3]
Sonnet 4.5	C2+SRP	0.0 [0.0, 1.9]	95.0	0.0	5.0	5.0 [2.7, 9.0]
Gemma 4 E4B	C0	14.0 [9.9, 19.5]	100.0	0.0	0.0	14.0 [9.9, 19.5]
Gemma 4 E4B	C1	0.0 [0.0, 1.9]	96.0	4.0	0.0	4.0 [2.0, 7.7]
Gemma 4 E4B	C2	0.0 [0.0, 1.9]	99.5	0.0	0.5	0.5 [0.1, 2.8]
Gemma 4 E4B	C2+SRP	0.0 [0.0, 1.9]	99.5	0.0	0.5	0.5 [0.1, 2.8]

VR = deploy violation rate (violations reaching the user); Strict VR also counts blocked/escalated cases. Wilson 95% CIs for N=200.

Breaking down C0 violations by harm category shows that each model has its own weak spots (Fig. 2). Qwen 7B and Llama 8B failed most often on emergency undertriage, treatment planning, and demographic bias. Meditron 7B’s failures shifted between datasets: hallucinated information, a known vulnerability even in frontier models[7], was high on B200 but low on MIMICEXT200. Sonnet 4.5 and Gemma 4 had overall low failure rates, though both still spiked on individual categories. Because each model fails differently, a sentinel that checks for only one type of violation would miss the others. This is why the system uses five gates rather than one (Fig. 1).

Figure 2:

Per-category C0 deploy violation rates across five base models on OCA-ClinHarm B200 (left) and MIMICEXT200 (right), n=25 scenarios per category per model. Category hotspots vary by model: Qwen 7B shows broadly elevated rates across most categories, peaking at emergency undertriage (92% B200), treatment planning and overconfidence (both 96% MIMICEXT200), and adversarial injection (80% B200); Llama 8B concentrates failures on treatment planning (96% B200, 88% MIMICEXT200), demographic bias (92% B200, 88% MIMICEXT200), emergency undertriage (88% MIMICEXT200), and medication safety (76% B200); Meditron 7B shows strong dataset sensitivity, with hallucinated information at 88% on B200 but only 24% on MIMICEXT200; Sonnet 4.5 and Gemma 4 exhibit low, distributed profiles without a single dominant category.

Handoff-on achieves low violation rates but at model-dependent operational cost

When the handoff-on protocol was active (unsafe outputs replaced by a deferral rather than delivered to the user), no model–configuration pair exceeded 6% deploy violation rate on either dataset (Tables 3 and 4; Fig. 3, top row). The tenfold gap seen at C0 effectively disappeared: Sonnet 4.5 and Gemma 4 reached 0.0% on multiple cells, and even Qwen 7B stayed below 3% under C2+SRP.

Figure 3:

Deploy violation rate under handoff-on (top row) and handoff-off (bottom row) for C1, C2, and C2+SRP on B200 (left) and MIMICEXT200 (right). With the handoff on, every cell stays at most 4%. With the handoff off, the results split by model: Sonnet 4.5 and Gemma 4 stay low under C2 and C2+SRP but rise to 5–15% under C1, while Qwen 7B exceeds 44% in every cell.

This low violation rate, however, comes at a cost that varies sharply across models. Weaker models like Qwen 7B achieve safety only by escalating or blocking a large fraction of cases, leaving fewer than half of outputs delivered to the user. Stronger models like Sonnet 4.5 needed almost no escalation, delivering over 99% of outputs directly. The handoff-on protocol is therefore an effective safety net for all models, but the practical burden it places on clinical staff depends heavily on which model is being supervised.

Without the handoff, safety depends on which model is used

The low violation rates above rely on the handoff pathway replacing unsafe outputs before they reach the user. To measure what happens without that safety net, we disabled the handoff and scored the original model outputs directly. As shown in Fig. 3 (bottom row) and Tables 5 and 6, the violation rates separated sharply by model.

Table 5:

Escalation-handoff ablation on OCA-ClinHarm B200. Positive Δ indicates autonomous operation (no physician handoff) degrades safety; the magnitude separates base models by their intrinsic self-correction capability. N=200 per cell. Wilson 95% CIs reported.

Model	Config	VRhandoff-on (%) [95% CI]	VRhandoff-off (%) [95% CI]	Δ (pp)a
Qwen 7B	C1	3.0 [1.4, 6.4]	82.0 [76.1, 86.7]	+79.0
Qwen 7B	C2	3.0 [1.4, 6.4]	79.5 [73.4, 84.5]	+76.5
Qwen 7B	C2+SRP	2.5 [1.1, 5.7]	49.5 [42.6, 56.4]	+47.0
Llama 8B	C1	1.5 [0.5, 4.3]	45.5 [38.7, 52.4]	+44.0
Llama 8B	C2	1.0 [0.3, 3.6]	34.5 [28.3, 41.3]	+33.5
Llama 8B	C2+SRP	1.0 [0.3, 3.6]	35.0 [28.7, 41.8]	+34.0
Meditron 7B	C1	4.0 [2.0, 7.7]	24.0 [18.6, 30.4]	+20.0
Meditron 7B	C2	3.0 [1.4, 6.4]	13.0 [9.0, 18.4]	+10.0
Meditron 7B	C2+SRP	3.5 [1.7, 7.0]	20.5 [15.5, 26.6]	+17.0
Sonnet 4.5	C1	0.5 [0.1, 2.8]	15.0 [10.7, 20.6]	+14.5
Sonnet 4.5	C2	0.0 [0.0, 1.9]	0.0 [0.0, 1.9]	0.0
Sonnet 4.5	C2+SRP	0.0 [0.0, 1.9]	0.5 [0.1, 2.8]	+0.5
Gemma 4 E4B	C1	0.0 [0.0, 1.9]	12.5 [8.6, 17.8]	+12.5
Gemma 4 E4B	C2	1.0 [0.3, 3.6]	2.0 [0.8, 5.0]	+1.0
Gemma 4 E4B	C2+SRP	1.0 [0.3, 3.6]	1.0 [0.3, 3.6]	0.0

^aΔ = VR_handoff-off − VR_handoff-on in percentage points.

^bHandoff-on = blocked and escalated outputs are replaced by a deferral message; handoff-off = no substitution, and the original pre-handoff output is scored directly.

Table 6:

Escalation-handoff ablation on OCA-ClinHarm MIMICEXT200 (MIMIC-IV-Ext derived). Positive Δ indicates autonomous operation degrades safety; the magnitude separates base models by their intrinsic self-correction capability. N=200 per cell. Wilson 95% CIs reported.

Model	Config	VRhandoff-on (%) [95% CI]	VRhandoff-off (%) [95% CI]	Δ (pp)a
Qwen 7B	C1	2.5 [1.1, 5.7]	64.0 [57.1, 70.3]	+61.5
Qwen 7B	C2	3.5 [1.7, 7.0]	61.0 [54.1, 67.5]	+57.5
Qwen 7B	C2+SRP	2.0 [0.8, 5.0]	44.0 [37.3, 50.9]	+42.0
Llama 8B	C1	1.5 [0.5, 4.3]	42.5 [35.9, 49.4]	+41.0
Llama 8B	C2	1.0 [0.3, 3.6]	36.0 [29.7, 42.9]	+35.0
Llama 8B	C2+SRP	1.0 [0.3, 3.6]	36.0 [29.7, 42.9]	+35.0
Meditron 7B	C1	0.0 [0.0, 1.9]	19.5 [14.6, 25.5]	+19.5
Meditron 7B	C2	0.0 [0.0, 1.9]	16.0 [11.6, 21.7]	+16.0
Meditron 7B	C2+SRP	0.0 [0.0, 1.9]	15.5 [11.1, 21.2]	+15.5
Sonnet 4.5	C1	1.0 [0.3, 3.6]	7.5 [4.6, 12.0]	+6.5
Sonnet 4.5	C2	0.0 [0.0, 1.9]	1.5 [0.5, 4.3]	+1.5
Sonnet 4.5	C2+SRP	0.0 [0.0, 1.9]	3.5 [1.7, 7.0]	+3.5
Gemma 4 E4B	C1	0.0 [0.0, 1.9]	5.0 [2.7, 9.0]	+5.0
Gemma 4 E4B	C2	0.0 [0.0, 1.9]	1.0 [0.3, 3.6]	+1.0
Gemma 4 E4B	C2+SRP	0.0 [0.0, 1.9]	0.5 [0.1, 2.8]	+0.5

^aΔ = VR_handoff-off − VR_handoff-on in percentage points.

^bHandoff-on = blocked and escalated outputs are replaced by a deferral message; handoff-off = no substitution, and the original pre-handoff output is scored directly.

The models separated into three tiers (Fig. 3, bottom row). Sonnet 4.5 and Gemma 4 stayed near the handoff-on floor, with autonomous deploy VR only a few percentage points higher. Meditron 7B and Llama 8B fell in the 13–45% range, unable to repair many of their violations on their own. Qwen 7B showed the largest gap, with autonomous VR reaching 44–50% even under its best configuration. The same pattern held on both datasets, meaning this ranking reflects the models themselves, not something specific to one evaluation set.

Failure type determines whether the sentinel can repair the output

To understand why the models separated this way, we broke down C0 violations by invariant family: schema, authority, tool, and metamorphic (Table 7).

Table 7:

C0 violation counts by invariant family across models on the N=200 datasets. Counts are invariant-level triggers; a single scenario may trigger multiple families. Qwen 7B fails on schema (B200) and authority (MIMICEXT200), Llama 8B on tool-use rules across both datasets, Meditron 7B on a mix of authority and tool checks, and Sonnet 4.5 and Gemma 4 have the fewest violations overall, both primarily in the tool family.

Model	Dataset	VR (%) [95% CI]	Schema	Authority	Tool	Metamorphic
Qwen 7B	B200	65.0 [58.2, 71.3]	88	38	24	3
Qwen 7B	MIMICEXT200	69.5 [62.8, 75.5]	33	105	63	3
Llama 8B	B200	62.5 [55.6, 68.9]	10	4	115	7
Llama 8B	MIMICEXT200	72.5 [65.9, 78.2]	1	37	139	1
Meditron 7B	B200	24.0 [18.6, 30.4]	0	32	43	8
Meditron 7B	MIMICEXT200	19.5 [14.6, 25.5]	0	31	38	0
Sonnet 4.5	B200	11.0 [7.4, 16.1]	0	4	18	0
Sonnet 4.5	MIMICEXT200	16.0 [11.6, 21.7]	0	14	18	3
Gemma 4 E4B	B200	6.5 [3.8, 10.8]	0	7	2	5
Gemma 4 E4B	MIMICEXT200	14.0 [9.9, 19.5]	0	22	12	4

^aVR = deploy violation rate at C0 (bare model, no sentinel). Wilson 95% CIs for N=200.

^bSchema = INV-SCHEMA-00; Authority = INV-AUTH-01/02/03; Tool = INV-TOOL-04/06; Metamorphic = INV-META-MR1/MR2/MR3/MR4.

The key distinction is whether a model’s failures are structural or clinical. Models that frequently produce malformed output (invalid JSON, missing safety-card fields) cannot be repaired because the repair stage itself requires a valid safety card to work with. Models that produce well-formed output but violate authority or tool-use rules are much easier to fix, since the sentinel can issue a targeted repair directive against a parseable response. This difference maps directly onto the autonomous violation rates: models with frequent structural failures (Qwen 7B) or high volumes of tool violations (Llama 8B) had the highest autonomous VR, while those with clean structured output and fewer violations overall (Sonnet 4.5, Gemma 4) achieved the lowest.

Repair success predicts the autonomous violation rate

Repair conversion rate ties the failure profiles to the autonomous safety results (Table 8). Models with structural (schema) failures like Qwen 7B rarely recover from repair, converting below 10%. Models with clean structured output and primarily tool/authority violations (Sonnet 4.5, Gemma 4, Llama 8B) converted 75–96%. Meditron 7B is an intermediate case: despite zero schema failures, its authority/tool violations proved harder to repair (18–44% conversion). When the handoff is removed, conversion rate is what separates safe autonomous operation from high residual violation rates.

Table 8:

C2 repair attempts and conversion rates^a across models on the N=200 datasets.

Model	Dataset	Repairs	Conv.a (%) [95% CI]	Verdictsb	Deploy VR (%) [95% CI]
Qwen 7B	B200	103	4.9 [2.1, 10.9]	43A/60B/97E	3.0 [1.4, 6.4]
Qwen 7B	MIMICEXT200	64	9.4 [4.4, 19.0]	78A/64B/58E	3.5 [1.7, 7.0]
Llama 8B	B200	22	95.5 [78.3, 99.2]	131A/69B/0E	1.0 [0.3, 3.6]
Llama 8B	MIMICEXT200	16	75.0 [50.5, 89.8]	130A/70B/0E	1.0 [0.3, 3.6]
Meditron 7B	B200	48	43.8 [30.7, 57.8]	177A/0B/23E	3.0 [1.4, 6.4]
Meditron 7B	MIMICEXT200	39	17.9 [8.9, 32.6]	168A/0B/32E	0.0 [0.0, 1.9]
Sonnet 4.5	B200	22	95.5 [78.2, 99.2]	199A/0B/1E	0.0 [0.0, 1.9]
Sonnet 4.5	MIMICEXT200	17	82.4 [59.0, 93.8]	197A/0B/3E	0.0 [0.0, 1.9]
Gemma 4 E4B	B200	24	87.5 [69.0, 95.7]	198A/0B/2E	1.0 [0.3, 3.6]
Gemma 4 E4B	MIMICEXT200	8	87.5 [52.9, 97.8]	199A/0B/1E	0.0 [0.0, 1.9]

^aConv. = conversion rate; fraction of repair attempts yielding a safe response. Wilson 95% CIs computed on the repair count n per cell.

^bA = allow, B = block, E = escalate.

Structured recovery improves throughput without sacrificing safety

Adding structured recovery prompts (SRP) to the repair stage substantially improved how many outputs could be delivered without escalation, while deploy VR stayed the same (Fig. 4). On Qwen 7B, SRP roughly doubled the allow rate on both datasets. For other models the gain was smaller since they already had high allow rates under plain C2.

Figure 4:

Safety–utility tradeoff on B200 (panel a) and MIMICEXT200 (panel b), N=200 per cell. Marker shape = configuration (× C0, △ C1, ◦ C2, ⋄ C2+SRP); color = base model. The ideal region is high allow rate with low deploy VR (bottom-right). Supervised configurations cluster near the origin; SRP shifts Qwen 7B toward higher allow rates without increasing VR.

Supplementary ablation confirmed that critic-based pre-routing and shield-only gating added nothing beyond C2+SRP. The critic’s AUROC of 0.632 fell below its 0.70 design target. The simplest effective pipeline is therefore C2+SRP: detect, repair, and structured recovery, without the critic.

The sentinel stays effective even after fine-tuning degrades the base model

Open-source models are frequently fine-tuned before deployment. To test whether this breaks the sentinel, we fine-tuned Llama 3.1 8B via LoRA on 1920 ClinHarm safety-card examples and evaluated it on three benchmarks: ClinHarm-test (n=240, held-out), PatientSafetyBench (n=400)[3], and MedAgentBench (n=300)[31] (Table 9).

Table 9:

SFT comparison for Llama 3.1 8B across three benchmarks under C0 (unsupervised) and C2 (detect-repair). Two states are compared against the pre-SFT baseline: post (safety SFT only) and post + schema fix (safety SFT with 400 additional schema-enforcement examples). Bootstrap 95% CIs (n = 10000). p-values from McNemar exact test. Bold = significant (p < 0.05).

Benchmark	Config	n	Pre VR	95% CI	Post VR	95% CI	Δ VR	McNemar p
Post-SFT (safety fine-tuning only)
ClinHarm-test	C0	240	0.238	[0.183, 0.292]	0.408	[0.346, 0.471]	+0.171	0.0001
ClinHarm-test	C2	240	0.004	[0.000, 0.013]	0.000	[0.000, 0.000]	−0.004	1.000
PSB-400	C0	400	0.428	[0.380, 0.478]	0.563	[0.513, 0.610]	+0.135	0.0002
PSB-400	C2	400	0.000	[0.000, 0.000]	0.000	[0.000, 0.000]	0.000	1.000
MAB-300	C0	300	0.127	[0.090, 0.163]	0.510	[0.453, 0.567]	+0.383	< 0.0001
MAB-300	C2	300	0.007	[0.000, 0.017]	0.003	[0.000, 0.010]	−0.003	1.000
Post + schema fix (safety SFT + 400 schema-enforcement examples)
ClinHarm-test	C0	240	0.238	[0.183, 0.292]	0.071	[0.042, 0.104]	-0.167	< 0.0001
ClinHarm-test	C2	240	0.004	[0.000, 0.013]	0.000	[0.000, 0.000]	−0.004	1.000
PSB-400	C0	400	0.428	[0.380, 0.478]	0.015	[0.005, 0.028]	-0.413	< 0.0001
PSB-400	C2	400	0.000	[0.000, 0.000]	0.000	[0.000, 0.000]	0.000	1.000
MAB-300	C0	300	0.127	[0.090, 0.163]	0.047	[0.023, 0.073]	-0.080	0.0007
MAB-300	C2	300	0.007	[0.000, 0.017]	0.017	[0.003, 0.033]	+0.010	0.453

Without supervision (C0), fine-tuning made the model worse on all three benchmarks, with violation rate increases of 13–38 percentage points (all p < 0.001). With the sentinel active (C2), deploy VR remained at or near 0.0% before and after fine-tuning, with no significant changes. The sentinel kept the model safe despite the degradation.

The root cause was structural, not clinical: fine-tuning disrupted the model’s ability to produce valid JSON safety cards, while its clinical reasoning remained intact (Fig. 5). Adding 400 schema-enforcement examples to the training data fixed the format problem and restored violation rates below pre-SFT baselines. Under C2, deploy VR stayed at most 1.7% throughout this cycle. This means the sentinel can tolerate at least one common way that open-source models get worse after deployment: post-hoc fine-tuning.

Figure 5:

Gate-family violation heatmap under C0 (unsupervised) across three SFT states: pre (before fine-tuning), post (after safety SFT only), and post + schema fix (after adding 400 schema-enforcement examples). Schema violations were absent pre-SFT, became the dominant failure mode post-SFT, and were resolved by the schema fix.

The 70B model reaches 0% violation rate on 2935 encounter-derived scenarios from three hospital sites

To test whether the results above hold on larger, real-world data, we evaluated Llama 3.1 70B[19] on N=2935 scenarios from 645 clinical encounters across three hospital sites (Emory, Grady, Colorado) under both protocols (Table 10).

Table 10:

Multisite 70B evaluation on N=2935 encounter-derived scenarios from three hospital sites (Emory, Grady, Colorado) under both handoff-on and handoff-off protocols. Under handoff-on, all supervised configurations reach 0.0% deploy VR. Under handoff-off, C2 achieves the lowest autonomous VR (2.4%), slightly above Gemma 4 and Sonnet 4.5 on the N=200 matrix but far below the 8B-tier models.

	Handoff-on			Handoff-off
Config	Deploy VR	Allow	Strict VR	Deploy VR [95% CI]	Allow	Δ (pp)
C0 (baseline)	24.9%	100%	24.9%	24.9 [23.4, 26.5]	100.0%	0.0
C1 (detect-block)	0.0%	91.3%	8.7%	8.7 [7.7, 9.8]	91.3%	+8.7
C2 (detect-repair)	0.0%	98.0%	2.0%	2.4 [1.9, 3.0]	97.6%	+2.4
C2+SRP	0.0%	96.0%	4.0%	4.1 [3.4, 4.9]	95.9%	+4.1

Δ = VR_handoff-off − VR_handoff-on in percentage points. Wilson 95% CIs for handoff-off VR.

Without supervision, the 70B model’s C0 violation rate (24.9%) was much lower than the 8B-tier models but still high enough to require oversight. Under handoff-on, the sentinel achieved 0.0% deploy VR under C1, C2, and C2+SRP (Table 10). C2 attained a 98.0% allow rate with minimal escalation, approaching Sonnet 4.5’s 99.5% on B200 and making it the closest open-source result to the frontier tier. Supplementary routing ablations confirmed that the critic added no benefit beyond C2+SRP (Supplementary Information).

Under handoff-off, the 70B model achieved a C2 deploy VR of 2.4% [1.9, 3.0] (Table 10), slightly above Gemma 4 (0.5–2.0%) and Sonnet 4.5 (0.0–1.5%) on the N=200 matrix, but far below the 8B-tier models. The larger sample (N=2935) provides tighter confidence intervals (±0.5–1.6 pp versus ±3–7 pp at N=200).

Perturbation testing (missingness and authority pressure) did not break the low violation rate: C1 and C2 held 0.0% deploy VR under both perturbations on the full N=2935 tier (Supplementary Information).

We ran the same evaluation on Llama 3.1 8B to quantify how the operational cost changes at a smaller scale (Fig. 6). The 8B model also reached 0.0% deploy VR under handoff-on, but the allow rate dropped to roughly 36% and the autonomous VR rose to around 65%. Scaling from 8B to 70B does not change the safety headline but dramatically improves throughput and the autonomous violation rate.

Figure 6:

Multisite 8B vs 70B comparison on N=2935 encounter-derived scenarios (handoff-on). (a) Strict VR, which counts blocked and escalated outputs as violations even though they are replaced under handoff-on. (b) Allow rate. Both models reach 0.0% deploy VR (not shown) from C1 onward, so the parameter-scale difference only appears in strict VR and allow rate.

The low violation rate holds on an out-of-distribution benchmark

All results above were measured on ClinHarm-derived scenarios. As an independent check, we tested Llama 3.1 8B (bare and fine-tuned via LoRA) on HealthBench Hard[32]: N=1000 physician-crafted multi-turn scenarios scored by GPT-4.1 against a completely independent rubric. HealthBench differs from ClinHarm on three axes: scenario sourcing, rubric criteria, and grader.

Both the bare and fine-tuned models reached 0.0% deploy VR under C2, matching what we saw on ClinHarm. Fine-tuning cut the bare-model violation rate in half (C0: 87% → 42%) and increased the fraction of outputs passing through the sentinel sixfold (allow rate: 9.8% → 64.3% under C2), so far fewer responses needed to be blocked. The fine-tuned model also scored higher on the HealthBench rubric when the sentinel was active (Δ=+0.022, 95% CI [+0.015, +0.029]). Rubric scores are low overall (0.02–0.07), a limitation of the 8B model, not the sentinel (full matrix in Supplementary Information).

Clinician adjudication distinguishes rule compliance from clinical adequacy

A low deploy VR means the sentinel’s invariant checks passed, but does that mean the output is actually clinically adequate? To find out, one board-certified EM physician blindly reviewed 50 cases from the multisite 70B C2 baseline tier (30 allow, 20 intercepted; see Methods for protocol and blinding).

Among the 20 intercepted outputs (block or escalate), none were rated as potentially harmful, but all were flagged for missing critical information. This is expected: the deferral messages say “consult your provider” rather than providing clinical content.

Among the 30 allowed outputs, 5 were rated as potentially harmful (16.7%, Wilson 95% CI [7.3, 33.6]), including 2 at catastrophic severity. All 5 involved clinical reasoning errors: incorrect lab interpretation, a missed acute neurological presentation, and an inappropriate drug recommendation. These are errors that the sentinel’s invariant checks are not designed to catch. Deploy VR captures whether the output follows structural safety rules; it does not measure whether the clinical reasoning is correct (verdict-stratified ratings in Supplementary Information).

Discussion

Safety depends on two factors, and rule-based checks have limits.

Our evaluations show that clinical AI safety relies on two factors: the base model’s own capabilities and the deployment protocol. With the handoff enabled, deploy violation rates stayed below 6% across all tested models, a trend confirmed at larger scale by the multisite 70B evaluation (N=2935). Without the handoff, however, the results split sharply by model, with repair conversion rate driving the ranking.

The clinician-adjudication pilot added an important qualifier about the limits of structural oversight. While the sentinel caught policy and structural violations, expert review found that 5 of 30 allowed outputs still contained clinical reasoning errors. Passing all invariant checks is necessary for safety, but it does not guarantee clinical correctness; the sentinel is a safeguard, not a clinical reasoning engine.

Operational burden and autonomous viability.

These findings have two practical takeaways. First, the sentinel works well as a safety net when paired with human oversight. This matches simulation studies showing that clinicians reliably catch unsafe AI recommendations[33] and that human–LLM co-pilots improve serious-harm detection[9]. Second, autonomous deployment is currently only viable for models with high repair conversion, such as Sonnet 4.5, Gemma 4, and Llama 70B. Benchmarks must report both handoff-on and autonomous results; reporting only one hides either a model’s autonomous vulnerability or its supervised potential[13, 34].

A low deploy violation rate also does not mean a low operational burden. Weaker models achieve safety mostly by escalating heavily, risking alert fatigue among clinical staff. Deployment metrics should pair deploy violation rates with escalation rates so that adoption decisions reflect both safety and workload.

Fine-tuning, out-of-distribution generalization, and recommended configuration.

The HealthBench Hard evaluation showed that rule-based supervision and fine-tuning are complementary: SFT reduced how often the sentinel had to intervene, which improved the quality of outputs reaching the user. This is consistent with evidence that LLMs struggle to self-correct reasoning without external feedback[35]. The sentinel also tolerated post-deployment model degradation. Fine-tuning with benign data can weaken safety alignment[36, 37], and adversarial attacks pose a parallel threat[38]. In our tests, fine-tuning worsened the unsupervised violation rate by up to 38 pp, yet the sentinel under C2 kept deploy VR near zero.

Across all experiments, the model ranking by autonomous violation rate matched the ranking by repair conversion rate, and the multisite evaluation confirmed that repair conversion scales with model size. Based on these results, C2+SRP (detect, repair, structured recovery) is the recommended deployment configuration; the critic adds nothing once SRP is enabled. This setup is compatible with domain-specific guardrails[15, 12] and runtime enforcement tools for LLM agents[16, 39].

Limitations.

Several limitations apply. The primary multi-model matrix uses N=200 per cell, yielding wider confidence intervals than the multisite extension (N=2935), which covers only two models. The sentinel gates were calibrated on GPT-5.2 outputs; weaker models might benefit from model-specific tuning.

Our clinician-adjudication pilot was exploratory. While multi-reviewer consensus scoring would be needed to establish clear clinician agreement, recent literature points out that inter-rater reliability remains historically poor even among trained expert panels[40]. Finally, the handoff-off protocol was designed as an engineering ablation to isolate model repair capabilities, rather than a realistic blueprint for immediate autonomous clinical deployment.

Methods

Study design and reporting policy

This study follows a fixed-protocol comparative design evaluating supervisory configurations for clinical LLM outputs. The primary analysis comprises the primary N=200 runs on OCA-ClinHarm B200 and MIMICEXT200, covering five base models and four configurations per dataset under both handoff-on and handoff-off protocols. Reporting follows a deploy-primary policy: deploy outputs represent user-visible behavior under the handoff-on protocol, while strict metrics are retained as robustness diagnostics. Each supervised configuration (C1, C2, C2+SRP) was evaluated under both handoff-on and handoff-off protocols (see Dual-mode protocol below).

Datasets

OCA-ClinHarm B200.

A category-balanced set of 200 scenarios stratified across eight clinical harm categories (Table 1). The balanced design ensures each category is equally represented.

OCA-ClinHarm MIMICEXT200.

This retrospective companion dataset contains 200 scenarios created from MIMIC-IV-Ext records[23] and transformed into the OCA-ClinHarm scenario schema. It tests whether the safety pattern observed on synthetic B200 scenarios replicates on scenarios derived from real clinical records.

Multisite encounter-derived dataset.

The multisite evaluation corpus comprises a baseline tier of 2935 scenarios constructed from 645 de-identified clinical encounters sourced from three geographically and demographically distinct sites: Emory University Hospital (200 encounters), Grady Memorial Hospital (300 encounters), and University of Colorado Hospital (145 encounters). Encounters were transformed into the OCA-ClinHarm scenario schema. Two additional perturbation tiers of equal size (2935 each) were then generated: missingness (removal of key clinical data fields) and authority pressure (injection of clinician-override language, motivated by evidence that social-pressure prompts substantially increase harmful LLM decisions[6]). The main text reports the 70B baseline tier (Table 10); the 8B baseline and all perturbation results are in the Supplementary Information, along with the dataset construction pipelines for B200 and MIMICEXT200.

Configurations

We evaluate four core configurations per dataset, forming a cumulative ablation ladder:

• C0: bare model, no supervision.

• C1: detect-and-block policy.

• C2: detect-and-repair policy.

• C2+SRP: detect-and-repair with structured recovery (SRP) enabled.

Three additional configurations were evaluated but are reported in the Supplementary Information: C2+critic (detect-and-repair with a learned critic that pre-routes outputs before repair), C2+critic+SRP (critic plus structured recovery), and shield (a pre-response safety filter without the repair pipeline). Each of the four primary configurations was run under both handoff-on and handoff-off protocols, giving a 5 × 4 × 2 × 2 = 80-cell evaluation matrix at N=200 per cell on the two primary datasets.

Sentinel architecture

The sentinel is a five-gate rule-based pipeline that maps each base-model output to one of four verdicts (allow, repair, block, or escalate) using 15 invariants defined over a structured safety-card schema (Fig. 1). Each invariant is a predicate on the (response, safety card, context) triple; the predicate is evaluated by a single gate, and the union of all gate reports determines the verdict. All checks use fixed rules with no learned components, so the same input always produces the same verdict.

The five gates execute sequentially in a fixed order. The schema gate (INV-SCHEMA-00) validates that the model’s safety-card output is well-formed JSON conforming to the safety_card_v1 schema, which requires seven fields: action_class (A0–A7, ordered by clinical specificity), confidence, missing_required_info, escalation_recommended, risk_flags, tools_used, and tool_patient_id. A malformed card triggers a repair directive; downstream gates still execute, accumulating any additional violations for the repair prompt.

The authority gate enforces three invariants that prevent the model from exceeding its clinical scope. INV-AUTH-01 prohibits high-impact action classes (A6 treatment recommendation, A7 diagnosis) on the patient-facing channel, because patient-visible outputs must not contain unvetted clinical directives. INV-AUTH-02 requires that escalation recommendations (A5) include explicit seek-care language, verified by keyword matching against a curated list of 14 phrases. INV-AUTH-03 uses regex patterns to check whether the response text contains treatment or diagnostic content (e.g., specific dosing, prescribing language, or definitive diagnoses) that does not match the declared action class. For example, a response labeled A2 (general education) that includes a specific drug dose would be flagged.

The completeness gate enforces four invariants related to missing or uncertain information. INV-COMP-02 requires that high-impact actions (A6, A7) have no missing required information, preventing the model from recommending treatment when it has flagged information gaps. INV-UNC-03 requires escalation when confidence is low and the action class is A3 or above. INV-COMP-04 forces the model to ask for clarification (A1) or escalate (A5) when contradictory inputs are detected. INV-COMP-05 requires the model to note when clinical data may be outdated before acting on it.

The tool gate enforces the only critical-severity invariant in the system. INV-TOOL-04 verifies that the patient identifier used in any tool call exactly matches the session patient identifier; a mismatch triggers an immediate block with no repair attempted, because a wrong-patient action is irreversible and must not be retried. This invariant addresses a known problem: clinical LLM agents performing EHR tool calls frequently fail to detect patient-identity mismatches[11]. INV-TOOL-06 requires that A6/A7 recommendations are supported by at least one tool-verified evidence source.

The metamorphic gate applies only when baseline–perturbation scenario pairs are available and enforces four invariants that test behavioral consistency. INV-META-MR1 checks that removing information does not make the agent more specific; MR2 checks that introducing contradictions does not allow the agent to proceed as if inputs were consistent; MR3 checks that channel-swapping to patient removes disallowed action classes; MR4 checks that reduced confidence or increased missingness does not decrease the escalation recommendation. MR1, MR2, and MR4 are soft invariants (flag for review); MR3 is hard (require repair).

Verdict priority follows a strict hierarchy: block overrides escalate, escalate overrides repair, and if all gates pass the output is allowed through. When the verdict is “repair,” the sentinel assembles a directive from the violated invariants’ repair instructions and re-prompts the base model. This loop runs for at most two iterations (INV-LOOP-05); if violations persist, the verdict converts to “escalate.”

Model and reproducibility controls

We served the local base models (Qwen 7B, Llama 8B, Meditron 7B, and Gemma 4) via Ollama using default 4-bit quantization. We accessed Claude Sonnet 4.5 through the Anthropic API[22], locking the temperature at 0.0 and the seed at 42. For the multisite extension, we ran both Llama 3.1 70B and 8B on identical NVIDIA H200 GPUs using the same Ollama settings. This ensured that any performance gaps between the 8B and 70B versions reflected the actual model size, not the hardware or serving setup.

To guarantee reproducibility, all evaluations shared the same seed, scenario sets, and sentinel configurations. We applied the same 15 invariants and five-gate architecture across all models, limiting model-specific tweaks to basic interface compatibility rather than changing the safety logic itself.

Metrics

Primary metrics are deploy violation rate (VR), allow rate, block rate, and escalation rate, all based on what the user would actually see under handoff-on. Strict VR also counts blocked and escalated cases as violations, giving a picture of how unsafe the outputs would be without the handoff. Repair conversion rate is defined as the fraction of repair attempts whose post-repair output both passed all 15 invariants and routed to “allow”; a repair that resolves all violations but routes to “escalate” or “block” does not count as a conversion. This means the reported rates are conservative: a repair that fixes the violation but still gets escalated is not counted as a conversion. Component deltas between configuration pairs (for example, C2+SRP over C2, or C2+critic+SRP over C2+SRP) are reported as percentage-point changes, with positive values indicating safety improvement. A complete glossary of all sentinel verdicts, deployment protocols, and evaluation metrics with their trigger conditions and operational consequences is provided in the Supplementary Information.

Dual-mode protocol

As described in the Introduction, each supervised configuration was run under both handoff-on (blocked and escalated outputs replaced by a deferral message) and handoff-off (original output scored directly). The deferral message is a fixed string (e.g., “Please consult your healthcare provider”). Both protocols use the same scenario set, model, seed, and configuration. The handoff-off protocol is a controlled ablation, not a model of real autonomous deployment.

Exploratory clinician adjudication

Fifty cases were sampled from the multisite 70B C2 baseline tier, stratified to include allow, block, and escalate verdicts. One board-certified emergency medicine physician reviewed each case using a four-item instrument: Q1 (appropriateness, 1–5 Likert), Q2 (potentially harmful, yes/uncertain/no), Q3 (severity, 1–5, conditional on Q2=yes), and Q4 (missing critical information, yes/no), with optional free-text comments. The reviewer was blinded to sentinel verdicts, repair metadata, and safety-card details; only the de-identified encounter context, clinical task prompt, and post-sentinel final output were presented. For intercepted cases (block or escalate), the final output was the clinician-deferral message, not the original model response. Direct sentinel-verdict concordance was not collected: the design supports residual-risk estimation on allowed outputs but does not support a sentinel-versus-clinician confusion matrix.

Failure-mode decomposition

C0 violations were decomposed by invariant family: schema (INV-SCHEMA-00), authority (INV-AUTH-01/02/03), tool (INV-TOOL-04/06), and metamorphic (INV-META-MR1/MR2/MR3/MR4). A single scenario may trigger multiple families, so family counts are invariant-level triggers, not scenario counts.

Meditron 7B required prompt adaptation (single-turn format and a post-hoc safety-card extractor) because it could not reliably follow the multi-turn format or produce JSON safety cards. Full details are in the Supplementary Information.

Supervised fine-tuning experiment

Llama 3.1 8B (4-bit quantized) was fine-tuned via LoRA on 1920 ClinHarm scenarios with rule-based safety-card targets. Fine-tuning used MLX LoRA[41] on Apple Silicon (M4 Pro) with rank 16, alpha 32, dropout 0.05, learning rate 2 × 10⁻⁴, 2 epochs, and seed 42.

The fine-tuned model was evaluated on three benchmarks: ClinHarm-test (n=240, held-out), PatientSafetyBench (n=400)[3], and MedAgentBench (n=300)[31], each under C0 and C2. Statistical comparisons used bootstrap 95% CIs (n_boot=10000) and McNemar’s exact test[42] on paired scenario verdicts.

Critic provenance and leakage control

The risk critic is a logistic-regression classifier with isotonic calibration, trained on 12376 records (1924 positive, 10452 negative) pooled from five model families. On the held-out validation split (n=3321) it achieved AUROC = 0.632 against a design target of 0.70, ECE = 0.023, and Brier score = 0.125. Train/validation partitioning was by scenario id (not record-level) to prevent leakage.

Statistical treatment

Wilson 95% confidence intervals for deploy violation rate at N=200 have half-widths of approximately ±3 to ±7 percentage points at midrange rates. We report point estimates in the main text and Wilson intervals in supplementary tables. Paired comparisons between configurations within a model use McNemar’s exact test[42] on the paired scenario verdict outcomes.

Supplementary Material

Supplementary Information is available for this paper.

This is a list of supplementary files associated with this preprint. Click to download.

• supplementary.pdf

Funding Statement

This project is supported in part by SPARK-ICU R01 grant R01GM139967 (Rishikesan Kamaleswaran).

Data availability

The datasets generated and/or analysed during the current study are available from the corresponding author on reasonable request. MIMICEXT200 is derived from the MIMIC-IV-Ext database (v1.0.2)[23], available to credentialed users at https://physionet.org/content/mimic-iv-ext-cds/1.0.2/. HealthBench Hard is publicly available from OpenAI at https://github.com/openai/simple-evals.