Multi-Agent collaboration as a complementary architecture for AI-generated medical examination items

Abstract

Qian et al. showed single LLMs can generate acceptable knowledge-based questions but struggle with higher-order reasoning. We argue this is architectural: decomposing item development into specialized agents for drafting, critique, and iterative adversarial refinement improves quality. In blinded evaluation for China’s National Medical Licensing Examination, multi-agent outputs received 57.7% of expert preferences, compared with 42.3% for the single-model baseline, indicating a viable path to surpass current limits in AI-assisted item generation.

We read with interest the study by Qian et al.¹, which evaluated DeepSeek’s capacity to generate in-training examination questions for radiology resident education and concluded that single-model AI holds defined boundaries of applicability, particularly for items requiring higher-order reasoning. We wish to offer a complementary perspective suggesting that these boundaries may be substantially extended through architectural innovation.

Qian et al. observed what might be described as a performance boundary: while their LLM demonstrated parity with human experts in fact-based recall items (A1-type questions), it significantly underperformed in higher-order clinical reasoning questions (A3/A4-type questions). Specifically, they traced these deficiencies to the AI’s lack of genuine clinical experience and tacit knowledge, which causes generated scenarios to appear superficial, lacking narrative depth and authentic decisional tension. Our own experience developing items aligned with China’s National Medical Licensing Examination standards revealed the same pattern. We interpret these converging findings as pointing to a structural explanation: the deficiencies are not random model failures, but predictable consequences of a paradigm that asks a single generative pass to accomplish what, in human practice, requires multiple rounds of specialized review and adversarial critique.

In established high-stakes examination programs, item development is inherently a collaborative, multi-stage process. The USMLE pipeline used in the National Board of Medical Examiners^2,3 and the analogous workflow for China’s National Medical Licensing Examination share a common architecture: subject-matter experts draft items according to blueprints and item-writing guidelines, independent reviewers evaluate content accuracy and clinical plausibility from diverse perspectives, and editors mediate conflicting opinions through iterative revision before final approval. Each stage serves a distinct quality control function that the others cannot replicate. It is this division of cognitive labor, rather than the expertise of any single individual, that underpins the reliability of the overall process.

A Multi-Agent Architecture Mirroring Established Item Development Workflows

We developed an open-source multi-agent framework, MAID (Multi-Agent Item Development), that translates this collaborative workflow into a system of specialized, interacting AI agents. Each agent is equipped with carefully designed prompts specifying its role, duties, domain knowledge, and evaluation criteria. The core item development workflow proceeds as follows (Fig. 1). The architecture consists of one Author Agent, three Reviewer Agents, one Editor Agent, and optional multimodal Image Generation and Image Verification Agents. The Author Agent is responsible for generating an initial examination item according to blueprint specifications and item-writing guidelines. Reviewer Agent 1 evaluates factual and scientific accuracy; Reviewer Agent 2 evaluates clinical plausibility and reasoning quality; Reviewer Agent 3 evaluates item-writing quality, including distractor construction and adherence to assessment principles. The Editor Agent receives all reviewer feedback, identifies points of agreement and disagreement, synthesizes a consolidated revision directive, and mediates the final revision. The revised item is then either approved or returned for further iteration. The interaction mechanism explicitly mirrors human item-development practice: generation → independent review → editorial synthesis → revision → re-evaluation. Rather than relying on a single generation pass, quality control is decomposed into specialized stages, allowing distinct failure modes to be detected and corrected through adversarial review. Due to space limitations, complete prompts and implementation details are available in the MAID GitHub repository.

Fig. 1. The MAID multi-agent item development framework.

a Role architecture. Specialized agents assume distinct functions: an Author Agent drafts items, three independent Reviewer Agents provide adversarial critique from complementary evaluation perspectives, and an Editor Agent synthesizes feedback and mediates iterative refinement. Dedicated Image Generation and Verification Agents handle multimodal content. Multiple LLMs are randomly assigned to agent roles to mitigate systematic bias. b Collaborative workflow. An initial draft undergoes parallel independent review by three reviewer agents, each applying role-specific evaluation criteria. The editor agent consolidates feedback, resolves disagreements, and directs revision. Approved items are forwarded to the item bank; items failing quality thresholds are returned for further iteration. A screenshot of the MAID user interface is shown below the workflow diagram.

This architecture addresses the specific deficiencies identified by Qian et al. through direct structural correspondence. Superficial clinical scenarios, noted for lacking “narrative depth and authentic decisional tension“¹, are subjected to dedicated evaluation of clinical plausibility, compelling the author agent to enrich case complexity through iterative revision. Potential shortcomings in distractor quality, suggested by the lower perceived difficulty of AI-generated items¹, receive focused attention from a reviewer agent tasked specifically with evaluating option design. Furthermore, broader concerns regarding content accuracy—which a single model may perpetuate unchecked—are exposed to adversarial scrutiny from multiple reviewer agents that independently verify information. By decomposing quality control into these specialized functions, the multi-agent framework ensures that each known failure mode is addressed by a dedicated mechanism rather than left to the implicit judgment of a single model.

A critical design principle is the decoupling of agent roles from specific language models. Multiple LLMs (including GPT-4o, Claude 3.5, DeepSeek-R1, Qwen, Gemini, and MedSeek) are randomly assigned to agent roles across tasks, ensuring that no single model’s systematic biases or knowledge gaps propagate unchecked through the pipeline.

The framework also integrates multimodal capabilities. A dedicated image generation agent autonomously determines whether visual content should accompany a given item and produces medically relevant images (e.g., radiographs, CT scans, fundus photographs) via diffusion-based or API-based methods. A separate verification agent then evaluates the semantic consistency between visual and textual content, providing an additional adversarial check. This modular, agent-mediated approach to multimodality addresses the acknowledged limitation that current AI cannot reliably generate clinically accurate images integrated with examination items¹.

Expert Validation by Professional Item Writers

To assess whether this multi-agent architecture yields measurably superior item quality, we conducted a cross-sectional double-blind paired-comparison study involving 14 experts from 7 medical disciplines (neurosurgery, gastroenterology, physiology, general surgery, hematology, cardiology, and neurology). Experts were required to (1) hold an associate senior professional title or above, (2) possess at least three years of residency-training teaching experience, and (3) have prior experience participating in national or provincial medical examination item-writing or item-review activities. Detailed expert characteristics, disciplinary distribution, professional qualifications, and years of experience are reported in Supplementary Note 1. The evaluation employed a cross-sectional, double-blind, two-alternative forced-choice design: each expert compared 25 matched pairs of MCQs, yielding 350 item-level observations.

For each item pair, experts were presented with one item generated directly by the Author Agent (single-model baseline) and one item generated through the complete MAID workflow. Both approaches received identical blueprint information, item-writing instructions, and task requirements. The only difference was whether the generated item subsequently underwent multi-agent review and revision. Thus, the baseline condition represents the Author Agent operating independently, whereas the multi-agent condition represents the same initial generation process followed by structured collaborative refinement. Additional implementation details, models, prompts and evaluation procedures are provided in Supplementary Note 2.

Multi-agent-generated items received expert preference in 57.7% of comparisons (95% CI [52.5%, 62.8%]), significantly exceeding the 42.3% preference for single-model items (95% CI [37.2%, 47.5%]; χ²(1) = 8.33, p = 0.004). A paired t-test confirmed that individual experts consistently favored multi-agent output (mean difference = 3.86, SD = 5.12, t(13) = 2.81, p = 0.015). This advantage was consistent across all seven disciplines, suggesting that the benefit of multi-agent collaboration generalizes across medical domains.

Discipline-specific analyses suggested variation in preference magnitude across specialties. Significant preferences favoring MAID were observed in some disciplines, whereas others showed positive but non-significant trends. These findings suggest the potential value of multi-agent collaboration across multiple medical domains, although broader generalization warrants further validation. Detailed discipline-specific results are provided in Suppl. Note 3.

This evaluation paradigm is worth contextualizing. Professional item writers assess items against established technical standards including construct representativeness, adherence to item-writing principles, and clinical accuracy⁴, rather than perceived difficulty or realism. Such expert-level evaluation provides a complementary lens to the resident-based assessment reported by Qian et al., strengthening the evidence from different vantage points within the assessment development lifecycle.

Implications

These findings suggest that the performance boundary observed with single-model item generation can be substantially raised through multi-agent architectural design. Nevertheless, the present evaluation was conducted using a specific sample of experts, disciplines, and examination items, which may limit the generalizability of the findings. Both Qian et al.'s evidence and ours converge on the principle that human expertise remains essential; however, multi-agent systems may shift the boundary of what AI can contribute, generating higher-order items of sufficient quality that expert review can be redirected from basic structural correction toward finer-grained clinical judgment and pedagogical alignment.

We have made the complete multi-agent framework openly available at https://github.com/zjiang4/MAID to facilitate independent replication, adaptation across medical specialties and assessment contexts, and continued empirical validation.

Author contributions

Z.J. conceived the study, designed the research framework, developed the code, conducted the study, performed data analysis, and wrote the manuscript.

Data availability

The expert evaluation data that support the findings of this study are available from the corresponding author upon reasonable request.

Code availability

The multi-agent framework developed in this study is publicly available at: [https://github.com/zjiang4/MAID]

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41746-026-03187-z.