Using a large language model artificial intelligence agent to improve the efficiency of clinical quality measure evidence evaluation: a case study

Abstract

Objectives

To evaluate the feasibility and performance of a large language model (LLM)-based artificial intelligence (AI) agent, implemented within a structured Claim–Argument–Evidence System (CAES), for supporting the review of clinical quality measure (CQM) evidence in the Centers for Medicare & Medicaid Services Consensus-Based Entity (CBE) endorsement process.

Methods

The CBE conducted a pilot study using a previously endorsed measure. CAES extracted claims and citations from a submitted diagnostic performance measure for pneumonia, automatically retrieved additional relevant evidence from PubMed abstracts and assessed the quality, confidence and agreement of evidence supporting each claim. The system’s assessments were compared with the judgement of a subject matter expert (SME).

Results

CAES completed the assessment in approximately 5 hours. The SME agreed with the CAES-assigned claim statuses for 69% of claims, was neutral for 11% and disagreed for 14%. Disagreements primarily stemmed from the need for contextual interpretation beyond abstracts.

Discussion

Manual evaluation of CQM evidence requires significant time and resources, estimated at over 2400 labour hours per review cycle, limiting efficiency and transparency.

The AI agent evaluated 64 claims and 355 claim–evidence pairs related to the pneumonia diagnosis measure. It assigned claim statuses based on evidence strength and generated justifications.

Conclusion

This pilot demonstrated the feasibility and potential of LLM-based AI agents to improve the efficiency and transparency of evidence review for CQMs. Further development is needed to incorporate additional data sources and extend applicability across the measure development lifecycle.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Evaluation of clinical quality measure (CQM) evidence is labour-intensive and vulnerable to confirmation bias, and there is limited evidence on the use of large language model (LLM)-based tools to support structured evidence review in this context.

WHAT THIS STUDY ADDS

• This study demonstrates that an LLM-based artificial intelligence (AI) agent, implemented within a Claim–Argument–Evidence System, can systematically retrieve, assess and synthesise evidence with substantial agreement with expert review, while markedly reducing review time.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• AI-assisted evidence evaluation could improve the efficiency, transparency and consistency of CQM review processes and support earlier, more iterative evidence assessment across the measure development lifecycle.

Introduction

Battelle Memorial Institute is the current Centers for Medicare & Medicaid Services (CMS) contracted consensus-based entity (CBE).¹ The CBE, in collaboration with the Partnership for Quality Measurement (PQM),² conducts clinical quality measure (CQM) reviews under the Endorsement & Maintenance (E&M) process and the Pre-Rulemaking Measure Review (PRMR) process. E&M considers whether a CQM that a developer has submitted is safe and effective for use in an accountability programme and, if so, whether the CQM should be endorsed. PRMR considers whether a CQM that CMS has proposed is fit for purpose for use in a CMS quality payment or reporting programme and, if so, whether the CQM should be recommended.

The goal of the CBE processes is to assure trustworthy CQM for use in accountability programmes. To achieve that goal, the CBE must make all of the evidence relevant to a CQM explicit and explicitly evaluate that evidence. However, identifying, assessing and summarising evidence from published literature and other sources is time and resource intensive. Assuming 40 measures per cycle (The E&M process includes two measure review cycles each year: spring and fall. Each cycle lasts 6 months from the initial measure submission through the final endorsement determination and appeal. The PRMR process begins with publication of the Measures under Consideration (MUC) list no later than 1 December, and recommendations must be posted by 1 February of the following year.), 20 provided and 100 found citations per measure, and 30 min per citation, human review would consume at least 2400 labour hours and cost over US$300 000 per cycle. The potential for artificial intelligence (AI) is to complete the same work in less than 10% of the time and resources.

Beyond efficiency, however, the CBE must guard against the potential for ‘confirmation bias’, or the tendency to process and interpret information in a manner that is consistent with existing beliefs (eg, that the measure is valid). Understandably, the evidence submitted to substantiate measure claims tends to be in support of those claims, rather than not in support of those claims.

Through the contract with CMS, the CBE conducted a pilot study (AI Pilot) to consider the potential of large language model (LLM) Artificial Intelligence Agents (AI Agents) to address the challenge of explicitly evaluating the evidence and guarding against confirmation bias. The intent of the AI Pilot was to evaluate how well the AI Agent assessed the evidence submitted to substantiate claims about CQM properties.

Gordon and Betty Moore diagnostic excellence initiative

For the AI Pilot, we selected a measure of diagnostic performance developed under the Gordon and Betty Moore Foundation Diagnostic Excellence Initiative.³ The Initiative identified clinical quality measurement as vital for creating accountability and necessary for informing quality improvement efforts. Projects were funded in cohorts to bring together diverse groups of experts working to address critical issues in diagnosis.⁴ For the AI Pilot, we collaborated with a Moore Foundation grantee at the University of Utah (Principal Investigator: Barbara Jones). The Utah team addressed the diagnostic process for pneumonia.

Methods

Claim–argument–evidence system (CAES)

The AI Agent is a component of the Battelle CAES. Beginning with the measure developer submitted documentation (The measure documentation might be submitted through the E&M Submission Tool and Repository Measure Database (STAR) or the PRMR MUC Entry/Review Information Tool (MERIT).), the CBE staff extract each instance of a claim along with the published literature evidence cited in support of that claim, including the PubMed ID (PMID) and hyperlink (see online supplemental appendix A). The CAES then automatically performs the following tasks using an AI Agent:

1. Assigns each claim a claim type: (importance, validity association (a validity-association claim is that there is an association between the person or entity response to the measure and the measure focus), validity-mechanism (a validity-mechanism claim is that there is an explicit articulation of the mechanisms (resources and response to those resources) responsible for the association) or usability).

2. For each claim, CAES retrieves from PubMed the abstract for the citations provided by the measure developer (‘provided’ evidence).

3. For each claim, CAES uses natural language processing (NLP) and a LLM to retrieve up to five additional citations/abstracts related to the claim (‘found’ evidence). The number of found evidence citations is a parameter that may be set higher or lower.

4. For each citation, including provided and found evidence, CAES evaluates the quality level of the evidence and the confidence level of the evidence (online supplemental appendix B), and generates a justification.

5. For each claim–evidence pair, including provided and found evidence, CAES evaluates whether the evidence agrees, disagrees or neither agrees nor disagrees with the claim and generates a justification.

6. For each claim, CAES assigns a claim status (table 1) based on the body of provided and found evidence, the agreement or disagreement of the claim–evidence pairs, and the quality level and confidence level of the evidence and generates a justification.

7. For each claim type, CAES determines the status of the claim type as the lowest status of the individual claims.

8. For each measure, CAES assesses the likelihood that the CQM would be endorsed (endorsable, potentially endorsable or unlikely endorsable) based on the claim type status, and generates a justification. Note that this likelihood is not a prediction but rather an argument.

Table 1. Status of a claim.

Status	Interpretation	Quality level	Confidence level
Established	Community standards are met for adding the claim to the body of evidence	High	High
Provisionally established	Standards partially met	Moderate	High
Arguably true	Standards minimally met	Moderate	Claim more likely than not
Speculative	None of the other categories
Arguably false	Standards minimally met	Moderate	Negation more likely than not
Provisionally ruled out	Standards partially met	Moderate	High
Ruled out	Community standards are met for adding the negation of the claim to the body of evidence	High	High

Source: Parkkinen et al. Evaluating evidence of mechanisms in medicine: principles and procedures. Springer International Publishing, 2018.

The CAES AI Agent completes these tasks in about 5 min per claim. For example, a measure submission with 20 claims and 120 claim–evidence pairs completes in about 100 min (1 hour 40 min). A more detailed description of the CAES is forthcoming.⁵

Pneumonia diagnosis use case

Currently, there is no gold standard for the diagnosis of pneumonia, and many conditions have similar signs and symptoms. The diagnosis of pneumonia can ‘emerge’ over the course of the episode as new information becomes available. The Utah team developed an NLP system to extract pneumonia diagnosis from the text of emergency department notes, chest imaging reports and discharge summaries.⁶ Measures were developed to identify discordances between pneumonia diagnoses found in the emergency department, chest imaging report and discharge summaries. The ‘Percent of Hospitalised Pneumonia Patients with Chest Imaging Confirmation’ is a process measure of inpatient hospitalisations that identifies the proportion of adult hospitalised patients with a discharge diagnosis of pneumonia and who received systemic or oral antimicrobials at any time during admission who received chest imaging that supported the diagnosis of pneumonia, as recommended by clinical practice guidelines. The measure was submitted to E&M in the Spring 2024 cycle and received endorsement (CBE ID 4440e).⁷

Claim extraction and assessment

Beginning with the pneumonia diagnosis measure STAR submission, the CBE staff extracted the claims and citations that were provided by the measure developer and included in PubMed into an evidence table. The verbatim text from the claim in the measure submission was sometimes edited to spell out acronyms (eg, community acquired pneumonia/CAP), to differentiate a single claim with multiple citations, and to ensure the inclusion of the measure focus and/or target population in the claim, as appropriate. The purpose of these edits was to support the AI Agent identification of additional relevant ‘found’ evidence. This evidence table is the input to CAES.

For each claim, the CAES retrieved the abstract for the provided citations and the abstracts for up to five found citations. The output export from CAES includes three worksheets (all the content of the export table is either retrieved from PubMed or generated by the AI Agent). The ‘evidence’ worksheet includes one row per claim–evidence pair. Each row contains the citation title, author list, verbatim abstract text, a summary of the abstract, the evidence source (provided, found), the study type (clinical, mechanistic), the study quality level, the study confidence level, justification for the quality/confidence level, the associated claim, the agreement between the claim and the evidence (agree, neither, disagree), the justification for the agreement, the journal title, the publication date and PubMed hyperlink, and the PMID. The ‘Claims’ worksheet includes the claim, the claim type, the claim status (table 1), the GRADE rating⁸ and the justification for the claim status. The ‘Document’ worksheet includes the measure name, the measure endorsability, the claim summary, provided evidence summary and found evidence summary.

The output export worksheets were provided to the human reviewer. The AI Pilot assessment of CAES focused largely on the plausibility of the AI-generated claim status justifications as judged by a human reviewer subject matter expert (SME) (Jones) and a member of the CBE technical staff (Geppert).

Evaluation of CAES revisions

To evaluate the capability of CAES to improve performance, the method described was repeated twice based on one minor and one major revision, both with improvements to accuracy, response quality and consistency. The minor revision (CAES V.1.0 to V.1.1) added functionality to (1) generate the input ‘Claim Type’ (Validity, Importance, Association) so the user need not evaluate it manually before submission, (2) generate ‘Aggregate Claims’ to aid evaluation of high level trends across arguments of a single type, (3) upload and leverage document level data, such as the measure description and (4) determine when evidence is not relevant and remove it from the processing pipeline, improving downstream evaluation while decreasing runtime and cost. To support this, there were also significant improvements to the LLM prompt design, further improving performance. The major revision (CAES V.1.1 to V.2.0) overhauled the user input system by implementing a preprocessing system to automatically extract claims, arguments and evidence directly from the natural language submission and bibliography. Therefore, while the claims overall share the same concepts, the specific phrasing of claims changes considerably between CAES V.1.1 and V.2.0. All versions used OpenAI’s GPT-4o as the underlying LLM for claim extraction/classification, evidence retrieval support and evidence evaluation.

Ethical considerations

This project was conducted as a quality improvement initiative under the auspices of the Centers for Medicare & Medicaid Services (CMS) CBE contract and was determined not to involve human subjects research. The analysis used publicly available data from PubMed and deidentified measure submission documents; therefore, Institutional Review Board (IRB) review was not required. No patient-level data were used in this study. All data handling adhered to applicable privacy, security and confidentiality standards.

Results

The CBE staff extracted 64 claims and citations. Of the 64 claims, there were 32 (50%) importance claims, 13 (20%) validity-association claims, 11 (17%) validity-mechanism claims and 8 (13%) usability claims. That distribution is typical for E&M measure submissions, where the evidence tends to focus on the association between the measure focus and a material outcome (importance). For these 64 claims, the CAES found up to five additional related citations for a total of 355 claim–evidence pairs (not every claim had five found citations; otherwise, there was no missing data).

Overall, the CAES assessed the measure as ‘potentially endorsable’ (table 2). For the provided evidence, the summary focused on potential issues for some claims with the confidence level of the evidence (eg, robustness or limited data). For the found evidence, the summary focused on potential issues for some claims with the quality level of the evidence (eg, specific benefits of certain diagnostic tools or treatments). For the claim status, the CAES assessed six (9.4%) claims as ‘established’, 33 (51.6%) claims as ‘provisionally established’, 17 (26.6%) claims as ‘arguably true’, 7 (10.9%) claims as ‘speculative’ and 1 (1.6%) claim as ‘arguably false’. Table 3 contains the distribution of claim status by claim type.

Table 2. Measure status for CBE 2024-4440E.

Claim summary	Provided evidence summary	Found evidence summary
The document presents various claims regarding the diagnosis, treatment and management of pneumonia, emphasising the importance of accurate diagnosis, appropriate antibiotic use and the integration of clinical guidelines into practice. It also discusses the impact of socioeconomic factors and racial disparities on pneumonia outcomes.	The provided evidence includes clinical studies, systematic reviews and expert opinions that support the claims made in the document. However, some evidence lacks robustness or is based on limited data, which affects the overall confidence in these claims.	The found evidence through systematic searches largely supports the claims made in the document, particularly regarding the effectiveness of clinical decision support systems, the role of imaging in pneumonia diagnosis and the impact of socioeconomic factors on outcomes. However, some claims, such as the specific benefits of certain diagnostic tools or treatments, are supported by moderate-quality evidence that requires further research.

Source: Claim–Argument–Evidence System.

CBE, consensus-based entity.

Table 3. Claim status by claim type.

Claim status	Number of claims	Claim status	Number of claims
Importance	32	Validity-association	13
Established	5	Established	1
Provisionally established	12	Provisionally established	6
Arguably true	10	Arguably true	5
Speculative	4	Speculative	1
Arguably false	1	Arguably false	0
Validity-mechanism	11	Usability	8
Established	0	Established	0
Provisionally established	10	Provisionally established	5
Arguably true	1	Arguably true	1
Speculative	0	Speculative	2
Arguably false	0	Arguably false	0

Source: Claim–Argument–Evidence System.

For the claim–evidence pairs, CAES (V.1.0) assessed the quality level and confidence level. For quality level, CAES assessed 3 (0.8%) pairs as ‘high’, 300 (84.5%) pairs as ‘moderate’, 42 (11.8%) pairs as ‘low’ and 9 (2.5%) pairs as ‘very low’. For confidence level, CAES (V.1.0) assessed 3 (0.8%) pairs as ‘high’, 308 (86.8%) pairs as ‘more likely than not’ and 43 (12.1%) pairs as ‘low’. In contrast, for quality level, CAES (V.2.0) assessed 28 (18.3%) pairs as ‘high’, 106 (69.3%) pairs as ‘moderate’, 5 (3.3%) pairs as ‘low’ and 14 (9.2%) pairs as ‘very low’. For confidence level, CAES (V.2.0) assessed 29 (19.0%) pairs as ‘high’, 101 (66.0%) pairs as ‘more likely than not’ and 23 (15.0%) pairs as ‘low’. Table 4 contains the distribution of quality level and confidence level by claim source (provided or found).

Table 4. Claim–evidence pair quality level and confidence level.

Quality level	Claim–evidence pairs			Quality level	Claim–evidence pairs
	V.1.0	V.1.1	V.2.0		V.1.0	V.1.1	V.2.0
Provided	53	64	44	Found	302	225	109
High	2	22	10	High	1	23	18
Moderate	37	35	29	Moderate	263	183	77
Low	7	0	0	Low	35	8	5
Very Low	6	7	5	Very Low	3	11	9

Confidence Level	Claim–evidence pairs			Confidence Level	Claim–evidence pairs
	V.1.0	V.1.1	V.2.0		V.1.0	V.1.1	V.2.0
Provided	53	64	44	Found	302	225	109
High	2	26	10	High	1	47	19
More likely than not	38	31	29	More likely than not	270	153	72
Low	12	7	5	Low	31	25	18

Source: Claim–Argument–Evidence System.

Finally, the CAES assessed the claim–evidence agreement level. CAES (V.1.0) assessed 177 (49.9%) pairs as ‘agree’, 165 (46.5%) pairs as ‘neither agree nor disagree’ and 13 (3.7%) pairs as ‘disagree’. In contrast, CAES (V.2.0) assessed 114 (74.5%) pairs as ‘agree’, 24 (15.7%) pairs as ‘neither agree nor disagree’ and 15 (9.8%) pairs as ‘disagree’. Notably, the per cent ‘agree’ was similar for the provided and found evidence (73% vs 75%). Table 5 contains the distribution of the agreement level by claim source (provided or found).

Table 5. Claim–evidence pair agreement level.

Agreement level	Claim–evidence pairs			Agreement level	Claim–evidence pairs
	V.1.0	V.1.1	V.2.0		V.1.0	V.1.1	V.2.0
Provided	53	64	44	Found	302	225	109
Agree	31	53	32	Agree	146	177	82
Neither	21	8	12	Neither	144	24	12
Disagree	1	3	0	Disagree	12	24	15

Source: Claim–Argument–Evidence System.

The human reviewer SME and CBE technical staff agreed with the CAES claim status for 44 (69%) claims, neutral for 7 (11%) claims, disagreed for 9 (14%) claims and did not express an opinion for 4 (6%) claims. Specific areas of disagreement related to the prevalence of CAP diagnoses without imaging, the causal link between pneumonia and sepsis, the relative prevalence of CAP and hospital-acquired pneumonia in terms of hospitalisations and discordance with direct quotes from a single source.

Discussion

The AI Pilot demonstrated that a LLM AI Agent was feasible to implement and efficiently provided assessments of measure developer provided claims that were largely accurate and deemed useful to a SME. In addition, the results improved through revisions that identified and removed not relevant evidence, and that improved the LLM prompt design. Areas for further anticipated improvement are mentioned below. We are not aware of any comparable prior studies on LLM AI Agents used in evidence reviews for CQM.

The intent of the AI Pilot was not to evaluate the pneumonia diagnosis measure per se but rather to demonstrate the potential for AI to support human review of evidence for measures submitted for endorsement or proposed for federal programmes. Identifying, assessing and summarising over 350 abstracts would require weeks or months of work for a human reviewer. CBE staff and an AI Agent completed the task in a little over 5 hours with a promising degree of accuracy (almost 70% agreement with a SME/CBE reviewers). As discussed below under limitations, the areas of SME/CBE disagreement had more to do with how the CAES might be improved than with any disagreement of justification. Although a use case for CAES may certainly be to support the efficiency and effectiveness of E&M or PRMR CBE staff and committee reviews, the greatest benefit might be to support the entire measure lifecycle (conceptualisation, specification, testing and implementation and use). Assessing evidence early in the life cycle would accelerate and guide evidence generation activities to focus on areas with the greatest uncertainty. Such support could reduce the time and resources required for measure development on measure developers (a rule of thumb is that measures take 18 months–2 years to develop and cost US$750 000 per measure).

Limitations

A limitation of this case study is the evaluation of a single use case with two human reviewers, although the AI Pilot did include additional measures and reviewers not reported here. As the AI Pilot continues, the CBE will collect similar feedback on the performance of CAES to monitor performance and to inform future functional improvements.

The AI Pilot identified several opportunities for functional improvements for CAES. CAES currently only uses abstracts rather than full text of articles in part because full text is not always publicly available. However, some of the human reviewer SME claim status disagreements were because the claim was based on secondary outcomes or populations that were contained in the full text and not in the abstract. Similarly, CAES currently only uses PubMed rather than other bibliographic sources (eg, Embase, CINAHL). Adding the capability to retrieve full text of articles through subscriptions from multiple bibliographic sources would enhance the usefulness of the assessment.

CAES currently does not distinguish between empirical studies and other types of published literature. Retrieving ‘found’ evidence may be less useful for some types of claims. For example, assessing a claim that is a direct quote from a guideline or from a review article may only require agreement with the ‘provided’ evidence. Also, in searching for relevant ‘found’ evidence, CAES currently does not give more weight to more recent evidence or apply any type of evidence hierarchy. Both of these considerations might be user options in future versions of the CAES.

Sometimes claims may be based on a synthesis of evidence across several studies. CAES does have the functionality to conduct such meta-analyses using an LLM and the PICO framework⁹ to extract quantitative estimates and uncertainty of those estimates from published sources, and then to apply a statistical model (fixed-effect ANOVA) to provide an aggregate estimate and uncertainty. We will report those results in a separate paper.

Finally, CAES currently only considers published evidence and does not consider expert evidence, unpublished evidence, guidelines, systematic reviews or empirical evidence from implementation. In that sense, the measure endorsement status is only provisional, and CBE staff and committee reviewers must consider all of the evidence to explicitly evaluate the evidence and guard against confirmation bias. AI Agents are able to assess and summarise such evidence, however, and future versions of CAES will have this functionality.

CAES is currently a proprietary internal system developed by Battelle for use under the CMS CBE contract and is not presently available as open-source software. However, CAES output tables can be made available to PQM members at no cost on request. Accordingly, the generalisability of this work pertains primarily to the CAES approach and workflow rather than to an openly deployable software implementation.

Data availability statement

No data are available.