Skip to main content
JAMIA Open logoLink to JAMIA Open
. 2026 Jul 30;9(4):ooag152. doi: 10.1093/jamiaopen/ooag152

Large language model-based extraction of rheumatoid arthritis clinical disease activity index from clinical notes

Reid Weisberg 1, Ruoqi Yang 2, Iram Kamdar 3, Justin Chestang 4, Joan Bathon 5, Jon T Giles 6, Chunhua Weng 7, Elizabeth Park 8,✉
PMCID: PMC13424428  PMID: 42539747

Abstract

Objectives

We extracted a validated disease activity measure in rheumatoid arthritis (RA), the Clinical Disease Activity Index (CDAI), from a large tertiary academic medical center electronic health record (EHR) using an automated large language model (LLM)-based approach without requiring model pretraining.

Materials and Methods

The New York Presbyterian/Columbia University Medical Center Clinical Data Warehouse contains EHR data for over 4.5 million patients. RA patients were identified using International Classification of Disease-9 (ICD-9) and ICD-10 codes. Expert-curated CDAI keywords were extracted from unstructured notes using an automated natural language processing (NLP) pipeline leveraging GPT-4o API, a HIPAA-compliant, institutionally approved LLM platform. Performance was evaluated against expert chart review.

Results

Among 2756 RA patients with notes, 1038 (37.7%) were seropositive, 796 (28.9%) were seronegative, and 922 (33.4%) had unknown serostatus. Clinical Disease Activity Index and its components were extracted in 15.4% (160/1038) of seropositive patients indicating remission or low disease activity. Clinical Disease Activity Index documentation was more frequent among patients with multiple notes and among faculty, with high extraction accuracy (precision/recall/F1 = 0.97).

Discussion

This represents the first attempt to employ a zero-shot, ChatGPT-powered LLM platform to extract RA disease activity measures from real-world EHR data. Although a low prevalence of documentation was noted, important distinctions were observed when patients were subgrouped by serostatus, level of training, and number of visits.

Conclusion

An LLM-based pipeline accurately extracted CDAI from a single large academic EHR, revealing infrequent real-world documentation.

Keywords: rheumatoid arthritis, electronic health records, large language models

Introduction

Treat-to-target (TTT) approaches for rheumatoid arthritis (RA) have been promulgated by professional rheumatology associations like the American College of Rheumatology (ACR) and the European League Against Rheumatism (EULAR) for over a decade.1–2 TTT requires regular patient monitoring and the use of validated disease activity metrics to evaluate disease activity at each clinic visit, toward low disease activity or remission.3 Official RA disease activity measures endorsed by the ACR include the Clinical Disease Activity Index (CDAI), simplified disease activity index (SDAI), Routine Assessment of Patient Index-3 (RAPID-3) and Disease Activity Score 28 (DAS28).4 In practice, there is likely variability in preference for the use of these measures. This may, in part, be due to inconsistencies between electronic health records (EHRs) in how disease metrics are put into the system and a lack of structured, standardized ways of entering these data depending on the site of practice, EHRs utilized, and individual clinician preference. Therefore, it is more likely that these disease activity measures are found in free-text notes generated by clinicians.

The rapid integration of generative artificial intelligence (AI) in health care research has brought large language models (LLMs) for example, ChatGPT to the forefront.5 Compared with rule-based natural language processing (NLP) systems utilized in the past, LLMs can rapidly interpret and summarize massive quantities of heterogeneous free text with high accuracy and contextual understanding, with fewer needs for prompting and training, enabling scalable extraction of complex clinical concepts from unstructured EHR data6

Examples of applications of LLMs in clinical research include enhancing diagnostic accuracy, generating treatment recommendations, triaging and care coordination, literature review, and data synthesis, to name a few.6 However, only a handful (5%) of all published studies using LLMs have used real patient data.7

Two prior publications explored the documentation of RA disease activity measures in free-text notes using traditional NLP algorithms to extract Routine Assessment of Patient Index Data 3 (RAPID-3) from the Rheumatology Informatics System for Effectiveness (RISE) registry,8 and DAS28 from the Veterans Administration EHR system.9 These methods required extensive pre-training and validation with clinical data, which is a potential downside of NLP. CDAI is considered to be a valid and equally important RA disease activity metric alongside DAS28, correlating well with physician opinion and being predictive of change in therapy.10–13 Additionally, because of the absence of the need for laboratory measures, CDAI is theoretically more readily usable in clinical practice.

Objective

Thus, we employed a university approved, HIPAA-compliant GPT-4o API14 to automatically extract CDAI and related parameters from a large batch of clinical notes drawn from a large, tertiary academic center inflammatory arthritis clinic.

We add novelty to the few existing studies on this topic by: (1) focusing on CDAI extraction, which has been previously unreported in the literature, (2) using GPT-4o enabled LLMs to perform zero-shot automated extraction without extensive pretraining with prior data, and (3) examining documentation rates from a diverse, heterogeneous academic EHR representative of real-world practice.

Methods

Data source and cohort construction

Automated extraction of disease activity measures from EHR

The study was approved by the Columbia University Irving Medical Center (CUIMC) Institutional Review Board protocol #AAAU6757.

This study met institutional criteria for a full waiver of consent based on the following factors: (1) no direct diagnostic or therapeutic interventions were performed given that this was an EHR-based observational study, (2) obtaining individual consent across potentially thousands of patients would be impractical and unfeasible, and (3) this full waiver would not adversely affect the rights or welfare of participants. The overall risk was minimal, with potential for loss of confidentiality of participants. However, extensive steps were taken to maintain confidentiality of data. All patient data was assigned a code number, separated from their name or other identifiable information. All electronic study data was stored in certified, secure, Health Insurance Portability and Accountability Act (HIPAA)-compliant environments, including the CUIMC-IT certified server, as well as password-protected computers. Prior to uploading data into Columbia ChatGPT Education and associated LLMs, NLP, and/or Machine Learning (ML) packages, 18 HIPAA identifiers described according to national standards (https://www.dhcs.ca.gov/dataandstats/data/Pages/ListofHIPAAIdentifiers.aspx) were manually deidentified within the CUIMC-IT certified server. All PHI data and associated outputs were retained within and not used for model training outside of this secured and approved environment.

Figure 1 outlines the LLM based extraction from The New York Presbyterian/Columbia University Irving Medical Center Clinical Data Warehouse (NYP/CUIMC CDW), containing over 4.5 million patients’ data stored in structured and unstructured formats since 2000.

Figure 1.

Data Extraction Process.

Data extraction from NYP/CUIMC EHR and CDW.

Clinical notes were de-identified, converted from HTML to plain text using an automated NLP pipeline, and processed under HIPAA-compliant, (CUIMC) IT–approved protocols. Notes with missing or empty text were excluded, patient identifiers were standardized, note dates were parsed into a uniform format, and notes were sorted chronologically within each patient. Text was normalized by collapsing repeated whitespace, line breaks, and tab characters introduced during EHR export. For each patient, cleaned notes were concatenated into a longitudinal document while preserving temporal order to enable extraction of disease activity measures across visits. These preprocessing steps were performed solely for data cleaning, standardization, and quality control and thus did not assign CDAI values, joint counts, serostatus, or disease activity categories.

A standardized prompt template was then applied to each patient-level document, with only the patient-specific clinical text varying between requests. GPT-4o (OpenAI API) was used to extract structured RA disease activity measures, including CDAI, DAS28, tender and swollen joint counts (SJCs), patient and physician global assessments, disease activity descriptors, and rheumatoid factor (RF)/anti-cyclic citrullinated peptide (anti-CCP) serostatus (Figure S1). Prompt development was iterative, with multiple templates evaluated on a pilot set of rheumatology notes to optimize extraction of numeric disease activity measures and corresponding clinical labels. The final prompt explicitly defined each variable and instructed the model to extract only documented information without inference.

To maximize reproducibility, the model was configured for deterministic extraction (temperature = 0; all other decoding parameters at default settings). Responses were required to conform to a predefined JavaScript Object Notation (JSON) schema. If output failed schema validation, it was automatically resubmitted for reformatting without modifying the extracted content. JSON outputs were compiled as JSONL files while preserving source metadata for auditing. Deterministic post-processing parsed the JSON, flattened longitudinal disease activity timelines into tabular records, standardized dates, and sorted observations by patient and encounter date. The initial JSON formatting failure rate was approximately 3%. Most errors were corrected through automated JSON repair and schema validation, with fewer than 1% of outputs remaining unparsable and excluded from downstream analyses. A complete overview of the preprocessing, extraction, and post-processing workflow is shown in Figure 1.

Components of CDAI [SJC, tender joint count (TJC), patient global assessment (PGA), evaluator global assessment (EGA)], and serostatus (RF and anti-CCP antibodies (eg, RF+ (<15 IU/mL), CCP+ (<19 U/mL), seronegative), as well as all the phrase variations were assessed (Table 1).

Table 1.

CDAI/serostatus terminology extraction.

CDAI components CDAI score phrasing Serostatus terms
  • Swollen joint count: Swollen; SJC: S

  • Tender joint count: Tender; TJC; T

  • “CDAI is…”

  • “CDAI was…”

  • “CDAI was … today is..”

  • “CDAI went from … to..”

  • Seropositive; Sero+ Rheumatoid factor (RF) positive; RF+; +RF; RF++; ++RF

  • RF “Number”

Patient global assessment score: Patient global assessment of disease activity (PGA); Pt global score; Pt score; Pt
  • “CDAI is <number”

  • “CDAI is >number”

  • “CDAI is number-number”

  • Seropositive; Sero+ 

  • Anti-citrullinated protein antibodies (CCP) positive;

  • CCP+; CCP++; +CCP; ++CCP

  • CCP “Number”

  • Physician global assessment score:

  • Physician global score; physician score; evaluator global assessment of disease activity (EGA); provider score (Pro); Pr

  • CDAI:

  • Pt/PGA: “Number”; Pr/Pro/EGA: “Number”; SJC/S: “Number”; TJC/T: “Number” = Total number

  • Seronegative; Sero−; 

  • RF negative; negative RF; RF−; −RF

  • CCP negative; negative CCP; CCP−; −CCP

Each note was deconstructed and separately analyzed for the composite score as well as every component of the CDAI: SJC, TJC, PGA, EGA. CDAI components were computed automatically into a composite CDAI score if they were not already summarized in the note.

RA case identification

All patients who were coded with RA using the International Classification of Disease (ICD) 9 or 10 (at least 1 code; Table S1) in the context of a rheumatology clinic visit and had at least 1 outpatient rheumatology clinic visit underwent note extraction. Previous literature demonstrated a positive predictive value (PPV) of 88% for a single ICD 9 or 10 code for RA.15 Our own internal chart review of 100 random patients demonstrated a PPV of 82% for a single ICD 9 or 10 code for RA, against a gold standard of expert review of the diagnosis from charts.

RA serologic validation

RA patients were defined as seropositive or seronegative if these phrases were explicitly mentioned and extracted from the notes. Serostatus unknown meant that neither seropositive nor seronegative phrases could be initially extracted from the notes.

Approximately 10% of all RA patients with LLM extracted serostatus were randomly selected without replacement and their documented serologic status was verified by linkage with available laboratory data by the authors (E.P., J.C., and J.B.) who were blinded to the LLM output. Precision, recall, and F1 scores were calculated accordingly. Cohen’s κ was not calculated as each author reviewed a unique batch of patients without overlap.

For the “serostatus unknown” patients, key diagnostic terminology that may suggest RA or alternate diagnoses were extracted (including ankylosing spondylitis, psoriatic arthritis, gout, and juvenile idiopathic arthritis) and summarized accordingly, but these terms were not integrated into CDAI analyses. For the subset identified as RA from the LLM extraction, available serologies were manually reviewed by authors and their data were summarized accordingly and their CDAI scores were then re-integrated into analyses.

CDAI validation

Approximately 10% of all seropositive and seronegative RA patients with LLM-extracted CDAI were randomly chosen without replacement and their scores were reviewed by a trained rheumatologist (E.P.) who performed correlation with manual chart review, while blinded to LLM outcomes. Precision, recall, and F1 scores were calculated accordingly. Additionally, we randomly selected 30 patients (roughly representing 10% of available CDAI extractions) whose CDAI scores had been automatically computed from components and reviewed the calculations for accuracy.

Statistical analysis

Seropositive and seronegative RA subgroups were compared using descriptive analyses. Frequencies and medians were calculated for demographics, serologies, medication classes, and CDAI score categories (remission, low, moderate, and high activity). CDAI scores were calculated by GPT-4o if only components (eg, TJC) were present but scores were not directly mentioned in free text. We further stratified data by (1) trainee vs faculty author (fellows vs faculty) and (2) documentation frequency (single-note vs multi-note defined as >1). Continuous variables were compared using 2-sample t-tests, and categorical variables—including prevalence (percentage with the outcome)—were compared between groups using Pearson’s χ2 test or Fisher’s exact test when expected cell counts were <5 (2-sided α = .05).

Results

A total of 2996 RA patients were identified based on ICD codes and clinic visits. Of this group, 240 were excluded as they did not have associated notes. The remaining 2756 patients with RA with 14 285 associated notes were extracted. Of the 2756 patients, 1038 (37.7%) were seropositive, 796 (28.9%) were seronegative. After the RA group (n = 294) was separated out from the serostatus unknown group (n = 1216) by employing the same LLM pipeline on the notes, remaining 922 patients were documented as having ankylosing spondylitis (n = 467), other/unspecified conditions (n = 368), psoriatic arthritis (n = 73), spondyloarthropathy (n = 17), gout (n = 7), and juvenile idiopathic arthritis (n = 7), with some of these categories overlapping with each other. These non-RA patients were not ultimately included in the CDAI analyses. The RA subgroup (n = 294) underwent chart mining for serologies and n = 102 were found to be seropositive and n = 109 found to be seronegative, and a small portion (n = 83) did not have serologies available in EHR and were ultimately omitted from analyses. Their respective CDAI scores were re-integrated into analyses. The respective breakdown of all patients and analytic work flow is summarized in Figure 2.

Figure 2.

RA patients.

RA patient breakdown.

Table 2 summarizes the characteristics of the seropositive and seronegative RA patients. The majority of seropositive patients had RF or CCP positive (or both), and almost all of the seronegative patients had a negative RF and CCP. Serologic validation revealed a precision of 0.98, recall 1.0, F1 score of 0.99 for seropositives and precision of 0.90, recall 1.0, F1 score of 0.95 for seronegatives, respectively (Table S2a and 2b).

Table 2.

Seropositive vs seronegative RA clinical characteristics.

Total RA (n = 1834)c Seropositive % (n = 1038) Seronegative % (n = 796) P-value a
RA characteristics b
RF positive 71.2% (739) 0.4% (3) <.001
CCP positive 51.6% (536) 0.0% (0) <.001
RF and CCP positive 39.0% (405) 0.0% (0) <.001
RF and CCP negative 0.0% (0) 66.8% (532) <.001
Erosive (eg, erosions) 49.8% (517) 30.7% (244) <.001
Destructive 0.1% (1) 0.0% (0) N/A
Nodular (eg, nodules) 14.3% (148) 8.5% (68) <.001
Age (median years) 63.0 59.0 N/A
Sex (female) 84.2% (874) 82.2% (654) .246
RA disease duration (median years) 8.0 7.0 N/A
DMARD use a
csDMARDs (methotrexate, leflunomide, sulfasalazine, or hydroxychloroquine) with or without other DMARDs 77.2% (801) 65.6% (522) <.001
bDMARDs (abatacept, anakinra, adalimumab, certolizumab, etanercept, golimumab, infliximab, rituximab, or tocilizumab) with or without other DMARDs 27.1% (281) 17.0% (135) <.001
tsDMARDs (tofacitinib or upadacitinib) with or without other DMARDs 7.8% (81) 7.5% (60) .832
Combination therapy:
csDMARD + bsDMARD 21.2% (220) 12.3% (98) <.001
csDMARD + tsDMARD 6.3% (65) 6.3% (50) .986
Prednisone or methylprednisolone 39.5% (410) 37.4% (298) .369
a

Pearson’s Chi-squared test used for comparing categorical variables; Fisher’s exact test if < 5 observations in cell.

b

Some proportions may overlap.

c

All variables reflect extractions done from most recent note to year 2024 (current notes).

Both groups were predominantly female, middle aged, with long-standing disease. Compared to seronegative RA, seropositive RA patients had more erosions and nodules (as written in free text). Compared with seronegative RA, a higher proportion of seropositive RA patients were currently using conventional synthetic disease modifying anti-rheumatic drugs (csDMARDs), biologic DMARDs (bDMARDs), and combination therapy, at the time of note documentation.

Table 3 shows the recorded CDAI (composite and by individual category) and DAS28, as well as score ranges (remission, low/moderate/high disease activity) divided by serostatus. Of the seropositive patients, only 15.4% (n = 160) had CDAI recorded, compared with 9.9% (n = 79) for seronegative RA, and this difference was statistically significant (P ≤ .001). All individual components (SJC, TJC, PGA, EGA) were documented more frequently in seropositives compared to seronegative patients (P < .001). A statistically higher proportion of seropositive patients were documented with low, moderate and high CDAI categories compared to seronegative patients. DAS28 was sparsely documented in both seropositive and seronegative RA, in less than 2% of both groups. For CDAI score validation, a precision of 0.97, recall of 0.97, and F1 score of 0.97 were calculated, suggesting high accuracy of this LLM pipeline (Table S2c). Additionally, all 30 of the CDAI scores which were computed from components were noted to be accurate in summation (data available upon request). When stratified by faculty or fellow status (Table S3), among seropositive and seronegative RA, a higher proportion of faculty had CDAI recorded compared to fellows (P = .0034 and P = .0499, respectively). Each component of CDAI (SJC, TJC, PGA, and EGA) was documented with significantly higher frequency in faculty compared to fellows among seropositive patients. A higher proportion of fellows’ patients were in remission compared to faculty (P = .0049). When stratified by single or multi note status (Table S4), a higher proportion of patients with multiple notes had CDAI recorded in both seropositives and seronegatives (P < .001). Each component of CDAI (SJC, TJC, PGA, and EGA) was documented with significantly higher frequency in multi-note patients than in single-note seropositive patients. A higher proportion of multi-note seropositive and seronegative patients were in remission compared to single-note.

Table 3.

RA disease activity measures (composite group).

Total RA (n = 1834)c Seropositive % (n = 1038) Seronegative % (n = 796) P-value a
CDAI score complete 15.4% (160) 9.9% (79) <.001
CDAI score components b
SJC 14.3% (148) 5.8% (46) <.001
TJC 11.6% (120) 4.3% (34) <.001
PGA 8.5% (88) 4.1% (33) <.001
EGA 7.8% (81) 3.4% (27) <.001
CDAI score ranges
CDAI remission 4.1% (43) 4.8% (38) .5143
CDAI low 7.4% (77) 4.1% (33) .0034
CDAI moderate 2.0% (21) 0.8% (6) .0253
CDAI high 1.8% (19) 0.3% (2) .0013
DAS28 1.5% (16) 0.4% (3) .018
a

Pearson’s chi-squared test used for comparing categorical variables; Fisher’s exact test if < 5 observations in cell.

b

Some patients may be counted more than once if they have >1 component documented.

c

All variables reflect extractions done from most recent note to year 2024.

Discussion

In this paper, we demonstrated the feasibility and accuracy of using a university medical center approved, HIPAA-compliant ChatGPT-4o-enabled NLP/LLM pipeline14 to extract CDAI scores of RA patients at our academic institution. To our knowledge, this is the first study to extract RA disease activity measures using a zero-shot ChatGPT-based approach. We ultimately found that CDAI or its components were documented in only a minority (15%) of the RA patients whose notes were extracted, which is in concordance with at least 1 prior work that also demonstrated relatively low prevalence of CDAI documentation (36% of the whole cohort).8 This is strikingly low given that CDAI is a relatively user-friendly point of care documentation tool, as it does not require any laboratory metrics or complex calculations. Moreover, in our institution, CDAI documentation is theoretically enabled by the use of a digital homunculus integrated into the EHR system. However, as we could not directly extract information from the homunculus, we had to assume that our providers are opting to free text this information instead, though this exact proportion could not be ascertained. DAS28 was even more poorly documented (<2%), which speaks to its limited applicability outside of clinical trials and cohort studies. Although a statistically higher percentage of seropositive RA patients compared with seronegative RA had CDAI and its components documented, this difference could arise from several confounding influences that we could not take into account, such as severity of disease (with seropositives likely having more severe disease). Nonetheless, this finding is interesting and underscores potential clinical differences between seropositive and seronegative RA, the latter of which may encompass other inflammatory arthritis diagnoses, such as spondyloarthropathies.16

Faculty members were more likely to document CDAI elements than fellows (in seropositive patients), highlighting a potential generational difference in documenting RA disease activity. Additionally, documentation was more likely in patients with multiple notes vs single note which underscores the value of continuous care in terms of incentivizing and optimizing disease documentation. However, it is possible there were differences in RA disease and co-morbidity severity between the two groups, as well as EHR interface shifts (such as the transition to EPIC in 2020 and beyond) that may affect documentation. Because we did not specifically subgroup, stratify, or factor in these potential confounders, these differences should be interpreted with caution.

For assessing accuracy, 10% of each subcohort (seropositive and seronegative) for serostatus and CDAI scores were manually chart reviewed, with high precision scores. However, we could not reliably and consistently calculate inter-rater reliability (eg, Cohen’s κ) due to only one author validating CDAI scores and 3 authors separately reviewing seropositive and seronegative serostatus datasets. Therefore, these precision scores should be interpreted in light of these limitations. In assessing precision and accuracy, we followed conventional evaluation metrics for LLMs, which have largely centered around accuracy and comprehensiveness.17–19 Other important dimensions such as fairness, bias, and deployment have not yet been standardized in LLM evaluations, which represents a future direction for research. Interestingly, in almost a quarter of patients (294/1216) whose serostatus could originally not be automatically extracted (and yet were ICD coded with RA), application of the same LLM pipeline extracted a diagnosis of seropositive and seronegative RA, which highlights the discordant and erroneous nature of individual clinician ICD coding.

Our study’s unique strengths include: (1) using ChatGPT-powered LLMs- with limited prompting and no prior training (ie, zero-shot)- to perform RA outcome extraction from an academic EHR representative of real-world practice, (2) though initially ICD code generated, our RA cohort definition was refined by using an LLM to extract serostatus with manual verification, an approach that is known to increase accuracy of RA detection from EHRs,15 and (3) stratifying analyses by serostatus, fellow vs faculty visit, and number of visits delineated important differences in documentation among these groups.

The limitations include: (1) Non-generalizability: since extraction was performed in a single institution’s EHR, results are not generalizable to other academic practices with different patient or provider populations and relative portability of this LLM approach has yet to be validated; (2) Missing data: up to 240 patients had missing notes and for 83 patients serostatus could not be extracted at all. Additionally, due to incomplete linkage with laboratory data, not all patients designated as seropositive or seronegative (via LLM extraction) actually had serologies available for confirmation, though the proportion we randomly confirmed (∼10%) were highly accurate. These missing data may stem from failure to establish initial care at our institution (hence no notes in system), some providers not documenting serostatus in free-text notes, or significant lab data loss from migration during EHR system transitions; (3) Demographics (age, sex) and other RA characteristics including erosions, nodules, disease duration, and DMARD use were not systematically linked to EHR data for verification, as the primary focus of this manuscript was demonstrating the prevalence of disease activity measures, though these characteristics will be fully explored and validated in future manuscripts; and (4) Other conventional RA disease activity measures eg, RAPID-3, SDAI, were not extracted and analyzed. All of these gaps represent key improvement steps for further refining this EHR-based RA cohort for application of other LLM-based methodologies in the future.

Conclusion

Despite the explosive onset of LLM-fueled approaches for data generation, extraction, and analyses across all specialties, only 5% of recently published studies7 have utilized real clinical data. To our knowledge, this study represents one of the first successful attempts to employ LLMs to extract real-world clinical outcomes in RA, which could be expanded to many other complex disease concepts and phenotypes. The relatively low frequency of CDAI documentation highlights an opportunity for institution-based EHR systems and professional societies to collaborate on more user-friendly and clinically deployable RA disease activity measures.

Supplementary Material

ooag152_Supplementary_Data

Contributor Information

Reid Weisberg, Division of Rheumatology, Department of Medicine, UT Southwestern Medical Center, Dallas, TX 10032, United States.

Ruoqi Yang, Data Science Institute, Columbia University, New York, NY 10027, United States.

Iram Kamdar, Data Science Institute, Columbia University, New York, NY 10027, United States.

Justin Chestang, Columbia University Vagelos College of Physicians & Surgeons, New York, NY 10032, United States.

Joan Bathon, Division of Rheumatology, Department of Medicine, Columbia University Medical Center, New York, NY 10032, United States.

Jon T Giles, Division of Rheumatology, Kao Autoimmunity Institute, Cedars-Sinai Medical Center, Los Angeles, CA 90048, United States.

Chunhua Weng, Department of Biomedical Informatics, Columbia University, New York, NY 10032, United States.

Elizabeth Park, Division of Rheumatology, Department of Medicine, Columbia University Medical Center, New York, NY 10032, United States.

Author contributions

Reid Weisberg (Writing—original draft, Writing—review & editing), Ruoqi Yang (Data curation, Formal analysis, Methodology, Software, Visualization), Iram Kamdar (Data curation, Formal analysis, Software), Justin Chestang (Data curation, Formal analysis), Joan Bathon (Writing—review & editing), Jon T. Giles (Writing—review & editing), Chunhua Weng (Supervision, Writing—review & editing), and Elizabeth Park (Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing—original draft, Writing—review & editing)

Supplementary material

Supplementary material is available at JAMIA Open online.

Funding

This work was supported by the Columbia Irving Institute for Clinical and Translational Research Training KL2 Mentored Career Development Award (KL2TR001874).

Conflicts of interest

There are no relevant conflicts of interest to declare on behalf of the authors.

Data availability

The data underlying this article cannot be shared publicly as they contain PHI derived from the EHR, subject to institutional privacy, security, and data-use restrictions. Deidentified aggregate data not already present in the manuscript, analytic prompts, and codes may be made available from the corresponding author upon reasonable request, subject to institutional approvals and applicable data-use agreements.

References

  • 1. Singh JA, Saag KG, Bridges SL, et al. ; American College of Rheumatology. 2015 American College of Rheumatology guideline for the treatment of rheumatoid arthritis: ACR RA treatment recommendations. Arthritis Care & Research. 2016;68:1-25. [DOI] [PubMed] [Google Scholar]
  • 2. Smolen JS, Breedveld FC, Burmester GR, et al.  Treating rheumatoid arthritis to target: 2014 update of the recommendations of an international task force. Ann Rheum Dis. 2016;75:3-15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Bakker MF, Jacobs JWG, Verstappen SMM, et al.  Tight control in the treatment of rheumatoid arthritis: efficacy and feasibility. Annals of the Rheumatic Diseases. 2007;66 (Suppl. 3):iii56-iii60. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. England BR, Tiong BK, Bergman MJ, et al.  2019 Update of the American College of Rheumatology recommended rheumatoid arthritis disease activity measures. Arthritis Care Res (Hoboken). 2019;71:1540-1555. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Webster P.  Six ways large language models are changing healthcare. Nat Med. 2023;29:2969-2971. 10.1038/s41591-023-02700-1 [DOI] [PubMed] [Google Scholar]
  • 6. Bedi S, Liu Y, Orr-Ewing L, et al.  Testing and evaluation of health care applications of large language models: a systematic review. JAMA. 2025;333:319-328. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Shah NH, Entwistle D, Pfeffer MA.  Creation and adoption of large language models in medicine. JAMA. 2023;330:866-869. [DOI] [PubMed] [Google Scholar]
  • 8. Humbert‐Droz M, Izadi Z, Schmajuk G, et al.  Development of a natural language processing system for extracting rheumatoid arthritis outcomes from clinical notes using the National Rheumatology Informatics System for Effectiveness Registry. Arthritis Care & Research. 2023;75:608-615. [DOI] [PubMed] [Google Scholar]
  • 9. Cannon GW, Rojas J, Reimold A, et al.  Extraction of rheumatoid arthritis disease activity measures from electronic health records using automated processing algorithms. ACR Open Rheumatol. 2019;1:632-639. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Smolen JS, Aletaha D.  Scores for all seasons: SDAI and CDAI. Clin Exp Rheumatol. 2014;32:75-79. [PubMed] [Google Scholar]
  • 11. Canhão H, Rodrigues AM, Gregório MJ, et al.  Common evaluations of disease activity in rheumatoid arthritis reach discordant classifications across different populations. Front Med (Lausanne). 2018;5:40. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12. Takanashi S, Kaneko Y, Takeuchi T.  SAT0093 CDAI and DAS28 in the management of rheumatoid arthritis in clinical practice. Ann Rheum Dis. 2020;79:980. [DOI] [PubMed] [Google Scholar]
  • 13. Salaffi F, Carotti M, Di Carlo M, et al.  High-resolution computed tomography of the lung in patients with rheumatoid arthritis: prevalence of interstitial lung disease involvement and determinants of abnormalities. Medicine. 2019;98:e17088. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Columbia University Information Technology (CUIT). ChatGPT Education. Columbia University. Accessed May 18, 2026. https://www.cuit.columbia.edu/content/chatgpt-education.
  • 15. Liao KP, Cai T, Gainer V, et al.  Electronic medical records for discovery research in rheumatoid arthritis. Arthritis Care Res (Hoboken). 2010;62:1120-1127. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Kimbrough BA, George RJ, Crowson CS, et al.  Long‐term outcomes in seronegative rheumatoid arthritis. Arthritis Care Res (Hoboken). 2025:acr.70011. Accessed May 18, 2026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Xie Y, Zhang S, Cheng H, et al. DOCLENS: Multi-aspect Fine-grained Evaluation for Medical Text Generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 1:649–679.
  • 18. Zhong M, Liu Y, Yin D, et al. Towards a unified multi-dimensional evaluator for text generation. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing [Internet]. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics; 2022:2023–38.
  • 19. Bommasani R, Liang P, Lee T.  Holistic evaluation of language models. Ann N Y Acad Sci. 2023;1525:140-146. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

ooag152_Supplementary_Data

Data Availability Statement

The data underlying this article cannot be shared publicly as they contain PHI derived from the EHR, subject to institutional privacy, security, and data-use restrictions. Deidentified aggregate data not already present in the manuscript, analytic prompts, and codes may be made available from the corresponding author upon reasonable request, subject to institutional approvals and applicable data-use agreements.


Articles from JAMIA Open are provided here courtesy of Oxford University Press

RESOURCES