Extracting Pain Care Quality Indicators from U.S. Veterans Health Administration Chiropractic Care Using Natural Language Processing

Abstract

Background  Musculoskeletal pain is common in the Veterans Health Administration (VHA), and there is growing national use of chiropractic services within the VHA. Rapid expansion requires scalable and autonomous solutions, such as natural language processing (NLP), to monitor care quality. Previous work has defined indicators of pain care quality that represent essential elements of guideline-concordant, comprehensive pain assessment, treatment planning, and reassessment.

Objective  Our purpose was to identify pain care quality indicators and assess patterns across different clinic visit types using NLP on VHA chiropractic clinic documentation.

Methods  Notes from ambulatory or in-hospital chiropractic care visits from October 1, 2018 to September 30, 2019 for patients in the Women Veterans Cohort Study were included in the corpus, with visits identified as consultation visits and/or evaluation and management (E&M) visits. Descriptive statistics of pain care quality indicator classes were calculated and compared across visit types.

Results  There were 11,752 patients who received any chiropractic care during FY2019, with 63,812 notes included in the corpus. Consultation notes had more than twice the total number of annotations per note (87.9) as follow-up visit notes (34.7). The mean number of total classes documented per note across the entire corpus was 9.4 (standard deviation [SD]  =  1.5). More total indicator classes were documented during consultation visits with (mean  =  14.8, SD  =  0.9) or without E&M (mean  =  13.9, SD  =  1.2) compared to follow-up visits with (mean  =  9.1, SD  =  1.4) or without E&M (mean  =  8.6, SD  =  1.5). Co-occurrence of pain care quality indicators describing pain assessment was high.

Conclusion  VHA chiropractors frequently document pain care quality indicators, identifiable using NLP, with variability across different visit types.

Background and Significance

Musculoskeletal pain is highly burdensome, carrying significant direct health care costs and indirect costs through functional impacts, disability, and lost productivity. 1 2 3 It is among the most common reasons individuals in the United States seek health care, 4 especially in the U.S. military Veteran population receiving health care in the Veterans Health Administration (VHA). 5

In 2009, VHA issued Directive 2009-053 to promote quality pain care consistent with the VHA National Pain Management Strategy. 6 The directive outlined essential components of high-quality pain care to include (1) timely and appropriate pain assessment, (2) development and enactment of a pain treatment plan, and (3) subsequent reassessment of the effectiveness of the plan. The directive also reaffirmed VHA's commitment to providing pain care consistent with the Stepped Care Model for Pain Management, 7 which promotes early self-management with stepwise progression to primary care management and increasingly advancing specialty care management when clinically appropriate. The earliest steps of the Stepped Care Model promote the use of many guideline-recommended nonpharmacologic approaches for pain, including psychological/behavioral therapies, exercise/movement therapies, and manual therapies. 8

One VHA specialty frequently utilizing many of these nonpharmacologic approaches for pain is chiropractic services. 9 The VHA is the largest integrated health care system offering chiropractic services, first as a covered service in 2000 and later as an integrated service at VHA facilities beginning in 2004. 10 Since 2018, VHA chiropractic care has undergone transformative expansion following legislative and executive action. 11 12 VHA Chiropractic Program Office data show a growth of over 250% in facilities offering on-site chiropractic services from Fiscal Year (FY) 2017 (87 facilities) to FY2022 (221 facilities). Similarly, the total number of visits and patients receiving chiropractic care increased substantially from FY2017 (208,400 visits by 47,486 Veterans) to FY2022 (393,532 visits by 93,360 Veterans).

Among the backdrop of rapid expansion, a scalable, dynamic solution to monitoring the quality of chiropractic care delivered across the national VHA enterprise is needed. Current state quality assessment through the VHA Ongoing Professional Practice Evaluation (OPPE) requirements is typically measured via time-intensive, peer-conducted chart review to evaluate predetermined OPPE minimum standards. While easier to access and use, structured electronic health record (EHR) data present limitations to support in-depth evaluation of clinical care delivered across a national health system.

Natural language processing (NLP) of clinic documentation offers a potential opportunity to extract care quality data from unstructured EHR data on a national scale. 13 NLP is a diverse, integrated field utilizing computational techniques for linguistic analysis to achieve human-like language processing. 14 Information extraction from unstructured text using NLP has been successful across multiple sources, including physician reports, 15 radiology reports, 16 and patient experience surveys, 17 and across different settings, including VHA. 18 NLP has been used to evaluate pain care quality (PCQ) in VHA primary care clinics, including developing an extraction tool, 19 a PCQ indicator score, 20 and artificial intelligence methods to detect instances of pain assessment. 20 21 The defined PCQ indicators represent essential elements of guideline-concordant, comprehensive pain assessment, treatment planning, and reassessment, and their documentation in the clinical text note may offer a measurable representation of the quality of pain care being provided.

Objectives

While the previously developed NLP algorithms were built from VA primary care progress notes, the lexicon describing PCQ indicators should theoretically be consistent across primary and specialty care clinic settings. We propose in this work to explore the application of the previously developed NLP algorithm to identify PCQ indicators in chiropractic clinics, a VHA specialty care clinic frequently evaluating and managing musculoskeletal pain conditions. The primary objective of the study was to assess PCQ patterns documented by chiropractors, describe their distributions across different clinic visit types, and compare the co-occurrence of documented pain assessment classes to a previous sample from primary care visits for pain.

Methods

Study Design, Setting, Participants, and Data Sources

This observational study was a secondary analysis of the Women Veterans Cohort Study 22 —an EHR cohort of post-9/11 men and women Veterans who served during Operation Enduring Freedom, Operation Iraqi Freedom, and Operation New Dawn (OEF/OIF/OND). Study reporting was informed by the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE). 23 This study was approved by the VA Connecticut Healthcare System Institutional Review Board.

Patients with at least one visit to a chiropractic clinic at a VHA facility from October 1, 2018 to September 30, 2019 (FY2019) were included in the sample. Patient demographic characteristics were obtained from the EHR, including age (as of October 1, 2018), sex, race, and ethnicity.

All chiropractic care visits during the study period were identified using a VHA administrative clinic stop code identifier for “Chiropractic Care.” Access to chiropractic care in the VHA most often requires the placement of a referral order to initiate a consultation. Visits were identified as consult visits if linked to an ordered consultation request in the EHR and follow-up visits if not. Any visit with an “Evaluation and Management” (E&M) current procedural terminology (CPT) code (99201–99205, 99211–99215) were identified as examination visits. Four groups of clinic visits were defined for comparison of documentation across types of visits—consult visits with an E&M code, consult visits without an E&M code, follow-up visits with an E&M code, and follow-up visits without an E&M code.

All clinical text note documents (VHA text integration utilities documents) were extracted for the identified visits to build a corpus of clinic visit notes (and sub-corpora for each clinic visit type). Only notes describing face-to-face (i.e., in-person) patient encounters (“ambulatory” or “in-hospital” visits) were retained for the NLP analysis, based on a structured visit type identifier in the EHR. Telehealth visits were uncommon in this corpus describing visits from FY2019 (pre-COVID-19 pandemic) and were not included. 24 Other notes linked to the identified visits but describing historical events, clinical reminders, telecommunications, electronic consultations, or nursing notes were excluded from the analysis.

Analysis

We applied a previously developed and described rule-based NLP algorithm using Python 3.5 to extract PCQ indicators from the corpus. 20 The algorithm performed strongly in a study of 1 year of primary care notes from pain visits (F-measure  =  91.9%, Precision  =  93.0%, Recall  =  90.9%). 20

The algorithm uses a line-by-line analysis of text documents paired with rule-based token regular expression matching to identify annotation spans—or tagged snippets of text—mapped to PCQ indicator classes defined by a vocabulary. The vocabulary was reviewed by the study team to review appropriateness for application in chiropractic clinic documentation. A random sample of 100 chiropractic clinic notes was reviewed after the NLP algorithm was applied to generate a qualitative impression of the model's face validity 25 in the unseen corpus. Quantitative evaluation of this process was beyond the objectives of this study.

PCQ indicator classes included those describing pain assessment, treatment planning, and reassessment, for a total of 25 indicators ( Supplementary Table S1 , available in the online version). We also examined the subset of 11 classes describing pain assessment to compare documentation by chiropractors to a reference sample derived from primary care documentation. 21

Study Outcomes

As an exploratory analysis, descriptive statistics were calculated for the presence of PCQ indicators across the entire corpus and across notes for each of the four unique clinic visit types.

The number of PCQ indicator classes present was calculated for each note, with a greater number of present PCQ indicator classes presumed to reflect a broader array of the quality of pain care delivered. The prevalence of notes with each individual indicator class and each total number of indicator classes was calculated as the percentage of the entire corpus and each visit type.

The within note frequency of each individual PCQ indicator class was identified, with a greater number describing more mentions of the indicator class. The mean number of mentions of each class per note were calculated for all notes and the four clinic visit types.

The presence or absence of 11 PCQ indicator classes describing pain assessment was evaluated for co-occurrence of classes—or simultaneous documentation of individual PCQ indicator classes in the same note. The percentages of all notes and notes of each visit type in the chiropractic clinic note corpus were calculated for varying levels of co-occurrence, and aggregate percentages were compared to previously published data from primary care visits for musculoskeletal pain diagnoses by Fodeh et al. 17

Results

There were 11,752 patients who received any chiropractic care during FY2019 ( Table 1 ), with 11,416 having at least one ambulatory or in-hospital visit and 336 patients excluded. Characteristics of the NLP corpus of ambulatory or in-hospital visits are shown in Table 2 , across all notes and by visit type. The entire corpus included 63,812 notes from 52,117 visits across 80 VHA facilities, with over 2.6 million annotations of PCQ indicators present across the corpus. The most common text spans that were annotated across the entire corpus and their respective PCQ indicator classes are shown in Supplementary Table S2 (available in the online version). The most common text span by far was “pain” (Pain Mention) accounting for 12.6% of all annotations, with “low back” (Pain Site) the second most common (2.0% of all annotations).

Table 1. Characteristics of Women Veterans Cohort Study patients receiving VHA chiropractic care during FY2019.

Variable	Total
N	11,752
Age, a median [Q1, Q3], y (%)	39 [34, 48]
<40 y	52.6
40–49 y	25.6
50–64 y	20.7
65 years and over	1.1
Sex (%)
Female	15.8
Male	84.2
Race (%)
White	73.5
Black or African American	15.7
Asian	2.2
Native Hawaiian or Other Pacific Islander	1.2
American Indian or Alaska Native	1.0
Mixed race	1.3
Other/unknown	5.1
Ethnicity (%)
Hispanic or LatinX	12.0
Not Hispanic or LatinX	88.0

Abbreviation: VHA, Veterans Health Administration.

^aAge as of October 1, 2018.

Table 2. Corpus characteristics from ambulatory or in-hospital only chiropractic clinic visits from FY2019, by visit type.

		Visit type
Variable	Total	Consult, E&M	Consult, no E&M	Follow-up, E&M	Follow-up, no E&M
Patients	11,416	4,339	2,980	5,020	7,890
Facilities	80	72	72	75	80
Visits	52,117	4,421	3,022	13,690	33,017
Notes	63,812	4,425	3,024	15,884	40,479
Annotations	2,608,565	418,148	236,736	612,413	1,341,268

Abbreviation: E&M, evaluation and management.

Follow-up visit notes made up 88.3% of the entire corpus and were more likely to not have an E&M code present (63.4% of all notes). Consult notes were more likely to have an E&M code associated with the visit (6.9% of all notes) than not (4.7% of all notes). Consult notes had more than twice the total number of annotations per note (87.9) as follow-up visit notes (34.7) on average. Across all chiropractic care notes, only 3.8% did not include any documentation of PCQ indicators. These notes were most commonly for follow-up visits and frequently included telecommunications visit notes that were miscoded as ambulatory or in-hospital visits upon manual review.

The distributions of the total number of PCQ indicator classes across the entire corpus and by visit type are shown in Fig. 1 . The mean number of total classes documented per note across the entire corpus was 9.4 (standard deviation [SD]  =  1.5). More total indicator classes were documented during consult visits with or without E&M CPTs (consult with E&M: mean  =  14.8, SD  =  0.9; consult without E&M: mean  =  13.9, SD  =  1.2) compared to follow-up visits with or without E&M (follow-up with E&M: mean  =  9.1, SD  =  1.4; follow-up without E&M: mean  =  8.6, SD  =  1.5).

Fig. 1.

Prevalence of chiropractic clinic notes with total number of pain care quality indicator classes, by visit type (with mean and standard deviation).

Fig. 2 shows the relationship between PCQ indicator class frequency within a note (as the mean number of mentions) and the class prevalence across the entire corpus. Across all notes and across notes for each visit type, classes with higher prevalence across the corpus (or sub-corpus) were also mentioned more frequently, on average, within a note. The most common classes were consistent across all notes and visit types and included Pain Mention, Chiropractic, Pain Site, Physical Diagnosis, and Etiology. The least common classes were similarly consistent across all notes and visit types, describing Diurnal Variation, Implantables, and Mental Health. The remaining classes were mentioned with similar within note frequencies across all notes and visit types. However, they had a greater prevalence across the corpora of consult visits with or without E&M than within the corpora of follow-up notes.

Fig. 2.

Pain care quality indicator class mentions per note by class prevalence across corpus, by visit type.

When examining only the subset of 11 pain assessment classes ( Fig. 3 ), over 50% of all chiropractic clinic notes documented at least four different classes (consult with E&M  =  94.8%; consult without E&M  =  87.2%; follow-up with E&M  =  46.9%; follow-up without E&M  =  46.5%). Greater co-occurrence of pain assessment classes was present in all chiropractic care visit types compared to the reference sample of 1,058 primary care pain visit notes by Fodeh et al. 21

Fig. 3.

Co-occurrence of subset of pain care quality indicators describing pain assessment, organized by visit type. The height of each stacked bar represents proportion of notes with at least the specified lower bound number of pain care quality indicators. *Comparison data from primary care pain visit notes from Fodeh SJ, Finch D, Bouayad L, et al. Classifying clinical notes with pain assessment using machine learning. Med Biol Eng Comput 2018;56(7):1285–1292

Discussion

Monitoring and evaluating pain care to ensure quality is a high priority for the VHA. A challenge presents in unifying and applying solutions across multiple specialties providing health care services for musculoskeletal pain in the VHA's Stepped Care Model for Pain Management. In this study, we applied a rule-based NLP algorithm developed in primary care documentation to chiropractic care documentation as one specialty delivering musculoskeletal pain care in the VHA. A scalable solution for monitoring PCQ in VHA chiropractic clinics nationally is essential given the rapid expansion of VHA chiropractic services nationally since 2017, with adoption of modernized processes that move beyond current-state time-intensive manual chart review methods for quality monitoring. Operationalizing this process, including determining the computational infrastructure needed to support real-time (or near real-time) monitoring, is important future work.

The application of the existing PCQ indicator NLP algorithm in an unseen corpus of notes from a separate clinical discipline was based on the premises that (1) indicators of quality pain care would be consistent across disciplines managing musculoskeletal pain and (2) the terminology describing how pain care is documented is reasonably similar across disciplines. While formal validation of the algorithm's performance in the corpus of chiropractic clinic notes was not performed in this study, we took steps to ensure it was appropriate for the study's primary purpose before comparing PCQ indicator patterns across different visit types. The rule-based NLP model used a lookup vocabulary that was reviewed by the study team before application to review the PCQ indicator classes and the terminology linked to each class. After applying the NLP algorithm, we also reviewed a sample of model-annotated notes to confirm face validity and appropriateness for use in a corpus of chiropractic clinic notes. While applied in only a single VHA specialty pain care discipline in this study, application of the NLP algorithm in other nonprimary care settings is likely reasonable after ensuring appropriateness with manual review.

Our findings show that VHA chiropractors frequently document multiple PCQ indicators across notes and visits. We identified a direct relationship between the most prevalent indicator classes across the corpus and the average number of mentions of a class per note. When classes were mentioned more frequently across the entire collection of notes, they were also mentioned more frequently within individual notes. The individual classes mentioned most frequently across the corpus and within individual notes were generally expected. Their high per note prevalence may be attributable to the depth of evaluation of a patient's pain in a thorough pain history with respect to location and etiology and the use of multiple physical diagnosis procedures as part of a comprehensive musculoskeletal and neuromuscular examination to evaluate a patient's pain and functional status and inform the management plan. 26 The most common text spans also support attribution to chiropractic practice as they described pain, especially spinal pain (e.g., low back or neck), range of motion (e.g., flexion, rotation), and chiropractic treatment (e.g., spinal manipulation). The most common text spans may be useful in developing a standardized vocabulary and ontology of chiropractic care to support future NLP evaluation in the clinical discipline.

Of interest was Mental Health among the classes mentioned least frequently, given comorbid pain and mental health conditions are common in the OEF/OIF/OND Veteran population 27 28 and those receiving VHA chiropractic care. 29 Further, recognition of the contributions of psychosocial factors, and mental health conditions as an extension of those factors, to a patient's pain experience is an expected best practice of chiropractic care consistent with a biopsychosocial model of care. 26 We suspect this finding is due to chiropractors not considering or not documenting mental health conditions or psychosocial factors related to mental health, given the vocabulary was viewed as appropriate by the study team. However, a disconnect between the algorithm's vocabulary for the Mental Health class and how chiropractors are documenting mental health considerations in their notes could also be contributory and is a potential area for future formal evaluation. Our findings suggest an opportunity may exist for quality improvement related to consideration of mental health conditions and symptoms by VHA chiropractors, while also demonstrating the generalizability of a scalable, NLP quality monitoring solution to identify potential areas for targeted quality improvement initiatives.

Our findings regarding the documentation of PCQ indicators by visit type showed expected but interesting documentation patterns. Consult visits consistently had many more PCQ indicator classes documented than follow-up visits, which was expected. However, the presence of an E&M CPT code was not necessarily reflective of more PCQ indicators documented in either consult or follow-up visits. We suspect this may reflect the “minimum components” requirements of E&M coding in use during FY2019 or may reflect a known lack of fidelity in outpatient clinician coding, of which under-coding is common. 30 31

When evaluating the subset of pain assessment classes for co-occurrent documentation in comparison to the reference sample of primary care notes, chiropractic clinic notes showed consistent documentation of more assessment classes across all visit types. We suspect this is likely due to the focused nature of a chiropractic care visit on one or a few musculoskeletal pain conditions—while pain may be one of many conditions being managed in a typical primary care visit. Repeating our methods in corpora of notes from other VHA pain care specialties would likely yield similar patterns of pain assessment. This highlights the benefits of the collaborative structure of the VHA in providing integrated and interdisciplinary team-based pain care with facilitated access to maximize the quality of pain assessment. 32 Important future work should also evaluate how patient factors, such as demographic characteristics, and visit factors, such as visit type or method of health care delivery (e.g., face-to-face vs. telehealth), influence documentation patterns across different VHA pain care settings.

As in any NLP study, our study is ultimately limited by the contents and quality of the text documents that are processed. While NLP of clinical notes can allow analyses that move beyond structured data elements captured in the EHR, secondary analysis is limited by the contents collected and entered at the point of care for clinical purposes by the document writer and may not fully reflect what is performed in the visit. In some cases, documentation may under-represent details of the patient–clinician encounter, while in others, historical documentation may be carried forward for reference as an over-representation. Other limitations include the use of templated text within clinic notes. For example, templated statements documenting patient consent may include text spans related to PCQ indicator classes (e.g., “…I advised patient on the benefits and risks of spinal manipulation < Treatment >…”), but not confirm an indicator was present during the visit. Our approach was to broadly explore the application of the existing algorithm in chiropractic clinic notes and thus we did not aim to discern the intentions of PCQ indicator mentions using assertion or negation.

We also recognize limitations related to short communication notes, most often due to inaccurate coding of note types or multiple notes linked to an individual visit. We attempted to limit our corpus to only ambulatory and in-hospital note types to focus our analysis on notes documenting clinical encounters. However, it was evident during review that inaccurate coding of note types led to the inclusion of other types of notes in the corpus, for example telecommunications notes describing scheduling or telephone follow-up on patient status or diagnostic test findings. Further, multiple notes could be linked to an individual visit identifier that often were addenda describing trainee participation in notes or other short, administrative communications. As our analysis was performed at the note level, each of these were considered distinct notes. Future work should consider alternative strategies, such as only evaluating notes authored by the attending chiropractor or concatenating all notes related to the same visit identifier to resolve to a single collected document per visit.

Conclusion

Our study demonstrates that a PCQ indicator NLP algorithm developed from primary care pain visit notes may have utility in VHA pain specialty care for the purposes of quality care monitoring on a national scale. We identified patterns of frequently documented PCQ indicators by VHA chiropractors that can be used to inform quality improvement initiatives related to pain evaluation and management. NLP may be a useful approach in future work to study features of pain care, including chiropractic care, not evident in structured EHR data. Developing standardized, comprehensive ontologies describing pain care and chiropractic care may enhance validity and reproducibility of results. Future applications of this NLP algorithm should investigate relationships between PCQ indicators and patient characteristics as part of EHR-based phenotyping efforts, which may inform clinical decision making and ensure delivery of high-quality and equitable pain care in VHA pain specialty clinics, including chiropractic clinics.

Clinical Relevance Statement

Natural language processing is an approach that may facilitate evaluation of pain care quality in unstructured data from specialty pain care, including chiropractic clinic documentation. Veterans Health Administration chiropractors frequently document many pain care quality indicators, with different patterns noted across consultation and follow-up visit types and visits with or without evaluation and management procedural codes. These results can inform education and quality improvement initiatives related to evaluation and management of pain in clinical practice.

Multiple-Choice Questions

1. The greatest average number of pain care quality indicator classes was documented in which type of VHA chiropractic care visit?

   1. Consultation visit with an evaluation and management CPT

   2. Consultation visit without an evaluation and management CPT

   3. Follow-up visit with an evaluation and management CPT

   4. Follow-up visit without an evaluation and management CPT

   Correct Answer: The correct answer is option a. Consultation visit with an evaluation and management CPT. In consultation visits with an evaluation and management CPT, an average of 14.8 pain care quality indicator classes were documented. Consultation visits without an evaluation and management CPT had 13.9 classes documented on average. Follow-ups with (9.1) or without (8.6) an evaluation and management CPT had fewer pain care quality indicator classes documented.

2. VHA chiropractors infrequently documented which of the following pain care quality indicator classes?

   1. Pain Site

   2. Physical Diagnosis

   3. Mental Health

   4. Etiology

   Correct Answer: The correct answer is option c. Mental Health. Chiropractors infrequently documented “Mental Health” indicators. It is unclear if our finding is due to chiropractors not considering or not documenting mental health conditions or psychosocial factors related to mental health, a disconnect between the algorithm's vocabulary for the Mental Health class and how chiropractors are documenting mental health considerations in their notes, or some combination of these and other factors.

Funding Statement

Funding This study was funded by the Health Services Research and Development, (grant numbers: CIN-13-407 and IIR-12-118) and U.S. Department of Health and Human Services, National Institutes of Health, National Center for Complementary and Integrative Health, (grant number: K08AT011570), and NCMIC Foundation.