Utilizing a learning health system to capture real-world patient data: Application of the reliable change index to evaluate and improve the outcome of a pain rehabilitation program

Abstract

Background and Objectives:

The learning healthcare system (LHS) has been developed to integrate patients’ clinical data into clinical decisions and improve treatment outcomes. Having little guidance on this integration process, we aim to explain a) an applicable analytic tool for clinicians to evaluate the clinical outcomes at a group and an individual level and b) our quality improvement (QI) project, analyzing the outcomes of a new outpatient pain rehabilitation program (“Back-in-Action”: BIA) and applying the analysis results to modify our clinical practice.

Methods:

Through our LHS (CHOIR; https://choir.stanford.edu), we administered the Pain Catastrophizing Scale (PCS), Chronic Pain Acceptance Questionnaire (CPAQ), and Patient-Reported Outcomes Measures (PROMIS)^® before and after BIA. After searching for appropriate analytic tools, we decided to use the Reliable Change Index (RCI) to determine if an observed change in the direction of better (improvement) or worse (deterioration) would be beyond or within the measurement error (no change).

Results:

Our RCI calculations revealed that at least a 9-point decrease in the PCS scores and 10-point increase in the CPAQ scores would indicate reliable improvement. RCIs for the PROMIS measures ranged from 5 to 8 T-score points (i.e., 0.5–0.8 SD). When evaluating change scores of the PCS, CPAQ, and PROMIS measures, we found that 94% of patients showed improvement in at least one domain after BIA and 6% showed no reliable improvement.

Conclusions:

Our QI project revealed RCI as a useful tool to evaluate treatment outcomes at a group and an individual level, and RCI could be incorporated into the LHS to generate a progress report automatically for clinicians. We further explained how clinicians could use RCI results to modify a clinical practice, to improve the outcomes of a pain program, and to develop individualized care plans. Lastly, we suggested future research areas to improve the LHS application in pain practice.

The learning healthcare system (LHS) framework has been introduced “to generate and apply the best evidence for the collaborative healthcare choices of each patient and clinician, to drive the process of discovery as a natural outgrowth of patient care, and to ensure innovation, quality, safety, and value in health care.” ¹ In daily practice, data collected via LHS at the point of care help clinicians assess individual patients’ health status, responses to interventions, and guide treatment decisions. ¹ LHS data are also used to examine the effectiveness of the standard of care and to identify the gaps between treatment outcomes from clinical trials and pragmatic settings. In acknowledging these values, LHSs have been adopted eagerly by stakeholders such as patients, clinicians, leaders of hospitals, payers, and policymakers. ^1–3 Subsequently, more clinicians are expected to use LHS.

Since the LHS concept was introduced and the infrastructure was developed, we and others have been utilizing LHS for pain care and pain research. ^4–10 However, there has been relatively less discussion on how a clinician should use LHS data to improve clinical outcomes. With little guidance about integrating LHS data into practice, LHS may have limited impact on daily clinical practice and treatment outcome. ¹¹ Different from randomized clinical trials (RCTs) where researchers compare outcomes between a new treatment group and a control group, clinicians need to analyze and interpret the change scores of patients receiving one type of treatment, often without a comparison group, and to evaluate a single patient’s response to a treatment. ¹² Subsequently, we have searched for an appropriate analysis method when conducting a quality improvement (QI) project to evaluate the effectiveness of a newly developed, multidisciplinary pain rehabilitation program for patients with chronic back pain, named ‘Back-in-Action’ (BIA).

To evaluate the outcome of the BIA program, we collected patient-reported outcome (PRO) data via our LHS and previously compared the outcome between the new BIA program and ongoing Cognitive Behavioral Therapy (CBT). ⁷ The results indicated BIA was better in improving mobility and pain behaviors, but both significantly improved pain catastrophizing, pain interference, fatigue, depression, anxiety, and social role satisfaction. Having collected more data for the BIA program, the current QI project evaluated the outcomes without a comparison group, as this is a more plausible scenario in real-world practice. In selecting an analytic tool, we looked for tools that could be used to evaluate the outcomes at a group and an individual level. The primary goal of the current paper was to explain an analytic approach for outcome data obtained from patients with chronic back pain who attended a pain rehabilitation program. The second goal was to explain how the analysis results could be applied to improving the outcomes of a pain program and informing a clinical decision about individual care plan.

To evaluate PROs, clinicians can use the minimally important difference or minimal clinically important difference, which are used interchangeably (hereafter MID). MID is a measure of the smallest detectable change in PROs that may match with a patient’s perception of beneficial improvement from a treatment. ¹³ There has been no standard approach to calculate MID. A rule of thumb approach is one-half SD to assess a moderate effect. ¹⁴ Clinicians can evaluate whether a patient’s scores are improved at least one-half SD, which is at least 5.0T score increase when using the Patient Reported Outcomes Measurement Information System (PROMIS) measures (M = 50.0T, SD = 10.0). ¹⁵ Although the one-half SD rule is easily applicable in daily practice, this rule may not match well with the perceivable smallest benefit from a treatment. Subsequently, a third of SD is sometimes used as it is a small effect and approximates better to the concept of MID. ¹⁶ Some researchers use an anchor-based method to identify MIDs that match with patients’ perceived “better/mild improvement” or “worse/deterioration” from a treatment and are beyond the measurement error. ¹⁷ In applying this approach to PROMIS measures, identified MID scores for improvement were 2.4–4.9T for pain interference, ^18–20 3.0–3.1T for depression, 2.3–3.4T for anxiety, ¹⁹ and 1.3–5.0T for fatigue ²¹ after a medical treatment or movement program for several chronic pain conditions. However, MID values are not always available for certain measures, pain conditions, and treatments (e.g., surgery, medication, or rehabilitation program) that a clinician needs. Thus, a universal approach to analyzing and interpreting change scores is needed.

Reliable change index (RCI) is a universal approach to analyzing a change score. ^22,23 RCI addresses inherent measurement errors in assessments, ensuring that observed changes reflect genuine progress rather than random fluctuations. RCI determines if an observed change in the better or worse direction is beyond or within the measurement error. Clinically significant change is another useful index to determine if a reliable improvement also meets subclinical symptom severity at post-treatment. ²² With RCI, responses to a treatment can be classified into a) deterioration, b) no change, and c) reliable or clinically significant improvement. We will explain how we have utilized RCI to analyze PROs obtained before and after the BIA program and how we have applied the analysis results to clinical care and quality improvement.

METHODS

The local institutional review board approved this QI project (IRB No. 28435). Patients were asked to complete questionnaires at their initial and subsequent clinic visits via our LHS, CHOIR (https://choir.stanford.edu). ²⁴ Patients voluntarily completed CHOIR questionnaires at home or in clinic before their visit with their clinician(s) (e.g., pain physicians, physical therapists, psychologists), and the results were available to the clinicians at the point of care. We extracted CHOIR data completed within one or two weeks of the first and the last session of BIA. The BIA program consists of 2-hour pain psychology [acceptance and commitment therapy (ACT) and CBT] and 2-hour physical therapy (pain neuroscience education, mindful movement, and individualized gym exercise programs). ⁷

Measures

We extracted demographic data (sex, age, race/ethnicity, marital status, education levels) to describe the characteristics of patients attending BIA. The 13-item Pain Catastrophizing Scale (PCS) was administered to assess the changes in catastrophic cognitive and emotional responses to pain. ^25,26 The total scores range from 0 to 52, with higher scores indicating higher levels of pain catastrophizing. The 20-item Chronic Pain Acceptance Questionnaire (CPAQ) was administered to assess the degree of pain acceptance. ^27–30 The total scores range from 0 to 120, with higher scores indicating higher levels of pain acceptance. PROMIS pain interference, pain behaviors, fatigue, sleep disturbance, depression, anxiety, social isolation, and social role satisfaction^18,31–34 items were administered using computer-adaptive testing (CAT) to assess multiple health domains with minimal patient burden and reasonable accuracy. ^35–39 PROMIS-social role satisfaction T scores were inverted so higher T-scores on all PROMIS measures would indicate worse symptoms on each measure. The numerical pain rating scale (0–10) was administered to assess average and worst pain in the past 7 days at baseline and post-treatment. ⁴⁰

Primary outcome measures were the PCS, ²⁵ CPAQ, ²⁷ and PROMIS measures of pain interference¹⁸ and mobility. ⁴¹

Secondary outcome measures were the PROMIS pain behaviors, fatigue, sleep disturbance, depression, anxiety, social isolation, and social role satisfaction. ^18,31–34

BIA Program

Detailed content of the BIA program was previously published. ⁷ Briefly, BIA is an intensive outpatient pain program, consisting of pain neuroscience education, physical therapy, CBT, ACT, and mindfulness. ⁷ All patients were seeking treatment for chronic pain at a tertiary pain clinic and diagnosed with chronic back pain. A psychologist and physical therapist evaluated patients to assess their eligibility for the 6-week program. BIA patients attended two sessions per week; each session consisted of two hours of pain psychology and PT. In total, patients received 24 hours of pain psychology and PT sessions. Current data were obtained from patients who attended BIA from January 2017 to October 2019.

Analysis

To calculate RCI, ⁴² the standard deviation and reliability of the measures are needed. These values are generally available with validated measures. For the PCS and CPAQ, we have used published M, SD, and reliability values. ⁴² The mean of PCS total scores is 20, the SD is 12, and the test-retest reliability is 0.75. ^25,26 The mean of CPAQ total scores is 47, the SD is 19 and the test-retest reliability values are > 0.79. ^27–30 The mean of the PROMIS measures is 50T with SD of 10. Reliability values have been calculated for various PROMIS item banks or short-forms, ^32,43,44 but our CAT administration stops when the score standard error (SE) reaches ≤ 0.3, which corresponds to a reliability coefficient of 0.91. ³⁸ With these values, RCI can be calculated using a formula (RCI = $\frac{X_2-X_1}{S{E}_{diff}}$; $S{E}_{diff}$: standard error of difference, $X_2$: post-test score, $X_1$: Pre-test score). In this formula, $S{E}_{diff}$ can be calculated with standard error of measure (SEM) or estimated SEM using a standard deviation and the reliability of a measure ($S{E}_{diff}=\sqrt{2{\left(SEM\right)}^2}$ or $\sqrt{2(SD{\sqrt{\left(1-r\right)}}^2}$); r = a reliability coefficient). Reliable change is determined if the RCI is > 1.96, which is the Z score for a 95% confidence interval. ²²

To determine clinically significant improvement, ⁴² RCI and a clinical cutoff score are needed. ^22,42 The cutoff score should be clinically meaningful to differentiate changes from being “symptomatic” to “non-symptomatic” or from “severe or moderate” to “mild or normal” at post-treatment. ⁴² We have used the PCS score of 16 as a clinical cutoff score because the median of healthy subjects’ PCS total score is 16 (M = 16.56, SD = 7.8). ⁴⁵ The cutoff score for the CPAQ total should be clinically meaningful to capture the change from “maladaptive” to “adaptive.” A cluster analysis have been conducted to identify a high pain acceptance group and the high acceptance group’s mean CPAQ total score is 72.8 (SD = 11.5). ²⁸ Therefore, CPAQ score of 72.8 can be used for a clinical cutoff score. We have used the T score of 60 as a clinical cutoff score for the PROMIS measures because PROMIS scores of < 60T (1 SD) indicate mild or normal symptoms. ^33,46

Finally, we computed the number of improvements, deteriorations, and no changes on the above patient-reported outcomes per patient. Then, we evaluated how many patients showed reliable or clinical improvement in the primary and secondary outcome measures and how many showed no improvement.

RESULTS

We summarize characteristics of patients who attended BIA in Table 1. Mean age was 56.0 years. Patients were predominantly female, White/Caucasian, and married/partnered. The majority obtained at least high school-level education (94%). Means of worst and average pain in the past seven days were 6.8 (SD = 2.1) and 4.7 (SD = 2.0). Patients attended at least 9 (75%) out of the 12 sessions.

Table 1.

Demographics (n = 33)

	M	SD	Min	Max
Age (years)	56.0	14.5	25	80
Worst pain in the past 7 days	6.8	2.1	2	10
Average pain in the past 7 days	4.7	2.0	0	9
The number of sessions attended	11.1	1.0	9	12
	n	%
Sex
Female	27	81.8
Male	6	18.2
Race/Ethnicity
Non-Hispanic
White/Caucasian	20	60.6
Asian	6	18.2
Black/African American	1	3.0
American Indian or Alaska Indian	1	3.0
Other	3	9.1
Missing	1	3.0
Hispanic	1	3.0
Marital Status
Married/Partnered	25	75.8
Unpartnered	8	24.2
Education
Some college or more	25	75.8
High School	6	18.2
Less than high school	1	3.0
Missing	1	3.0

RCI and Clinically Significant Change

Means of the primary and second outcome variables at baseline were calculated (Table 2). Mean T-scores of PROMIS pain interference, mobility, fatigue, and social role satisfaction measures ranged from 60.3 to 64.8, which were 1 SD above the mean, moderate severity. Mean T-scores of the other PROMIS measures ranged from 52.3 to 59.1, which were less than 1 SD of the population mean, mild severity. Mean PCS total score of the BIA attendees was 16.2 (SD = 9.5), which was somewhat lower, but not significantly different from that of patients visiting our clinic for pain treatment (M = 20.0, SD = 13.0, n = 1,794, t = 1.38, p = .095). ⁴⁷ Notably, 10 patients (30%) had PCS total scores < 10 and 9 patients (27%) had PCS total scores between 10 and 19. The mean CPAQ total score of our sample was 52.5 (SD = 11.9), which was not significantly different from that of patients at another tertiary pain clinic (M = 47.2, SD = 18.8, n = 641, t = 1.60, p = .110). ²⁸ The highest CPAQ total score of our sample was 71 so there was room for improvement up to 120.

Table 2.

The baseline M and SD of the primary and secondary outcomes and RCIs (n = 33)

Primary Outcome	Baseline		Deteriorate		No Change		Improvement Reliability Change
							Reliable Change only		Clinically Significant
	M	(SD)	N	(%)	n	(%)	n	(%)	n	(%)
1. PCS	16.2	(9.5)	2	(6.1)	22	(66.7)	2	(6.1)	7	(21.2)
2. CPAQ	52.5	(11.9)	0	(0.0)	3	(9.1)	12	(36.4)	18	(54.5)
PROMIS: 3. Pain interference	64.8	(5.2)	0	(0.0)	20	(60.1)	5	(12.2)	8	(24.2)
4. Mobility	60.6	(6.0)	1	(3.0)	25	(75.8)	0	(0.0)	7	(21.2)
Secondary Outcomes
PROMIS: 5. Pain behavior	59.1	(2.8)	2	(6.1)	16	(48.5)	1	(3.0)	14	(42.4)
6. Fatigue	60.3	(9.2)	5	(15.2)	16	(48.5)	2	(6.1)	10	(30.3)
7. Sleep disturbance	57.5	(6.9)	3	(9.1)	23	(69.7)	0	(0.0)	7	(21.2)
8. Depression	55.6	(5.7)	4	(12.1)	16	(48.5)	1	(3.0)	12	(36.4)
9. Anxiety	57.5	(7.1)	2	(6.1)	22	(66.7)	1	(3.0)	8	(24.2)
10. Social isolation	52.3	(8.5)	3	(9.1)	23	(69.7)	0	(0.0)	7	(21.2)
11. Social role satisfaction	60.4	(6.7)	2	(6.1)	20	(60.6)	2	(6.1)	9	(27.3)

Table 2 summarizes the RCI and clinically significant change. Of the primary outcomes, the PCS, CPAQ, pain interference, and mobility scores were improved in 9 (27%), 30 (91%), 13 (39%), and 7 (21%) patients, respectively. Most noteworthy was the clinically significant improvement in CPAQ scores, observed in 54.5% of the sample. Of the secondary outcomes, reliable or clinically significant improvement was observed more frequently in pain behavior (n = 15, 46%) and depression (n = 13, 39%). In summary, most patients showed reliable or clinically significant improvement in pain acceptance and about 40% showed improvement in pain interference, pain behaviors, and depression symptoms. Between 21 and 36% showed improvement in the other outcomes. The areas where patients showed most deterioration or no change were sleep disturbance and social isolation (n = 26 for both, 76%), followed by anxiety (n = 24, 73%), social role satisfaction (n = 22, 67%), and fatigue (n = 21, 55%).

We calculated minimal change scores for reliable improvement in each measure. At least a 9-point reduction in PCS total scores and a 10-point increase in CPAQ total scores would indicate reliable improvement. At least a 5- and 6-point reduction in T-scores would indicate reliable improvement in PROMIS-pain interference and mobility measures, respectively. Among the secondary PROMIS outcome measures, T-scores should be reduced by at least 4 points in pain behavior; 6 points in sleep disturbance, depression, and social role satisfaction; 7 points in anxiety and social isolation; and 8 points in fatigue to be considered as reliable improvement.

Evaluating the change score at a person-level

RCI and clinically significant improvement were examined at a person-level. On average, a patient had 4.0 improved areas (SD = 2.6). Of the 33 attenders, 31 patients (94%) had at least one area of improvement after BIA (Table 3). Among the 31 treatment-responders, four patients (no. 8, 9, 18, and 21) with improvement in PCS or CPAQ did not report improvement on PROMIS measures, and one patient (no. 17) with no change in PCS or CPAQ scores reported improvement in the PROMIS-pain behavior, fatigue, sleep disturbance, and depression. Of the two patients (6%) with no improvement on any outcome measures, one patient (no. 5) showed a significant increase in PCS scores (23 at pre- and 43 at post-treatment) and PROMIS-fatigue T-scores (54 at pre- and 62 at post-treatment) and the other patient (no. 31) had low PCS (7 at pre- and 2 at post-treatment) and high CPAQ total score at the baseline (85 at pre- and post-treatment).

Table 3:

Individual clinical outcome profile

		PCS	CPAQ	PROMIS
Cohort	Patient	1	2	3	4	5	6	7	8	9	10	11
1	1
1	2
1	3
1	4
1	5
2	6
2	7
2	8
2	9
2	10
2	11
3	12
3	13
3	14
4	15
4	16
5	17
5	18
5	19
5	20
5	21
5	22
6	23
6	24
6	25
7	26
7	27
7	28
7	29
8	30
8	31
8	32
8	33

Note Grey: Deterioration, White: No Change, Light blue: Reliable Improvement, Blue: Clinically Significant Improvement.

1: PCS-Pain Catastrophizing Scale, 2: CPAQ-Chronic Pain Acceptance Questionnaire, 3: PROMIS-pain interference, 4: mobility, 5: pain behavior, 6: fatigue, 7: sleep disturbance, 8: depression, 9: anxiety, 10: social isolation, 11: social role satisfaction

DISCUSSION

Our first aim was to demonstrate the utility of RCI in evaluating and improving clinical outcomes at a group or an individual level within the context of a pain rehabilitation program. Previously, we compared PROs between the BIA and CBT programs. ⁷ The results suggested no significant differences in most PROs between the two groups. Both groups showed significant improvement in pain interference, fatigue, depression, anxiety, social role satisfaction, and pain catastrophizing, with small to moderate effect sizes (ds = 0.29 – 0.73). Then, we found that mobility and pain behavior were significantly improved only after BIA (ds = 0.69 and 0.55, respectively). ⁷ With additional data collected for BIA, we conducted the current QI project to further evaluate the outcomes of the BIA program without a comparison group because a comparison group might not be available in some clinical settings nor available in daily practice. We previously compared the BIA and CBT outcomes in patients with chronic back pain, ⁷ but the BIA program was designed for patients who would need an intensive outpatient rehabilitation program (totaling 24 treatment hours), instead of the standard CBT program (totaling 16 treatment hours). As a result, selection bias and dose effects (24 hours vs 16 hours) were inevitable in our previous analysis. Expectedly, the BIA attendees had more comorbid health conditions than CBT attendees did. Another difference was that all patients interested in the free CBT could attend the sessions whereas some who were interested in the BIA could not attend the sessions due to insurance denials. Further, in the current QI project, we looked for an analytic tool that could be used to evaluate the outcomes both at a group and an individual level.

The BIA program consisted of two evidence-based pain psychology approaches, CBT and ACT. CBT and ACT target pain catastrophizing^48,49 and pain acceptances. ^50,51 Somewhat unexpectedly, the current QI project revealed that reduction of pain catastrophizing was observed in only 1 in 3 patients whereas improved pain acceptance was observed 9 in 10 patients. This unexpected result might be because 30% reported PCS total scores < 10 at baseline, and PCS scores should be reduced by at least 9 points to be considered as reliable improvement. If we apply one-half SD, a 6 point- reduction in PCS total scores would indicate MID, ^25,26 and subsequently more BIA attenders (i.e., 1 in 2 patients) would be considered as having symptom improvement. However, a 6-point reduction falls within the measurement error. There were 21 patients (64%) with improvement only in CPAQ scores, 9 patients (27%) with improvement in both PCS and CPAQ scores, and none with improvement only in PCS scores. These results suggest that most patients (91%) were better in accepting chronic pain after BIA. Reduction in the PCS and CPAQ scores after a pain psychology program or multidisciplinary treatment is considered to be an important psychological process, which may improve clinical outcomes months or years later. ^52–56 Consistent with this mechanism of change, out of 30 patients with improved PCS and/or CPAQ scores, the majority (87%) showed reliable improvement in at least one PROMIS measure, but some patients (13%) showed no improvement in any PROMIS measures. To understand the improvement only in the psychological process variable, we should also consider the timing of outcome evaluation. The current QI project evaluated the outcomes only within one or two weeks of the last session and our LHS was limited in evaluating a “sleeper effect” (i.e., delayed effect). The sleeper effect was reported in CBT and ACT as some people might need more time to understand the concept and make behavioral change. ^57–59 Additionally, one patient showed improvement in PROMIS measures without improvement in PCS or CPAQ scores. This might indicate that we did not assess all relevant psychological process variables that could be changed by pain psychology intervention (e.g., pain self-efficacy^60,61) or that other medical interventions might improve the PROMIS measures because patients were allowed to receive their usual pain care while engaging in the BIA program.

Our second aim was to demonstrate how the outcomes would guide clinical decisions about modifying a pain program or clinical practice. Our QI project revealed that BIA improved pain acceptance in most patients (94%) as it was designed to do so, but different from our expectation, it improved pain catastrophizing only in 27% of patients mainly because about a third reported low pain catastrophizing before the treatment. Our project also revealed that the BIA attendees reported, on average, four areas of improvement. Specifically, about 40% reported improvement in pain interference, pain behaviors, and depression; 36% reported improvement in fatigue; and 33% reported improvement in social role satisfaction. Overall, these results indicated that chronic pain acceptance and multiple clinical outcomes were improved after BIA.

Using these RCI results to inform clinical practice and improve BIA outcomes, we should add the CPAQ measure at the initial pain psychology assessment as the measure was previously administered only at pre- and post-BIA program. We found that one patient with the highest CPAQ and low PCS scores at baseline did not report any improvements, and therefore, the patient did not appear to benefit from BIA. Based on this observation, clinicians might need to consider different pain programs for people with high pain acceptance and low catastrophizing. Generally, clinicians assess treatable psychological targets like pain self-efficacy, lack of knowledge about chronic pain, fear of movement, and social support so clinicians can recommend appropriate individualized treatments targeting such variables (e.g., a single-session pain psychology, ^62,63 pain neuroscience education, ⁶⁴ movement-based programs, ^65,66 support groups, ⁶⁷ or internet-based pain programs^68–71). We also need to add questionnaires to assess psychological process variables such as pain self-efficacy, ^60,61 psychological flexibility, ^72,73 and fear of movement^74,75 before and after BIA. Furthermore, we need to evaluate the long-term outcomes, since four patients showed reliable improvement in CPAQ or PCS at post-treatment, but not in the PROMIS measures. Additionally, we need to examine whether the initial responders and non-responders at post-treatment will report better or worse outcomes months later.

Using these RCI results to inform individual care plans after the BIA program, patients who showed any improvement on the primary or secondary outcome measures were recommended to attend a monthly follow-up group or a monthly chronic pain support group. One patient whose PCS and CPAQ scores were in non-clinical ranges at post-treatment received the same recommendation (i.e., a follow-up group or a support group). One patient whose PCS score was significantly worse after the BIA was recommended to attend individual sessions with a pain psychologist. Four patients who reported significant depression or anxiety symptoms during or after a pain program were recommended to do psychotherapy targeting emotional distress or a psychiatrist consult to explore pharmacological treatment options.

Our QI results should be understood with several limitations. First, we used an LHS to collect the outcome data, and we did not have data from ten patients (out of a total 43 attendees, 23% missing). These ten patients did not complete the outcome measures at pre- and post-treatment, although clinicians who ran the BIA asked patients to complete the questionnaire several times, emphasized the importance, and informed patients that they would receive their progress report after the program. An LHS works only when all or almost all patients participate in this process. Therefore a future QI project will look to understand why some patients would not complete the online questionnaire and how to address the barriers. Secondly, our current LHS setup is limited in assessing 1) delayed responses in between medical appointments and 2) outcomes at 3 or 6 months post-treatment since the data is collected at a point of care. Some psychological or behavioral interventions might take a few weeks or months to impact PROs. Lastly, RCI might be a high standard that does not match patients’ perceived improvement. Using RCI analysis, we found at least 5-point reduction in the PROMIS-pain interference and 8-point reduction in the PROMIS-fatigue measures as reliable improvement. These RCIs are much higher than the MID that match patients’ perceived improvement, i.e., a reduction of 2.4–4.9 T-score points for PROMIS-pain interference^18–20 and 1.3–5.0 T-score points for PROMIS-fatigue. ²¹

Despite the limitations, RCI is a useful tool for clinicians to evaluate PROs for its several advantages. As long as clinicians select a validated and reliable measure, no additional research is needed to calculate RCI. Also, RCI as a universal approach is dependent on what questionnaires a clinician uses, not what type of treatments a patient receives (surgery, physical therapy, or pain psychology). Therefore, a clinician can calculate RCI for a certain measure and use the RCI to evaluate a patient’s response to different treatments. We have identified at least 9-point decrease in PCS total scores and 10-point increase in CPAQ scores as reliable improvement and these values can be used in evaluating the outcome of any pain treatments. Lastly, RCI as a conservative approach may guide clinicians to modify or improve their clinical practice or pain programs to its highest standard.

In summary, our QI project demonstrated RCI as a useful analytic tool in evaluating treatment outcomes at a group or an individual level. With no guidelines about using the LHS to inform clinical decisions, we demonstrated how to evaluate the outcome of a pain program without a comparison group and how to use RCI results to make changes in clinical practice and make individual care plans. RCI could be built into LHS to assist clinicians further in evaluating outcomes automatically. More discussion or consensus guidelines about methods to integrate the LHS data into clinical practice would be in a greater need to utilize the LHS to provide individualized care for patients with chronic pain. Future studies should also identify and address barriers for LHS at patient, clinician, and system levels to evaluate the clinical outcomes of all patients receiving pain treatments.

Data availability statement:

The datasets used and analyzed during the current study will be available from the corresponding author on reasonable request.