Customizing CAT Administration of the PROMIS Misuse of Prescription Pain Medication Item Bank for Patients with Chronic Pain

Abstract

Objective

The 22-item PROMIS^®-Rx Pain Medication Misuse item bank (Bank-22) imposes a high response burden. This study aimed to characterize the performance of the Bank-22 in a computer adaptive testing (CAT) setting based on varied stopping rules.

Methods

The 22 items were administered to 288 patients. We performed a CAT simulation using default stopping rules (CAT_PROMIS). In 5 other simulations, a “best health” response rule was added to decrease response burden. This rule stopped CAT administration when a participant selected “never” to a specified number of initial Bank-22 items (2–6 in this study, designated CAT_Alt2-Alt6). The Bank-22 and 7-item short form (SF-7) scores were compared to scores based on CAT_PROMIS, and the 5 CAT variations.

Results

Bank-22 scores correlated highly with the SF-7 and CAT_{PROMIS, Alt5, Alt6} scores (r=0.87–0.95) and moderately with CAT_{Alt2- Alt4} scores (r=0.63–0.74). In all CAT conditions, the greatest differences with Bank-22 scores were at the lower end of misuse T-scores. The smallest differences with Bank-22 and CAT_PROMIS scores were observed with CAT_Alt5 and CAT_Alt6. Compared to the SF-7, CAT_Alt5 and CAT_Alt6 reduced overall response burden by about 42%. Finally, the correlations between PROMIS-Rx Misuse and Anxiety T-scores remained relatively unchanged across the conditions (r=0.31–0.43, Ps < .001).

Conclusions

Applying a stopping rule based on number of initial “best health” responses reduced response burden for respondents with lower levels of misuse. The tradeoff was less measurement precision for those individuals, which could be an acceptable tradeoff when the chief concern is in discriminating higher levels of misuse.

Introduction

To minimize inappropriate prescribing and ensure patient safety, the Centers for Disease Control and Prevention (CDC) guideline recommends physicians to monitor opioid misuse behaviors in patients taking opioids over the long term [1]. Misuse refers to the use of prescription medications in ways that differ from medical advice [2]. When nonadherence is identified, clinicians can team with patients to identify plans for reducing misuse of prescription opioids. There are several instruments whose scores have been found to validly distinguish levels of opioid misuse by patients taking opioids for pain management: the Current Opioid Misuse Measure (COMM) [3], Pain Medication Questionnaire (PMQ) [4, 5], and Prescription Opioid Misuse Index (POMI) [6]. Unfortunately, these instruments are rarely used in daily practice, perhaps because of their response burden; the COMM comprises 17 items and the PMQ has 26 items. The six-item POMI is quite brief, but validity evidence has only been obtained using a small sample of 40 people receiving treatment for addiction and 34 people taking opioids for pain [6].

For clinicians to monitor and address misuse of prescription opioids, a brief instrument is needed that has proven validity for use in patients taking opioid medications for pain and seeking treatment at pain clinics. One candidate measure is NIH’s PROMIS^®-Rx Pain Medication Misuse item bank (PROMIS-Rx Misuse) [7]. The development and initial validity assessment of scores on this measure was based on a mixed sample of community dwelling adults and patients in outpatient addiction treatment programs [7]. In a subsequent study, responses to all 22 items of the PROMIS-Rx Misuse (Bank-22) were administered to a large sample of patients in a tertiary pain clinic as part of the Collaborative Health Outcomes Information Registry (CHOIR), an open-source registry to assess general and pain-specific health status [8]. Analyses supported the validity of Bank-22 to assess levels of prescription opioid misuse at outpatient pain clinics [8]. The Bank-22 maximizes score precision, but exacts a rather large response burden. Pilkonis and colleagues recommended a 7-item short form based on their instrument development work in community dwelling adults and patients in outpatient addiction treatment programs [7].

Another option for reducing response burden is to administer the PROMIS-Rx Misuse items using computer adaptive testing (CAT), a tailored approach to testing that greatly reduces response burden while maintaining measurement precision. One feature of CATs is the requirement for “stopping rules.” Stopping rules are the preselected conditions that trigger the end of a CAT. These most often include a minimum and maximum number of administered items and a standard error of measurement (SE) threshold (stop CAT once SE < 3 T-score points). Kallen and colleagues studied the impact of alternate CAT stopping rules [9]. One of these was a “highest health” response rule that stops the CAT after individuals have responded to a prespecified number of items by endorsing the response associated with highest health; that is, the response associated with highest function or lowest symptom level [9]. The impetus for developing this stopping rule was PROMIS user complaints that those with the highest function and lowest symptom levels were administered the maximum number of items; though such respondents were generally of the least clinical interest. This occurs because the SE stopping rule cannot be reached when there is no variation in item response pattern.

The primary goal of the current study was to evaluate the impact on scores and response burden of including a “highest health” stopping rule in the CAT administration of the PROMIS Rx-Misuse item bank and recommend rules for its use in the tertiary pain clinic setting.

Methods

The Institutional Review Board at the Stanford University School of Medicine approved the current study procedures.

Sample

The sample for this study was the same as used by You and colleagues [8]. Detailed sample characteristics have been published [8]. Briefly, the sample data for the current study were collected from 288 consecutively enrolled adult (>18 years) patients with any chronic pain condition who presented for initial medical evaluations at an outpatient pain clinic and endorsed taking opioid pain medication. The 22 items were administered via CHOIR between December 2014 and March 2015. Patients were predominantly female (62.2%), married (53.1%), middle-aged (M = 51.6 years, SD = 15.5), and White/Non-Hispanic (51.7%). Only 6.6% reported less than high school education (6.6%) and 54.8% obtained associate’s or higher degree (54.8%). The average duration of pain was 6 years (SD = 9.1).

Measures

The data set included responses to the PROMIS-Rx Misuse full-item bank (inclusive of the seven-item short form) [7], PROMIS-Anxiety [10], and PROMIS-Physical Function [11]. All PROMIS-measures were scaled on a T score metric (M = 50, SD = 10). Higher scores would indicate more misuse problems, more anxiety symptoms, and better physical function. The PROMIS-Rx Misuse items were rated on a 0–4 scale. Of the 22 items, 19 items assessed the frequency of opioid misuse behaviors (never, rarely, sometimes, often, almost always), and three items assessed the severity of adverse symptoms (not at all, a little bit, somewhat, quite a bit, very much). Subsequently, the best health responses correspond to “never” or “not at all.”

We obtained T scores of the PROMIS-Rx misuse measures using the free online scoring service (https://www.assessmentcenter.net/ac_scoringservice). The PROMIS-Anxiety and Physical Function measures were included to examine the relationships of their scores to scores on the PROMIS-Rx Misuse measure under different CAT stopping rules. According to our preliminary validation study [8], the PROMIS-Rx Misuse T-scores were most strongly associated with the PROMIS Anxiety T-scores and least strongly associated with the PROMIS Physical Function T-scores.

Analyses

Simulation of CAT Administration of the PROMIS-Rx Misuse Item Bank

CAT simulations were conducted using the Firestar program [12]. Firestar allows users to input a dataset in which responses were collected for all items of an item bank and then calculate scores based on user specifications including CAT stopping conditions. SPSS Version 26 was used for all statistical analyses.

The PROMIS default (CAT_PROMIS) stopping rules require a minimum of four answered items and a maximum of 12 items [9, 13]. Once four items are answered, the CAT algorithm continues until the SE for the estimated scores is < 3 on the T-score metric. If 12 items are administered but the SE stopping rule has not been reached, the CAT administration stops, and the score for the individual is the one obtained after the twelfth item. For example, under the CAT_PROMIS condition, all individuals complete the four items. After the four items, CAT administration stops for people with SE < 3 T score or continues for the others until SE becomes < 3 T score or until 12 items are administered.

For the current study, we simulated CATs using the default stopping rules (CAT_PROMIS) and also tested five variations of those rules. In the five variations, we maintained the PROMIS default of administering a maximum of 12 items administered and stopping when the SE < 3 but dropped the requirement to administer at least four items. The five simulations differed by how many items were required to invoke the “best health” response rule, which allowed two to six initial items answered “never” (or “not at all”) indicating the lowest level of prescription misuse. Hereafter, we refer to these CAT conditions as CAT_Alt2 through CAT_Alt6. For example, under the CAT_Alt2 conditions, CAT administration stops for people who answered “never” to the first two items and continues for the others until their SE is < 3 T-score points or 12 items are administered. Under the CAT_Alt6 conditions, CAT administration stops for people who answered “never” to the first six items and continues for the others until their SE is < 3 T-score points or 12 items are administered.

By dropping the requirement that four items be administered, we were able to evaluate the effectiveness of two- and three-item “best health” screens, an approach suggested in previously published research (i.e., CAT_Alt2) [9]. The range of the required minimum number of best health responses studied, allowed exploration of the full range of tradeoffs between precision and response burden. We judged it unreasonable to expect less than 2-items to have the required precision and unnecessary to require more than 6 such items.

Evaluation of CAT Simulation Conditions

The fidelity of scores based on CAT_PROMIS (the default stopping rules for simulation), and CAT_Alt2 through CAT_Alt6, was evaluated by comparing the obtained T scores to those based on the full item bank (Bank-22) and those based on the seven-item short form (SF-7). For fidelity analysis, the associations between T scores, obtained from Bank-22, SF-7, and CAT_Alt2-Alt6, were estimated using Pearson r correlation coefficients. Then, difference scores between simulated CAT T-scores and references T-scores (Bank-22) were computed for each individual. The mean difference scores were calculated to quantify divergence from mean reference scores that were obtained under different CAT simulation conditions.

The number of items administered under each condition was subtracted from 7 to quantify reduction in response burden under the studied CAT conditions compared to SF-7 scores. Next, for those who answered “never” in initial items, we counted “premature exits” in CAT_Alt2-Alt5 conditions compared to CAT_Alt6. We defined a premature exit as a score that deviated from the reference T-score by ≥ 5.0. A T-score of 5.0 (SD of 0.5) is sometimes used as a minimally important difference [14, 15]. Finally, T-score differences were plotted against Bank-22 T scores to examine score fidelity by score level. Additionally, difference scores between CAT_PROMIS and SF-7 T scores were plotted against Bank-22 T scores.

To evaluate the impact of CAT conditions on concurrent validity, we calculated Pearson r correlation coefficients between all study PROMIS-Rx Misuse scores (i.e., Bank-22, SF-7, and all CAT conditions) and scores on PROMIS Anxiety and Physical Function measures. Specifically, we compared the magnitude and P values of correlations coefficients.

Results

Table 1 shows the intercorrelations among PROMIS-Rx misuse scores obtained under the study conditions. SF-7, CAT_PROMIS, and CAT_Alt5-6 scores correlated highly with Bank-22 scores (r = 0.87–0.95). When only two, three, or four items were required in the “best health” rule, however, the correlations were substantially lower (r = 0.63–0.74).

Table 1.

Pearson’s correlation coefficients between all PROMIS^®-Rx Misuse scores (N = 288)

	Bank-22	SF-7	CATPROMIS	CATAlt2	CATAlt3	CATAlt4	CATAlt5
SF-7	0.94
CATPROMIS	0.95	0.92
CATAlt2	0.63	0.63	0.72
CATAlt3	0.72	0.72	0.80	0.88
CATAlt4	0.74	0.73	0.81	0.85	0.98
CATAlt5	0.87	0.88	0.94	0.75	0.84	0.86
CATAlt6	0.90	0.88	0.96	0.73	0.82	0.83	0.98

Note: SF-7 = The 7-item short form; CAT_PROMIS = CATs using the default stopping rules; CAT_Alt2 - CAT_Alt6 = CAT using the alternative stopping rule such as stopping the CAT after two to six initial items answered “never”; All P < .001.

Table 2 reports descriptive statistics for Bank-22, SF-7, and CAT scores. Though all mean scores were similar, the greatest similarity with Bank-22 scores was obtained in the CAT administrations (absolute differences in means of 0.2–1.2T). This compares with a difference in 2.5 points for SF-7 scores. When comparing the number of cases with at least 5.0 T-score difference between the Bank-22 and the other conditions, CAT_Alt2 had the most cases (36.1%) and CAT_PROMIS had the least (3.8%).

Table 2.

Changes in T scores obtained from the full item administration and CAT simulations under different stopping rules (N = 288)

	Bank-22	SF-7	CATPROMIS	CATAlt2	CATAlt3	CATAlt4	CATAlt5	CATAlt6
M	43.1	45.6	42.7	41.9	42.9	42.7	42.5	42.7
(SD)	(6.2)	(6.2)	(6.4)	(4.9)	(5.7)	(6.1)	(6.4)	(6.3)
Min	30.9	36.1	31.6	39.4	35.8	34.8	33.7	32.9
Max	63.0	68.1	62.5	62.5	62.5	62.5	62.5	62.5
Mean difference	Ref.	2.5	0.4	−1.2	−0.2	−0.4	0.6	0.4

Ref. = Reference T-score; Mean difference = T-scores from each CAT simulation—Reference T-score.

Response Burden and Pattern under Different CAT Conditions

Table 3 shows percentages of people completing 2–12 CAT items under different stopping conditions. The impact of the alternate stopping rules was observed in the mean number of items for each CAT condition. Comparison of the total number of items administered in the sample is also telling. As an example, for the SF-7, that number was 2,016 (7 × 288). For CAT_Alt5 and CAT_Alt6, the number was 865 and 863, respectively, amounting to a reduction in respond burden by about 42%. Though most of this reduction was the result of the “best health” rule, a small portion was due to lifting the minimum of four items required in PROMIS. Based on the results for CAT_Alt5-Alt6, we deduce that 2.4% of alternate CAT administrations stopped because the SE < 3 was reached. Under CAT_PROMIS, these administrations would have continued. When comparing response burdens between CAT_Alt5 (mean = 6) and CAT_Alt6 (mean = 7), there were on average 14% increase in CAT_Alt6.

Table 3.

Percentage of people completing the number of items administered under the different conditions (N = 288)

Number of items	CATPROMIS	CATAlt2	CATAlt3	CATAlt4	CATAlt5	CATAlt6
2	…	75.7	…	…	…	…
3	…	2.4	34.0	2.4	2.4	2.4
4	17.0	14.9	14.9	39.6	14.9	14.9
5	9.4	2.8	9.0	9.0	33.0	9.0
6	10.8	3.5	10.8	10.8	10.8	27.4
7	10.8	0.7	10.4	10.8	10.8	10.8
8	11.1	…	9.7	11.1	11.1	11.1
9	19.8	…	3.5	5.6	5.6	6.6
10	3.8	…	1.4	1.7	2.1	3.8
11	5.2	…	1.0	2.8	2.8	4.5
12	12.2	…	5.2	6.3	6.6	9.4
Mean CAT length	8	3	5	6	6	7

Potential for Premature Exit under Alternate Condition

To evaluate whether the CAT_Alt2 rule might result in a “premature exit” from the CAT, we evaluated the response histories of those whose assessment terminated because of the two-item “best health” rule. For this subset of the sample (n = 218, 75.7%), we compared the score they would have gotten under the CAT_Alt2 condition, (which, based on expected a priori estimation, is 39.4 T) to the score they would have obtained under CAT_Alt6. Of the 218 participants for whom the 2-item best health rule was invoked, 59.2% had a premature exit. Of the 91 participants who answered “never” to the first three items, only 2.2% had a premature exit. No premature exit was observed in CAT_Alt4 and CAT_Alt5.

Differential Impact of SF-7 and CAT Conditions at Score Levels

Figure 1 depicts score differences between CAT_PROMIS and all other conditions by Bank-22 T-score levels. Differences for most scores were close to zero and within the 5.0 T-score difference. For the alternate CAT conditions, each successive increase in required number of “best health” responses resulted in less variations from CAT_PROMIS scores. Patterns of discrepancy were similar across CAT conditions, with the largest discrepancies in the lower score ranges (i.e., below 50 in CAT_Alt2 and below 45 in the other conditions). Notably, the pattern of discrepancy was different in SF-7 scores, which were relatively evenly distributed across the Bank-22 T scores. Among CAT conditions, the largest difference was observed in CAT_Alt2 and the least in CAT_Alt6.

Figure 1.

T-score difference between CAT_PROMIS and SF-7 (SF7 Diff) and between CAT_PROMIS and different CAT conditions (CAT_Alt2-6 Diff). Dotted lines indicate no difference (center) and ± 5.0 T-score difference (upper and lower), which was considered a minimally important difference.

Changes in Estimated Relationships to Other Measures under Different Sets of Stopping Rules

Pearson correlations were computed to exam whether different stopping rules would alter the estimated relationships between PROMIS-Rx Misuse T scores and PROMIS Anxiety and Physical Function T scores (Table 4). Correlation coefficients were similar between T-scores obtained from Bank-22 and those of CAT_PROMIS. Correlation coefficient values with PROMIS-Anxiety T-scores were significant and small for Bank-22 (r = 0.43, P < .001), SF-7 (r = 0.38, P < .001), and all CAT T-scores (r = 0.38–0.31, P < .001). Except CAT_Alt2 T-scores (r = −0.07, P = .222), PROMIS-Physical Function T-scores had small, but significant negative correlations with Bank-22 (r = −0.18, P = .003), SF-7 (r = −0.13, P = 025), and the other CAT T-scores (r = 0.17–0.13, Ps < .029). Overall, all provided convergent and discriminant validity evidence for the measure.

Table 4.

Pearson correlation coefficients between T-scores of PROMIS-Rx Misuse and other most (Anxiety) and least (Physical Function) related PROMIS measures (N = 288)

CATAlt2

		PROMIS
		Anxiety T scores		Physical Function T scores
		r	P	r	P
PROMIS Rx Misuse T scores	Bank-22	0.43	<.001	−0.18	.003
	SF-7	0.38	<.001	−0.13	.025
	CATPROMIS	0.38	<.001	−0.17	.003
	0.31	<.001	−0.07	.222
	CATAlt3	0.37	<.001	−0.16	.008
	CATAlt4	0.35	<.001	−0.13	.029
	CATAlt5	0.36	<.001	−0.14	.020
	CATAlt6	0.37	<.001	−0.16	.008

Discussion

Screening for prescription opioid misuse is recommended to prescribing physicians, but efficient, sensitive, and low burden screening measures are lacking. The purpose of our current study was to quantify the impact on PROMIS-Rx Misuse scores when “best health” stopping rules were applied in simulated CATs. Our goal was to identify reasonable options for CAT administrations in clinical settings where the chief interest would be in discriminating well among individuals with higher self-reported prescription opioid misuse (e.g., tertiary pain clinic). Overall, we found that alternate stopping rules result in substantial reductions in response burden. This reduction in response burden was principally in the lower ranges of self-reported prescription opioid misuse which represents most patients taking opioids in a typical pain clinic. Consistent with research by Kallen and colleagues [9], the sacrifices in precision resulting from the “best health” response stopping rules were relatively small. The exception was the CAT_Alt2 condition, where we found the most cases (36.1%) with differences in score greater than 0.5 SD compared to Bank-22. Follow-up analyses indicated that, for many respondents, the CAT_Alt2 rule resulted in a premature exit for many participants who answered “never” to the first two items. Notably, the largest difference in mean scores were observed between SF-7 and Bank-22 (M = 2.5), followed by between CAT_Alt2 and Bank-22 (M = -1.2) and between all the other comparisons between CAT conditions and Bank-22 (M < 1.0). Hence, compared to Bank-22, more discrepant performance was observed in SF-7 and CAT_Alt2 than in any of the other CAT conditions.

Applying the “best health” response CAT stopping rules resulted in loss of precision in lower levels of reported prescription misuse. Thus, we do not recommend their use when there is a need to discriminate among these low levels. The need to discriminate these low levels would not be a typical goal in a usual clinical setting. However, if the primary purpose of collecting the data was to test an intervention whose purpose was to ameliorate prescription opioid misuse, applying the alternative rules could mask the treatment’s effect at the lower levels of opioid misuse. For such settings as these, the CAT_PROMIS would be a better choice. When the concern is chiefly to identify clinically concerning levels of prescription opioid misuse, however, the alternative stopping rules may be worth the tradeoff in precision for the resulting decrease in response burden. It is important to note, however, that the tradeoff between precision and burden occur only for those at lower levels of reported prescription opioid misuse. The response burden for those with higher levels remains the same when the alternative rules are applied.

In settings in which the applying a “best health” response stopping rule is appropriate, either CAT_Alt5 and CAT_Alt6 would be good choices. Both conditions performed well and reduced response burden by about 42% compared to SF-7. Though the CAT_Alt6 performed slightly better than CAT_Alt5, the differences were small and may not be worth the 14% increase in response burden. For use in our tertiary pain clinic, we judged the CAT_Alt5 to be the better choice.

Limitations

The study findings should be qualified by a number of limitations. Our patient population was derived from a tertiary pain clinic, which limits generalizability to other pain populations or settings (e.g., acute pain or perioperative settings). Additionally, our sample was predominantly White/Non-Hispanic, females, college-educated, and middle-aged adults. Future studies should evaluate whether our results generalize to samples with other conditions (e.g., medical and psychiatric), males, adults with less than college education, and different ages (e.g., adolescents and geriatric populations). The simulations were based on the assumption that patients would answer items in the same way whether administered by CAT or presented with the full item bank. The accuracy of this assumption needs to be examined in future studies by administering both the full item bank and the CAT items to patients and comparing their responses between the two administration methods.

Implications and Future Directions

Despite these limitations, our study supported the use of alternative CAT stopping rules that decrease response burden for those with lower reported prescription opioid misuse, with minimal impact on score precision for those with higher reported prescription opioid misuse. This work is responsive to the need for improved monitoring of patients receiving opioid pain medication [1]. The findings add to the body of knowledge that supports the monitoring of patients for prescription opioid misuse, and subsequently providing appropriate interventions to identified patients [16]. Future studies may seek to examine individual time assessment points within a longitudinal analysis to assess the degree to which “best health” response rules impact scores in treatment studies and as participants reach improved states of health. Finally, another future direction is the development of score thresholds for operationalizing clinically significant levels of prescription opioid misuse and Opioid Use Disorder (OUD). Once such thresholds are developed, alternative CAT conditions can be compared with respect to their ability to successfully classify individual patients. Eventually, our findings will be useful when a CAT survey is developed to assess prescription opioid misuse for specific clinical or research purposes or when CAT platform with user-defined stopping rules is available for clinical or research use.

Conclusion

For clinicians to identify, monitor and address misuse of prescription opioids, a brief instrument is needed that has proven validity for use in patients taking opioid medications for pain and seeking treatment at pain clinics. Applying a stopping rule based on number of initial “best health” responses was effective in reducing response burden for respondents with lower levels of prescription opioid misuse. The tradeoff came in less measurement precision for those individuals. This could be an acceptable tradeoff where the concern is in discriminating higher levels of misuse. In selecting from among the alternate CAT stopping rules, we concluded that the CAT_Alt5 and CAT_Alt6 performed well. Users should weigh whether the increase in response burden for those with lower PROMIS-Rx Misuse scores is worth the increment in score precision. For our tertiary pain clinic, we judged the CAT_Alt5 to be the better choice to reduce response burden with sufficient precision.

 

Funding sources: K23 DA048972 (DY), K23DA047473 (MZ), K23DA035302 (JH), R01AT008561 (BD), K24DA029262 (SM), and Redlich Endowment (SM).

Conflicts of interest: There are no conflicts of interest to report.