They Don’t Know What They Don’t Know: Internal Medicine Residents’ Knowledge and Confidence in Urine Drug Test Interpretation for Patients with Chronic Pain

ABSTRACT

BACKGROUND

Urine drug testing (UDT) can help identify misuse or diversion of opioid medications among patients with chronic pain. However, misinterpreting results can lead to false reassurance or erroneous conclusions about drug use.

OBJECTIVE

To examine the relationship between resident physicians’ knowledge about UDT interpretation and confidence in their ability to interpret UDT results.

DESIGN

Cross-sectional survey.

PARTICIPANTS

Internal medicine residents in a university health system in the Bronx, from 2010 to 2011.

MAIN MEASURES

We assessed knowledge using a 7-item scale (UDT knowledge score), and confidence in UDT interpretation using a single statement (“I feel confident in my ability to interpret the results of urine drug tests”). We conducted chi-square tests, t-tests, and logistic regression to determine the association between knowledge and confidence, and in exploratory analyses to examine whether resident characteristics (gender, training level, and UDT use) moderated the relationship between knowledge and confidence.

KEY RESULTS

Among 99 residents, the mean UDT knowledge score was 3.0 out of 7 (SD 1.2). Although 55 (56 %) of residents felt confident in their ability to interpret UDT results, 40 (73 %) of confident residents had a knowledge score of 3 or lower. Knowledge score was not associated with confidence among the full sample or when stratified by training level or UDT use. The association between knowledge and confidence differed significantly by gender (interaction term p < 0.01). Adjusting for training level and UDT use, knowledge was positively associated with confidence among females (AOR 1.79, 95 % CI: 1.06, 3.30), and negatively associated with confidence among males (AOR 0.47, 95 % CI: 0.23, 0.98).

CONCLUSIONS

Despite poor knowledge about UDT interpretation, most resident physicians felt confident in their ability to interpret UDT results. Gender differences warrant further exploration, but even confident physicians who use UDT should evaluate their proficiency in interpreting UDT results. Educational initiatives should emphasize the complexities of UDT interpretation.

INTRODUCTION

Prescription opioid addiction and overdose have skyrocketed over the past decade.^1–3 Therefore, it is important that physicians who prescribe opioids monitor their patients closely to identify signs and symptoms of risky use.^4–8 When interpreted correctly, urine drug testing (UDT) can provide objective data that complement a patient’s self-report, and can help to identify patients who are at increased risk of addiction or overdose due to concurrent use of illicit drugs or undisclosed controlled substances.^9–15 In addition, inappropriately negative UDT results can help to identify diversion, the sale or transfer of prescribed opioids to others.^16,17

Interpreting urine drug test results is complicated, and mistakes can have serious consequences for patient care.^18,19 Correct interpretation of UDT requires knowledge of the sensitivity and specificity of the assay for detecting drugs of interest, causes of false positive results, metabolic pathways, and detection times in urine for each drug.²⁰ Use of UDT among physicians who lack knowledge or are overconfident in UDT interpretation can lead to mistaken conclusions about the risks of prescribing opioids for their patients. For example, misinterpretation can lead to false reassurance that a patient is adhering to prescribed medications, or to an erroneous conclusion that a patient is abusing or diverting prescribed opioids.

Adoption of UDT for patients treated for chronic non-malignant pain (CNMP) in primary care settings has increased rapidly in recent years.²¹ However, studies suggest that primary care physicians have poor proficiency in interpreting UDT results.^22,23 We are not aware of any studies investigating whether physicians’ confidence in their own ability to interpret UDT results reflects their actual knowledge about interpretation. Additionally, studies have found that physician self-assessment of knowledge in other clinical areas is often inaccurate.^24,25 Therefore, as part of a needs assessment about internal medicine residents’ management of CNMP, we sought to explore the relationship between residents’ knowledge and confidence in interpreting UDT results. We hypothesized that residents’ knowledge and confidence in UDT interpretation would be low, and that they would be positively associated with each other. Secondarily, because previous studies have found that physician self-assessment differs by gender and experience,^25–28 we sought to explore whether differences in residents’ gender, training level, or frequency of using UDT moderate the association between knowledge and confidence.

METHODS

Setting and Participants

As part of a needs assessment for developing a curriculum in management of CNMP, we invited all 148 internal medicine residents in their first, second, or third post-graduate year at Montefiore Medical Center to participate in a cross-sectional study from October 2010 to July 2011. Residents were in categorical, primary care, or social internal medicine tracks, practiced at three clinics without standardized UDT protocols, and received variable training on UDT. The Montefiore Institutional Review Board reviewed and exempted the study.

Data Collection and Measures

The survey instrument consisted of 53 items about residents’ attitudes, behaviors, and knowledge about several aspects of managing CNMP, including use of UDT. Items used Likert scale, multiple-choice, and true-false response options and were derived from previously developed instruments.^22,29,30 The instrument was piloted for clarity by 12 recently graduated residents. The final instrument was administered by an emailed link to a web-based survey, followed by additional emails to nonresponders every 2–4 weeks for 3 months, then in person at educational sessions to residents who had not responded to the web-based survey. Participants were not remunerated, but a single digital MP3 player was distributed to a randomly selected resident to incentivize participation.

The main variables of interest were knowledge and confidence in interpreting UDT results. Knowledge about UDT interpretation was assessed using seven multiple choice questions developed by Reisfield and colleagues.²² The questions assessed knowledge of UDT assays, opioid metabolism, and potential explanations for unexpectedly positive or negative results. The few (<1 %) missing responses were considered incorrect. We report the percent of participants answering each question correctly, as well as the mean number of items answered correctly out of seven (the UDT knowledge score). Confidence was assessed using a single item statement (“I feel confident in my ability to interpret the results of urine drug tests”), in which residents rated their level of agreement on a 5-point Likert Scale. We dichotomized responses to either confident (agree or strongly agree) or not confident (neutral, disagree, or strongly disagree).³⁰ In the questionnaire, the confidence item preceded the knowledge items.

Other variables of interest were residents’ sociodemographic and clinical characteristics. Sociodemographic characteristics included gender and training level (post-graduate year [PGY]), dichotomized to either PGY-1, or PGY-2 or PGY-3). Clinical characteristics included residents’ self-reported percent of patients with CNMP, percent of patients with CNMP who were prescribed opioids, confidence managing CNMP in general, and UDT use. UDT use was assessed using two items: (1) self-reported percentage of current patients on long term opioid analgesics for whom they had ever ordered a UDT (continuous variable), and (2) response to a multiple-choice question, “Which of the following best describes how often you order urine drug testing?” Response options were: regularly for most or all patients, occasionally for most or all patients, only when I’m concerned about misuse or my preceptor requests it, or never. We dichotomized responses to “routine UDT use,” defined as regularly or occasionally for most or all patients; and “rare UDT use,” defined as never or only when the resident is concerned about misuse or a preceptor requests it.

Analysis

For the current analysis, we restricted the sample to residents who provided a response to the confidence item and to at least three of the seven knowledge items. To determine the association between knowledge and confidence, we first conducted chi-square tests and t-tests to determine if the proportion of residents answering each knowledge item correctly, or the mean UDT knowledge score, differed significantly among residents who were confident versus not confident. We constructed an unadjusted logistic regression model (the “base model”) with confidence (yes/no) as the dependent variable, and knowledge score (a continuous variable) as the independent variable.

To evaluate effect modification by gender, training level, and UDT use (the “potential moderators”), we stratified the base model by each potential moderator (male, female; PGY-1, PGY-2 or PGY-3; routine UDT use, rare UDT use). After identifying differences by gender, we tested the significance of the knowledge*gender interaction term in a logistic regression model that included gender, training level, and UDT use as covariates. The final logistic regression models testing the association between knowledge and confidence were stratified by gender, and adjusted for training level and UDT use.

To test the sensitivity of our findings to alternate definitions of the key variables, we conducted adjusted logistic regression models with training level as an ordinal variable (PGY-1, PGY-2, or PGY-3) and as a continuous variable (estimated days into training from July 1^st of PGY-1 year); and we conducted linear regression models with confidence as a continuous variable (1 through 5 on the Likert scale). The conclusions remained the same, so these findings are not presented. All statistical analyses were conducted using Stata 10 software (College Station, TX).

RESULTS

Of 148 internal medicine residents, 109 (73.6 %) participated in the study. Ten residents were excluded because they did not provide data on confidence (n = 6) or knowledge (n = 4) about UDT interpretation. Of the 99 residents, most (60.6 %) were female, reflecting the gender distribution in the sample population (62.2 % female) (Table 1). Most residents reported that between 6 % and 30 % of their patients had CNMP and that 10 % or fewer were prescribed opioids. Self-reported UDT use varied markedly; 16.2 % routinely ordered UDTs for most or all patients on opioids for CNMP, 29.3 % occasionally ordered UDTs for most or all patients, 23.2 % ordered UDTs only when concerned about misuse or when a preceptor requested it, and 27.3 % never ordered UDTs. The median number of residents’ current patients for whom they reported having ever ordered a UDT was 5 (IQR 0, 80).

Table 1.

Sample Characteristics (n = 99)

	n	%*
Sociodemographic and clinical characteristics
Female	60	60.6
Training level
PGY-1	27	27.3
PGY-2	34	34.3
PGY-3	38	38.4
Percent of patients who have CNMP
5 % or fewer	4	4.3
6 % to 10 %	20	21.5
11 % to 30 %	35	37.6
More than 30 %	34	36.6
Percent of CNMP patients prescribed opioids
5 % or fewer	44	48.4
6 % to 10 %	33	36.3
11 % to 30 %	9	9.9
More than 30 %	5	5.5
Confidence managing CNMP
Median score (IQ range)†	3 (2,3)
I feel confident, n (%) agree or strongly agree	13	13.4
Confidence interpreting urine drug test results
Median score (IQ range)†	4 (3,4)
I feel confident, agree or strongly agree	55	55.6
Use of urine drug testing
How often do you order urine drug testing?
Routinely	16	16.2
Occasionally	29	29.3
Only when concerned or preceptor requests	23	23.2
Never	27	27.3
Not applicable/No patients prescribed opioids	4	4.0
Median percent of my CNMP patients prescribed opioids for whom I have ever ordered a urine drug test (IQ range)	5 (0, 80)

* Missing data for percent of patients who have CNMP (n = 6), percent of CNMP patients prescribed opioids (n = 8), and confidence managing CNMP (n = 2). For each item, residents who did not provide a response are not included in denominator for calculated percentages

† On a 5-point Likert Scale, where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree

Knowledge

Residents’ mean UDT knowledge score, the number of seven items answered correctly, was 3.0 (SD 1.2). Individual residents’ scores ranged from zero to six, and 27 residents (27.3 %) answered four or more questions correctly. The percent of residents who answered each item correctly is presented in Table 2. The item that was answered correctly by the greatest proportion of residents (74.8 %) was a true-false question evaluating knowledge that secondhand marijuana smoke exposure would not cause a positive urine drug screen. The items that were answered correctly by fewest residents (<30 %) were those that required knowledge of opioid metabolic pathways (Q1, 3, and 4), and about half of residents demonstrated knowledge that semi-synthetic and synthetic opioids are not reliably detected on a screening test for opiates (Q6 and 7). Mean UDT knowledge score was significantly lower among PGY-1 residents vs. PGY-2 or PGY−3 (2.4 vs. 3.3, p < 0.001), but did not differ significantly by gender (3.1 vs. 2.9 among females vs. males, p = 0.55), or by UDT use (3.2 vs. 2.9 among routine vs. rare UDT users, p = 0.14).

Table 2.

Knowledge About Urine Drug Test Interpretation Among Confident and Not Confident Residents

	Total n = 99	Confident n = 55	Not confident n = 44	p value*
	n (%)	n (%)	n (%)
Knowledge questions, % of residents who answered correctly†
Q1-4. Identify substance(s) expected to be detected in urine of a patient who is:
prescribed acetaminophen/codeine	29 (29.3)	14 (25.5)	15 (34.1)	0.35
prescribed morphine	68 (68.7)	36 (65.4)	32 (72.7)	0.44
using heroin	10 (10.1)	6 (10.9)	4 (9.1)	0.77
ingesting poppy seed danish	20 (20.2)	12 (21.8)	8 (18.2)	0.65
Q5. Identify that second-hand marijuana smoke exposure would not cause a positive urine drug screen for cannabis	74 (74.8)	43 (78.2)	31 (70.5)	0.38
Q6. Identify valid reasons for a negative urine opiate screen in a patient prescribed opioids	42 (42.4)	27 (49.1)	15 (34.1)	0.13
Q7. Identify need for confirmatory testing	58 (58.6)	30 (54.6)	28 (63.6)	0.36
UDT Knowledge Score
Score of 4 or greater, n (%) of residents	27 (27.3)	15 (27.3)	12 (27.3)	1.00
Mean score (SD)	3.0 (1.2)	3.1 (1.2)	3.0 (1.2)	0.90

*Chi-square test to evaluate differences in proportions of residents answering each question correctly, and t-test to evaluate differences in mean percent of questions answered correctly

†Knowledge questions from Reisfield GM et al., J Opioid Management, 2007 ²²

Confidence

Although only 13 (13.4 %) of residents reported feeling confident managing CNMP in general, 55 (55.6 %) felt confident in their ability to interpret the results of UDT, with a median score of 4 (“agree”) on the 5-point Likert scale. In unadjusted analyses, confidence did not vary significantly by gender, training level, or UDT use. We considered residents to be “overconfident” if they were confident interpreting UDT results, but answered fewer than half of the knowledge questions correctly (3 or fewer correct out of 7). According to this definition, 40 residents were overconfident (40.4 % of the full sample and 72.7 % of residents who were confident). Overconfidence was identified in 46.1 % of male vs. 36.7 % of female residents (p = 0.35), 51.8 % of PGY-1 vs. 36.1 % of PGY-2 or PGY-3 residents (p = 0.15), and 40.0 % of routine UDT users vs. 41.5 % of rare UDT users (p = 0.88); these differences were not statistically significant.

Association Between Knowledge and Confidence

The mean UDT knowledge score was not different among residents who were confident vs. not confident (3.1 vs. 3.0, p = 0.90) (Table 2), and the percent of residents who answered correctly did not differ significantly by confidence for any of the seven individual items. In unadjusted stratified analyses, the association of knowledge with confidence did not differ by training level or UDT use (Table 3). However, there was significant effect modification by gender that persisted in models adjusting for training level and UDT use (p < 0.01 for the knowledge*gender interaction term). In gender-stratified adjusted models, UDT knowledge score was positively associated with confidence among females (adjusted odds ratio for each additional knowledge item answered correctly [AOR] was 1.79, 95 % CI: 1.06, 3.03, p = 0.03) and negatively associated with confidence among males (AOR 0.47, 95 % CI: 0.23, 0.98, p = 0.04).

Table 3.

Effect of Gender, Training Level, and Urine Drug Test Use on the Association Between Resident Knowledge and Confidence in Interpretating Urine Drug Test Results

	I feel confident in my ability to interpret UDT results
Unadjusted Models*	OR	95 % CI	P value
Knowledge Score, each correct answer
Full sample	1.02	(0.74, 1.41)	0.90
Stratified by gender
Females	1.72	(1.05, 2.84)	0.02
Males	0.45	(0.23, 0.88)	0.007
Stratified by training level
PGY-1 (intern)	1.01	(0.45, 2.28)	0.98
PGY-2 or PGY-3	1.06	(0.73, 1.55)	0.75
Stratified by UDT use
Routine use	1.02	(0.63, 1.66)	0.92
Rare use	0.98	(0.63, 1.54)	0.95
Gender-stratified Adjusted Models†	AOR	95 % CI	P value
Females
Knowledge Score, each correct answer	1.79	(1.06, 3.03)	0.03
Training level (PGY-1 vs. PGY-2 or PGY-3)	1.33	(0.32, 5.56)	0.69
Frequency of UDT use (routine vs. rare)	0.96	(0.30, 3.04)	0.94
Males
Knowledge Score, each correct answer	0.47	(0.23, 0.98)	0.04
Training Level (PGY-1 vs. PGY-2 or PGY-3)	1.83	(0.30, 11.07)	0.51
Frequency of UDT use (routine vs. rare)	2.65	(0.51, 13.84)	0.25

*Unadjusted logistic regression models included confidence as the dependent variable and UDT knowledge score as the sole independent variable

†Adjusted logistic regression models included confidence as dependent variable, UDT knowledge score as the main independent variable, and two dichotomous covariates (training level, and frequency of UDT use)

DISCUSSION

In this cohort of internal medicine residents, we found that despite poor knowledge about urine drug test (UDT) interpretation, the majority of residents felt confident in their ability to interpret UDT results. Most (73 %) of those who reported feeling confident were identified to be overconfident, because they answered fewer than half of the knowledge items correctly. In exploratory analyses of gender differences, knowledge was positively associated with confidence among female residents, but negatively associated with confidence among male residents. Taken together, our findings indicate that resident physicians’ knowledge of UDT interpretation was poor, self-assessment of their knowledge was often inaccurate, and confidence in interpreting UDT results often did not reflect underlying knowledge.

Previous studies of practicing physicians have identified poor knowledge about UDT interpretation among emergency physicians, pediatricians, and family physicians,^22,23 and our findings extend this to internal medicine residents. In our study, only 27 % of residents answered at least half of the knowledge items correctly, compared to 20 % of family physicians²² and 32 % of physicians from diverse fields (pain, primary care, or addiction),³¹ using the same UDT knowledge scale. These similarly low knowledge scores among practicing physicians indicate that our finding of poor knowledge among residents might reflect a lack of knowledge among attending physicians who supervise and teach them, and this warrants further research. We did find that knowledge scores were greater among residents in their second or third year of training, compared to interns, suggesting a positive effect of medical education. Our finding that knowledge about interpreting UDT results was not greater in physicians who use UDT routinely supports similar findings by Reisfield and colleagues.^22,31

To our knowledge, this is the first study to examine how well confidence in interpreting UDT results reflects knowledge among physicians at any training level. Our finding of high confidence is corroborated by a study by Yanni and colleagues, in which only 7 % of internal medicine residents reported feeling uncomfortable interpreting UDT results, but knowledge was not reported.³⁰ In this study, we identified not only that knowledge was poor and that most residents felt confident, but that knowledge was poor even among those who felt confident. This coupling of poor knowledge and feeling confident is likely be more detrimental than poor knowledge alone, because physicians who do not identify their lack of knowledge are likely to be less motivated to seek verification and more likely to follow through with errors of misinterpretation.³²

There are several explanations for why residents might be overconfident in their ability to interpret UDT results. First, interpreting results of an immunoassay urine drug screen, the default test in most laboratories, seems straightforward because a simple positive or negative result is reported for each drug class (e.g., opiates or benzodiazepines). However, there are hidden complexities to consider, including the sensitivity for different drugs within the drug class, substances that might cause a false positive result, detection time in the urine, and how to identify tampering.^18,33,34 Second, most residents are accustomed to using UDT as a simple screen for drug use (e.g., to determine whether chest pain is cocaine-related or altered mental status is due to intoxication). For these purposes, unlike in pain management, it is not usually important to determine which opioid is present or the possible causes of negative results. Third, training in substance abuse and pain management is lacking in medical school and residency, and the subtleties of UDT interpretation are omitted from standard curricula.^35–40

Physician overconfidence can lead to diagnostic errors,³² in this case leading to either under-diagnosis or over-diagnosis of abuse or diversion, with potentially negative consequences for patients.¹⁸ One way overconfidence can cause diagnostic error is through premature closure, for example, concluding without further investigation that an opiate-positive urine drug screen reflects a patient’s prescribed oxycodone use.^32,41 Because semi-synthetic opioids like oxycodone do not reliably yield a positive result on immunoassay screens for opiates,²⁰ additional confirmatory testing (for example, with gas chromatography and mass spectrometry) might be warranted to identify use of other opiates like heroin or morphine. Failure to identify use of non-prescribed opioids may represent a missed opportunity to identify addiction or at least, to recognize the increased risk of overdose in a patient who is combining prescribed and non-prescribed opioids.

Physician overconfidence in interpreting the results of UDT could also lead to inaccurate diagnoses of opioid analgesic abuse, addiction, or diversion where none exists.^18,19 It has been described that overconfident physicians may make errors because of inappropriate use of heuristics;^32,41 for example, assuming that an opiate-negative result on a urine drug screen must mean that no opioids are present. This error might lead a physician to incorrectly conclude that a patient is diverting medications, which can undermine the therapeutic alliance or prompt the patient to leave the physician or practice. In primary care settings, these outcomes put a patient at risk for deterioration of other chronic illnesses. If the physician discontinues opioids, the patient may have worsened pain or opioid withdrawal, and future barriers to adequate analgesia if inaccurate diagnoses follow them to subsequent providers.

Although this is the first study to identify discordance between knowledge and confidence in UDT interpretation, similarly poor self-assessment by physicians has been identified in other clinical areas.^24,25 Our finding that gender modified the association between knowledge and confidence is supported by previous findings that among undergraduate students, medical students, and residents, females tended to under-assess, and males tended to over-assess their own competence.^26–28 Importantly, other studies have found that inaccurate self-assessment is most common among physicians with the least skills and greatest confidence,²⁵ raising concern that physicians and trainees who are newly adopting UDT may be ineffective at determining when they lack knowledge. This could decrease motivation to learn and lead to failure to seek guidance or improve practice.^24,32

There are several limitations to this study. Our sample was comprised of resident physicians who do not make clinical decisions independently. We could not evaluate the role of their supervising physicians in supplementing residents’ knowledge, encouraging or discouraging use of UDT, or interpreting test results during clinical care. The study had a small sample size, which limited statistical power, particularly for stratified analyses. The study took place at a single university medical center, limiting generalizability. The response rate of 74 % was fair, but there is a possibility of sampling bias. In addition, our measure of confidence was a single item derived from previous surveys, but not validated as a self-assessment construct. Finally, UDT use was self-reported and is subject to recall bias.

This study calls attention to a critical concern impacting efforts to curb the public health threat of opioid analgesic misuse, addiction, and overdose. UDT has been increasingly promoted and embraced to minimize the risk of opioid misuse in patients with chronic pain,^4–7 and when interpreted correctly, it provides valuable data that can help identify addiction or diversion. However, it is important that physicians, educators, and public health officials consider that interpreting UDT results may be more complex than many physicians understand. Efforts to promote general use of UDT must simultaneously address knowledge gaps in interpreting the results and provide physician learners with feedback about their own knowledge. Importantly, even physicians who feel confident in their ability to interpret UDT should seek laboratory guidance about available assays, become familiar with opioid metabolic pathways,⁴² and take care to minimize misinterpretation. Further studies are needed to evaluate the effectiveness of education and policy initiatives to improve physicians’ use and interpretation of UDT.