Abstract
Context: Narcotics are frequently prescribed in the Emergency Department (ED) and are increasingly abused. Prescription monitoring programs affect prescribing by Emergency Physicians (EPs), yet little is known on how EPs interpret prescription records.
Objective: To assess how EPs interpret prescription narcotic history for patients in the ED with painful conditions.
Design/Main Outcome Measures: We created an anonymous survey of EPs consisting of fictitious cases of patients presenting to the ED with back pain. For each case, we provided a prescription history that varied in the number of narcotic prescriptions, prescribing physicians, and narcotic potency. Respondents rated how likely they thought each patient was drug seeking, and how likely they thought that the prescription history would change their prescribing behavior. We calculated κ values to evaluate interobserver reliability of physician assessment of drug-seeking behavior.
Results: We collected 59 responses (response rate = 70%). Respondents most suspected drug seeking in patients with greater than 6 prescriptions per month or greater than 6 prescribing physicians in 2 months. Medication potency did not affect physician interpretation of drug seeking. Respondents reported that access to a prescription history would change their prescribing practice in all cases. κ values for assessment of drug seeking demonstrated moderate agreement.
Conclusion: A greater number of prescriptions and a greater number of prescribing physicians in the prescription record increased suspicion for drug seeking. EPs believed that access to prescription history would change their prescribing behavior, yet interobserver reliability in the assessment of drug seeking was moderate.
Introduction
Pain is a common problem for which patients seek care in the Emergency Department (ED), accounting for up to 42% of all visits.1,2 As pain control in the ED is often inadequate, increasing emphasis has been placed on pain relief.2,3 Narcotic prescribing has increased to the point that the US, constituting 4.6% of the world population, uses 80% of the world's opiate supply, including 99% of the world's hydrocodone supply.4
In addition, the use of prescription medications for nonmedical reasons has increased. Approximately 7 million Americans over the age of 12 years use prescription medications for non-therapeutic reasons each year, with nearly 50 million Americans having used prescription drugs recreationally during their lifetime.4,5 Prescription drug abuse has become an epidemic in the US, and not without major consequences. Adverse events from the nonmedical use of prescription medication account for more than 700,000 ED visits annually, and deaths due to narcotic overdose have tripled since 1996, with nearly 14,000 opiate-related deaths in 2006.5–7 Deaths resulting from prescription opiate medication account for three-fourths of deaths due to all prescription medications.8 Patients who present to the ED to obtain medication for nontherapeutic reasons are common, estimated to be as high as 20% of all ED patients.9 Most of the literature on “drug-seeking patients” in the ED consists of anecdotal data or small studies, giving Emergency Physicians (EPs) little information on how to identify these individuals.10–15
The increasing prevalence of prescription medication abuse, particularly narcotic abuse, makes it important for EPs to be able to identify patients presenting to the ED seeking medication for nontherapeutic purposes. Numerous scoring systems have been developed to identify patients with narcotic addiction, but they were developed outside the ED and often involve long, complex surveys that are too cumbersome to use in the ED.16–22 Prescription monitoring programs have emerged as a means of identifying patients trying to obtain medication for nontherapeutic reasons.23–26 The use of prescription monitoring programs in the ED has been shown to affect prescribing behavior,25,27 and in one study such a program changed EP prescribing practice in more than 60% of cases.25 Although most states in the US have prescription monitoring programs, little is known about how to clinically interpret the information obtained in prescription records in the ED.
The goals of this investigation were to assess, using a survey instrument, how EPs interpret prescription narcotic history when assessing patients presenting to the ED with pain. We aimed to understand how variations in the number of narcotic prescriptions, number of prescribing physicians, and potency of the narcotic in the prescription history affected how likely EPs perceived a patient as demonstrating drug-seeking behavior. We also sought to assess how useful EPs believed that access to prescription history was in assessing patients in the ED with pain, and how well EPs agreed on whether a patient was drug seeking.
Methods
Study Design and Setting
This study consisted of an anonymous voluntary online survey at one academic medical center in California. The ED in this suburban area sees approximately 51,000 patients per year. Demographically, 54% of patients are between the ages of 18 and 65 years, and 27% of patients are age 17 years or younger; 48% of patients are white, and 29% of patients are Latino. The survey consisted of 6 fictitious case vignettes, each presenting the same case of a 35-year-old man in the ED with a chief complaint of low back pain. This patient reported that he had a history of a “slipped disk” and routinely took narcotic pain medication but had run out of medication and could not see his regular physician for a few days.
For each of the six cases, a prescription history for the prior two months was provided. Each prescription history contained the following information for each case: date that each prescription was filled, name of the medication, dose of the medication, number of pills prescribed, and the name of the prescribing physician. This is comparable with data provided by the California Prescription Drug Monitoring Program, except that the California program also provides the patient's date of birth and address, the name of the pharmacy at which the patient filled the prescription, and the prescribing physician's license and Drug Enforcement Administration numbers. As we believed that most of the assessment of whether a patient exhibits drug-seeking behavior lies in the number of prescriptions, the potency of medication, and the number of prescribing physicians, we did not include this additional information in the data provided in our survey.
Each case varied in the number of prescriptions per month, the number of prescribing physicians, and the potency of the narcotic. A summary of the prescription histories from each of the 6 cases appears in Table 1. The low-potency narcotic in our survey was hydrocodone-acetaminophen at a dose of 5 mg and 500 mg, respectively, and the high-potency narcotic was oxycodone, 5 mg. The order of the 6 cases was the same for each respondent, and the survey software allowed respondents to review questions in any order and to modify previously submitted answers before submission.
Table 1.
To the best of our knowledge, there has been no earlier study of how the availability of a prescription history affects assessment by an EP of whether a patient is exhibiting drug-seeking behavior. We created six clinical vignettes on the basis of what we considered were likely scenarios that might be encountered in the ED. We also tried to vary the number of prescribing physicians prescriptions and medication potency in an attempt to find a threshold at which EP suspicion for drug-seeking behavior would change. These vignettes were not validated, although they were reviewed extensively and pilot-tested with our research advisers.
Respondents were asked to read each case, including the prescription history. After reviewing each case, respondents were asked 2 questions: “How likely do you think this patient is drug seeking?” and “How much would having the information in the prescription record change your prescribing practice?” For each question for each case, respondents were asked to rate their answers on a scale from “1” to “4,” with “1” being very unlikely, “2” being unlikely, “3” being likely, and “4” being very likely.
Although it is controversial and lacking standard definition, we chose to use the term drug seeking because it is used frequently in clinical practice and in the medical literature. Every physician has a different understanding of what constitutes drug-seeking behavior. Because our research objective was to learn how EPs assess prescription history in patients presenting with a painful condition, we believed that using this term accurately reflects what occurs in clinical practice. Additionally, our research aim was to determine whether EPs consider a prescription history useful rather than to identify how this information affects prescribing practice. As such, we did not ask our respondents what medication they would prescribe in each case, or if the prescription history would make them more or less likely to prescribe opiate medications.
We created the survey using online survey software (SurveyMonkey, www.surveymonkey.com; Palo Alto, CA; 2011). It was available for responses for a one-month period. We intended for the survey to take less than five minutes, and respondents were given this information. There was no financial incentive for participating, and no penalty for not participating.
We sent out a link to the online survey via recruitment e-mail to all resident physicians in the Emergency Medicine (EM) residency affiliated with an academic West Coast Medical Center. We sent the same recruitment e-mail to all attending EPs working more than two shifts per month at this same Medical Center. One reminder e-mail with a link to the online survey was sent two weeks after the initial recruitment e-mail.
… we chose to use the term drug seeking because it is used frequently in clinical practice and in the medical literature. Every physician has a different understanding of what constitutes drug-seeking behavior.
Data Collection and Analysis
We collected data using SurveyMonkey online survey software and imported the raw data from the online software to a computer spreadsheet (Excel 2007, Microsoft, Redmond, WA) for review and analysis.
For data analysis, we chose Case 1 as the index case, because it represented what we perceived to be a prescription history for a patient receiving regular care from a primary care physician or pain management physician. We then made individual comparisons between the physician responses for each of the remaining cases and Case 1, to isolate the effect of changes in each of the 3 variables (number of prescriptions per month, number of prescribing physicians in the record, and strength of the narcotic). For example, Case 1 and Case 2 differ in that Case 1 involved 3 prescriptions per month, whereas Case 2 involved only 1.5 prescriptions per month. Comparing Cases 1 and 2 allowed us to analyze how a change in the number of prescriptions per month affected physician assessment. Each comparison consisted of comparing the average response value for Case 1 to the average response value for each of the remaining cases. For reference, a summary of the prescription history in each case is provided in Table 1. Because categorical data were collected, p values for each comparison were calculated using the Wilcoxon rank sum test. To assess the interobserver reliability within our responses, we calculated the Fleiss κ values. For each scenario, we calculated the κ values to assess the likelihood of drug seeking.
Statistical analysis was performed using Microsoft Excel 2007 and open-source online statistical software (Statcato 0.9.10).
Results
We received responses from 30 attending EPs out of 48 attending EPs who worked at this Medical Center, and from 29 EM residents of 36 EM residents in the program, giving us a total response rate of 70%. The response rate for residents was 81%, and that for attendings was 63%. Of the resident respondents, 28% were Postgraduate Year (PGY) 3, 38% were PGY 2, and 34% were PGY 1.
The first question for each case asked, “How likely do you think this patient is drug seeking?” The average response score for the presentation in the index case (3 prescriptions per month, 1 prescribing physician, and a low-potency narcotic) was 3.1 of 4. A decrease in the number of prescriptions per month to 1.5 in Case 2 decreased the average response to 2.4 of 4 (p < 0.001), whereas an increase in the number of prescriptions per month to 6 in Case 3 increased the average response to 3.5 of 4 (p = 0.02). Increasing the number of prescribing physicians per month to 6 in Case 5 increased the average response to 3.7 of 4 (p < 0.001). A summary of the average response scores and p values for all cases is in Table 2.
Table 2.
The second question for each case asked, “How much would having the information in the prescription record change your prescribing practice?” The average response score for the presentation in the index case was 3.4 of 4. Decreasing the number of prescriptions per month to 1.5 in Case 2 decreased the average response to 3.0 of 4 (p < 0.001). A summary of the average response scores and p values for all cases is in Table 3.
Table 3.
All κ values of the assessment by our respondents as to whether a patient was drug seeking were between 0.40 and 0.57. Our respondents demonstrated only fair to moderate agreement in their assessments of these cases. A complete list of the κ values for the assessment of drug seeking for each case, as well as reference values for the interpretation of κ values,28 are in Table 4.
Table 4.
To assess if a difference existed in the assessment of drug seeking between resident and attending physicians, we calculated the average suspicion for drug seeking for all cases for both resident and attending physicians. For the 6 cases, the average suspicion for drug seeking by resident physicians was 3.16 of 4.0, whereas the average suspicion for drug seeking by attending physicians was 3.12 of 4.0 (p = 0.86).
Discussion
In our anonymous online survey of resident and attending EPs at a single academic Medical Center, we found that a greater number of prescriptions per month and a greater number of prescribing physicians per month increased physician suspicion for drug-seeking behavior. On the basis of the data in Table 2, the scenarios in which our respondents most suspected drug-seeking behavior were Cases 3 (6 prescriptions per month) and 5 (6 prescribing physicians in a 2-month period). In both cases, our respondents predicted the likelihood of drug seeking greater than or equal to 3.5 of 4. This finding suggests that our respondents believed that the patient in these cases was very likely to be drug seeking. A greater number of prescribing physicians in the prescription history increased suspicion for drug-seeking behavior slightly more than did a greater number of prescriptions per month. Despite our study's limitations, these data suggest that 6 or more prescriptions per month and 6 or more prescribing physicians on a prescription history should raise concern that a patient presenting to the ED is trying to obtain prescription medication for nontherapeutic reasons.
We decided a priori not to assess in what fashion the information in a prescription history affects prescribing behavior. Rather, our survey determined that EPs believed that access to prescription records was helpful and would change their prescribing practice when assessing patients presenting to the ED with pain in all of our fictitious cases. Therefore, whether EPs should access a patient's prescription history each time a patient presents to the ED with an acute painful episode is worth considering. Although it may take a few extra moments on a busy shift, routinely accessing such information (if available) may be a powerful tool to screen for patients with concerning narcotic use.
Pseudoaddiction has been defined as an iatrogenic condition resulting from inadequate pain management, in which patients exhibit drug-seeking behaviors to obtain medication to relieve their pain.8 When patients receive adequate analgesia, these behaviors typically cease. The behaviors exhibited by patients attempting to obtain prescription medication who suffer from pseudoaddiction are nearly identical to those suffering from addiction, making it extremely difficult to differentiate between the two conditions. These conditions are even more difficult to differentiate in the acute care setting.9 Our survey did not attempt to differentiate between addiction and pseudoaddiction. Instead, we focused our study on drug-seeking behavior regardless of the cause, because we felt that this more likely reflects what is encountered in the ED.
The assessment of whether a patient is drug seeking has been completely subjective before the development of prescription monitoring programs. It would seem that the provision of an objective prescription history would allow physicians to better determine whether a patient is drug seeking. However, our survey demonstrated that our respondents had only moderate agreement in this assessment. It is unclear exactly why the interobserver reliability of this assessment was so poor. Although this may be the result of variation in how EPs choose to assess pain in their clinical practice or variable clinical experience with drug-seeking patients, it is possible that some of our respondents did not have enough experience using a prescription history. However, our comparison of the assessment of drug-seeking behavior between attending and resident physicians did not show a significant difference. This lack of experience appeared more likely to be related to overall inexperience with prescription monitoring programs rather than inexperience related to level of training. Further experience with prescription monitoring programs and research on drug-seeking behavior may improve EP use and interpretation of such programs in the future.
Limitations
Our study had several limitations. First, our survey had a limited number of respondents and was confined to one academic Medical Center, which limits the generalizability of these data. Our survey consisted of physicians of varying experience levels. Nearly 50% of our respondents were EM residents, which may limit the ability of our data to be generalized to the general EP population as well. Another limitation is that our cases were simulated cases only. Although we found statistically significant differences in EP concern for drug seeking based on differences in a patient's prescription history, these differences may not be clinically significant, because seeing a patient in person is very different from reading a case presentation on a computer. Additionally, the use of nonvalidated clinical vignettes may be a source of potential bias. Furthermore, the initial case presentation of a young man with chronic back pain presenting to the ED requesting narcotic pain medication may be a source of bias, as it is by itself concerning for drug-seeking behavior.
We also looked at physician suspicion of whether a patient is drug seeking and whether physicians thought the information in the prescription record would be helpful; we did not survey our respondents to see what they would actually prescribe to the patient in each case. Furthermore, asking our respondents whether they thought a patient was drug seeking might have increased their suspicion in all cases for drug-seeking behavior. Along this same line, both authors have been involved with research on drug-seeking behavior in the past, and perhaps knowing that the research study was being performed by our research group increased suspicion for drug-seeking behavior in all cases.
None of our survey cases included a patient who had zero previous narcotic prescriptions, which might have provided additional information on how prescription histories are interpreted. Finally, the term drug seeking lacks a uniform definition, which might lead to inconsistencies related to a physician's response to an online, anonymous survey.
Conclusion
In our anonymous survey of 6 fictitious cases of a 35-year-old man presenting to the ED requesting narcotics for low back pain, a greater number of prescriptions per month and a greater number of prescribing physicians listed on the prescription record increased EP suspicion for drug-seeking behavior. In addition, our survey demonstrated EPs believed that access to narcotic prescription history was likely to change their prescribing behavior. However, there was only moderate interobserver reliability in the EP assessment of whether patients were demonstrating drug-seeking behavior.
Disclosure Statement
The author(s) have no conflicts of interest to disclose.
Acknowledgments
Kathleen Louden, ELS, of Louden Health Communications provided editorial assistance.
References
- 1.Pletcher MJ, Kertesz SG, Kohn MA, Gonzales R. Trends in opioid prescribing by race/ethnicity for patients seeking care in US emergency departments. JAMA. 2008 Jan 2;299(1):70–8. doi: 10.1001/jama.2007.64. DOI: http://dx.doi.org/10.1001/jama.2007.64. [DOI] [PubMed] [Google Scholar]
- 2.O'Connor AB, Zwemer FL, Hays DP, Feng C. Outcomes after intravenous opioids in emergency patients: a prospective cohort analysis. Acad Emerg Med. 2009 Jun;16(6):477–87. doi: 10.1111/j.1553-2712.2009.00405.x. DOI: http://dx.doi.org/10.1111/j.1553-2712.2009.00405.x. [DOI] [PubMed] [Google Scholar]
- 3.Rupp T, Delaney KA. Inadequate analgesia in emergency medicine. Ann Emerg Med. 2004 Apr;43(4):494–503. doi: 10.1016/j.annemergmed.2003.11.019. DOI: http://dx.doi.org/10.1016/j.annemergmed.2003.11.019. [DOI] [PubMed] [Google Scholar]
- 4.Manchikanti L, Singh A. Therapeutic opioids: a ten-year perspective on the complexities and complications of the escalating use, abuse, and nonmedical use of opioids. Pain Physician. 2008 Mar;11(2 Suppl):S63–88. [PubMed] [Google Scholar]
- 5.Rockville, MD: Substance Abuse and Mental Health Services Administration; 2010 Sep. Results from the 2009 national survey on drug use and health: Volume I. Summary of national findings. HHS Publication No. SMA 10-4586 Findings. [Google Scholar]
- 6.Drug Abuse Warning Network, 2006: National estimates of drug-related emergency department visits. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2008 Aug. DHHS Publication No. SMA 08-4339. [PubMed] [Google Scholar]
- 7.Webster LR, Cochella S, Dasgupta N, et al. An analysis of the root causes for opioid-related overdose deaths in the United States. Pain Med. 2011 Jun;12(Suppl 2):S26–35. doi: 10.1111/j.1526-4637.2011.01134.x. DOI: http://dx.doi.org/10.1111/j.1526-4637.2011.01134.x. [DOI] [PubMed] [Google Scholar]
- 8.Centers for Disease Control and Prevention (CDC). Vital signs: overdoses of prescription opioid pain relievers— United States, 1999–2008. MMWR Morb Mortal Wkly Rep. 2011 Nov 4;60(43):1487–92. [PubMed] [Google Scholar]
- 9.Hansen GR. The drug-seeking patient in the emergency room. Emerg Med Clin North Am. 2005 May;23(2):349–65. doi: 10.1016/j.emc.2004.12.006. DOI: http://dx.doi.org/10.1016/j.emc.2004.12.006. [DOI] [PubMed] [Google Scholar]
- 10.Hawkins SC, Smeeks F, Hamel J. Emergency management of chronic pain and drug-seeking behavior: an alternative perspective. J Emerg Med. 2008 Feb;34(2):125–9. doi: 10.1016/j.jemermed.2007.07.001. DOI: http://dx.doi.org/10.1016/j.jemermed.2007.07.001. [DOI] [PubMed] [Google Scholar]
- 11.McNabb C, Foot C, Ting J, Breeze K, Stickley M. Diagnosing drug-seeking behaviour in an adult emergency department. Emerg Med Australas. 2006 Apr;18(2):138–42. doi: 10.1111/j.1742-6723.2006.00821.x. DOI: http://dx.doi.org/10.1111/j.1742-6723.2006.00821.x. [DOI] [PubMed] [Google Scholar]
- 12.Zechnich AD, Hedges JR. Community-wide emergency department visits by patients suspected of drug-seeking behavior. Acad Emerg Med. 1996 Apr;3(4):312–7. doi: 10.1111/j.1553-2712.1996.tb03443.x. DOI: http://dx.doi.org/10.1111/j.1553-2712.1996.tb03443.x. [DOI] [PubMed] [Google Scholar]
- 13.Carlson MJ, Baker LH. Difficult, dangerous, and drug seeking: the 3D way to better patient care. Am J Public Health. 1998 Aug;88(8):1250–2. [PubMed] [Google Scholar]
- 14.McNabb C, Foot C, Ting J, Breeze K, Stickley M. Profiling patients suspected of drug seeking in an adult emergency department. Emerg Med Australas. 2006 Apr;18(2):131–7. doi: 10.1111/j.1742-6723.2006.00820.x. DOI: http://dx.doi.org/10.1111/j.1742-6723.2006.00820.x. [DOI] [PubMed] [Google Scholar]
- 15.Chan L, Winegard B. Attributes and behaviors associated with opioid seeking in the emergency department. J Opioid Manag. 2007 Sep–Oct;3(5):244–8. doi: 10.5055/jom.2007.0011. [DOI] [PubMed] [Google Scholar]
- 16.Gianutsos LP, Safranek S, Huber T. Clinical inquiries: is there a well-tested tool to detect drug-seeking behaviors in chronic pain patients? J Fam Pract. 2008 Sep;57(9):609–10. [PubMed] [Google Scholar]
- 17.Chou R, Fanciullo GJ, Fine PG, Miaskowski C, Passik SD, Portenoy RK. Opioids for chronic noncancer pain: prediction and identification of aberrant drug-related behaviors: a review of the evidence for an American Pain Society and American Academy of Pain Medicine clinical practice guideline. J Pain. 2009 Feb;10(2):131–46. doi: 10.1016/j.jpain.2008.10.009. DOI: http://dx.doi.org/10.1016/j.jpain.2008.10.009. [DOI] [PubMed] [Google Scholar]
- 18.Moore TM, Jones T, Browder JH, Daffron S, Passik SD. A comparison of common screening methods for predicting aberrant drug-related behavior among patients receiving opioids for chronic pain management. Pain Med. 2009 Nov;10(8):1426–33. doi: 10.1111/j.1526-4637.2009.00743.x. DOI: http://dx.doi.org/10.1111/j.1526-4637.2009.00743.x. [DOI] [PubMed] [Google Scholar]
- 19.Wu SM, Compton P, Bolus R, et al. The addiction behaviors checklist: validation of a new clinician-based measure of inappropriate opioid use in chronic pain. J Pain Symptom Manage. 2006 Oct;32(4):342–51. doi: 10.1016/j.jpainsymman.2006.05.010. DOI: http://dx.doi.org/10.1016/j.jpainsymman.2006.05.010. [DOI] [PubMed] [Google Scholar]
- 20.Butler SF, Budman SH, Fernandez KC, et al. Development and validation of the Current Opioid Misuse Measure. Pain. 2007 Jul;130(1–2):144–156. doi: 10.1016/j.pain.2007.01.014. DOI: http://dx.doi.org/10.1016/j.pain.2007.01.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Butler SF, Fernandez K, Benoit C, Budman SH, Jamison RN. Validation of the revised Screener and Opioid Assessment for Patients with Pain (SOAPP-R) J Pain. 2008 Apr;9(4):360–72. doi: 10.1016/j.jpain.2007.11.014. DOI: http://dx.doi.org/10.1016/j.jpain.2007.11.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Belgrade MJ, Schamber CS, Lindgren BR. The DIRE score: predicting outcomes of opioid prescribing for chronic pain. J Pain. 2006 Sep;7(9):671–81. doi: 10.1016/j.jpain.2006.03.001. DOI: http://dx.doi.org/10.1016/j.jpain.2006.03.001. [DOI] [PubMed] [Google Scholar]
- 23.Gilson AM, Kreis PG. The burden of the nonmedical use of prescription opioid analgesics. Pain Med. 2009 Jul;10(Suppl 2):S89–100. doi: 10.1111/j.1526-4637.2009.00668.x. DOI: http://dx.doi.org/10.1111/j.1526-4637.2009.00668.x. [DOI] [PubMed] [Google Scholar]
- 24.Barrett K, Watson A. Physician perspectives on a pilot prescription monitoring program. J Pain Palliat Care Pharmacother. 2005;19(3):5–13. DOI: http://dx.doi.org/10.1300/J354v19n03_03. [PubMed] [Google Scholar]
- 25.Baehren DF, Marco CA, Droz DE, Sinha S, Callan EM, Akpunonu P. A statewide prescription monitoring program affects emergency department prescribing behaviors. Ann Emerg Med. 2010 Jul;56(1):19–23.e1–3. doi: 10.1016/j.annemergmed.2009.12.011. DOI: http://dx.doi.org/10.1016/j.annemergmed.2009.12.011. [DOI] [PubMed] [Google Scholar]
- 26.Yokell MA, Green TC, Rich JD. Prescription drug monitoring programs. JAMA. 2012 Mar 7;307(9):912. doi: 10.1001/jama.307.9.912-b. DOI: http://dx.doi.org/10.1001/jama.307.9.912-b. [DOI] [PubMed] [Google Scholar]
- 27.Grover CA, Garmel GM. How do emergency physicians interpret prescription narcotic history when assessing patients presenting to the emergency department with pain? [abstract] Ann Emerg Med. 2011 Oct;58(4 Suppl):S319–20. doi: 10.7812/tpp/12-038. DOI: http://dx.doi.org/10.1016/j.annemergmed.2011.06.450. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005 Mar;85(3):257–68. [PubMed] [Google Scholar]