ABSTRACT
BACKGROUND
Patient hand-offs at physician shift changes have limited ability to convey the primary team’s longitudinal insight. The Patient Acuity Rating (PAR) is a previously validated, 7-point scale that quantifies physician judgment of patient stability, where a higher score indicates a greater risk of clinical deterioration. Its impact on cross-covering physician understanding of patients is not known.
OBJECTIVE
To determine PAR contribution to sign-outs.
DESIGN
Cross-sectional survey.
SUBJECTS
Intern physicians at a university teaching hospital.
INTERVENTIONS
Subjects were surveyed using randomly chosen, de-identified patient sign-outs, previously assigned PAR scores by their primary teams. For each sign-out, subjects assigned a PAR score, then responded to hypothetical cross-cover scenarios before and after being informed of the primary team’s PAR.
MAIN MEASURE
Changes in intern assessment of the scenario before and after being informed of the primary team’s PAR were measured. In addition, responses between novice and experienced interns were compared.
KEY RESULTS
Between May and July 2008, 23 of 39 (59 %) experienced interns and 25 of 42 (60 %) novice interns responded to 480 patient scenarios from ten distinct sign-outs. The mean PAR score assigned by subjects was 4.2 ± 1.6 vs. 3.8 ± 1.8 by the primary teams (p < 0.001). After viewing the primary team’s PAR score, interns changed their level of concern in 47.9 % of cases, their assessment of the importance of immediate bedside evaluation in 48.7 % of cases, and confidence in their assessment in 43.2 % of cases. For all three assessments, novice interns changed their responses more frequently than experienced interns (p = 0.03, 0.009, and <0.001, respectively). Overall interns reported the PAR score to be theoretically helpful in 70.8 % of the cases, but this was more pronounced in novice interns (81.2 % vs 59.6 %, p < 0.001).
CONCLUSIONS
The PAR adds valuable information to sign-outs that could impact cross-cover decision-making and potentially benefit patients. However, correct training in its use may be required to avoid unintended consequences.
Electronic supplementary material
The online version of this article (doi:10.1007/s11606-012-2257-4) contains supplementary material, which is available to authorized users.
KEY WORDS: hospital medicine, medical education assessment methods, communication
INTRODUCTION
By nature of 24 h care, inpatients must be cross-covered by alternate physicians, an error-prone process that requires extensive communication.1,2 Unfortunately, miscommunication is a well-established source of medical errors, especially during inhospital handoffs.3,4 Numerous problems have been identified in written physician sign-outs, including lack of critical information and variability in use and format within departments.4–9 Furthermore, poor sign-outs have been associated with uncertainty during medical decision making and potential patient harm.5,10 In addition, during oral handoffs, even when senders believe the information is transmitted, receivers are often unable to discern critical information necessary for patient care.11 Recommendations to improve handoffs focus on including some level of anticipatory guidance, particularly related to how ill the patient is.5,12 This is particularly important in light of the recent Accreditation Council for Graduate Medical Education (ACGME) requirements for decreased resident duty hours, which inherently result in increased handoffs.13
Recent research suggests that nonanalytical reasoning strongly influences time-pressured complex clinical decisions.14 Moreover, intuitive clinician concern for patient stability has been shown to be a strong predictor of cardiac arrests on general inpatient wards.15 Thus, providing covering physicians with accurate intuition of patient stability may improve clinical decision-making by the covering physicians. One such potential tool is the Patient Acuity Rating (PAR), a quantitative summary of clinical judgment regarding patient risk.16 The PAR has been prospectively validated among physicians and midlevel practitioners to predict impending cardiac arrest and intensive care unit (ICU) transfer, with an area under the receiver operator curve of 0.82 (95 % CI 0.77, 0.87). However, it is not known if the PAR supplements or merely duplicates existing sign-out information.
Additionally, intuitive physician concern for patient stability has been previously linked with cardiac arrests. We hypothesized that the addition of the PAR to sign-outs would convey novel patient information and impact medical decision-making by physician trainees. We further hypothesized that the benefit of the PAR would vary by experience level.
METHODS
Design
We designed a survey study of interns at a university teaching hospital to assess the impact of the addition of PAR scores to a sign-out. The PAR score is a 7-point ordinal scale that measures provider intuition regarding the likelihood of a patient transferring to the ICU or suffering a cardiac arrest in the next 24 h. This measure was previously validated in a sample of 1,663 patients and shown to have an area under the receiver operating characteristic curve of 0.82 (95 % CI 0.77, 0.87) for predicting impending clinical deterioration.16 From this study of 2,095 written inpatient sign-outs,16 we identified 114 sign-outs that were given unanimous PAR scores by the interns, residents, and/or attending physicians of the respective primary teams. These sign-outs were stratified by PAR score, and ten total sign-outs were randomly chosen from the stratified sign-out groups to ensure variation in risk levels (see Online Appendix Figure 1). Sign-outs in this study were exactly as they were recorded electronically each day during patient care, and were considered updated for purposes of the study. We used a threshold of one point for clinical significance when analyzing a change in PAR score.
Intern physicians in the General Internal Medicine residency program at the University of Chicago completed a written survey in which they were asked to read each of the ten sign-outs and then answer a series of questions for each patient (Fig. 1). The survey was developed by expert consensus of the study team through multiple iterations and discussions, then pilot tested with attending physician colleagues. All study subjects had previously received formal training on using the PAR as part of their required sign-out curriculum. Additionally, an explanatory cover sheet was included with each survey that graphically and textually described the PAR. In the written survey, subjects were asked to assign a PAR and respond to a hypothetical cross-cover scenario involving an increase in heart rate from 89 to 111 bpm both before and after revealing the primary team’s PAR. Subjects then opened a sealed portion of the sign-out page to reveal the primary team’s PAR for each sign-out and were asked to respond to the same hypothetical scenario. Study subjects were also asked to score the helpfulness of the PAR for each patient and answer demographic questions about themselves.
Figure 1.
Survey tool. PAR, Patient Acuity Rating. *X represents the PAR score given by the primary team.
The written survey was administered by research personnel during dedicated lunch sessions to both “experienced” interns, who were surveyed during May and June 2008, and “novice” interns, who were surveyed during July 2008. Volunteers were recruited with flyers and general announcements. Responses to the scenarios were stratified by intern experience level and compared. All data were double-keyed, cross-compared, and corrected against the original questionnaires to ensure accurate data entry.
Statistical Analysis
Mixed-effects regression models were utilized to compare subject PAR scores with primary team PAR scores and to compare novice and experienced interns’ patient assessment and management. Fixed effect variables controlled for repeated sign-outs, intern experience level, and repeated primary team PAR scores, as appropriate, and a random effect variable controlled for multiple subject responses. Ratings of how helpful the PAR was to interns were analyzed with ordinal logistic regression with clustering at the subject level. Demographic data of subject groups was compared using chi-square and independent two-sided t-tests, as appropriate. All statistical analyses were performed using Stata/MP 11.0 (Stata Corp., College Station, TX), with a p-value of <0.05 deemed significant.
RESULTS
Twenty-three of 39 eligible experienced interns (59 %) and 25 of 42 eligible novice interns (60 %), volunteered to participate and provided a total of 480 evaluations. There were no significant differences between the groups with respect to gender, intended specialty or number of months worked in the ICU as a medical student (Table 1). As expected, experienced interns had significantly more ICU experience as an intern (1.5 ± 0.5 vs. 0.1 ± 0.2 months, p < 0.001).
Table 1.
Baseline Respondent Characteristics by Experience Level
| Characteristic | Novice interns n = 25 | Experienced interns n = 23 | p-value |
|---|---|---|---|
| Male gender, n (%) | 11 (46 %)* | 11 (48 %) | 0.89 |
| Intensive care months, mean ± SD | |||
| As a medical student | 0.7 ± 0.6 | 0.7 ± 0.6 | 0.93 |
| As an intern | 0.1 ± 0.2 | 1.5 ± 0.5 | <0.001 |
| Intended specialty, n (%) | 0.19 | ||
| Primary care† | 4 (16 %) | 4 (17 %) | |
| Medical sub-specialty‡ | 8 (32 %) | 13 (57 %) | |
| Nonmedical specialty§ | 5 (20 %) | 4 (17 %) | |
| Undecided/unknown | 8 (32 %) | 2 (9 %) | |
*Gender missing for one novice intern †Includes internal medicine, medicine-pediatrics, and geriatrics ‡Includes cardiology, gastroenterology, hematology/oncology, infectious disease, and pulmonary/critical care §Includes anesthesia, ophthalmology, dermatology, and radiology
The ten patient-day sign-outs selected included one each with a provider assigned PAR of 1, 4, 6, and 7, and two each with a PAR score of 2, 3, and 5, resulting in a mean provider PAR score of 3.8 ± 1.8 (see Online Appendix Table 1). The subjects rated the patients 0.4 ± 2.1 higher on the PAR scale than the primary providers did (p < 0.001). Subjects’ concern for the patient stability and the perceived importance of immediate bedside evaluation decreased after learning the provider PAR scores (−0.1 ± 0.9 on a scale of 1–5, p < 0.001 for both). In addition, subject confidence in their assessment and management increased after learning the provider PAR (0.2 ± 0.9, p = 0.004).
As demonstrated in Figure 2, subject concern for patient stability increased linearly when the subjects underestimated the patient acuity in comparison to the primary team, but decreased when they overestimated it (r = 0.74, p < 0.001). For example, when a subject gave a patient a PAR score of 3 and then saw the primary team’s PAR assignment was a 5, the subject was likely to report more concern about the patient, compared to the level of concern reported prior to seeing the primary team score. There was a similar proportional relationship for the change in perceived importance of immediate evaluation (r = 0.72, p < 0.001). Revealing the primary team PAR score led to differences in the responses to the assessment questions in novice interns (p < 0.001 for level of concern and p = 0.001 of importance of immediate evaluation), but not in experienced interns (Table 2).
Figure 2.
Change in level of concern for patient by discrepancy in PAR score. PAR, Patient Acuity Rating. The x-axis represents the difference between the subject assigned PAR score and the primary team assigned PAR score for a given sign-out. A positive number indicates that the primary team assigned a higher PAR score, and a negative number indicates that the subject assigned a higher PAR score. The y-axis represents the difference in the subject’s level of concern before and after revealing of the primary team assigned PAR score. A positive number indicates the subject increased their level of concern after revealing the primary team PAR score, while a negative number indicates a decrease in level of concern. The size of the circle represents the frequency weighted distribution, and the straight line is the linear best fit regression. R2 = 0.547.
Table 2.
Patient Assessment Pre and Post Provision of the Patient Acuity Rating
| Assessment | Novice evaluations n = 250 | Experienced evaluations n = 230 | Adjusted p-value |
|---|---|---|---|
| Concern for patient stability | |||
| Pre-PAR | 3.7 ± 1.0 | 3.4 ± 1.0 | 0.70 |
| Post-PAR | 3.5 ± 1.1 | 3.4 ± 1.0 | 0.75 |
| Importance of immediate bedside evaluation | |||
| Pre-PAR | 3.8 ± 1.0 | 3.5 ± 1.1 | 0.49 |
| Post-PAR | 3.6 ± 1.1 | 3.5 ± 1.1 | 0.81 |
| Confidence in assessment | |||
| Pre-PAR | 3.3 ± 1.0 | 3.8 ± 0.7 | 0.04 |
| Post-PAR | 3.6 ± 1.0 | 3.9 ± 0.7 | 0.02 |
PAR Patient Acuity Rating. Results are given as mean ± standard deviation using a five-point ordinal scale in which higher numbers indicate increased concern, importance, and confidence. Data were missing for two experienced intern evaluations and one novice intern confidence assessment
When stratified by intern experience level, interns assigned similar PAR scores (4.3 ± 1.6 vs. 4.0 ± 1.6, p = 0.07, for novice and experienced interns respectively). They also reported a similar concern for patient stability and importance of immediate bedside evaluation, both before and after revealing the primary team PAR (Table 2). Experienced interns, on the other hand, were significantly more confident in their assessments, both before and after seeing the provider PAR (p = 0.04 and 0.02, respectively).
Table 3 demonstrates the impact of revealing the provider PAR on the interns’ assessments. While both groups changed their level of assessment regarding patient stability, importance of immediate bedside evaluation, and confidence in their assessments after revealing the provider PAR, novice interns did so to a larger degree (p = 0.03, p = 0.009 and p < 0.001, respectively). Likewise, while 59.6 % of experienced interns found the PAR to be helpful, novice interns were much more likely to find the PAR helpful (81.2 % of cases, p < 0.001) (Fig. 3).
Table 3.
Frequency of Change in Assessment Associated with Provision of the Patient Acuity Rating
| Assessment | All interns n = 480 | Novice interns n = 250 | Experienced interns n = 230 | Adjusted p-value |
|---|---|---|---|---|
| Change in concern for patient’s stability | 229 (47.9 %) | 133 (52.2 %) | 96 (42.1 %) | 0.03 |
| Change in importance of immediate bedside evaluation | 233 (48.7 %) | 137 (54.8 %) | 96 (42.1 %) | 0.009 |
| Change in confidence regarding assessment | 206 (43.2 %) | 131 (52.6 %) | 75 (32.9 %) | <0.001 |
Results shown as n (%). Data were missing for two experienced intern evaluations and one novice intern confidence assessment
Figure 3.
Subjective evaluation of the Patient Acuity Rating by experience level.
DISCUSSION
We have demonstrated that the PAR score, when added to a written sign-out in a simulated scenario, changes intern perception of a cross-covered patient’s clinical stability, increases confidence in that assessment, and is generally deemed helpful in the care of that patient. An underestimation of the patient acuity leads to a proportional increase in the interns’ concern for patient stability and the importance of immediate bedside evaluation, while an overestimation results in a decrease. In addition, novice interns found the PAR more useful and were more likely to change their assessment compared to experienced interns.
Given these findings, it is important to consider why the PAR may be helpful to interns in considering care of cross-coverage patients. One of the obvious benefits of the PAR is the ability to synthesize intuitive clinical judgment into a single number, thus adding minimal bulk to the sign-out. Previous work has shown that written sign-outs can become cluttered with information that is not helpful to the receiver.5 Because of this, the key features of a sign-out are accuracy, currency, and limitation to only relevant or pertinent information.5 Also, as electronic health records become more available, the danger of copy and paste makes it possible that sign-outs may suffer from too much information, making it harder to tease out critical information.17,18 The PAR fits neatly within this paradigm. Thus, a standardized quantitative score that is easily incorporated into sign-outs and exported to providers may be very useful in prioritizing those patients that are sickest.
We found significant differences between end-of-year and beginning-of-year interns, wherein the PAR was more likely to result in a change in assessment and confidence, and be deemed helpful for the latter group. This is likely a product of the differential experience level in cross-covering patients between the groups. Prior work has shown that expert physicians rely on nonanalytic reasoning using pattern recognition much more than novice physicians in making diagnoses.19 It is possible that communicating the PAR, which conveys the nonanalytical reasoning ability of the primary team, eliminates some of this disparity.
Although the novice and experienced interns differed in their confidence levels, they did not appear to differ greatly in their initial patient assessments as measured by their PAR scores, which were both different from the provider-assigned PAR scores. Such an observation was not a primary goal of this study, but it brings to attention the possible need to formally evaluate existence of a disparity between observed and self-assessed skill that varies with level of training.
An additional interesting study observation was the directly proportional relationship between the response to the vignettes and the difference in initial judgment between the subject and primary team. Subjects who underestimated the primary team PAR were more likely to increase their level of concern and assessment of need for immediate bedside evaluation when the PAR was revealed. However, the opposite was true when they overestimated the primary team’s PAR. In fact, at a per-patient level, the variation in their responses mostly decreased or remained the same after revealing the primary team assigned PAR score (see Online Appendix Table 2). Importantly, special consideration of this point during physician training may be necessary to avoid the unintended consequence of overconfidence in a patient’s stability since a patient’s condition can change at any time. Moreover, since a known hazard of sign-outs is that they are not routinely updated and interns may rely even more heavily on a PAR, it is crucial to ensure that the PAR is updated, and ideally a date and time stamp should accompany the PAR.
There are several potential limitations to this study. First, there may be a sample bias in the interns who chose to participate. While the overall participation rate was 59 %, it is possible that the subject pool was not representative of the intern classes as a whole. Furthermore, since it is a single center study, the results may not be generalizable to other institutions or non-intern providers. Also, though the PAR scores are up to date, the sign-outs may not be. Finally, although the use of vignettes followed by theoretical questions to predict actions has been validated in several applications,20,21 it is not yet clear how these results translate into clinical practice. Future research is needed to investigate if PAR is helpful in actual clinical settings. In addition, due to the non-linear nature of the PAR in practice16 and the variability in clinical stability of the patients represented in the study, it is difficult to set an optimal threshold for clinical significance. As a result, we cannot rule out the possibility that some of our results may not be clinically significant, despite being statistically significant.
In summary, the PAR is a useful and brief, structured way to communicate patient acuity on a typical resident sign-out. This tool may be particularly helpful for inexperienced interns charged with cross-covering patients.
Electronic Supplementary Material
(PDF 68 kb)
Acknowledgments
Contributors
We would like to thank Elizabeth Weidman and Jefferson Cua for their help in data collection and entry, Seo Young Park, PhD for statistical support, and Cara V. Phillips for graphics support.
Funders
This study was supported by the National Heart, Lung, and Blood Institute (K23 HL097157-01; PI: Edelson) and by a National Institute of Aging (5 T35 AG029795-02) grant for the Pritzker Summer Research Program.
Registry Number
clinicaltrials.gov NCT01082991
Prior Presentations
Prior poster presentation at the 2009 Society of Hospital Medicine Annual Meeting in Chicago, IL.
Conflict of Interest
Dr. Arora has received research support from Picker Institute (Grant #86) and National Institute of Aging (K23 AG033763), and is the on editorial board of Agency for Healthcare Research and Quality Web M&M. Dr. Edelson has received research support from the National Heart, Lung, and Blood Institute (K23 HL097157-01; PI: Edelson). Dr. Philips, Mr. Yuen, Dr. Retzer, and Dr. Woodruff declare they have no conflict of interests
REFERENCES
- 1.Petersen LA, Brennan TA, O’Neil AC, Cook EF, Lee TH. Does housestaff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med. 1994;121(11):866–72. doi: 10.7326/0003-4819-121-11-199412010-00008. [DOI] [PubMed] [Google Scholar]
- 2.Vidyarthi AR. Fumbled handoff. AHRQ Web M & M. March 2004. Available at: http://webmm.ahrq.gov/case.aspx?caseID=55. Accessed: October 4, 2012.
- 3.Singh H, Thomas EJ, Petersen LA, Studdert DM. Medical errors involving trainees: a study of closed malpractice claims from 5 insurers. Arch Intern Med. 2007;167(19):2030–6. doi: 10.1001/archinte.167.19.2030. [DOI] [PubMed] [Google Scholar]
- 4.Sutcliffe KM, Lewton E, Rosenthal MM. Communication failures: an insidious contributor to medical mishaps. Acad Med. 2004;79(2):186–94. doi: 10.1097/00001888-200402000-00019. [DOI] [PubMed] [Google Scholar]
- 5.Arora V, Johnson J, Lovinger D, Humphrey HJ, Meltzer DO. Communication failures in patient sign-out and suggestions for improvement: a critical incident analysis. Qual Saf Health Care. 2005;14(6):401–7. doi: 10.1136/qshc.2005.015107. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Borowitz SM, Waggoner-Fountain LA, Bass EJ, Sledd RM. Adequacy of information transferred at resident sign-out (in-hospital handover of care): a prospective survey. Qual Saf Health Care. 2008;17(1):6–10. doi: 10.1136/qshc.2006.019273. [DOI] [PubMed] [Google Scholar]
- 7.Horwitz LI, Krumholz HM, Green ML, Huot SJ. Transfers of patient care between house staff on internal medicine wards: a national survey. Arch Intern Med. 2006;166(11):1173–7. doi: 10.1001/archinte.166.11.1173. [DOI] [PubMed] [Google Scholar]
- 8.Solet DJ, Norvell JM, Rutan GH, Frankel RM. Lost in translation: challenges and opportunities in physician-to-physician communication during patient handoffs. Acad Med. 2005;80(12):1094–9. doi: 10.1097/00001888-200512000-00005. [DOI] [PubMed] [Google Scholar]
- 9.Vidyarthi AR, Arora V, Schnipper JL, Wall SD, Wachter RM. Managing discontinuity in academic medical centers: strategies for a safe and effective resident sign-out. J Hosp Med. 2006;1(4):257–66. doi: 10.1002/jhm.103. [DOI] [PubMed] [Google Scholar]
- 10.Horwitz LI, Moin T, Krumholz HM, Wang L, Bradley EH. Consequences of inadequate sign-out for patient care. Arch Intern Med. 2008;168(16):1755–60. doi: 10.1001/archinte.168.16.1755. [DOI] [PubMed] [Google Scholar]
- 11.Chang VY, Arora VM, Lev-Ari S, D’Arcy M, Keysar B. Interns overestimate the effectiveness of their hand-off communication. Pediatrics. 2010;125(3):491–6. doi: 10.1542/peds.2009-0351. [DOI] [PubMed] [Google Scholar]
- 12.Horwitz LI, Moin T, Green ML. Development and implementation of an oral sign-out skills curriculum. J Gen Intern Med. 2007;22(10):1470–4. doi: 10.1007/s11606-007-0331-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Accreditation Council for Graduate Medical Education (ACGME). Task Force on Quality Care and Professionalism. Proposed Standards: Common Program Requirements. 2010. Available at: http://www.acgme-2010standards.org/pdf/Common_Program_Requirements_07012011.pdf. Accessed: October 4, 2012.
- 14.Byrne A. Mental workload as a key factor in clinical decision making. Adv Health Sci Educ Theory Pract. 2012. doi:10.1007/s10459-012-9360-5. [DOI] [PubMed]
- 15.Hodgetts TJ, Kenward G, Vlachonikolis IG, Payne S, Castle N. The identification of risk factors for cardiac arrest and formulation of activation criteria to alert a medical emergency team. Resuscitation. 2012;54(2):125–31. doi: 10.1016/S0300-9572(02)00100-4. [DOI] [PubMed] [Google Scholar]
- 16.Edelson DP, Retzer E, Weidman EK, Woodruff J, Davis AM, Minsky BD, et al. Patient acuity rating: quantifying clinical judgment regarding inpatient stability. J Hosp Med. 2011;6(8):475–9. doi: 10.1002/jhm.886. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Hirschtick RE. A piece of my mind. Copy-and-paste. JAMA. 2006;295(20):2335–6. doi: 10.1001/jama.295.20.2335. [DOI] [PubMed] [Google Scholar]
- 18.Wohlauer MV, Arora VM, Horwitz LI, Bass EJ, Mahar SE, Philibert I. The patient handoff: a comprehensive curricular blueprint for resident education to improve continuity of care. Acad Med. 2012;87(4):411–8. doi: 10.1097/ACM.0b013e318248e766. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Norman G, Young M, Brooks L. Non-analytical models of clinical reasoning: the role of experience. Med Educ. 2007;41(12):1140–5. doi: 10.1111/j.1365-2923.2007.02914.x. [DOI] [PubMed] [Google Scholar]
- 20.Bachmann LM, Muhleisen A, Bock A, ter Riet G, Held U, Kessels AG. Vignette studies of medical choice and judgment to study caregivers’ medical decision behaviour: systematic review. BMC Med Res Methodol. 2008;8:50. doi: 10.1186/1471-2288-8-50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Cooksey RW. Judgment Analysis: theory, Methods, and Applications. San Diego: Academic; 1996. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
(PDF 68 kb)