Summary
Background
Patient and surgical case complexity are important considerations in creating appropriate clinical assignments for trainees in the operating room (OR). The American Society of Anesthesiologists (ASA) Physical Status Classification System is the most commonly used tool to classify patient illness severity, but it requires manual evaluation by a clinician and is highly variable. A Risk Stratification System for surgical patients was recently published which uses administrative billing codes to calculate four Risk Stratification Indices (RSIs) and provides an objective surrogate for patient complexity that does not require clinical evaluation. This risk score could be helpful when assigning operating room cases.
Objectives
This is a technical feasibility study to evaluate the process and potential utility of incorporating an automatic risk score calculation into a web-based tool for assigning OR cases.
Methods
We created a web service implementation of the RSI model for one-year mortality and automatically calculated the RSI values for patients scheduled to undergo an operation the following day. An analysis was conducted on data availability for the RSI model and the correlation between RSI values and ASA physical status.
Results
In a retrospective analysis of 46,740 patients who received surgery in the year preceding the web tool implementation, RSI values were generated for 20,638 patients (44%). The Spearman’s rank correlation coefficient between ASA physical status classification and one-year mortality RSI values was 0.404.
Conclusion
We have shown that it is possible to create a web-based tool that uses existing billing data to automatically calculate risk scores for patients scheduled to undergo surgery. Such a risk scoring system could be used to match patient acuity to physician experience, and to provide improved patient and clinician experiences. The web tool could be improved by expanding the input database or utilizing procedure booking codes rather than billing data.
Key words: Risk stratification index, RSI, clinical assignments, operating room scheduling, patient acuity, clinical decision support
1. Introduction
Matching patient complexity with human resources is an important but challenging aspect of hospital management. Studies on the effects of human resource allocation have primarily focused on the benefits of increased clinician-to-patient ratios [3, 17]. Comparatively little information is available about efforts to match case complexity with clinicians’ skill sets when creating clinical assignments [4, 21, 24].
Previous research has focused on matching patient acuity with healthcare workers’ abilities and assignments in a variety of fields: rehabilitation nursing [9], case management [2], pharmacies [15], emergency departments [7], and the perianesthesia environment [6]. Although patient complexity is recognized as an important variable in human resource workload calculations [18], efforts to incorporate this factor into clinical assignments have been limited by difficulties quantifying case complexity [11]. Currently the most widely used perioperative patient illness scoring system is The American Society of Anesthesia (ASA) Physical Status Classification System. After evaluation by an anesthesiologist patients are assigned a score between 1, which represents a normal healthy patient, and 5, which represents a moribound patient not expected to survive without the operation [1]. The ASA Physical Status Classification System is widely used but cases must be manually reviewed by an clinician, which is time consuming, subjective, and highly variable [12]. Attempts have been made to adapt existing tools such as the Therapeutic Intervention Scoring System-28 [8] or to develop new metrics with site-specific analyses based on direct observation [14]. However, these tools have narrow scope or require significant time for data collection.
An ideal tool for calculating case complexity would be readily available, broadly applicable, and easily automated. A Risk Stratification System (RSS) for surgical patients was recently published which uses administrative billing codes to calculate four Risk Stratification Indices (RSIs) – length of stay, in-hospital mortality, 30-day mortality, and one-year mortality – and provides a surrogate for patient complexity [22]. The RSS was derived from Medicare Provider Analysis and Review (MEDPAR) data from 2001-2006 by analyzing the aggregate risk associated with procedure and diagnostic International Classification of Diseases 9 (ICD-9) billing codes. The most influential codes for the short term models tended to reflect acute medical conditions, while the one-year mortality model was influenced more heavily by underlying patient complexity from chronic medical conditions.
We created a web service implementation of the RSI model for one-year mortality and incorporated it into a local clinical decision support tool for resident clinical assignments, Rhinos (Residents Helping In Navigating OR Scheduling) [25], in order to provide automated assessment of underlying patient complexity when assigning OR cases.
2. Methods
2.1 Study Population
The Rhinos and RSS models use patient data from a variety of sources, as shown in ► Figure 1. The diagnostic and procedure billing code input data were extracted from a database of inpatient encounters at Massachusetts General Hospital from 1999 to 2012. This database is repopulated and updated monthly. The list of patients scheduled to receive operations is obtained daily from our electronic OR scheduling system. Both data sources are accessed daily to compile the RSI values.
Fig. 1.
RHIINOS and RSI web service data availability and flow
2.2 Web Service
The original RSS model was written in SPSS (IBM, Armonk, NY) and published online for general use. We created a web service implementation of the one-year mortality RSI model using Visual Basic .NET and SQL Server (Microsoft, Redmond, WA). The web service was validated against the original SPSS model using an example patient set of 1,020 hypothetical patients published by Sessler et al [23]. The RSI values of the original SPSS model matched the web service values for each of the 1,020 patients in the example patient set. Additional error-checking was successfully conducted with 50 unique sets of ICD-9 billing codes to ensure model congruity.
A process was established to automatically calculate the RSI values for patients scheduled to undergo an operation the following day. The web implementation accepts the first 10 diagnostic codes and 6 procedure codes from the billing database in order to mimic the data set on which the original RSS model was validated. In the web service, as in the original SPSS model, a list of up to 16 patient ICD-9 billing codes are collected for each patient and compared against an established database of ICD-9 codes and associated hazard ratios. The patient’s hazard ratios are aggregated to form an overall risk score for one-year mortality. The RSI model for one-year mortality was chosen because it included hazard ratio data for predominately chronic conditions, appropriately reflecting underlying patient complexity. ► Table 1 shows the ICD-9 codes with the five highest and lowest one-year mortality hazard ratios. The one-year mortality RSI value is displayed alongside additional patient information for use by residents and preceptors when selecting case assignments.
Table 1.
ICD-9 codes with the five largest and smallest associated hazard ratios for one-year mortality.
Hazard Ratios | ICD-9 Code | Description | Coefficient (Beta) | Covariate Mean | Hazard Ratio [95% CI] |
---|---|---|---|---|---|
5 smallest | P605 | Radical prostatectomy | -2.688 | 0.003 | 0.068 [0.061 0.075] |
P8154 | Total knee replacement | -2.017 | 0.037 | 0.133 [0.129 0.136] | |
P8180 | Total shoulder replacement | -1.778 | 0.001 | 0.169 [0.152 0.188] | |
P8363 | Rotator cuff repair | -1.619 | 0.003 | 0.198 [0.184 0.212] | |
P7050 | Repair of cystocele and rectocele | -1.590 | 0.004 | 0.204 [0.186 0.223] | |
5 largest | D1579 | Malignant neoplasm of pancreas, part unspecified | 1.155 | 0.001 | 3.174 [3.124 3.224] |
D155 | Malignant neoplasm of liver and intrahepatic bile ducts | 1.173 | 0.002 | 3.231 [3.178 3.285] | |
D163 | Malignant neoplasm of pleura | 1.259 | 0.000 | 3.521 [3.388 3.658] | |
D20500 | Acute myeloid leukemia, without mention of having achieved remission | 1.448 | 0.002 | 4.256 [4.186 4.328] | |
D191 | Malignant neoplasm of brain | 1.671 | 0.001 | 5.315 [5.217 5.414] |
2.3 Model Application
A database of patients and their associated ICD-9 billing codes was created from all inpatient encounters at Massachusetts General Hospital from 1999 to 2012. To determine the availability of input data for the RSI web service at our institution, we looked for ICD-9 data for all patients undergoing surgery in the main OR during the year prior to model implementation. If ICD-9 data were available for an operative patient then the first 10 diagnostic codes and the first 6 billing codes listed were used in the RSI web service.
The RSI value was displayed within Rhinos, a tool which helps anesthesia residents and administrators create OR assignments. Residents are shown the following day’s surgical schedule in graphic format and are asked to fill out their preferences for case assignment. The RSI value is incorporated into the patient’s information when displaying the schedule, allowing residents to consider the risk score when determining their selection, as shown in ► Figure 2.
Fig. 2.
One-year mortality RSI value displayed within the OR scheduling web tool, Rhinos
3. Results
3.1 Data Availability
821,608 inpatient encounters were retrieved from patient hospitalizations between 1999 and 2012 to populate the ICD-9 billing code dataset. In a retrospective analysis of 46,740 patients in the main OR who received surgery from 3/1/2011 to 3/1/2012, one-year mortality RSI values were generated for 20,638 patients (44%) when all available ICD-9 billing data were used. When billing data was restricted to the year prior to surgery in order to ensure recent model input data, RSI values were calculated for 12,686 patients (27%).
3.2 RSI Validation
The one-year mortality RSI values were compared to the ASA physical status assigned by anesthesia staff during pre-operative evaluation. Treating the ASA physical status as a continuous variable, the Spearman’s rank correlation coefficient between RSI values and ASA physical status is, -0.480 for length of stay RSI, 0.341 for in-hospital mortality RSI, 0.374 for 30-day mortality RSI, and 0.404 for one-year mortality RSI. Spearman’s rank correlation coefficients between all RSI values and ASA values are shown in ► Table 2.
Table 2.
Spearman’s Rank Correlation Coefficient for ASA Physical Status and RSI Values
RSI one-year mortality | RSI length of stay | RSI in-hospital mortality | RSI 30-day mortality | ASA physical status | |
---|---|---|---|---|---|
RSI one-year mortality | 1.000 | ||||
RSI length of stay | -0.590 | 1.000 | |||
RSI in-hospital mortality | 0.674 | -0.629 | 1.000 | ||
RSI 30-day mortality | 0.810 | -0.559 | 0.768 | 1.000 | |
ASA physical status | 0.404 | -0.480 | 0.341 | 0.374 | 1.000 |
The RSI values for one-year mortality were found to have a normal distribution in our data set, with mean -0.466 and standard deviation (SD) of 1.185.
4. Discussion
We have incorporated a risk stratification model into an existing clinical decision support tool to enable automated calculation of surgical patient complexity. We created a web service for the one-year mortality RSI, but the process is generalizable to RSI for length of stay, in-hospital mortality and 30-day mortality and other risk classification systems based on administrative data.
There are numerous potential applications for this use of the model. OR administrators could use the tool to ensure that there is an appropriate congruence between case complexity and staff experience, in an effort to improve patient outcomes. A recent surgical workflow redesign that took into account patient acuity, among other things, resulted in a mortality reduction [20]. Additionally, studies of academic training programs have found that resident learning is affected not only by patient load but also patient acuity [10]. Improved the match between patient acuity and clinician skills could lead to better experiences for residents and patients alike.
One limitation to the web service is the input data relevance and validity. The RSS model was developed on MEDPAR data from 2001 to 2006 that contained up to ten diagnostic codes and six procedure codes. The MEDPAR dataset was advantageous because of its large size of over 79 million patient stays, and because it is a national rather than regional or local database. An updated version of the model validated on more recent MEDPAR data would be more applicable to the current web implementation. It would also be constructive to validate the use of the RSS model with more than ten diagnostic and six procedure codes as inputs.
There are significant potential pitfalls in applying administrative billing data to clinical decision-making. First and foremost, administrative data are frequently incorrect; a 1985 study of 1,829 VA patient medical records found an average of 0.81 coding errors per discharged patient [16]. Similarly, a 1988 review of 7,050 medical records from 239 hospitals found that 21% of ICD-9 codes were incorrect [13]. Although strategies exist for analyzing and improving the accuracy of ICD-9 code assignments, there are many potential sources of error which are likely to persist [19]. Approaches which use administrative billing data for clinical purposes should be validated and utilized with caution.
Another limitation is the inaccuracy of using potentially outdated billing codes from previous hospitalizations. For example, a patient admitted and successfully treated for pneumonia five years prior to her operation should not have a diagnosis of pneumonia incorporated into her risk score. Our web implementation utilizes data from the most recent hospitalization and is therefore unable to distinguish between current and resolved diagnoses. This limitation could be mitigated by only using input data from recent visits. However, this results in decreased data availability for the web tool. As described above, while RSI values were calculated for 20,638, or 44%, of patients when all available inpatient billing data was used, the RSI values were calculated for only 12,686 patients, or 61.5% of that subset, if billing data is limited to one year prior to surgery.
One potential method of increasing the accuracy of billing data is to include outpatient encounters. Our institution has access to data from over 27 million such outpatient visits, compared to 821,608 inpatient encounters. Expanding the database to include outpatient data would increase its size by over 30 times and could provide more recent and accurate input data. This approach would also increase the yield of the billing code database. The patient must have had a previous inpatient visit at our hospital in order to calculate his or her RSI value. By including outpatient visits in the billing database we could increase the percentage of patients for whom the RSI value can be calculated. However, the original RSS model was not developed on outpatient data, so their inclusion in the web service would need to be validated.
Given the frequent unavailability of input data for the RSS web model, a model which uses procedure booking data instead of billing data could be a viable alternative. Such a system has been developed [5]. However, this system utilizes patient age and American Society of Anesthesiology Physical Status and Current Procedural Terminology codes to generate these scores. These data are not all available in a coded format pre-operatively at Massachusetts General Hospital, but may be available at other institutions. Also, as mentioned previously, the ASA physical status classifications are highly variable; 1995 study surveyed 133 anesthesiologists about the ASA status of ten hypothetical patients. No case had complete agreement of ASA status and nine out of ten cases received at least three of the five ASA physical status grades. The significant subjectivity and variability in ASA scoring likely contributed to the low correlation between ASA status and RSI values in our study.
Lastly, further work is needed to delineate the impact of displaying risk indices on OR operations and resident OR case assignments. The RSI web service output values are currently displayed in numeric format within Rhinos, but the number alone may not be easily understood without context. It would be helpful to provide interpretation of the output value. Since the RSI values have a normal distribution, this could be done by organizing the RSI values into the qualitative categories ”low”, “medium”, or “high” risk based on whether the RSI value is less than one SD below the mean, within one SD of the mean, or greater than one SD above the mean, respectively. Administrators could consider the risk score when creating anesthesia case assignments, with the goal of improving resident satisfaction and patient safety. However, these approaches would also need to be tested and validated.
5. Conclusion
We have constructed a web-based decision support tool that automatically calculates a risk score for patients scheduled to undergo surgery. This risk score can be used when creating OR assignments to match patient complexity to physician experience, plan clinician staffing, or provide a ready and objective preoperative estimate of a patient’s complexity in addition to the more familiar ASA physical status. Advantages of this web-based tool include its automatic compilation, objectivity, ease of use, and ability to harness data from existing databases to create patient-specific information for improving clinical workflow. Further validations and refinements of the model are needed, but this approach represents a potentially powerful means for quickly analyzing patient complexity and improving human resource management in operating rooms.
6. Clinical Relevance Statement
Matching patient complexity with human resources is an important but challenging aspect of hospital management. We have created a web-based decision support tool that automatically calculates a risk score for patients scheduled to undergo surgery. This risk score can be used to match patient acuity to physician experience, plan clinician staffing, or provide a ready and objective preoperative estimate of a patient’s complexity.
Conflicts of Interest
The authors declare that they have no conflicts of interest in the research.
Protection of Human and Animal Subjects
The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. All research was conducted with the approval by the Partners HealthCare Institutional Review Board (2012P001284).
References
- 1.ASA Physical Status Classification System. http://www.asahq.org/Home/For-Members/Clinical-Informa tion/ASA-Physical-Status-Classification-System [Google Scholar]
- 2.Balstad A, Springer P. Quantifying case management workloads: Development of the PACE tool. Lippincott's case management : managing the process of patient care 2006; 6: 291–302; quiz 3-4. [DOI] [PubMed] [Google Scholar]
- 3.Callaghan LA, Cartwright DW, O'Rourke P, Davies MW. Infant to staff ratios and risk of mortality in very low birthweight infants. Archives of disease in childhood Fetal and neonatal edition 2003; 2: F94–7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Curtin LL. An integrated analysis of nurse staffing and related variables: effects on patient outcomes. Online journal of issues in nursing 2003; 3: 5. [PubMed] [Google Scholar]
- 5.Dalton JE, Kurz A, Turan A, Mascha EJ, Sessler DI, Saager L. Development and validation of a risk quantification index for 30-day postoperative mortality and morbidity in noncardiac surgical patients. Anesthesiology 2011; 6: 1336–44 doi: 10.1097/ALN.0b013e318219d5f9 [DOI] [PubMed] [Google Scholar]
- 6.Dexter F, Wachtel RE, Epstein RH. Impact of average patient acuity on staffing of the phase I PACU. Journal of perianesthesia nursing : official journal of the American Society of PeriAnesthesia Nurses / American Society of PeriAnesthesia Nurses 2006; 5: 303–10 doi: 10.1016/j.jopan.2006.07.007 [DOI] [PubMed] [Google Scholar]
- 7.Gedmintas A, Bost N, Keijzers G, Green D, Lind J. Emergency care workload units: a novel tool to compare emergency department activity. Emergency medicine Australasia : EMA 2010; 5: 442–8 doi: 10.1111/j.1742-6723.2010.01322.x [DOI] [PubMed] [Google Scholar]
- 8.Goj K, Knapik P, Kucewicz-Czech E, Lubon D. [The TISS-28 scoring system for assessment of cardiac surgical postoperative intensive care]. Anestezjologia intensywna terapia 2009; 1: 37–40 [PubMed] [Google Scholar]
- 9.Gross JC, Faulkner EA, Goodrich SW, Kain ME. A patient acuity and staffing tool for stroke rehabilitation inpatients based on the FIM instrument. Rehabilitation nursing : the official journal of the Association of Rehabilitation Nurses 2001; 3: 108–13 [DOI] [PubMed] [Google Scholar]
- 10.Haney EM, Nicolaidis C, Hunter A, Chan BK, Cooney TG, Bowen JL. Relationship between resident workload and self-perceived learning on inpatient medicine wards: a longitudinal study. BMC medical education 2006: 35 doi: 10.1186/1472-6920-6-35 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Harper K, McCully C. Acuity systems dialogue and patient classification system essentials. Nursing administration quarterly 2007; 04: 284–99 doi: 10.1097/01.NAQ.0000290426.41690.cb [DOI] [PubMed] [Google Scholar]
- 12.Haynes SR, Lawler PG. An assessment of the consistency of ASA physical status classification allocation. Anaesthesia 1995; 3: 195–9 [DOI] [PubMed] [Google Scholar]
- 13.Hsia DC, Krushat WM, Fagan AB, Tebbutt JA, Kusserow RP. Accuracy of diagnostic coding for Medicare patients under the prospective-payment system. The New England journal of medicine 1988; 6: 352–5 doi: 10.1056/NEJM198802113180604 [DOI] [PubMed] [Google Scholar]
- 14.Innes GD, Stenstrom R, Grafstein E, Christenson JM. Prospective time study derivation of emergency physician workload predictors. Cjem 2005; 5: 299–308 [DOI] [PubMed] [Google Scholar]
- 15.Land MJ, Guzzetti PJ, May FJ. Balancing workload and staffing in satellite pharmacies by using a patient acuity index. Hospital pharmacy 1983; 6: 305–7, 10-1. [PubMed] [Google Scholar]
- 16.Lloyd SS, Rissing JP. Physician and coding errors in patient records. JAMA : the journal of the American Medical Association 1985; 10: 1330–6 [PubMed] [Google Scholar]
- 17.Mamaril ME, Sullivan E, Clifford TL, Newhouse R, Windle PE. Safe staffing for the post anesthesia care unit: weighing the evidence and identifying the gaps. Journal of perianesthesia nursing : official journal of the American Society of PeriAnesthesia Nurses / American Society of PeriAnesthesia Nurses 2007; 6: 393–9 doi: 10.1016/j.jopan.2007.08.007 [DOI] [PubMed] [Google Scholar]
- 18.O'Connor G, Geary U, Moriarty J. Critical care in the emergency department. European journal of emergency medicine : official journal of the European Society for Emergency Medicine 2009; 6: 296–300 doi: 10.1097/MEJ.0b013e32831090bd [DOI] [PubMed] [Google Scholar]
- 19.O'Malley KJ, Cook KF, Price MD, Wildes KR, Hurdle JF, Ashton CM. Measuring diagnoses: ICD code accuracy. Health services research 2005; 5 Pt 2: 1620–39 doi: 10.1111/j.1475-6773.2005.00444.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Ravikumar TS, Sharma C, Marini C, Steele GD, Jr., Ritter G, Barrera R, Kim M, Safyer SM, Vandervoort K, De Geronimo M, Baker L, Levi P, Pierdon S, Horgan M, Maynor K, Maloney G, Wojtowicz M, Nelson K. A validated value-based model to improve hospital-wide perioperative outcomes: adaptability to combined medical/surgical inpatient cohorts. Annals of surgery 2010; 3: 486–96; discussion 96-8. doi: 10.1097/SLA.0b013e3181f1c412 [DOI] [PubMed] [Google Scholar]
- 21.Rischbieth A. Matching nurse skill with patient acuity in the intensive care units: a risk management mandate. Journal of nursing management 2006; 5: 397–404 doi: 10.1111/j.1365-2934.2006.00622.x [DOI] [PubMed] [Google Scholar]
- 22.Sessler DI, Sigl JC, Manberg PJ, Kelley SD, Schubert A, Chamoun NG. Broadly applicable risk stratification system for predicting duration of hospitalization and mortality. Anesthesiology 2010; 5: 1026–37 doi: 10.1097/ALN.0b013e3181f79a8d [DOI] [PubMed] [Google Scholar]
- 23.Sessler DI, Sigl JC, Manberg PJ, Kelley SD, Schubert A, Chamoun NG. http://my.clevelandclinic.org/anesthesia/outcomes/risk-stratification-index.aspx. [Google Scholar]
- 24.Spence K, Tarnow-Mordi W, Duncan G, Jayasuryia N, Elliott J, King J, Kite F. Measuring nursing workload in neonatal intensive care. Journal of nursing management 2006; 3: 227–34 doi: 10.1111/j.1365-2934.2006.00609.x [DOI] [PubMed] [Google Scholar]
- 25.Wanderer JP, Charnin J, Driscoll WD, Balin MT, Baker K. Decision Support Using Anesthesia Information Management System Records and Accreditation Council for Graduate Medical Education Case Logs for Resident Operating Room Assignments. Anesth Analg 2013 [DOI] [PubMed] [Google Scholar]