Comparison of risk-adjusted cumulative quality control charts compared with standardized mortality ratios in critical care

Abstract

Purpose

The optimal method for monitoring intensive care unit (ICU) performance is unknown. We sought to compare process control charts using standardized mortality ratio (SMR), p-charts, and cumulative sum (CUSUM) charts for detecting increases in risk-adjusted mortality within ICUs.

Methods

Using data from 17 medical-surgical ICUs that included 29,592 patients in Ontario, Canada, we created risk-adjusted p-charts and SMRs on monthly intervals and CUSUM charts. We defined positive signals as any data point that was above the 3-sigma limit (approximating a 99% confidence interval [CI]) on a p-chart, any data point whose 95% CI did not include 1 for the SMR charts, and when a data point exceeded control limits for an odds ratio of 1.5 for CUSUM charts. We simulated increases in mortality of 10%, 30%, and 50% for each ICU to determine the sensitivity of each method. We calculated sensitivity as the number of positive signals divided by the number of ICUs (equal to number of simulated events).

Results

Cumulative sum charts generated 31 signals in 12 different ICUs, while p-charts and SMR agreed in 10 and 6 of these signals, respectively, followed by 21 signals from p-charts across 14 ICUs (agreement in 10 of these signals for both CUSUM and SMR) and 15 signals from SMR charts across eight ICUs (agreement from p-charts and CUSUM in 10 and six signals, respectively). The p-chart had a sensitivity of 88% (95% CI, 73 to 104) for a 50% simulated increase in ICU mortality followed by CUSUM at 71% (95% CI, 49 to 102) and SMR at 59% (95% CI, 35 to 82). Performance with lower simulated increases was poor for all three methods.

Conclusions

P-charts created with risk-adjusted mortality at monthly intervals are potentially useful tools for monitoring ICU performance. Future studies should consider usability testing with ICU leaders and application of these methods to additional clinical domains.

Monitoring the performance of intensive care unit (ICUs) is challenging but important as these are complex systems where mortality is a common outcome measure. Performance improvement requires not just measurement but also action, based on approaches such as the “Model for Improvement.”¹ Completion of an improvement cycle is complex and dependent on many factors. The purpose of data visualization is to gain understanding and insight into the data, to appreciate variation, and to help recognize outliers, trends, patterns, and relationships between indicators.

Since many factors may influence patient outcomes, adjustment for risk is often used. The standardized mortality ratio (SMR), which is the ratio of observed to expected mortality, is commonly used with Canadian examples such as the hospital SMR produced by the Canadian Institute for Healthcare Improvement² and hospitals participating in the National Surgical Quality Improvement Program.³ Nevertheless, the utility of the SMR is unclear.^4–8 Statistical process control charts have been used extensively by industry for quality control and are becoming increasingly common in health care.^9,10 When applied in real or near real time, this method provides an appropriate basis for action while avoiding “tampering” (acting when not warranted by the data). These charts are commonly used with unadjusted rates (p-charts) or counts plotted at regular intervals, but even incomplete adjustment with a small number of factors can improve the performance of a chart.^11,12 Risk-adjusted cumulative sum (CUSUM) charts were recently reported to identify potential performance issues in Danish hospitals.¹³ Cumulative sum charts track cumulative differences between observed and expected events and may detect small but persistent trends. This method is, however, more complex and sensitive to initial parameters and specific methods.¹⁴ A previous study compared CUSUM and p-charts, along with other methods, and concluded that p-charts were simplest to implement and interpret while CUSUM may be useful to detect small changes.¹⁵ Another study used simulated data to compare different risk-adjusted methods including the SMR.⁵ Nevertheless, the optimal method to use is unknown in the absence of direct comparison of various methods using actual patient data and including the SMR.

Thus, the objective of our study was to use a population-based cohort of ICU patients to compare risk-adjusted p-charts, CUSUM charts, and SMR and their effectiveness at detecting simulated increases in mortality.

Methods

The Western University Research Ethics Board (London, ON, Canada) granted approval for this study on 15 February 2017. Since this study used deidentified data that was collected previously (see Data Source below), patient consent was not required. The reporting of this study adheres to the Strengthening the Reporting of Observational Studies in Epidemiology statement for cohort studies.¹⁶

Study design and patient population

We conducted a retrospective study of the 17 medical-surgical ICUs in teaching hospitals in Ontario, Canada that can provide life support interventions and mechanical ventilation for more than 48 hr. We included patient arrivals between July 2015 and December 2016. We used this population-based cohort to compare the effectiveness of three different types of run charts at detecting different levels of changes in unit-level risk-adjusted ICU mortality.

Data source

We used data from the Ontario Critical Care Information System (CCIS) for this study. We performed risk adjustment using a previously published and externally validated ICU mortality prediction model.^17,18 An explanation of the CCIS, and information on exclusions and data definitions, can be found in the external validation publication.¹⁸ Briefly, the model adjusts for age (18–39, 40–79, 80 yr or older), sex (male, female), source of ICU admission (operating room/postanesthetic care unit, emergency department, ward, other hospital, other), ICU admission diagnosis (cardiovascular, respiratory, gastrointestinal, neurologic, trauma, other), readmission to ICU within the same hospital stay, the Nine Equivalents of Nursing Manpower Score¹⁹ (0–22, 23–29, 30 or greater), and the Multiple Organ Dysfunction Score²⁰ (MODS) (0, 1–4, 5–8, 9–12, 13 or greater).

Chart development

P-charts

P-charts present the proportion of an event over time. Control limits are calculated as three standard deviations (SDs) from the mean (3-sigma). When plotted on the chart, these limits allow for a statistical-based determination of special cause variation. Specifically, data that lie above the average and beyond the three-sigma line represent less than 1% of all data points and are therefore not likely because of common cause variation. In this study, we based calculation of the control limits on admissions occurring between July and December 2015 (initiation period). We used the subsequent year of data to plot the risk-adjusted mortality rate each month. We calculated risk-adjusted mortality rates as:

$$(observeddeaths/predicteddeaths)\times observedmortalityrate,$$

where predicted deaths were the sum of individual predicted risks of death.

For this study, we defined points above 3-sigma to be positive signals.¹ We used SQCpack version 7 (PQ Systems Inc., Dayton, OH, USA) to produce the p-charts.

Standardized mortality ratio

We calculated the SMR with a correction applied since the chance of a small number of deaths in any given month was likely.³ The formula used was:

$$(sumofobservedmortality+\mathit{1})/(sumofpredictedriskofmortality+\mathit{2}).$$

The 95% confidence interval (CI) around the SMR was calculated as:

$$\{sumofobservedmortality\pm [\mathit{1.96}\times \surd sum(predictedriskofmortality\times \{\mathit{1}-predictedriskofmortality\})]\}/sumofpredictedriskofmortality.$$

We defined a positive signal if the 95% CI did not overlap an SMR equal to 1.0.^2,3 All calculations and plots were produced using SAS version 9.4 (SAS Institute Inc., Cary, NC, USA).

Cumulative sum

Implementation of risk-adjusted CUSUM charts followed the methodology and calculations published in a previous study.¹³ Risk-adjusted CUSUM functions were calculated and plotted per patient by the date of ICU discharge or death.

We set CUSUM charts to detect when the mortality rate had increased by an odds ratio of 1.5 or more, defined as a positive signal, compared with the reference period. This represented a balance between the ability to detect clinically relevant increases in ICU mortality and avoiding signals when small changes occurred.¹³ The value of the CUSUM function on the last day of discharge or death during the reference period was used as the initiation value of the risk-adjusted CUSUM chart on 1 January 2016. We determined the control limits using the first six months of data (July–December 2015) using simulation so that false signals should only occur once every 25 years for each unit. Once a signal had fired, we reset the CUSUM value to 0.

All calculations and plots were produced using SAS version 9.4 (SAS Institute, Cary, NC, USA).

Statistical analyses

We describe the ICUs in the cohort with summary data presented as mean (SD). We examined the charts created using the 12 months that followed the initiation period for signals (baseline scenario) and looked for agreement across the three methods whenever a signal was detected.

To determine the sensitivity for each method, we introduced simulated increases in mortality by 10%, 30%, and 50%, rounded to the nearest whole number, during specific months for each ICU. We determined the targeted month in which to increase mortality based on the p-charts. Specifically, to avoid imposing a simulated signal on top of an actual signal, we used the first three-month period without a baseline signal and the increase in mortality was made to the third month. In the case of the p- and SMR charts, the deaths were simply added to the monthly totals. For the CUSUM charts, the last number of survivors in the month was changed to nonsurvivors.

Since an event was simulated for each ICU, we calculated sensitivity for each level of simulated change as the number of times each method generated a signal in the ICUs following a simulated increase in mortality (true positives) divided by the number of ICUs (true positives + false negatives). We calculated the 95% CI around the sensitivity as ± 1.96 × √{[sensitivity × (1 − sensitivity)]/number of ICUs}. We also report the average time it would take risk-adjusted CUSUM charts to signal an alarm (ARL_A) when mortality had increased by an OR of 1.5. We minimized bias by including all available patients in the data set. A sample size was not calculated.

Results

There were 29,592 patients admitted to the 17 ICUs during the 18 months used for our analysis. These patients had a mean (SD) age 61 (18) yr, 43% were female, and the ICU admission sources were operating room (25%), emergency department (34%), ward (17%), other hospital (13%), and other (11%). During the 12 months that followed the six-month initiation period and that was used for the simulations, the 17 ICUs admitted a mean (SD) of 1,168 (182) patients with a mortality rate of 16.5 (4.6)% and a MODS of 4.4 (0.8). The median (SD) ICU length of stay (LOS) 3.2 (0.9) days (Table 1).

Table 1.

Description of included intensive care units in teaching hospitals in Ontario, Canada

ICU code	Annual volume, n/total N (%)	Mortality, n/total N (%)	MODS, mean (SD)	ICU length of stay (days), median [IQR]
A	1,251/19,855 (6.3%)	133/1,251 (10.6%)	3.8 (3.2)	2.4 [1.0–5.6]
B	1,226/19,855 (6.2%)	96/1,226 (7.8%)	2.9 (2.3)	2.0 [0.9–6.0]
C	973/19,855 (4.9%)	169/973 (17.4%)	5.1 (3.2)	3.3 [1.6–7.1]
D	811/19,855 (4.1%)	122/811 (15.0%)	4.1 (2.8)	3.0 [1.6–6.0]
E	1,248/19,855 (6.3%)	242/1,248 (19.4%)	4.7 (2.7)	4.1 [2.1–8.2]
F	1,065/19,855 (5.4%)	259/1,065 (24.3%)	4.9 (2.8)	4.6 [2.2–8.9]
G	1,010/19,855 (5.1%)	127/1,010 (12.6%)	4.1 (3.2)	2.1 [1.1–4.2]
H	1,043/19,855 (5.3%)	80/1,043 (7.7%)	3.4 (2.9)	1.9 [0.9–3.8]
I	1,034/19,855 (5.2%)	211/1,034 (20.4%)	4.7 (2.5)	3.5 [1.5–7.5]
J	1,462/19,855 (7.4%)	250/1,462 (17.1%)	4.3 (3.1)	3.7 [1.7–8.0]
K	1,090/19,855 (5.5%)	173/1,090 (15.9%)	6.3 (3.3)	3.8 [1.9–8.2]
L	1,029/19,855 (5.2%)	177/1,029 (17.2%)	3.8 (2.6)	4.5 [1.9–8.9]
M	1,405/19,855 (7.1%)	242/1,405 (17.2%)	3.8 (3.3)	2.0 [0.9–4.9]
N	1,159/19,855 (5.8%)	255/1,159 (22.0%)	5.0 (2.9)	3.0 [1.5–6.7]
O	1,438/19,855 (7.2%)	273/1,438 (19.0%)	5.0 (3.0)	3.2 [1.5–6.3]
P	1,305/19,855 (6.6%)	211/1,305 (16.2%)	4.3 (3.1)	2.8 [1.2–5.2]
Q	1,306/19,855 (6.6%)	269/1,306 (20.6%)	5.3 (3.1)	4.7 [2.2–9.2]

Bold values represent minimum and maximum for each metric across all 17 ICUs. Annual volume represents the % of total volume for the 17 ICUs. The minimum and maximum mortality are based on the mortality rate.

ICU = intensive care unit; IQR = interquartile range; MODS = multiple organ dysfunction score; SD = standard deviation

We used the graphical display method of charts in small multiples to show all the ICUs on a single page or graphic (Figs 1–3). The baseline scenarios for risk-adjusted mortality for the ICUs are shown in p-charts (Fig. 1), CUSUM charts (Fig. 2), and SMR charts (Fig. 3). While establishing the baseline parameters for the CUSUM charts, we determined the median [interquartile range] estimated average time to detect a 1.5-times higher odds ratio of ICU mortality with CUSUM was 73 [61–87] days, with a minimum and maximum of 46 and 126 days, respectively. Cumulative sum generated 31 signals in 12 ICUs followed by 21 signals from p-charts although the two methods agreed on only 10 signals (Table 2). Since p-charts are a type of statistical process control chart, Fig. 1 shows additional signals that could be considered important if another rule (2 of 3 sequential points beyond 2-sigma) is implemented. While the SMR method generated the fewest signals at 15, the wide CIs at each point reflect the challenge that is created by the relatively small number of events during each monthly period (Fig. 3). Only one ICU did not have a signal for increased mortality by any of the methods.

Fig. 1.

P-charts in small multiples. Each chart represents one of the 17 ICUs (A–Q; for details, see Table 1). The x-axis is time (months) and the y-axis is the risk-adjusted mortality rate. In this visual representation, the units are not shown so that the amount of data shown can be maximized. Shaded areas represent 1-, 2-, and 3-sigma levels. A single point above the 3-sigma level is considered a signal for increased mortality rate. Using additional rules such as 2 of 3 sequential points beyond 2-sigma can produce further signals (square points).

ICU = intensive care unit

Fig. 3.

Standardized mortality ratio charts in small multiples. Each chart represents of the 17 ICUs (A–Q; for details, see Table 1). The x-axis is time (months) and the y-axis is the standardized mortality rate. In this visual representation, the units are not shown so that the amount of data shown can be maximized. Points are shown with calculated 95% confidence intervals (CIs). A point with a 95% CI that does not overlap 1.0 is considered a signal for increased mortality rate.

ICU = intensive care unit

Fig. 2.

Cumulative sum charts in small multiples. Each chart represents one of the 17 ICUs (A–Q; for details, see Table 1). The x-axis is time (months) and the y-axis is the risk-adjusted mortality rate. In this visual representation, the units are not shown to maximize the amount of data shown. The dashed line represents the control limit for that ICU for a 1.5-times higher odds of mortality. A point above the control limit is considered a signal for increased mortality rate.

ICU = intensive care unit

Table 2.

Agreement of the three methods for detecting a signal during the study period

	P-chart	SMR	CUSUM
P-chart	21	10	10
SMR	10	15	6
CUSUM	10	6	31

The values should be interpreted in reference to the headers in each column. Numbers along the diagonal are the total number of signals for each of the three methods. The other cells in each column are the number of signals detected during the same interval by the other two methods

CUSUM = cumulative sum; SMR = standardized mortality ratio

Figure 4 shows a representative ICU with the baseline scenario and simulated increases in mortality for the three chart types. In simulations, the p-chart had the best sensitivity of the three methods at 88% (95% CI, 73 to 104) with a 50% simulated increase in ICU mortality (Table 3).

Fig. 4.

Charts showing baseline data and simulated 10%, 30%, and 50% increases in mortality for ICU A. The arrows indicate when the simulated increase in mortality was introduced to the baseline actual data. The charts are presented and interpreted as described for Figs 1–3 with the addition of units on the axes. The p-charts also have tables displayed with the actual values for each monthly period. For this ICU, the simulated increases of 30% and 50% resulted in a positive signal in the p-chart and additional positive signals in the CUSUM chart. The SMR charts did not result in any signals.

CUSUM = cumulative sum; ICU = intensive care unit; SMR = standardized mortality ratio

Table 3.

Comparison of sensitivity for the three methods based on simulated 10%, 30%, and 50% increases in mortality

Simulated mortality increase	Sensitivity (95% CI)
	P-chart	SMR	CUSUM
10%	18% (−1 to 36)	12% (−4 to 27)	0% (0 to 0)
30%	59% (35 to 82)	35% (13 to 58)	29% (8 to 51)
50%	88% (73 to 104)	59% (35 to 82)	71% (49 to 92)

Sensitivity (with 95% CI) was calculated as the number of times the method generated a signal in the 17 ICUs following a simulated increase in mortality divided by the number of simulated events (equal to the number of ICUs)

CI = confidence interval; CUSUM = cumulative sum; ICU = intensive care unit; SMR = standardized mortality ratio

Discussion

Risk-adjusted ICU mortality using data that is relatively simple and routine to collect can be displayed graphically with CUSUM charts, p-charts, or SMR. Each type of chart uses a different method to detect a signal in the mortality. Using simulation, we found that none of the methods were sensitive at detecting small increases in mortality. Although SMR is commonly used, it had the lowest sensitivity. The p-chart has the best sensitivity, although it is modest for a 30% increase in observed mortality. Interpretation of p-charts may be more intuitive than CUSUM for most health care professionals and could be incorporated within sets of quality indicators.

Intensive care unit mortality is a challenging quality indicator to implement.^8,18,21 Patient and system factors will influence crude mortality rates. Risk adjustment requires valid data and statistical models that may not be feasible for all institutions. The goal of statistical process control charts is to detect changes in performance in real time so that corrective action can be explored while not chasing false signals (normal variation). A recent study applied this approach with CUSUM charts to identify when the hospital mortality had increased by an OR of 1.5.¹³ Similar methods were evaluated for cardiac surgery,²² stroke care,²³ and surgical care.²⁴ In contrast to our findings, others reported good results with CUSUM but agree that the p-chart is simplest to implement and interpret.¹⁵ Other methods, such as mortality reviews of cases with low probability of death or machine learning can be considered. The former is not recommended since the prediction models’ performance are optimized at the population level and not for individual cases. The latter has shown promise for risk-adjustment^25,26 but was beyond the scope of the current study. Our study shows the potential application to intensive care using data for risk adjustment that is relatively simple and accurate to collect.¹⁸

Notably, all three methods were poor at detecting a small increase in mortality of 10% over baseline. The p-charts had the highest sensitivity among the three. The combination of more signals generated by CUSUM in the baseline scenario but lower sensitivity in the simulations suggests that CUSUM has less specificity as well. While we followed one approach for setting the initial parameters for CUSUM,¹³ this method is influenced by these choices.¹⁴ The sensitivity of CUSUM charts could be increased by setting a lower control limit but that would reduce the specificity. The format of p-charts is likely more intuitive to the users of these reports. Importantly, we found that SMR, which is commonly used, had the lowest sensitivity. We presented the actual data for the three charts in small multiples,²⁷ which is a powerful visual method to compare different ICUs. Users who are responsible for multiple units could use this approach to regularly monitor and identify signals in individual units and direct quality improvement efforts based on this information.

In our simulation, we applied the increases to mortality during a period when p-chart signals were absent for three months to avoid applying a signal when one already existed. In practice, implementation of p-charts to monitor ICU mortality will require an initial period of at least six intervals (we used monthly) to establish the baseline control limits. An ICU with large volumes might consider a shorter interval such as weekly or biweekly although smaller event rates will produce more variation. We used the simple rule of one point beyond the 3-sigma line for p-charts for comparability with the other two methods. An ICU using p-charts could consider other rules for statistical process control, such as two of three sequential signals beyond 2-sigma, four of five sequential signals beyond 1-sigma, nine sequential signals above the average and others.²³ These additional rules may improve sensitivity, detecting small but persistent changes in mortality. It is important to recognize that all approaches will identify that a problem exists, but they will not pinpoint the cause. Additional steps such as detailed chart or process reviews will be necessary. An ICU with a low volume and consequently small number of deaths may elect to simply review all cases, but this could be challenging to do well if the ICU volume (and consequently number of deaths) is high.

As an example, to show the value of a process control methodology, we can consider one of the ICUs in our study with low volume and an average mortality of 6–7 patients monthly. Such an ICU might experience only 2–3 deaths for a few months followed by a jump to eight deaths. Unless they understand their usual variation, as shown by the control limits of a p-chart, it would be tempting to conclude that something is wrong and spend time pursuing an issue that may not exist. Conversely, an improvement in practice and outcomes might be reflected by a persistent decrease in mortality that would also be seen in the control chart. Criteria should be established in advance to justify a change to the process control limits that consider such changes in practice or patient population. The “Health Care Data Guide” is a comprehensive resource for using data and process control charts to guide improvement.¹

Strengths of our study include that we used the same validated risk-adjustment method for all three chart types. We used actual patient data from individual ICUs to which we applied simulated increases in mortality after a period of stable performance. While other studies evaluated or simulated increases in mortality of 25% and 50%,^5,13 we also simulated a 10% increase in mortality since we believe that clinicians might consider detecting such smaller changes to be important. Limitations of our study include the inability to evaluate specificity of signals that were detected in addition to those that we simulated. We also did not perform any usability testing with clinicians and administrators who ultimately would be responsible for reviewing, interpreting, and acting upon the information provided by these charts.

In conclusion, the results of our study indicate that p-charts created with risk-adjusted mortality at monthly intervals are potentially useful tools for monitoring ICU performance. Future studies should consider usability testing with ICU leaders and application of these methods to additional clinical domains.

Author contributions

Claudio Martin and Fran Priestap contributed to all aspects of this manuscript, including study conception and design; acquisition, analysis, and interpretation of data; and drafting the article. Raymond Kao contributed to study conception and design, interpretation of data, and revisions of the article.

Acknowledgements

The authors thank Bruce Harries and Leanne Couves for introducing them to Shewhart process control charts and their thoughtful comments on our manuscript.

Disclosures

The authors have no disclosures relevant to this article.

Funding statement

Academic Medical Organization of Southwestern Ontario grant.

Editorial responsibility

This submission was handled by Dr. Alexis F. Turgeon, Associate Editor, Canadian Journal of Anesthesia/Journal canadien d’anesthésie.