To the Editor
The use of surgical quality measures to target quality improvement efforts and evaluate hospital performance is now standard. Surgical quality in the VA is measured by the VA Surgical Quality Improvement Program (VASQIP),1 the Surgical Care Improvement Program (SCIP),2 and the Patient Safety Indicators (PSIs).3 Each approach has a different perspective on surgical quality and uses a different source of data. For example, VASQIP evaluates 30-day postoperative morbidity and mortality outcomes among other parameters, SCIP measures compliance with specific perioperative processes of care, and PSIs calculate the rates of potentially preventable, inpatient, surgical adverse events using administrative data. We explored the correlation between VASQIP, SCIP and PSI measures and how consistently they identified high- and low-performing VA hospitals.
Methods
We used FY09 quality indicator data (Oct. 1, 2008 – Sept. 30, 2009) from 67 VA hospitals with advanced surgical programs. We obtained hospitals' VASQIP morbidity and mortality observed/expected ratios and SCIP compliance scores from the 2010 VA Facility Quality and Safety Report.4 We ran the PSI software on hospital surgical discharge data to generate observed and risk-adjusted rates for each of the 7 postoperative PSIs. We then adapted the PSI Composite software to develop a PSI Surgery Composite score calculated for the postoperative PSIs using numerator-based weights.5 Using these 4 quality measures, we ranked hospitals and examined the correlation between ranks. We also identified the top and bottom 25% of hospitals, and calculated the number of hospitals with high or low performance on multiple indicators.
Results
Few comparisons yielded significant correlations. Only the hospital VASQIP morbidity observed/expected ratio and hospital PSI Surgery Composite had a significant, albeit weak, association (r=0.267, p=0.029) (see Table 1). Agreement on whether hospitals were high, average or low performers was similarly moderate: SCIP and the PSI Surgery Composite agreed on performance category for 45% of the hospitals (the highest agreement), while SCIP and VASQIP mortality O/E ratio agreed for only 37% (lowest agreement). Although none of the hospitals performed well on all 4 measures, 5 (7%) were in the top 25% on 3 of the measures. Seven hospitals (10%) were considered average and one hospital (1%) was in the bottom 25% on all 4 measures (see Table 2).
Table 1.
Spearman Correlation Coefficients and Agreement in Performance Rankings of FY09 Surgical Quality Measures
| Quality Measure | VASQIP Morbidity | VASQIP Mortality | SCIP Composite |
|---|---|---|---|
| VASQIP Mortality O/E Ratio | 0.127 (p=0.31) | - | - |
| SCIP Composite | 0.047 (p=0.71) | −0.141 (p=0.25) | - |
| PSI Surgery Composite | 0.268 (p=0.03) | 0.054 (p=0.66) | −0.134 (p=0.28) |
Note: Comparison of facility rankings using each of the surgical quality measures. Bolded value indicates significance of p 0.05
VASQIP=VA Surgical Quality Improvement Program
SCIP=Surgical Care Improvement Program
PSI=Patient Safety Indicator; we developed a PSI surgery composite score using numerator weights for the 7 postoperative PSIs (see Appendix 1)
Table 2.
Agreement in High, Average, and Low Hospital Surgical Performance using FY09 Surgical Quality Measures*
| Percent of Hospitals | Top 25% | Average 50% | Bottom 25% |
|---|---|---|---|
| Rated by all 4 measures | 0% | 10% | 1% |
| Rated by 3 measures | 7% | 21% | 6% |
| Rated by 2 measures | 22% | 33% | 19% |
| Rated by 1 measure | 34% | 27% | 39% |
| Rated by none of the measures | 36% | 9% | 34% |
| Total | 100% | 100% | 100% |
Measures include the VA Surgical Quality Improvement Program (VASQIP) 30-day morbidity and mortality observed/expected ratios, the Surgical Care Improvement Program (SCIP) Composite of compliance, and the Patient Safety Indicator (PSI) surgery composite score developed using numerator weights for the 7 postoperative PSIs.
Discussion
High performance on one type of surgical quality measure was not associated with high performance on another. The lack of correlation between the VASQIP measures, SCIP compliance score, and the PSI Surgery Composite measure suggests that these indicators measure different dimensions of surgical quality. Information from multiple quality measures is useful in directing individual facilities toward different quality improvement activities. However, from the perspective of comparing facilities these differences highlight the importance of examining more than one measure.
Our findings illustrate the potential confusion that may be associated with multiple, poorly correlated measures that purport to all measure quality. However, the confusion arises only when quality is conceptualized as an underlying latent construct that is reflected in the individual indicators. When quality is conceptualized as a construct created by combining individual indicators that reflect different dimensions of quality, the low correlation of individual indicators does not create a problem. In fact, as noted by Feinstein, combining uncorrelated dimensions into a composite measure is more consistent with clinical needs than a composite created from multiple dimensions of the same phenomena.6 We postulate that measures like the PSIs, VASQIP, and SCIP, could be used to develop a single composite measure of quality that encompasses several aspects of surgical quality. In the future, a single composite measure of surgical quality could provide more actionable information for patients, providers and policymakers as they attempt to differentiate hospital performance.
Acknowledgment
This research was supported by funding from the VA Health Services Research and Development Service grant number SDR 07-002 (P.I. Rosen).
Footnotes
Scheduled for an oral presentation at the AVAS Conference on April 7, 2014.
The authors report no conflict of interests. Dr. Mull had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
References
- 1.Khuri SF, Daley J, Henderson W, et al. The Department of Veterans Affairs' NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg. 1998 Oct;228(4):491–507. doi: 10.1097/00000658-199810000-00006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Bratzler DW. The Surgical Infection Prevention and Surgical Care Improvement Projects: promises and pitfalls. The American surgeon. 2006 Nov;72(11):1010–1016. 1133–1048. discussion 1021–1030. [PubMed] [Google Scholar]
- 3.Agency for Healthcare Research and Quality (AHRQ) [Accessed Mar 3, 2013];AHRQ Patient Safety Indicators Overview. AHRQ Homepage. 2013 http://www.qualityindicators.ahrq.gov/Modules/psi_overview.aspx.
- 4.Department of Veterans Affairs 2010 VHA Facility Quality and Safety Report. http://www.va.gov/health/docs/HospitalReportCard2010.pdfOctober 2010.
- 5.Agency for Healthcare Research and Quality . AHRQ Quality Indicators: Composite Measures User Guide for the Patient Safety Indicators (PSI) Version 4.2 (September 2010) AHRQ; Rockville, MD: 2010. [Google Scholar]
- 6.Feinstein AR. Multi-item “instruments” vs Virginia Apgar's principles of clinimetrics. Arch Intern Med. 1999 Jan 25;159(2):125–128. doi: 10.1001/archinte.159.2.125. [DOI] [PubMed] [Google Scholar]