Table 2.

Analysis of the methodology of development of each composite quality measure

Instrument	Transparency in calculation11	Justified selection of individual measures11	Handling missing measure information11	Handling missing measure information11	Banding onto scales11	Justification for weights11	Justification for weights11	Uncertainty11
	Are all important methodological details easily accessible in a public document?	Are the measures used equally applicable across all rated hospitals?	Is missing measure information handled in a way that can introduce bias?	Are component measures adequately adjusted for case-mix?	Are measures standardized using banding?	Is there an apparent justification for the weights used?	Is any sensitivity analysis of the choice of weights reported?	Is the uncertainty in the final composite rating presented?
DAOH21	Yes	Yes. Insufficient clarity on process by which decisions made to choose measures. Length of stay dominates	Patients with missing data not included	Not discussed	Yes, mortality rate and length of stay banded onto a scale	Insufficient justification for calculations used	Yes	No
I-FEED22	Yes, however measures have a subjective element	Yes. Measures are related to procedure but insufficient justification for selection	No information	Not discussed	Yes, measures banded onto an arbitrary scale	Yes, developed by expert consensus	Yes	No
HARM3,24–25	Yes	Yes Insufficient clarity on process by which decisions made to choose measures	Patients with missing data not included	Not discussed	Yes, measures banded onto an arbitrary scale	LOS scaled based on normal distribution curve. No justification for other weights	No	No
SCOUT26	Yes, however measures have a subjective element	Yes. Measures chosen from existing list of complication types	Information collected for score	Not discussed	Yes, complications banded into four grades to get measure onto a consistent scale	Yes, developed by expert clinical opinion	No	Partially
MTL27	Yes	Yes. Insufficient clarity on process by which decisions made to choose measures	Patients with missing data not included	Authors state that analysis is not adjusted for case-mix	Yes, length of stay banded onto a scale	No	No	No
TO28	No, although measures included in this study are clear—there is no clear consensus of what measures should be included in future studies	No, measures may not be equally applicable to some hospitals	Hospitals without relevant data not included	Yes	Yes, measures banded into positive or negative result	Not applicable	No	No
TISS30	Yes	No, measures may not be equally applicable to some hospitals	Information collected for score	Not discussed	Yes, measures banded onto an arbitrary scale.	Yes, intensity of involvement	No	No
PQS31	Yes	Yes	Hospitals without relevant data not included	Not discussed	Yes, each measure banded as one ‘QI’	Yes, clinical opinion	Yes	Partially
DIMICK32	Yes, but in another paper. Insufficient clarity on how data from ‘other’ procedures incorporated	Yes	Hospitals without relevant data not included	Yes	No	Yes, well justified	Yes	Partially
PMI29	Yes	Yes	Hospitals without relevant data not included	Not discussed	Yes, severity of complications banded using Accordion Severity Grade	Yes, partially uses previously validated grading system	No	No

LOS, length of (hospital) stay; HARM, Hospital stay, Readmission, and Mortality; DAOH, Days Alive and Out of Hospital; SCOUT, Surgical Complication OUTcome; MTL, Mortality, Transfer, Length-of-stay; TO, Textbook Outcome; NSQIP, National Surgical Quality Improvement Program; TISS, Therapeutic Intervention Scoring System; I-FEED, Intake, response to nausea treatment, Emesis, Exam, and Duration; PQS, Patient Quality Score; DIMICK, Dimick et al. 2013; PMI, Post-operative Morbidity Index; QI, quality indicator.