Table 2.

Emergent Theory, Domains, and Supportive Quotations From Health System Leader Interviews.

Emergent Theory: Health System Leaders Want Meaningful Health Care Quality Data to Improve Performance. Supportive Quotations: “Would help us work together”; “The concept of SPS is to work together and not compete on safety”; “Without knowing where we are outliers, it’s hard to get a handle on where to get an aim.”
Domain	Theme	Categories	Example Quotations
Domain 1: Numerous barriers to meaningful benchmarking exist and are not insurmountable	Barriers	Data are difficult to collect	“It would be valuable to know are we an outlier in either direction—are we using far too much antibiotics compared to other hospitals or are we on the other extreme. I think that’s where data would be interesting and especially if you have access to the outpatient data. In the antibiotic stewardship world that’s a black hole” “We don’t even get data from the ED. I am only able to track antibiotic prescriptions inpatient. And its ever worse—I have no idea what happens in the community. And we think that’s important because if it’s true that antibiotic use affects rate of resistant in the community and if it’s true that most antibiotic use is in the community … then we are not doing a thorough job monitoring what’s been happening in our hospital. … So understanding how this relates to other communities would be very helpful” “The pharmacist worked with a group of other antibiotic stewardship programs. They looked at prescribing versus billing data and they found that it doesn’t always correlate well … medications get cancelled or people throw it up. Billing data doesn’t refer to what actually happened to the patient” “There’s people’s fears about the transparency because they’re concerned it could be used against them in some way, either through competitive marketing practices or just plain old they’ll worry about patients and families, seeing how they perform compared to others. There’s usual problems of risk adjustment if there’s a biased sample of patients at a particular hospital and there [sic] case mix is more severe, and that might not be adequately accounted for in any sort of benchmarking operation. There’s the effort involved in collecting the data”
		Complexities of the data	“I think physicians, a lot of them, and nurses get very excited when the [sic] go back to school to learn more of the stuff. … So I think that there’s a, there’s a results knowledge deficit that um, is not taught very well at medical school and residencies in all of these ways that we could probably do better at” “And it’s my reliability you know now, there’s some changes in SPS which is good in terms of not just picking out reliability with a bundle element, its reliability with the entire bundle … That is a lot of work, but that is the secret sauce. That is the way you should do it” “The biggest challenge for all of us is having case mix adjustment methods or varying adjustment methods or complexity adjustment methods that really reflect differences among patients. I think with children’s hospitals, the most important things we need to learn together about are less common diagnosis and conditions, and more complex conditions that we care for and um, and as soon as you get into those kinds of things, the numbers get smaller and the adjustments matter more”
		Metrics lack clear definitions	“… risks are like how apples to apples … operational definitions, are you really comparing performance?” “There’s lots of problems right now. There’s technical problems, you know being able to accurately collect the data to be able to accurately and consistently adhere to whatever metric definition has been put forward” “… surgeons saying ‘These definitions are useless!’ They mean nothing to us clinically, and don’t help with our improvement efforts’”
		There are perceived risks to benchmarking data	“There is another risk that people perceive that the patient safety organizations are addressing, which is I’d like to be able to share, I think the risk of transparency. I would like to be transparent and show you what we do and where our frailties are, and how do you get past them, but I’m afraid that if I do that it’s discoverable” “… there’s usual problems of risk adjustment if there’s a biased sample of patients at a particular hospital and there [sic] case mix is more severe, and that might not be adequately accounted for in any sort of benchmarking operation”
	Overcoming barriers	Leveraging the EHR	“It could be an adjustment or it could be stratification and I think um, I think our field is not quite there and we need to develop measures that use information from the EHR” “If we had a common data model using EHR search criteria, we would capture a lot of those individuals [currently being missed]” “I’m thinking about measures related to epilepsy as well. Thinking about you know, time to, seizure free period. Um, that you know might be a good outcome measure that would be helpful to everybody in terms of understanding best practices” “Orthopedic injuries and return to function, return to play, return to sports and school. I think that potentially could be helpful” “I think a lot of that is you know the power of the electronic health record really is not so much ’I can electronically pull a claims data’ or something. It’s not just cleaning up claims data, it’s more of can you implement decision support tools”
		Clearly define metrics and condition	“Well I think having you know a definition that’s probably been vetted by subject matter experts. Like people who are closest to um, taking care of the children in these areas would be important” “The quality of the data is definitely something that can be of concern. I think having a clear definition of what it is, that the conditions that we’re trying to describe, and what does or does not meet that definition”
		Better quality assurance of data is needed	“… trying to have tight interrater variability, if possible, so that we are not having a lot of differences and interpretation of the definition or application of the definition” “… some time piloting, testing, kind of testing the validity of the data would be good because one of the things that we encounter when we start to take data back to stakeholders is kind of an immediate ‘oh that can’t possibly be correct’ and if you have enough phenology down you’re able to say, ’no, it’s been tested and this has been vetted,’ you know this really is and you know here’s the definition and if it makes sense to the people who you know, have the power to drive the measure then you, you don’t spend time on that part of the change management” “The only problem is I think we feel like this type of data needs to be better validated” “I think it’s very valuable. I think having pediatric benchmarks is incredibly important; having large data sets is incredibly important; I think being able to identify peer institutions within those networks is also really really helpful”
Domain 2: Data may be of limited value	To HSLs		“[Value in the] ambulatory setting, I just don’t know how much they’re gonna care” “I think benchmark measures are most valuable when you see vulnerabilities or opportunities for improvement. So actually they have less value when you see yourself at the top” “I don’t know how that helps me benchmark, you know, whether I should send my patient to children’s for their asthma care or whatever. It’s not the right metric” (reports of data chosen beyond hospital’s control) “I think a lot of this is only we don’t have a lot of resources to do marketing, I think it helps with those that do marketing, and it might provide a little bit of trying to do bragging rights. I’m not sure it goes a whole lot further yet” “… outcome first of all of course is gonna be there. I think population mixed, so I think describing the denominator would be extremely helpful, and if you can come up with sort of interquartile ranges of line days, of types of line, of underlying issues, if you could agree on certain diagnoses, if we could come up with more accurate measures of disruption of the gut lining, if we understood the complexity across the board, putting that in with the outcome measure and then, while it’s very difficult to measure, it is a process” “Most centers report somewhere in the neighborhood of 95% compliance rates but when tested those numbers seem to be, depending on the site, tend to be in the 50%-60% range on average. And I think that’s a pretty universal trend in the data I’ve seen. Of course the numbers vary but the trend stands up. I think there’s clearly a patient population that we’re missing”
	To parents		“I think there’s probably a smaller proportion of the population that really does have a choice if they have a child with a very serious heart condition or neurologic condition. If they really would have the choice nationally to go looking to figure out which children’s hospital they would want their child to be at. I think probably for most of our patient population, they may not have as much of a choice” “I think it depends on the measure and it depends on how families see the link to better care and their health … Take sickle cell for example, you know families see that okay, if my kid gets a transcranial Doppler it’s really important to see what the risk of stroke is. I think that’s there’s value in that. First line antibiotics for otitis media, I don’t think families care that much as long as their child gets better”
Domain 3: HSLs described a variety of formats for presenting benchmarking data	Data trends		“It’s actually kind of getting it down to an actionable level because I think just sort of knowing ’Wow you’re an outlier,’ well it’s almost like the Pareto level of data”
	Simple presentations of data		“… I think short, very streamlined so that not just me and the people very invested would assimilated the something that could be without a lot of tweaking and still be brought to other kinds of meetings or stakeholders to be accessible to a wide range of stakeholders” “A clear visual layout without it being over worded”
	Physician-level data rather than institutional		“We’re trying to identify variation of care and there are 25 providers. If you know that really your performance is driven by two real outside of the, you know standard providers, you know exactly where to deliver that. It allows for individual ownership of performance”
Domain 4: Additional quality metrics were identified as candidates for future benchmarking	Alternative metrics may be more meaningful		“I think this is just a huge challenge so that we are getting really good valid data with good integrity and then I think it’s challenging to select things that would be broadly appealing …” “Things that we have evidence based measures around I think are more palatable to people in general but I think that’s a pretty important step as well. Being able to explain to the front line you know, why were these measures chosen?”
	Systemic measures, not condition-specific		“Take all the measures you have and so what is your overall rate from a patient perspective of, for how many infections did I, did my patients receive overall. Instead of just saying here is my infection, here is my falls … what percent of your patients experience one of these. You know ten percent of our patients either had a medication error, a clotting, a CLASBI, fall, something” “Look at that rate overall, and that gives you an index almost … what percentage of patients received perfect care in your hospital?”
	Condition-specific candidate measures		Outcomes that relate to health-related quality of life; CLABSI; CAUTI; lead screening; CBC; depression screening; immunizations; for persons with sickle cell anemia: whether offered hydroxyurea, eye exams, presentation to ED with acute chest syndrome, fetal hemoglobin >20%

Abbreviations: CAUTI, catheter-associated urinary tract infection; CBC, complete blood count; CLABSI, central line-associated bloodstream infection; ED, emergency department; EHR, electronic health record; HSL, health system leader; SPS, Solutions for Patient Safety.