Use of EHR-Based Pediatric Quality Measures: Views of Health System Leaders and Parents

Abstract

Measures of health care quality are produced from a variety of data sources, but often, physicians do not believe these measures reflect the quality of provided care. The aim was to assess the value to health system leaders (HSLs) and parents of benchmarking on health care quality measures using data mined from the electronic health record (EHR). Using in-context interviews with HSLs and parents, the authors investigated what new decisions and actions benchmarking using data mined from the EHR may enable and how benchmarking information should be presented to be most informative. Results demonstrate that although parents may have little experience using data on health care quality for decision making, they affirmed its potential value. HSLs expressed the need for high-confidence, validated metrics. They also perceived barriers to achieving meaningful metrics but recognized that mining data directly from the EHR could overcome those barriers. Parents and HSLs need high-confidence health care quality data to support decision making.

Benchmarking utilizes data to compare standardized health care quality measures (eg, National Quality Forum [NQF],¹ National Committee for Quality Assurance²). Data sources commonly include administrative sources such as billing records, surveys designed to assess patient experience (eg, Consumer Assessment of Healthcare Providers and Systems³), and manual chart abstraction, among others. In addition, electronic health records (EHRs) and registries (eg, Pinnacle,⁴ Health Quality Measure Format⁵) are sometimes used. Nonetheless, though physician practices invest more than $1.4 billion annually in reporting quality measures, only 30% of physicians believe that these measures reflect the quality of care provided.⁶ As a result, such measures are often underutilized to drive improvement activities by physician practices. Moreover, patients often do not utilize reported data to make health care decisions.^7,8 In pediatrics, the problem is compounded as measures designed for adults can be inappropriate when applied to children. Valid and actionable data are necessary to inform health system quality improvement (QI) and patient decision making alike. QI projects typically include performance benchmarking and continuous measurement of the area of focus.

Nonetheless, health system leaders (HSLs) need metrics in which stakeholders have confidence in order to undertake and evaluate QI efforts, and parents need data benchmarking on health care quality to inform decisions for their children’s health care. Mining data directly from the EHR documents a rich set of primary data in computable form for measuring and reporting health care quality. The Patient-Centered Outcomes Research Institute (PCORI) recently developed a network of clinical data research networks (called PCORnet) whose infrastructure improves EHR interoperability and provides rapid, automated data extraction and shared data governance.⁹ PCORnet includes 13 health system clinical data research networks, 20 patient-powered research networks, and 2 health plan research networks. Members extract specified data elements from their EHRs and transform them to an interoperable common data model¹⁰ in order to facilitate analyses across health systems.

This study investigates the value of using network data mined directly from the EHR for benchmarking on health care quality to HSLs and parents of children with chronic conditions. In-context interviews were conducted with HSLs and parents in order to investigate what new decisions and actions benchmarking using data mined from the EHR may enable and how benchmarking information should be presented to be most informative.

Methods

To ground the qualitative interviews in examples from real-world scenarios, the interviews focused on 3 distinct and diverse model quality measures from the NQF: the proportion of children with sickle cell anemia receiving transcranial Doppler (TCD) ultrasound screening, the proportion of children with ear infections receiving appropriate antibiotics, and the proportion of children on antipsychotic medications receiving metabolic screening. Current performance on these metrics demonstrates considerable room for improvement.^11–14 This study was reviewed and approved by the Cincinnati Children’s Hospital Medical Center institutional review board.

Setting and Participants

HSLs and parents were recruited from 2 PCORI Clinical Data Research Networks: PEDSnet and OneFlorida. PEDSnet, a network of 8 children’s hospitals with access to EHR data from more than 6 million children, conducts multi-institutional research to inform clinical care and leverages data to improve health outcomes through QI initiatives.¹⁵ OneFlorida is a statewide clinical research network and database that aims to unite researchers, clinicians, patients, and stakeholders to address some of the nation’s biggest health challenges and serves as a state resource for facilitating health care research and improving health, health care, and health policy.¹⁶ The site principal investigator (PI) from each PEDSnet hospital and the OneFlorida PI identified the head of health system safety and QI (eg, VP safety or chief medical officer [CMO]) from each site to participate in the HSL interviews. The PIs also recruited up to 3 English-speaking parents of children with sickle cell anemia, with a history of ear infections, or on antipsychotic medications and determined their preferred method of contact (email or phone). Potential parent participants were approached by the study clinical research coordinator (SJ) or by the study PI (KEW).

Interview Guide

Two semi-structured interview guides (one each for HSL and parent interviews) were developed by 5 authors (KEW, SJ, JS, DMH, AK). A parent (AK) coauthored the parent interview guide and an HSL (JS) coauthored the HSL interview. The draft parent interview guide was revised based on input from a Parent Advisory Panel (comprising 1 parent of a child with sickle cell anemia, 2 of children on antipsychotics, and 1 of a child with frequent ear infections). Similarly, the draft HSL interview guide was revised based on input from an HSL Advisory Panel of hospital CMOs, VPs Safety, and Directors of Ambulatory Quality. Advisory panel members were not eligible for interviews.

The final parent interview guide included 14 items related to their experience trying to access data on health care quality for themselves, their children, or other family members, the type of data they accessed, and how they used data. Each group of parents was asked specific questions. For example, parents of children with sickle cell anemia were asked about data comparing rates of annual TCD screening among care sites and how such data might change decisions about their child’s care or the source of their child’s care. Parents were asked how data would change interactions with the child’s doctor as well as any barriers to using such benchmarking data. They also were asked about the value of such benchmarking data when comparing children’s hospitals. Probes were employed (eg, “What makes the data valuable or not valuable?”; “What could make the data more valuable?”) in order to add depth to responses.

The 11-item HSL interview guide asked about their leadership role in their institution and their current use of external benchmarks. HSLs were asked how valuable they found data in benchmarking reports and what could make the data more valuable. They also were asked questions about the utility of benchmarking with data drawn from the EHR, including the best types of data for this, who could use those data, and how such data would be used. HSLs were asked about preferred formats for benchmarking data. As in the parent interviews, probes were employed to add depth to interview responses.

Interview Procedure

Following Midanik et al,¹⁷ phone interviews were conducted to minimize participant need to provide socially desirable responses. Thirty-minute interviews were conducted by one of 4 study staff trained by the senior author. Interview recordings were transcribed by a commercial medical transcription vendor and de-identified and verified for accuracy by study staff.

Analysis

A qualitative immersion/crystallization approach^18,19 was used to analyze interview transcripts. An initial set of themes was predefined based on the discussion guide, and new themes were added as they emerged from reading the first transcripts. A codebook was developed by 2 study team members who did a thorough immersive reading of all remaining transcripts to identify relevant portions of the transcripts. Data were coded by associating passages with themes. As concepts in participant responses crystallized (ie, as relationships between codes and themes were found and articulated), preliminary codes were revised and concepts organized into several classes, including emergent theories (most general), domains, themes, and categories (most particular). This continued through the first 5 interviews until the team concurred that the coding classes were appropriate and sufficient, and at that point, the codebook was considered locked. Coders read transcripts independently thereafter, coding participant statements into codebook classes. Discrepancies were resolved by consensus. Coding proceeded until the themes were fully understood (saturated). The study team followed Hennink et al²⁰ in distinguishing between code saturation (the point at which no new classes were added to the codebook) and meaning saturation (the point at which interview response concepts are comprehensively understood and no additional nuances or insights occur). Samples were assessed for both code and meaning saturation. The qualitative analytic software NVivo 11.0 (QSR International Pty, Ltd, Doncaster, Victoria, Australia) was used for analysis.

Results

A total of 32 interviews were necessary to reach saturation: 21 with parents and 12 with HSLs. Eight (38%) of the parents had children with a history of otitis media, 7 (33%) had children previously or currently on antipsychotic medication, and 6 (29%) had children with sickle cell anemia. The majority of parent interviewees were female (n = 19, 90%). HSLs included 5 hospital CMOs or chief quality officers, 1 vice chair for quality for pediatrics, 4 division directors or directors of quality for the divisions (general pediatrics, infectious disease, hematology), 1 director for ambulatory quality for the institution, and 1 director of quality analytics. Four HSL interviewees (31%) were female.

Parent Interviews

Parent interviews suggested one overall, emergent perspective: although generally naïve to benchmarking, parents believed that the availability of such information would inform and empower them in discussions with their child’s physician. Four domains emerged (Table 1). The first was that parental experience with “accessing health care quality data” equated to word-of-mouth recommendations and online searches for provider reviews. For example, one participant said, “[I] have a friend who worked at [a] hospital [and] gave [a] background recommend[ation] for [finding a doctor].” Parents expressed modest familiarity with published quality measures, tending instead to rely on peer reviews of the kind commonly found online or through social media. The use of social media and online resources was frequent: “[I] … basically put a post on Facebook and [asked] … all my friends.” Parents “tried to look online and get reviews,” often from blogs (“online websites and … other parents’ blogs”) or specialty websites (“if we do searches, we use … KidsHealth.org”). However, parents did not uniformly believe that online resources were adequate. One parent noted, for example, that “[it was] really hard in the context of mental health [where information is almost] nonexistent.”

Table 1.

Emergent Theory, Domains, and Supportive Quotations From Parent Interviews.

Emergent Theory: Although Parents Are Generally Naïve to Benchmarking Health Care Quality Data, They Believe That It Would Empower Their Parent-Physician Relationship
Domain	Quote
Domain 1: Parents use online provider reviews as their primary experience with data on health care quality	“I did a Facebook post and asked some of my friends what they thought; you know who to see …” “When my kids were little there was no internet. I had no way to search. Today I search everything before I do anything. I absolutely 100% think people would use it. I think different hospitals are better than other hospitals and I would 100% research before I took my child someplace” “… before [N.] gets hospitalized and before when I was trying to find providers for her, I tried to look online and get reviews and things like that … It was really hard in the context of mental health to get a lot of information. So I got some basic information, but most of it was nonexistent” “If a hospital presented information only on its own hospital … I would never expect to find something not good about a hospital [ … on their webpage]”
Domain 2: Having access to benchmarking on health care quality could empower parents and increase their confidence in conversations with providers	“I think that it’s a win-win because it’s going to empower parents to know how important it is to follow through with the test. It’s going to put a little bit more responsibility on the doctors say, hey you haven’t gotten the tests done I’m going to give you another form, please do it in the next week” “… when I am supposed to do the follow-up care. You know, she is taking medication and they are administering that, is she supposed to have blood work done every six months to check levels or something, I’m going to do that.” “If I had that I would at least have a discussion with them as opposed to just take whatever … like the doctors gave me a referral and I didn’t take it because when I googled it wasn’t good enough for me. She had just given it to my son. If I had been there I would have discussed it with her” “I would be more than empowered to have that conversation with my doctor, again having private insurance is important in knowing that is going to be covered but then also I want to go to the hospital this could be most knowledgeable … ” “I think it might be a good thing if they can justify what they’re doing, but sometimes when you’re coming in with data, that can be an uncomfortable thing for a doctor, when people are coming in and saying, you know, ‘it says here you’re supposed to be doing XYZ, why aren’t you?’” “I don’t think it would cause discomfort at all. I mean I guess it depends on where you’re going but here the culture is family provided care so the family provider is considered a critical part of the care team” “… if I’m seeing a new doctor, or if I’m going to a medical school where there are fellows and things like that, I always want to see the doctor who has done it for “x” amount of years. It always makes me feel better when I know somebody who has had the experience and that data helps drive that”
Domain 3: Parents use or imagine using data to make decisions regarding providers	“I definitely make an effort to go to a children’s hospital or someone that’s more savvy with kids with special needs because I can’t just walk into a quick diagnostic center” “I’d just bring it to discuss like is this normal that it’s given this often, … how much longer would you wait for you to consider tubes …? How long would they wait? And just kind of give them the report and say well this kind of saying this … and I would probably get one from another, the hospital just stating like why are they doing this, like, what’s, what’s the difference between” “If I would have known ‘this is the best place to go, they’re not going to give me medicine or give my child medicine that may not be effective or it could be successful but it couldn’t,’ I would totally choose a practice or physician that has the best ranking and success”
Domain 4: Multiple formats and delivery methods suggested to provide benchmarking data	“… I like numbers and if it’s in the form of a chart, that is fine too. Simple enough where you can just look at it and have a clear visual without it being overly worded” “I look at my email constantly you could send me an email and I’m going to open it and I’m going to look at it, but if there was a site you can go to see this information as you need it would be great, like some type of portal that was relevant” “[Formatting should be] definitely written, definitely graphed, and I would like to see feedback from actual patients” “It should come from the hospital because the hospital has data … So, I would say coming from the hospital and feeding it through an existing resource that that they already have access to … feed that information to MyChart because I’m going to get a notification” “I think a website, or if you want more information about this … a QR code that can bring up more information on peoples’ phones, if they wanted more information. That’s something I’m starting to do at my work on some of our flyers and things is QR codes on some of it, so that if people want more information, it can direct them to our website, or an article, or whatnot”

The second domain was that benchmarked quality measures could empower parents by informing and increasing their confidence in partnering with providers. One parent noted that “as a parent, the more you know, the more you can communicate with [your] child’s MD.” Parents described being able to shift their role from being passive receivers of information from their child’s doctor to becoming engaged co-producers of their child’s care. One parent of a child with sickle cell anemia said, “I would ask [an] MD if [a] report showed [that] her hospital didn’t offer TCD screening [as often as] others do.” Another parent of a child with frequent ear infections noted that “if I would have known [that] this would be the best medicine … [I] would have requested it first time.” Some questioned if access to measures of health care quality might create physician discomfort. One mother commented, “I would be more empowered to have that conversation with my doctor [if I had accessed quality data]”; another said, “[I could] … at least have a discussion with them as opposed to just take whatever doctors gave me.”

The third domain was that parents would use, if available, published measures of health care quality from reputable sources to make decisions regarding providers. Although some comments in this domain were based on parental experience, many of these comments were projections of what they imagined doing if benchmarking data were available to them. One parent commented, “I definitely make an effort to go to a children’s hospital or someone that’s savvy with kids with special needs because I can’t just walk into a quick diagnostic center” and another stated, “I would totally choose a practice or physician that has the best ranking and success.”

The fourth domain reflected a lack of consensus on formatting and accessing benchmarked data. Some parents expressed the desire for “… a clear visual without it being overly wordy,” whereas others emphasized a written passage accompanying a visual representation. Parents expressed the desire to access benchmarking data by email, online hospital portals, or a “… QR code that can bring up more information on people’s phones.”

Health System Leader Interviews

A single emergent theory explained HSL perspectives: they want meaningful, accurate measures of health care quality for benchmarking to inform QI and ultimately improve institutional performance. Illustrative comments include, “Without knowing where we are [compared to other hospitals] … it’s hard to get a handle on where to aim [improvement efforts].” HSLs were generally interested in sharing data to improve child health outcomes and realized that sharing requires broad participation to understand the variation in practice necessary to achieve change. Four domains were identified (Table 2).

Table 2.

Emergent Theory, Domains, and Supportive Quotations From Health System Leader Interviews.

Emergent Theory: Health System Leaders Want Meaningful Health Care Quality Data to Improve Performance. Supportive Quotations: “Would help us work together”; “The concept of SPS is to work together and not compete on safety”; “Without knowing where we are outliers, it’s hard to get a handle on where to get an aim.”
Domain	Theme	Categories	Example Quotations
Domain 1: Numerous barriers to meaningful benchmarking exist and are not insurmountable	Barriers	Data are difficult to collect	“It would be valuable to know are we an outlier in either direction—are we using far too much antibiotics compared to other hospitals or are we on the other extreme. I think that’s where data would be interesting and especially if you have access to the outpatient data. In the antibiotic stewardship world that’s a black hole” “We don’t even get data from the ED. I am only able to track antibiotic prescriptions inpatient. And its ever worse—I have no idea what happens in the community. And we think that’s important because if it’s true that antibiotic use affects rate of resistant in the community and if it’s true that most antibiotic use is in the community … then we are not doing a thorough job monitoring what’s been happening in our hospital. … So understanding how this relates to other communities would be very helpful” “The pharmacist worked with a group of other antibiotic stewardship programs. They looked at prescribing versus billing data and they found that it doesn’t always correlate well … medications get cancelled or people throw it up. Billing data doesn’t refer to what actually happened to the patient” “There’s people’s fears about the transparency because they’re concerned it could be used against them in some way, either through competitive marketing practices or just plain old they’ll worry about patients and families, seeing how they perform compared to others. There’s usual problems of risk adjustment if there’s a biased sample of patients at a particular hospital and there [sic] case mix is more severe, and that might not be adequately accounted for in any sort of benchmarking operation. There’s the effort involved in collecting the data”
		Complexities of the data	“I think physicians, a lot of them, and nurses get very excited when the [sic] go back to school to learn more of the stuff. … So I think that there’s a, there’s a results knowledge deficit that um, is not taught very well at medical school and residencies in all of these ways that we could probably do better at” “And it’s my reliability you know now, there’s some changes in SPS which is good in terms of not just picking out reliability with a bundle element, its reliability with the entire bundle … That is a lot of work, but that is the secret sauce. That is the way you should do it” “The biggest challenge for all of us is having case mix adjustment methods or varying adjustment methods or complexity adjustment methods that really reflect differences among patients. I think with children’s hospitals, the most important things we need to learn together about are less common diagnosis and conditions, and more complex conditions that we care for and um, and as soon as you get into those kinds of things, the numbers get smaller and the adjustments matter more”
		Metrics lack clear definitions	“… risks are like how apples to apples … operational definitions, are you really comparing performance?” “There’s lots of problems right now. There’s technical problems, you know being able to accurately collect the data to be able to accurately and consistently adhere to whatever metric definition has been put forward” “… surgeons saying ‘These definitions are useless!’ They mean nothing to us clinically, and don’t help with our improvement efforts’”
		There are perceived risks to benchmarking data	“There is another risk that people perceive that the patient safety organizations are addressing, which is I’d like to be able to share, I think the risk of transparency. I would like to be transparent and show you what we do and where our frailties are, and how do you get past them, but I’m afraid that if I do that it’s discoverable” “… there’s usual problems of risk adjustment if there’s a biased sample of patients at a particular hospital and there [sic] case mix is more severe, and that might not be adequately accounted for in any sort of benchmarking operation”
	Overcoming barriers	Leveraging the EHR	“It could be an adjustment or it could be stratification and I think um, I think our field is not quite there and we need to develop measures that use information from the EHR” “If we had a common data model using EHR search criteria, we would capture a lot of those individuals [currently being missed]” “I’m thinking about measures related to epilepsy as well. Thinking about you know, time to, seizure free period. Um, that you know might be a good outcome measure that would be helpful to everybody in terms of understanding best practices” “Orthopedic injuries and return to function, return to play, return to sports and school. I think that potentially could be helpful” “I think a lot of that is you know the power of the electronic health record really is not so much ’I can electronically pull a claims data’ or something. It’s not just cleaning up claims data, it’s more of can you implement decision support tools”
		Clearly define metrics and condition	“Well I think having you know a definition that’s probably been vetted by subject matter experts. Like people who are closest to um, taking care of the children in these areas would be important” “The quality of the data is definitely something that can be of concern. I think having a clear definition of what it is, that the conditions that we’re trying to describe, and what does or does not meet that definition”
		Better quality assurance of data is needed	“… trying to have tight interrater variability, if possible, so that we are not having a lot of differences and interpretation of the definition or application of the definition” “… some time piloting, testing, kind of testing the validity of the data would be good because one of the things that we encounter when we start to take data back to stakeholders is kind of an immediate ‘oh that can’t possibly be correct’ and if you have enough phenology down you’re able to say, ’no, it’s been tested and this has been vetted,’ you know this really is and you know here’s the definition and if it makes sense to the people who you know, have the power to drive the measure then you, you don’t spend time on that part of the change management” “The only problem is I think we feel like this type of data needs to be better validated” “I think it’s very valuable. I think having pediatric benchmarks is incredibly important; having large data sets is incredibly important; I think being able to identify peer institutions within those networks is also really really helpful”
Domain 2: Data may be of limited value	To HSLs		“[Value in the] ambulatory setting, I just don’t know how much they’re gonna care” “I think benchmark measures are most valuable when you see vulnerabilities or opportunities for improvement. So actually they have less value when you see yourself at the top” “I don’t know how that helps me benchmark, you know, whether I should send my patient to children’s for their asthma care or whatever. It’s not the right metric” (reports of data chosen beyond hospital’s control) “I think a lot of this is only we don’t have a lot of resources to do marketing, I think it helps with those that do marketing, and it might provide a little bit of trying to do bragging rights. I’m not sure it goes a whole lot further yet” “… outcome first of all of course is gonna be there. I think population mixed, so I think describing the denominator would be extremely helpful, and if you can come up with sort of interquartile ranges of line days, of types of line, of underlying issues, if you could agree on certain diagnoses, if we could come up with more accurate measures of disruption of the gut lining, if we understood the complexity across the board, putting that in with the outcome measure and then, while it’s very difficult to measure, it is a process” “Most centers report somewhere in the neighborhood of 95% compliance rates but when tested those numbers seem to be, depending on the site, tend to be in the 50%-60% range on average. And I think that’s a pretty universal trend in the data I’ve seen. Of course the numbers vary but the trend stands up. I think there’s clearly a patient population that we’re missing”
	To parents		“I think there’s probably a smaller proportion of the population that really does have a choice if they have a child with a very serious heart condition or neurologic condition. If they really would have the choice nationally to go looking to figure out which children’s hospital they would want their child to be at. I think probably for most of our patient population, they may not have as much of a choice” “I think it depends on the measure and it depends on how families see the link to better care and their health … Take sickle cell for example, you know families see that okay, if my kid gets a transcranial Doppler it’s really important to see what the risk of stroke is. I think that’s there’s value in that. First line antibiotics for otitis media, I don’t think families care that much as long as their child gets better”
Domain 3: HSLs described a variety of formats for presenting benchmarking data	Data trends		“It’s actually kind of getting it down to an actionable level because I think just sort of knowing ’Wow you’re an outlier,’ well it’s almost like the Pareto level of data”
	Simple presentations of data		“… I think short, very streamlined so that not just me and the people very invested would assimilated the something that could be without a lot of tweaking and still be brought to other kinds of meetings or stakeholders to be accessible to a wide range of stakeholders” “A clear visual layout without it being over worded”
	Physician-level data rather than institutional		“We’re trying to identify variation of care and there are 25 providers. If you know that really your performance is driven by two real outside of the, you know standard providers, you know exactly where to deliver that. It allows for individual ownership of performance”
Domain 4: Additional quality metrics were identified as candidates for future benchmarking	Alternative metrics may be more meaningful		“I think this is just a huge challenge so that we are getting really good valid data with good integrity and then I think it’s challenging to select things that would be broadly appealing …” “Things that we have evidence based measures around I think are more palatable to people in general but I think that’s a pretty important step as well. Being able to explain to the front line you know, why were these measures chosen?”
	Systemic measures, not condition-specific		“Take all the measures you have and so what is your overall rate from a patient perspective of, for how many infections did I, did my patients receive overall. Instead of just saying here is my infection, here is my falls … what percent of your patients experience one of these. You know ten percent of our patients either had a medication error, a clotting, a CLASBI, fall, something” “Look at that rate overall, and that gives you an index almost … what percentage of patients received perfect care in your hospital?”
	Condition-specific candidate measures		Outcomes that relate to health-related quality of life; CLABSI; CAUTI; lead screening; CBC; depression screening; immunizations; for persons with sickle cell anemia: whether offered hydroxyurea, eye exams, presentation to ED with acute chest syndrome, fetal hemoglobin >20%

Abbreviations: CAUTI, catheter-associated urinary tract infection; CBC, complete blood count; CLABSI, central line-associated bloodstream infection; ED, emergency department; EHR, electronic health record; HSL, health system leader; SPS, Solutions for Patient Safety.

The first domain was that although barriers to meaningful benchmarking exist, they are not considered insurmountable. This domain had 2 themes: barriers and overcoming barriers. HSL comments described barriers in 4 categories. The first was difficulty of data collection. Some noted that some data are only available in specific clinical settings (eg, “I can only track antibiotic prescriptions inpatient”). Another noted that billing data do not capture what actually happens to the patient, “The pharmacist worked with a group of other antibiotic stewardship programs. They looked at prescribing versus billing data and they found that it doesn’t always correlate well … medications get cancelled or people throw it out.” A second category was data complexity and accuracy. Some noted the need for adjustment to account for differences in patient acuity between institution and differences in the use of billing data across institutions, “metric[s] … measuring [guideline] compliance at other institutions … [are not necessarily] apples to apples [comparisons].” A third category was that existing quality metrics, as currently defined, are not perceived as relevant. This category was affirmed by nearly all interviewees. One HSL noted the difficulty with “operational definitions” and asked whether “you [are] really comparing performance.” HSLs believed that definitions need to be clinically meaningful to be helpful in QI efforts. For example, one person noted that “if you ask somebody [to] ‘change your practice’ they’ll say ‘what’s the evidence?’” and another observed that “the more objective [the metric, and] the more quantitative, the better.” The fourth category was perceived risks in benchmarking on health care quality. One was the concern that data on health care quality are potentially discoverable in legal proceedings. Another concern was that transparent disclosure of data to patients or the public could decrease an institution’s reputation or be used by competitors to enhance their marketing.

The second theme was overcoming barriers and had 3 categories. The first category was using the EHR to overcome barriers. HSLs discussed utilizing the EHR to capture clinically meaningful data such as the “seizure-free period in epilepsy” or the time between “orthopedic injury and return to function.” They also observed that the EHR can provide data currently missed because of “… a lack of a common data model using EHR search criteria.” The second category was credible and clinically meaningful measure definitions. They discussed that definition credibility could be accomplished by having the definition created or vetted by “subject matter experts, like people who are closest to taking care of the children in these areas, would be important.” The third category was improved data quality assurance. HSLs believed that inconsistent data quality could be overcome by pilot testing the validity of data drawn from the EHR to ensure a “statistically sound data set.”

The second domain was that currently available health care quality data are of limited value to HSLs. They questioned the accuracy and meaningfulness of current measures, the applicability of existing measures to different settings (eg, inpatient vs outpatient), and the relevance of measures to current hospital performance. For example, one HSL said, “In an ambulatory setting, I don’t know how much they’re going to care [about current measures],” and another stated that measures “… provide bragging rights. I’m not sure it goes a whole lot further yet.” In contrast to parental views, HSLs did not see the value of benchmarking quality data to parents given the potentially limited freedom to choose providers or health care institutions (eg, because there is only one local children’s hospital). One HSL observed, “… for otitis media, I don’t think families care that much as long as their kid gets better.”

The third domain was that HSLs, like parents, described a variety of formats for presenting benchmarking measures. HSLs emphasized the need for simple data displays that are “short … [and] very streamlined” and that emphasize visual over text. Some HSLs wanted to see trends. Some HSLs were interested in provider-level data being available, in addition to their institutional data, to enhance “ownership of performance” by clinicians.

HSLs also were asked to suggest quality metrics for future benchmarking efforts. Candidate measures included proper screening for risk of lead poisoning, developmental delay, and symptoms of depression. Others related to medical decision making such as appropriate antimicrobial selection. Some HSLs advocated creating more global measures of care quality (eg, “What percentage of patients received perfect care in your hospital?”) rather than more specific individual measures. The difficulty in metric definition and the desire for meaningful measures led some to suggest that benchmarking quality metrics should be based only on “things we have evidence-based measures around.”

Discussion

This study presents an analysis of HSL and parent interviews regarding the perceived usefulness of data mined from the EHR for benchmarking on health care quality. Interviews with parents revealed a desire for information helpful to those seeking outpatient care for their children. Parents in this study had little experience using such data, possibly because at present there is little information accessible or guidance on use when it is available. If available, parents imagined using quality measures to make decisions, provided that insurance does not limit their decision-making autonomy. HSLs also expressed a need for meaningful, validated metrics in which they have confidence to support effective QI programs. They saw barriers to achieving meaningful metrics for their own use but recognized that such barriers could be overcome by mining EHR data and careful metric definition. Moreover, they suggested potential future benchmarks. HSLs and parents preferred simple, precise metrics to inform their decision making and valued graphic representations.

Parent interviews suggested that quality measures can play a role in provider selection, increasing confidence in conversations with providers and informing care and treatment decisions. How these relate to a parent’s engagement in their child’s health care, their health care literacy, and other factors is unclear. Nor is it clear that parents have the autonomy to act on such information if available, given variation in insurance policies and coverage.

Present study findings are broadly consistent with previous research.²¹ A qualitative study of reporting of health care-associated infection rates by Mazor and Dodd⁸ found that consumers, who were largely unaware of the problem of health care–associated infection or the availability of consumer data on rates of infection, did not believe that the availability of consumer reports on infection rates would change their decisions regarding location of care. The present study also found that parents did not envision changing health systems based on measures of care quality. Similar to Mazor and Dodd, this study found parents to have a variety of needs regarding reporting content and format. The study team is not aware of prior studies of HSL views on reporting of quality metrics. HSL views on the need for high-confidence, credible quality measures are consistent with prior research that shows that health systems spend $15.4 billion reporting quality metrics, but only 30% believe that the metrics actually reflect quality of care, and only 30% are used in QI initiatives.⁶ HSL views also are consistent with an editorial by Panzer et al²² highlighting that current quality metrics based on billing data are limited and narrow, with variation in billing data hindering direct comparisons.

The quotes summarized in the tables also suggest practical approaches to potential implementation strategies designed to engage patients, leaders, and clinicians to make further improvements in patient care. For example, Table 1 suggests that increasing benchmarking data on mental health care quality (“It was really hard in the context of mental health to get a lot of information”), educating parents on quality metrics relevant to their child’s care (“when I am supposed to do the follow-up care … is she supposed to have blood work done every six months to check levels or something …”), making data relevant to current health care decision making available (“I’d just … discuss like is this normal that it’s given this often … how much longer would you wait for you to consider tubes?”), and providing such information in different formats for best uptake (“[Formatting should be] definitely written, definitely graphed …”) may all be beneficial and well received by parents. Structured interviews with such stakeholders can inform the evolution of quality measures and benchmarking.

Although this study included participants from 10 sites nationally and from 2 important stakeholder groups (parents and HSLs), qualitatively it has limitations. The study utilized English-speaking participants, but non–English-speaking parents may have different objectives and perceptions of health care. Similarly, although 2 critical stakeholder groups—HSLs and parents—were interviewed, the study team did not obtain the perspective of all stakeholders.

These findings provide a foundation for future studies regarding the use of EHR data by health systems to improve health outcomes collaboratively. Such future studies may explore the development of an engagement community focused on transparency and use of quality measurement from clinical data. Such a community would support testing, development, and dissemination of valid quality metrics that are understood by, and perceived as useful to, all stakeholders.