Integration of Clinical Decision Support and Electronic Clinical Quality Measurement: Domain Expert Insights and Implications for Future Direction

Abstract

The integration of clinical decision support (CDS) and electronic clinical quality measurement (eCQM) could help improve consistency, reduce redundancy, and ultimately help improve value. To guide efforts in this area, 15 leading experts in CDS and eCQM were interviewed to obtain insights on how CDS and eCQM could be better integrated. Four main themes emerged: cultural and business considerations for CDS, eCQM, and their integration trump the technical considerations; the purpose and goals of CDS and eCQM differ, and these differences must be accounted for; there is an oftentimes invisible overlap between CDS and eCQM, and with the larger domain of quality improvement; and despite the differences, synergies between CDS and eCQM should be pursued to amplify the effectiveness of each approach. A key implication of these findings is the need to bridge cultural and business differences between CDS and eCQM, with informaticists potentially playing a critical role.

Introduction

Clinical decision support (CDS) systems and electronic clinical quality measurement (eCQM) are two important computer-based strategies aimed at improving the quality of healthcare.¹ Integrating these two approaches could help enable Learning Health Care System transformations.² Despite the similarities of their respective goals, efforts to integrate CDS and eCQM have met with difficulties. The purpose of this paper is to explore this issue through the lens of those individuals who have experienced the process directly.

For the purposes of this study, we define CDS as the provision of pertinent knowledge and person-specific information to clinical decision makers to enhance health and healthcare.³ Examples of CDS tools include alerts, order sets, care plans, protocols, documentation templates and tools, relevant data summaries, and dashboards. Such CDS tools can help clinicians provide evidence-based care for a specific individual or for a population of patients.⁴

In turn, we define eCQM as the measurement and tracking of the quality of healthcare services using electronic data. Clinical quality measurement results are used in reports and feedback on clinician performance, accreditation review and institutional performance metrics. Clinical quality measurement (QM) has a longer history than CDS, and it focuses on systems and the processes of care and traditionally uses manual chart abstraction. Institutional QM is transitioning towards electronic data extraction and assessment to facilitate ongoing performance measurement.

CDS and eCQM fundamentally address the same issue of identifying patients who should receive particular health or administrative interventions and monitoring the delivery of those interventions.^5–7 Both CDS and eCQM can also help improve quality by providing feedback to relevant stakeholders.⁸ Coordinating the vision, processes, and technologies associated with each of these domains (i.e., integration) of CDS and eCQM has the potential to improve healthcare value.^9–11 CDS can facilitate the collection of data elements needed for quality measures, and eCQM results can support iterative, data-driven refinement and implementation of CDS initiatives. Other potential positive outcomes of integrating CDS and eCQM include reducing duplication of effort and minimizing inconsistencies in guidance recommendations.

Recognizing the importance of such integration, groups including the U.S. federal government and the Institute of Medicine (IOM) are advising healthcare providers to tighten the feedback loop between CDS and eCQM.^12,13 An improved ability to establish such a virtuous feedback loop between quality improvement (QI) and continuous performance measurement is an important enabler for becoming a Learning Health Care System. Notably, the U.S. Office of the National Coordinator (ONC) for Health Information Technology (IT) and the Centers for Medicare & Medicaid Services (CMS) sponsored the public-private Clinical Quality Framework (CQF) initiative to harmonize health IT standards for CDS and eCQM to facilitate their integrated implementation.¹⁴ Several standards and technological solutions have been suggested to enable integration of CDS and eCQM.^15–20

Despite this recognition of the importance of integrating CDS and eCQM, fully-integrated QI approaches are still rarely used and only sporadically reported in the literature. For example, only three out of 160 randomized clinical trials included in a review of CDS systems by Lobach et al. were accompanied by periodic performance feedback.⁴ Additionally, when an integrated approach is used, it is often incomplete. For example, coordinated CDS and eCQM efforts based on commercial EHR systems oftentimes use different tools for CDS and for eCQM implementations.^11,21 Moreover, family physicians report a lack of QI infrastructure to co-deliver CDS and eCQM in their practices.²² Finally, aspects of organizational culture and structure that inhibit integration of CDS and eCQM are poorly described in the literature. In summary, there is a need for research to characterize how CDS and eCQM can be better integrated to improve care. To address this need, we interviewed experts in the field to identify potential approaches for advancing the integration of CDS and eCQM moving forward.

Objective

This study aimed to explore the beliefs and perceptions regarding the integration of CDS and eCQM functionality and activities within healthcare organizations, using qualitative methods that engage subject matter experts (SMEs). Our specific objectives were to (1) identify expert insights into the integration of CDS and eCQM and (2) describe implications of those insights on future work in this area.

Materials and Methods

Study Design and Research Team

A qualitative study was conducted using in-depth semi-structured interviews with SMEs. The critical incident technique was used to identify the goals, mental models, and behavioral strategies of the SMEs when engaging in this work.²³ In addition, general questions were used to identify the components related to the key challenges in CDS and eCQM integration.

The study was conducted by a multidisciplinary research team with experience in CDS, eCQM, clinical and public health informatics, standards-based interoperability, qualitative methods, cognitive task analysis, biostatistics, and information technology. The study was approved by the University of Utah Institutional Review Board (Protocol # 00077948).

Subjects

SMEs were enrolled through an open invitation for participation made via email to relevant email listservs sponsored by the American Medical Informatics Association (AMIA) CDS and Implementation work groups, the Health Level 7 (HL7) Clinical Quality Information and CDS work groups, and the Clinical Quality Framework (CQF) initiative.¹⁴ In order to maximize the representation of relevant expert insights, invitations were also sent to individuals identified as being key SMEs based on literature review and by the study personnel. Participation was open to all interested professionals in the field who had both of the following qualifications: a) Experience developing or using a quality measurement system; and b) Experience developing or using CDS interventions

Thirty individuals responded initially and 15 individuals met the criteria and proceeded with the interview (Table 1). It has been previously shown that 12-13 interviews could be sufficient to gather a majority of insights.²⁴ Thus, we did not send any new invitations after conducting the 15 interviews.

Table 1.

Participants

Name of participant	Role/title	Name of Organization	Type of organization
Howard Bregman, MD, MS	Director, Clinical Informatics	Epic, WI	EHR vendor
Nathaniel Weiner, MS	Co-Founder, Chief Operating Officer	Avhana Health, MD	CDS vendor
Clement J. McDonald, MD	Director	U.S. National Library of Medicine Lister Hill National Center for Biomedical Communications, MD	Government, (and previous employment: academia, healthcare provider)
Samson Tu, MS	Senior Research Scientist	Stanford University, CA VA Palo Alto Healthcare System, CA	Academia, government, healthcare provider
Art Wallace, MD, PhD	Chief, Anesthesia Service	San Francisco Veterans Affair Medical Center, CA	Government, healthcare provider
Adam Wright, PhD	Associate Professor of Medicine	Harvard Medical School, MA	Academia, healthcare provider
Keith Marsolo, PhD	Associate Professor	Cincinnati Children′s Hospital Medical Center, OH	Academia, healthcare provider
Keith F. Woeltje, MD, PhD	Vice president, Chief Medical Informatics Officer	BJC HealthCare, MO	Academia, healthcare provider
Hojjat Salmasian, MD, MPH, PhD	Program Director of Research Science	Value Institute, NewYork-Presbyterian Hospital, NY	Healthcare provider
Joseph Kunisch PhD, RN-BC, CPHQ	Enterprise Director for Clinical Quality Informatics- Regulatory Performance	Memorial Hermann Hospital System, TX	Healthcare provider
Benjamin Brown, MRCGP, MSc	General Practitioner and Research Training Fellow	University of Manchester, UK	Healthcare provider
Jerome A. Osheroff, MD	Founder/Principal	TMIT Consulting, LLC, FL	Consultant
Michelle Currie, RN, MSN, CPHQ, CPHIMS	Founder and Healthcare Solution Architect	Savant Solutions4HIT, LLC, CA	Consultant
William Salomon, MD, MS MPH	Senior Medical Informatician	Clinical Metrics, Limited liability company, ME	Consultant
James McCormack, PhD	Instructor - Health IT Project Management	Oregon Health & Science University, OR	Consultant

At the beginning of each interview, verbal consent was obtained to participate in the study, record and transcribe the interview, and include participants’ names in publications. A financial incentive ($40) was offered to the participants for their time, but some participants declined.

Fifteen SMEs with diverse backgrounds and organizational experience participated, including executives and other leaders from academia, government, healthcare provider organizations, consulting companies, and CDS and electronic health record (EHR) system vendors (Table 1). SMEs were geographically distributed throughout the U.S. and included one participant from the United Kingdom.

For 11 of the SMEs, CDS was encountered first in their career. Among these 11 participants who encountered CDS first, five remain currently more experienced in CDS, one is now more experienced in QM, and five reported being equally experienced in both domains. In contrast, among the four participants who encountered QM first in their career, three remain currently more experienced with QM, while one is now more experienced with CDS.

Interviews

One-hour in-depth semi-structured interviews included three parts: (1) questions about the participants’ background and experience with CDS and eCQM, (2) critical incident questions on a challenging project in which the participant had used both CDS and eCQM, and (3) general questions about the integration of CDS and eCQM (Table 2). We did not include a pre-specified and constrained definition of the “integration” construct in the interview script so as to provide the respondent with flexibility to discuss any aspects of potential integration that they felt were important.

Table 2.

Questions about a Critical Incident and Integration of CDS and eCQM

Topic	Question
Critical Incident	What were your overall goals during the project? What CDS was involved in the project? What eCQM? How many groups were involved in this project? What roles did other people have in the project? What were the main decision points in this project? Let me verify the sequence of events… Do I have the sequence and the details right so far? What was challenging about the project? Were there any technical challenges? How much time pressure was involved? How much social pressure was involved? Did you have any communication problems? Were there any internal politics involved? Did events occur as expected? What surprised you? What was successful about the project? How would have things happened differently if there was coordinated governance which was addressing both CDS and eCQM in your institution? Would it make your job easier if it was possible to implement Clinical Decision Support and electronic Clinical Quality Measurement using the same standards and approaches?
CDS-eCQM Integration	Please describe benefits to an integrated approach to CDS and eCQM. Please describe similarities in CDS and eCQM implementation or use. Please describe differences in CDS and eCQM implementation or use. Please describe technical barriers to integrating CDS and eCQM. Please describe non-technical barriers to integrating CDS and eCQM. Please describe guiding principles for CDS-eCQM integration in your opinion. Do you have any other insights or thoughts on the integration of CDS and eCQM?

Questions about the participants’ background and experience concerned their current organizational role, the type of organization, whether they encountered CDS or eCQM first in their career, whether they had more experience with CDS or eCQM, and the degree of integration between CDS and eCQM in their organization.

The critical incident technique a) allows for the collection of rich data from the respondents’ perspectives and in their own words without forcing them into any given framework, b) facilitates recall of an event to support related contextual information, c) allows identification of goals and mental models regarding workflow, and d) allows identification of rare events that might be missed by other methods that only focus on common and everyday events.²⁵ The critical incident methodology was adapted from Crandall et al.’s cognitive work analysis methods where a four-phase format was used: (1) incident identification, (2) timeline verification, (3) deepening, and (4) ‘what-if’ queries (Table 2). First, we asked the interviewee to identify a specific project where he/she used both CDS and eCQM. Second, we asked the participant to provide a time-based description of the sequence of tasks to create an explicit timeline. Third, we asked a set of more specific questions to identify and verify project goals, social context, organizational issues, challenges, and decision points. Finally, a few ‘what-if’ questions were posed to explore what could have been done differently under critical relevant conditions.

General questions about integration of CDS and eCQM included questions about similarities and differences between CDS and eCQM implementation and use, technical and non-technical barriers to the integration of CDS and eCQM, and recommendations to reach a higher degree of integration (Table 2).

Interviews were recorded and transcribed verbatim. All responses were used for the analysis. The research was approved by the University of Utah Institutional Review Board.

Data analysis

The interviews, including answers to critical incident questions and general questions about integration of CDS and eCQM, were analyzed using content analysis techniques described by Patton.²⁶ Thematic analysis of the transcripts was conducted by three faculty-level clinical informaticists with significant experience in both CDS and eCQM (KK, CRW, CS). Relevant responses were reviewed at the paragraph level, and the team converted responses to condensed descriptions that preserved the meaning of the response. Then, related condensed descriptions and corresponding responses were discussed to identify underlying constructs. The research team discussion was iterative, with condensed descriptions discussed, reviewed, and then reviewed again until no new constructs emerged. Constructs were then aggregated into thematic statements in order to elucidate the “gist” of the content.

Insights were not tied to any specific individual study participant. In fact, exemplar statements supporting each theme were usually found among the responses from multiple participants. Based on identified themes, the study authors formulated key study implications by consensus. Because these implications represent the opinions of the authors, they are described in the Discussion section of this manuscript.

Results

Theme 1: Cultural and business considerations for CDS, eCQM, and their integration trump the technical considerations.

While many of the recent informatics efforts in the integration of CDS and eCQM have focused on the technical aspects of such integration, SMEs repeatedly noted that in pursuing CDS, eCQM, and the integration of these two aspects of QI, technology was secondary to cultural and business considerations. For example, one SME noted:

These are not technology projects. These are people projects… The challenges that you run into, the opportunities that you have to move things forward follow along those categories: people issues, process issues and technology issues. And I think the hardest and most important thing is people stuff… It has to do with incentives, and business cases, and the real needs in the real world. So, I think there are challenges across all those dimensions: people, processes and technology. But the most complicated ones, the most difficult ones are the people, followed by the processes, followed by the technology. I think the technology is probably the least challenging which is not the same as they’re not significant, but I think compared to others in those three categories they are less significant.

Similarly, another SME noted:

Always, always, always approach it from a business perspective. I think that the business or the clinical leadership should be leading and facilitating the conversations, not the technical people. The technical people are there to enable and to support operations and patient care.

SMEs also noted that eCQMs are often driven by regulatory or payment incentives. While CDS can be driven by the need to support improved performance on such eCQMs, CDS tends to focus on facilitating the practice of evidence-based medicine goals that emerge from clinical practice leaders. While, ideally, the driving forces between eCQM and CDS are aligned, SMEs noted that challenges arise when they are not aligned. For example, one SME noted:

A lot of our quality measures for better or worse right now come from the federal government or from insurance companies, or they’re from national quality measurement programs we participate in. The CDS historically comes from clinical needs that we felt. So, if things are going right, the national quality measure is well aligned with the clinical needs that we’re feeling, but if they’re not, then I think that could be an important difference.

Echoing this sentiment, another SME noted:

You can’t have the CDS driven by guidelines, if it is discrepant from the payment mechanisms because you may be chasing unobtainable standards that you’re not being paid for which at the end of the day you need to have— you need to be paid to deliver the care that you can, otherwise the system will collapse.

Another common insight from the SMEs was that the individuals engaged in CDS and the individuals engaged in eCQM often worked in distinct teams, each with distinct social foundations, cultural norms, and perspectives. Indeed, CDS has emerged from clinical interests in evidence-based medicine and are largely led by clinician informaticists, whereas eCQM emerged from a history of QI and performance measurement. CDS and eCQM teams often are distinct groups, and this separation in terms of teams and culture can lead to challenge in integration and coordination. As noted by one SME:

There were some interesting political dynamics which I think is very common. I see a lot where a certain group wants to be the group that solves the problem, so they can either get the recognition or they can substantiate their position. Or they can substantiate asking for additional resources. [Nursing] would have liked to say that they solved it all by themselves. I think the IT department would really like to say that they’ve solved this problem by themselves. But it really took everybody coming together and instead of worrying about who is going to solve the problem, we really just focus on the patient and what was best for the patient.

Two other SMEs similarly noted:

They have IT people who work on the CDS and then they have quality people who work on the quality metrics. And these two groups are kind of siloed and separate from each other.

Well, the decision support [people] tend to come up through the IT or the Informatics side of things and then there’s quality measurement, those people come from the health services research or analytical background. So, you have two ways of viewing the world, different terminology, and just different ways of talking and things like that. It does take some active work to make sure that those groups really work together.

In describing this cultural and organizational gap between the CDS and eCQM worlds, many SMEs identified the need for individuals who understand both worlds and can serve as a bridge. For example, a SME noted:

If at all possible it’s important to have somebody who can translate between the two sides. And although that is a difficult skill set to find, I would even go as far as to say hiring an outside consultant who has the ability to translate between the two departments will get you results much quicker and at a much less cost than trying to do it internally, if there isn’t somebody who can really translate between the two sides. Because you really need someone who understands both sides as well, who understands the workflow.

Moreover, a SME specifically noted that informaticists are ideally situated to play this critical bridging role:

That’s part of what informaticians are supposed to do. It’s to basically be bi-functional because none of these people have the slightest idea of what each other’s talking about. We have to figure this out.

Theme 2: The purpose and goals of CDS and eCQM differ, and these differences must be accounted for.

The SMEs agreed that CDS and eCQM have much in common, including the high-level goal of improving care quality. However, they identified important ways in which they differ, sometimes in ways that the SMEs found surprising. As one important difference, many SMEs noted that because eCQMs are focused on enabling meaningful comparison across patient populations, they tend to have many exclusion criteria to remove clinical outliers, whereas CDS is focused on supporting clinical decision making for as many relevant patients as possible, and therefore have less restrictive exclusion criteria. For example, one SME noted:

What was surprising about the project was how different these performance measures are from the CDS guidelines. This project really helped us to sharpen the difference between the goals of performance measures and the goals of the CDS. So for performance measures… it defines the denominator very strictly… broadly speaking they are the same, they are patients with heart failure. But the denominator definition of the heart failure performance measures is different from the target population for whom the CDS is supposed to provide management assistance… What we found was that the target populations of a performance measure tend to be much narrower than the target population of the analogous CDS.

While one potential approach to addressing these discrepant population definitions would be to simply use the eCQM definition, which may be tied to regulatory or payment requirements, one SME noted that this would be confusing and frustrating to providers, and should therefore be avoided. The source of authority is different and may produce conflicts within an institution:

We have talked earlier about performance measures for defining target population quality in a way that is different from the CDS, different from the guidelines upon which the CDS are based. So you have the problem that – so let’s say that your performance measures, say, that your hemoglobin A1c needs to be less than nine. But the CDS is a different goal. They have different targets. So you end up giving your clinician multiple, inconsistent messages… That is a challenge when trying to integrate CDS and performance measures. Because CDS and performance measures rely on different sources, right? The performance measures are defined by Meaningful Use, by National Quality Forum, and so on and so forth. And the CDS may be based on recommendations from professional societies, from NIH groups and so on. So you have different knowledge sources and they may give somewhat different recommendations and when you are trying to integrate the two you somehow need to reconcile them.

In addition to differences in actual definitions, SMEs noted that CDS and eCQM differ in their timing, as CDS is designed for providing guidance prospectively and at the point of care, whereas eCQMs are designed for retrospective evaluation. Because of this difference, SMEs noted that data available for eCQMs (e.g., administrative billing data) may not be available for CDS. As a result, a CDS system may need to find alternative data sources. As noted by a SME:

Let’s say [the] patient isn’t going to get [into] the denominator until you code the visit … You need the CDS before that, right? You got the patient in the room, [but] you haven’t put in any encounter diagnoses yet. So, …. the measure [criterion] isn’t going to put them in the denominator and then it’s kind of useless to you.

SMEs also noted that there were different needs for CDS and eCQM with regard to system execution speed and the number of patients requiring evaluation. Thus, a technical solution acceptable to one may not be acceptable when applied to the other:

You want to make sure that your CDS engine is optimized for being fast. Anything that slows down the process is a big no-no for the front end. That, again, does not necessarily apply to your clinical quality metrics. Hell is not going to break loose if you have to wait 20 minutes for a quality metric query to run. But that does not apply to the clinical decision support query.

There was the issue of performance, because the performance measures are applied to large cohorts of patients, whereas the CDS systems is usually applied to the individual patients. So, [there is a] performance issue of how to make the technical infrastructure which needs to work in computing complex recommendations for individual patients, and how to efficiently convert it for applying these inclusion/exclusion criteria on the large cohort of the patients.

Theme 3: There is an oftentimes invisible overlap between CDS and eCQM, and with the larger domain of QI.

SMEs noted that the overlap between CDS and eCQM was often invisible to the stakeholders involved in the processes, which may help explain why it is so difficult to effectively integrate between the two approaches. For example, a SME noted:

And I also think there’s a lot of people who were doing CDS who have no idea they’re doing it. They really don’t know that.

Moreover, several SMEs noted that both CDS and eCQM are components of the larger domain of quality improvement, and that having a more comprehensive perspective can help optimally improve care. For example, one SME noted:

Clinical decision support can be used to educate people about what to do and speed up this very prolonged timeframe. The clinical decision support speeds up the implementation of quality improvement and then you can use the system to see how well people are doing with it. Because what you’ll find when you try to do clinical improvements is that you need many, many, many layers of improvement. You have to educate people, you have to have standardized protocols, you have to audit and feedback the data to see how they’re doing, you have to computerize reminders, and you have to tell people why they’re doing it.

Similarly, another SME argued that eCQM should be considered a part of a larger QI practice of audit and feedback:

So quality measurement is a part of audit. So audit and feedback is like this generalized term about how you analyze the data then feed it back to a clinician, whereas I would argue the quality measurement is only one aspect of audit. Audit is kind of a wider behavior change quality improvement technique … that takes in lots of other contextual factors that lead to effectiveness or ineffectiveness in a way.

Another SME gave a personal experience of how, despite engaging in CDS for years, he did not realize he was doing so until recently:

I really didn’t realize that I was doing CDS or medical informatics until [I] took the AMIA 10*10 class a few years ago. I’ve been probably working on this kind of stuff since ‘77 and I didn’t really realize I was doing medical informatics.

Theme 4: Despite the differences, synergies between CDS and eCQM should be pursued to amplify the effectiveness of each approach.

While identifying important differences between CDS and eCQM, as well as challenges to their integration, SMEs identified several key benefits to their integration. While one SME was cautious of pursuing the integration of CDS and eCQM given the important differences, the other SMEs were highly supportive of integration. Indeed, many SMEs noted that CDS and eCQM should be considered essential to each other’s success:

I think quality and CDS, they go hand in hand. You really can’t have one without the other to do it successfully.

It’s the only way to do it. (in response to the question – would it make your job easier if it was possible to implement CDS and eCQM using the same standards and approaches?)

In explaining why they felt integration of CDS and eCQM was important, SMEs identified several reasons. Table 3 summarizes these identified benefits, along with sample SME responses.

Table 3.

Reasons CDS and eCQM integration should be pursued

Reason	Sample SME statement
Ensure consistency between CDS and eCQM for clinicians	The information you convey based on your performance measure calculations, and the information that you convey based on the CDS, they need to be consistent in some way, or their difference needs to be explained. It’s almost unfair, right. It seems funny that we would hold users accountable for performance on a quality measure and not support their decision. And especially it seems unfair that we would have inconsistencies between our CDS and our quality measures … We owe it to our users to harmonize those approaches. And I think we are doing it wrong, if we’re not.
CDS can facilitate collection of data required for eCQM	So not 100% of the eCQMs require CDS, but many of them that are direct patient care related, in my opinion, you really need CDS if you’re going to accurately capture what is happening. If it seems like certain data aren’t there for the quality measure then clinicians will be alerted with CDS to enter some data elements to make sure that all the data are captured.
Enable positive feedback loop	So, you got to have the whole package. You’ve got to help the doctor do it, you’ve got to check to see that they’re doing it, you have to be able to bug the crap out of them to get them to do it, and you have to be able to feedback the mortality rate of people to motivate them to change their behavior.
Reduce duplicate effort and enable resource allocation to best practices such as documentation and version management	The benefit is that it makes the production time incredibly shorter since you’re working from a common set of concepts. You’re basically creating your clinical content with the aim of doing CDS and quality measurement … Build small parts, use small libraries of things, test those out, then put them together in bigger parts. Modular is good. They didn’t document it. This is really hard because most administrators see no point in this. This is wasted time for them. This is a fatal mistake, because the amount of time that one spends going back and de-bugging this is stuff that you should’ve had either in separate documentation or, basically, comments in your code, can save you just oodles and oodles of time. It’s a front-end cost. But if you don’t do it, this becomes a nightmare.

Discussion

Building on prior studies examining the intersection of CDS and eCQM, which tended to focus on technical issues or a small sample of organizations,^5–7 we obtained insights from a broad group of 15 SMEs on both technical and non-technical considerations for the integration of CDS and eCQM. Through use of the critical incident technique to focus on specific projects, we sought to solicit insights grounded on concrete experiences in the use of CDS and eCQM together to improve care quality.

Thematic analysis of the SME interviews identified four main themes: (i) cultural and business considerations for CDS, eCQM, and their integration trump the technical considerations; (ii) the purpose and goals of CDS and eCQM differ, and these differences must be accounted for; (iii) there is an oftentimes invisible overlap between CDS and eCQM, and with the larger domain of quality improvement; and (iv) despite the differences, synergies between CDS and eCQM should be pursued to amplify the effectiveness of each approach. Based on the study findings, which resonate with our personal experiences engaging in CDS and eCQM for QI, we believe there are several key implications for ongoing and future work in the integration of CDS and eCQM.

First and foremost, the cultural and business differences between CDS and eCQM must be bridged. At the level of a healthcare system, such bridging may take the form of establishing governance and organizational structures that facilitate engagement and collaboration among the various groups engaged in both CDS and eCQM. At the level of individuals, such bridging may take the form of team members from the various groups actively making an effort to engage and learn from their colleagues in other teams. As noted by several of the SMEs, clinical informaticists could and should play a critical role in this type of bridging function, as they are often engaged in various aspects of CDS, eCQM, and QI and have backgrounds that are relevant to each of these groups. Moreover, because there may be overlaps and synergies between CDS, eCQM, and QI that are invisible to the stakeholders involved, it is important for clinical informaticists and others seeking to be as effective as possible in improving care quality to become familiar with each of these perspectives, cultures, and lexicons, so that they can more effectively identify these synergies for each of the groups involved.

A second key implication of the study findings is that synergies between CDS and eCQM are important and should be pursued, but should be undertaken with full awareness of the important ways in which they differ. In particular, while it is highly desirable for CDS and eCQM to be consistent with one another, it will often be necessary to use different population definitions or implementation technologies to ensure that the functional needs of each aspect of the QI process are met. In addition, the philosophical approach of CDS is often to advise clinicians through the provision of relevant information and guidance, whereas the approach of CQM may be more to enforce adherence to care standards and associated documentation requirements. These approaches may conflict and cause frustration.

Based on these key implications, we offer several recommendations for stakeholders engaged in CDS, eCQM, and their integration. These recommendations are outlined in Table 4.

Table 4.

Recommendations for Integration of CDS and eCQM

Stakeholder group	Recommendation
Healthcare Executives and Organizational Leadership	Establish governance and organizational structures that facilitate engagement and collaboration among the various groups engaged in both CDS and eCQM. Cross-train individuals who can serve as liaisons and facilitate collaboration among groups engaged in CDS and eCQM.
Clinical Informaticists and other Solution Implementers	Learn the perspectives, cultures, and lexicons of colleagues engaged in CDS, eCQM, and QI. Be mindful and respectful of important differences between CDS and eCQM.
EHR Vendors	Develop EHR capabilities to facilitate co-implementation of CDS and eCQM, such as common tools and resources that can be used to support both CDS and eCQM. Enable the CDS and eCQM solutions to share common, reusable components while efficiently diverging based on differing requirements, such as for population definitions or the timeframe of data availability.

Our study has several potential limitations. First, as a qualitative study, the results may be influenced by the researchers’ personal biases or by the phrasing of the interview questions. However, we used robust content analysis methodologies to help ensure the reliability of our findings.²⁶ Second, the inclusion and analysis of only 15 interviews may limit generalizability. Even though it has been previously shown that 12-13 interviews could be sufficient to gather a majority of insights,^24, more interviews may have provided more insights. However, the main themes were repeated across multiple SMEs, and we feel we reached saturation in terms of primary themes. Third, the self-selection recruitment strategy may have led to biases if the included SMEs strongly agreed or disagreed that CDS and eCQM should be integrated. However, the resulting sample included SMEs representing a broad spectrum of healthcare professionals from many geographical regions and with different past experiences, enhancing our ability to describe the breadth of issues. We therefore believe that our conclusions remain generalizable.

Conclusion

CDS and eCQM are important and highly related aspects of QI, with significant potential benefits associated with their tighter integration. In pursuing integration as we move forward, it will be critical to ensure that the cultural, business, and other non-technical challenges to integration be carefully considered and addressed.