Barriers and Facilitators to the International Implementation of Standardized Outcome Measures in Clinical Cleft Practice

Abstract

Objective:

To identify barriers and facilitators to international implementation of a prospective system for standardized outcomes measurement in cleft care.

Design:

Cleft teams that have implemented the International Consortium for Health Outcomes Measurement Standard Set for cleft care were invited to participate in this 2-part qualitative study: (1) an exploratory survey among clinicians, health information technology professionals, and project coordinators, and (2) semistructured interviews of project leads. Thematic content analysis was performed, with organization of themes according to the dimensions of the reach, effectiveness, adoption, implementation and maintenance (RE-AIM) framework: reach, effectiveness, adoption, implementation, and maintenance.

Results:

Four cleft teams in Europe and North America participated in this study. Thirteen participants completed exploratory questionnaires and 5 interviewees participated in follow-up interviews. Survey responses and thematic content analysis revealed common facilitators and barriers to implementation at all sites. Teams reach patients either via email or during the clinic visit to capture patient-reported outcomes. Adopting routine data collection is enhanced by aligning priorities at the organizational and cleft team level. Streamlining workflows and developing an efficient data collection platform are necessary early on, followed by pilot testing or stepwise implementation. Regular meetings and financial resources are crucial for implementing, sustaining, analyzing collected data, and providing feedback to health care professionals and patients. Fostering patient-centered care was articulated as a positive outcome, whereas time presented challenges across all RE-AIM dimensions.

Conclusions:

Identified themes can inform ongoing implementation efforts. Intentionally investing time to lay a sound foundation early on will benefit every phase of implementation and help overcome barriers such as lack of support or motivation.

Introduction

The use of various disease-specific outcome measures to capture what truly matters to patients is of increasing importance in daily clinical practice. Outcome measures can be used to enhance patient-centered care and evaluate treatment effects (Desomer et al., 2018). To facilitate the measurement of cleft-specific outcomes in clinical practice, the International Consortium for Health Outcomes Measurement (ICHOM) convened a Working Group of cleft experts including clinicians from various specialties, patients and parents, and academicians to establish international consensus on the outcomes that should be measured routinely as a standard part of cleft care. Emphasis was placed on including clinical indicators for all relevant disciplines and patient-reported outcome measures (PROMs) to incorporate patient and parent perspectives. The result was a holistic, patient-centered Standard Set of measures and guidelines for prospective data collection over the course of care, from birth to young adulthood (Allori et al., 2017; International Consortium of Health Outcomes Measurement, 2020). Within the Standard Set, satisfaction with appearance, speech function, psychosocial function, oral health, breathing, eating, and drinking are assessed by PROMs (CLEFT-Q scales, Nasal Obstruction Symptom Evaluation (NOSE) questionnaire, and Child Oral Health Impact Profile - Oral Symptoms Scale (COHIP-OSS)). Examples of clinical measures are tone audiometry for the assessment of hearing, Percent Consonants Correct for speech assessment, and screening for velopharyngeal incompetence. Recommended time points for collection of these measures are 5 years (only clinical measures), 8 years, 12 years, and 22 years of age (International Consortium of Health Outcomes Measurement, 2020).

The ICHOM Standard Set for the comprehensive appraisal of cleft care (hereafter, “Standard Set”) was designed for broad implementation, internationally and across cultures. Over the past 4 years, 4 cleft teams in North America and Europe have implemented Standard Set collection in their routine clinical practice, and implementation is ongoing at multiple other institutions. Collected outcome data are being used toward quality improvement (QI) efforts, research, and inter-center collaborations to identify and disseminate “best practices”. These endeavors are of special importance in cleft care, since research has shown that treatment protocols and quality of care vary widely throughout the world (Shaw et al., 2001; Russell et al., 2011).

The work of these pilot sites is important; however, before meaningful outcome comparisons can be made, widespread adoption and implementation of the Standard Set by more cleft centers is needed. Although many teams are keen to adopt the Standard Set, implementation is not easy. Many cleft teams are cautious about the myriad of challenges and obstacles that they will face. Factors hampering implementation efforts include lacking a defined strategy or a clear understanding of conditions that promote or hinder routine outcome measurement (Foster et al., 2018).

Following their collaboration with ICHOM to develop the Standard Set, 4 cleft teams including Boston Children’s Hospital, Duke Children’s Hospital, Erasmus University Medical Center, and Karolinska University Hospital served as pilots for implementing the Standard Set in clinical practice (Arora and Haj, 2016; Bittar et al., 2018). Their experiences can help inform ongoing implementation endeavors of other cleft teams. The comprehensive evaluation framework reach, effectiveness, adoption, implementation and maintenance (RE-AIM) is often used to evaluate implementation of an intervention or health care program focusing on 5 dimensions: reach, effectiveness, adoption, implementation, and maintenance (Glasgow et al., 1999; Glasgow et al., 2019). The purpose of this investigation is to use the RE-AIM framework to identify facilitators and barriers for implementing the ICHOM Standard Set for cleft care in routine clinical practice, based on the experiences of 4 pilot centers.

Methods

This study was conducted in 2 phases, beginning with an exploratory survey followed by in-depth interviews to understand the different centers’ experiences implementing the Standard Set. In this study, implementation was defined as “the continuous process of actively measuring, collecting, and analyzing outcomes according to the Standard Set in clinical practice”. Participants were recruited through purposive sampling from the authors’ personal networks, thus ensuring a diversity of stakeholders who could provide rich context and details regarding the implementation of the Standard Set by their teams. These stakeholders included clinicians, team coordinators, administrative personnel, IT professionals, project coordinators, and managers. Because the aim of this study was to provide an overview of facilitators and barriers from a health care provider’s perspective, we decided not to recruit patients and families. The pilot sites invited to participate included Boston Children’s Hospital, Duke Children’s Hospital, Erasmus University Medical Center, and Karolinska University Hospital. Informed consent prior to the survey or interview was provided by all participants. This qualitative analysis of the facilitators and barriers to implementation was designated by the institutional review board as exempt research (MEC-2020-0343).

Surveys and Interviews

A preliminary exploratory survey was constructed based upon the dimensions present in the RE-AIM framework: reach, effectiveness, adoption, implementation, and maintenance (Table 1; Online Appendix A). The comprehensive evaluation framework RE-AIM is often used to evaluate implementation of an intervention or health care program (Glasgow et al., 1999; Glasgow et al., 2019). Open-ended questions allowed each participant to expound on the implementation process and corresponding facilitators and barriers. The survey also included questions regarding numerical data such as response rates for the dimensions of reach and adoption. The survey was sent via email to all eligible participants followed by 2 reminders at biweekly intervals. Data collection for the exploratory survey took place between March 2, 2020, and April 6, 2020. Responses were transcoded according to overarching themes and tallied to discover what participants deemed the most important facilitators and barriers. Survey responses were described in frequencies of verbalization (n). Because survey respondents were able to name multiple factors in one answer, the total number of verbalizations could outraise the number of participants. The responses were used to elucidate relevant topics to include in subsequent in-depth interviews.

Table 1.

Original Definitions of the RE-AIM Framework Dimensions Adapted for This Study (Glasgow et al., 2019).

Dimension	Original definition (Glasgow et al., 2019)	Definition in current study
Reach	Proportion of the target population that participated in the program	Methods to reach participants (eg, patients and parents)
Effectiveness	Outcome effects of implementing the program as planned	Positive and negative effects of the implementation of the Standard Set in clinical practice
Adoption	Proportion of practices and individuals that adopted the program	Facilitators and barriers to reach adoption of the Standard Set among individuals involved in cleft care (eg, clinicians, organization, leadership, patients)
Implementation	Extent to which the program is implemented as intended	Facilitators and barriers in implementing the Standard Set in clinical cleft care as planned
Maintenance	Extent to which the program is sustained over time	Activities executed to sustain the (implementation of the) Standard Set over time

Following the completion of their exploratory surveys, the cleft team leaders or coordinators from each site were invited for in-depth, semistructured interviews to further explore various dimensions of implementation. Two researchers (I.A. and J.P.R.) conducted the interviews between April 3, 2020, and April 8, 2020. The researchers performing the interviews were not involved in the implementation process.

An interview guide (Online Appendix B) ensured the same questions were asked uniformly of all interviewees, but interviewees were allowed to follow their train of thought and bring up any issues that came to mind. Interviews were conducted in English, and all interviewers and interviewees were fluent in English; however, since the native language of some participants was different from English, they were offered the opportunity to add specific words or sentences in their own language to more accurately express feelings and perspectives. If needed, these parts could be separately translated by 2 additional objective researchers (Dutch and Swedish native speakers) with a good understanding of the English language.

Interviews were audio-recorded and transcribed verbatim in English using NVivo 12 Pro Software for Windows (QSR International, 2020). Thematic content analysis was performed by a main coder (I.A.), then reviewed by a second coder (J.P.R) who checked that transcripts were accurate and appropriately coded, and that no sections were missed during analysis (Green and Thorogood, 2018). All coded themes were then grouped according to the RE-AIM dimensions, and appropriately subcoded.

Results

Twenty participants were invited to complete the exploratory survey; 15 from Erasmus University Medical Center, 1 from Duke Children’s Hospital, 1 from Boston Children’s Hospital, and 3 from Karolinska University Hospital. Completion rate was 65% (n = 13). Five respondents were eligible for in-depth interviews. Every pilot center was represented by at least one interviewee, and one interviewee provided feedback on behalf of 2 centers, since he has been the implementation lead at both centers at different points in time. Interview duration ranged between 47 and 122 minutes. Survey respondent and interviewee characteristics are described in Table 2.

Table 2.

Survey Respondent and Interviewee Characteristics.

Characteristics	Survey respondents Count (%), Total n = 13	Interviewees Count, Total n = 5
Sex
Male	5 (38)	2
Female	8 (62)	3
Age
30-39	3 (23)	0
40-49	7 (54)	5
50-59	3 (23)	0
Institution
Erasmus University Medical Center	10 (76)	2a
Boston Children’s Hospital	1 (8)	1
Duke Children’s Hospital	1 (8)	1
Karolinska University Hospital	1 (8)	2a
Main job function
Clinician	11 (84)	5
Surgeon	6 (46)	5
Other	5 (38)	0
HIT	1 (8)	0
Management	1 (8)	0
	Mean (range)	Mean (range)
Years of working experience in cleft care	8.7 (0-19)	10.2 (7.5-12.5)

Abbreviation: HIT, health information technology.

^a Interviewees representing multiple institutions.

Findings from the survey and in-depth interviews are discussed per RE-AIM dimension below (Table 3).

Table 3.

Overview of Themes and Most Important Findings per RE-AIM Dimension.

Dimension	Methods	Response rates
Reach	Pen and paper	Labor-intensive, not utilized in the included institutions
	Electronically via clinic	Response rate 85%-99%
	Electronically via email	Response rate 75%-85%
	Positive outcomes	Negative outcomes
Effectiveness	Patient connection	Time
	Teambuilding
	Awareness parents and patients
	Focus for discussion
	Themes
Adoption	Creating importance and urgency
	Aligning motivation and priorities through regular meetings
	Securing resources
Implementation	Reorganizing the clinical workflows
	Developing an efficient HIT system
	Pilot testing and stepwise implementation
Maintenance	Analyzing and utilizing collected data

Abbreviation: HIT, health information technology.

Reach

To engage patients in providing PROMs, 3 different approaches were used. One center started by sending paper questionnaires with appointment letters to patients’ homes. Due to the amount of work (mailing questionnaires, sorting them, entering data in a digital system, storing paper forms), they switched to inviting patients to complete questionnaires on an iPad while waiting for their clinic appointment. Teams using the in-clinic iPad approach reached response rates of 85% to 99%. However, interviewees articulated that it was sometimes noted by clinicians that some patients and parents felt uncomfortable thinking about their appearance while surrounded by others in a waiting room. These concerns made one team change to a third approach of sending questionnaires, including information on how answers will be used for clinical care, by email a few days before the clinic visit so patients could answer in a quiet, private environment. Teams using the latter approach reached 75% to 85% of patients; some could not be reached due to incorrect or missing email addresses, encountered most often for 22-year-olds (as a result of moving from their family home, large time gap since last visit, and switching to their own email address, from that of their parents). The latter was verbalized 8 times by survey respondents as a barrier in reaching patients for PROM collection. Notably, interviewees mentioned that some patients and parents shared negative reactions about unsolicited emails with the team members. At the end of implementation, 2 teams used emailed invitations to complete PROMs at home, and 2 used an iPad to complete PROMs during the clinic visit.

Effectiveness

Interviewees mentioned the ultimate goal of comparing outcomes is not yet possible as individual centers need to reach more robust levels of data first. However, other effects of implementing the Standard Set in routine clinical practice became visible.

Positive effects

Survey respondents most frequently answered that the ability to quickly assess patient’s well-being (n = 8) is the main positive outcome of using the Standard Set. Interviewees and survey respondents (n = 5) added that using PROMs enables them to plan ahead of the clinic visit (n = 5) and provides a launching point for more focused and intentional discussions (n = 5). As a result, interviewees felt that using PROMs routinely fosters connection between patient and team. Additionally, interviewees mentioned that the use of PROMs gives the parents an opportunity to better prepare for the visit together with their child.

Two illustrative quotes about the positive effects of using the Standard Set are below:

Interviewee # 2: So, I think it [use of PROMs] is great…for making the parents…more aware of what the concerns are that the children might have, and it makes our work much easier because we can focus on those [concerns] and not miss out on them.

Interviewee # 3: The psychologists say, ‘Why haven’t we done this [collecting outcome data] before? This is so useful and we’re now reaching families, and parents who are struggling, and kids who are struggling, and we never asked these questions [CLEFT-Q psychosocial scales], they never raised it until it was too late.’ So, I think there is a true benefit of just using the set.

Furthermore, interviewees reported that introduction of the Standard Set has helped foster team solidarity by generating a common goal and giving the team an opportunity to self-evaluate.

Negative effects

Survey respondents listed time (n = 7) and extra work (n = 4) as negative effects of using the Standard Set. Interviewees were more nuanced about these limiting factors:

Interviewee # 4: In the beginning we had some people argue that [collecting outcomes] costs a lot of extra time but once you have everything up and running, and you’re used to it,…, it really fits. You find a way that it fits in the workflow of your team, we don’t experience it as a burden or extra administration or those kinds of things.

Adoption

Survey respondents emphasized that hospital leadership (n = 3) and cleft team coordinators (n = 3) are crucial stakeholders in successful adoption of the Standard Set. Motivation (n = 4) was most frequently mentioned by survey respondents as a facilitator for adoption, and time as a barrier (n = 4). Three themes were identified.

Theme 1: Creating importance and urgency

The hospital boards of all 4 centers were supportive of the initiative, and interviewees felt that lack of leadership support would hinder widespread implementation. To enhance adoption, interviewees advised teams to get on the hospital board’s agenda and explain the value of implementing the Standard Set, for example, to improve quality of care by having your own local outcome registry or positioning cleft teams to benchmark (inter)nationally. An additional advice of interviewees was to use cases from the literature and the experiences of pilot institutions to support this process. Interviewees also advised starting with a simplified implementation collecting only specific parts of the Standard Set, to show the feasibility, benefit, and value in expanding data collection. Demonstrating importance was not only found useful to garner commitment from leadership but also to convince other members of the cleft team, another key stakeholder, to adopt the Standard Set:

Interviewee # 5: I think the main person who we’re really talking about is the main team director, but it could also be the chair of a department or something like that. In any case, that person needs to convey to the team that this [measuring outcomes] is important, that this is creating a new sense of normal…, a new standard operating procedure. That this is not really voluntary, but this is what we as a team want to do, it fulfills our mission…So, you have to create a sense of urgency.

Informing patients and parents about the importance of the project varied by institution. When data collection was wrapped into a broader research program, patients underwent informed consent at the beginning of their clinic visit. If collection of Standard Set data was integrated into routine clinical practice for the purpose of QI, this advancement was announced to patients through newsletters, informational meetings, and on cleft team and/or scientific society websites. An interviewee articulated how prioritizing patient engagement in decision-making increased adoption:

Interviewee # 3: I think the fact that we ask questions from them [patients] and that we do something with these questions, increases the connection between the patient and the team, knowing that we look into it, that we care, that we listen to what they’re saying, and try to do something with it.

Theme 2: Aligning motivation and priorities through regular meetings

Interviewees reported that due to the multidisciplinary nature of cleft care, it is essential to ensure every specialty buys into measuring outcomes routinely. In addition, interviewees stated that interdisciplinary friction points should be discussed and incentives stated clearly, so the project will not be jeopardized later on because of competing priorities. All 4 participating centers held regular meetings to discuss feelings, visions, thoughts, challenges, and organizational matters regarding implementation of the Standard Set to keep everyone engaged. The most frequently discussed topics were how to organize different data collection workflows in clinical practice, what impact PROM questions might have on the child and how to deal with the answers, and what will ultimately happen with the data. Regular meetings also provided opportunities to build an overarching implementation strategy:

Interviewee # 4: They [cleft team members] were all taken along with what we [the implementation team] would do. We had regular meetings, to discuss what was the plan, what would be the next step, and everyone could have a say in that, what they thought about it. Then we did something and had a new meeting or evaluation. So, it was done as a team.

Theme 3: Securing resources

Interviewees articulated that teams that want to implement the Standard Set but lack financial resources and time will face barriers implementing health information technology (HIT) solutions and will more likely succeed by starting outcomes collection on paper. Two of the 4 hospitals partially financed their implementation projects through grants. Interviewees mentioned that Duke and Erasmus are now providing open-access platforms in collaborative networks, to decrease the startup time for teams wanting to adopt and implement the Standard Set.

Implementation

The Standard Set was implemented as planned at all 4 centers, but the initial implementation period was longer than anticipated (ranging between 6 and 24 months). Survey respondents most frequently (n = 7) answered that approximately 10 to 15 people were involved in the implementation team. One respondent reported a number of over 40 people. Interviewees articulated that a small core implementation team was preferred over a larger group because communication problems and staff turnover could disrupt the process. Crucial members of the implementation team included the clinical lead (n = 8), a HIT lead (n = 5), and a clinic coordinator (n = 9) or specialized nurse (n = 5). These members did not differ by teams. The participants felt that the implementation lead could come from any specialty, as long as they are enthusiastic and dedicated, familiar with workflows, and able to build good relationships. Representatives of every specialty could be invited to the team and HIT personnel were mostly included by consultation. Three unique themes were identified.

Theme 1: Reorganizing the clinical workflows

Interviewees frequently mentioned that evaluating and transforming workflows and clinical visits are important aspects of the implementation phase. Teams started by evaluating how data collection would best fit their current workflow, ensuring all outcomes are collected. The 4 centers already worked as a multi- or interdisciplinary team, making it easier for them to streamline workflows of the various specialties involved. Developing flowcharts of treatment protocols including designated Standard Set outcome time points and measurements was very useful to gain insights on how to seamlessly integrate data collection into the existing workflow. Awareness of the extra time needed for speech and language therapists to perform additional testing, and of possible increase in patient volume for the psychologist was necessary. Furthermore, assessing patient’s answers, providing feedback to them, and recording clinical outcomes resulted in an additional 5 minutes on average per clinical visit per patient.

Three teams reported that each specialty records their own clinical outcome measures in their HIT system for tracking outcomes, which interfaces directly with the patient’s electronic medical record (EMR) in 2 of the 4 teams. One team mentioned having a dedicated person who collects all outcomes from the clinicians in a standardized form, and then registers them in the system. At all 4 teams, after the completion of PROMs by the patient, both at home as in clinic, the answers were directly stored in the HIT system without the intervention of a person. Scoring algorithms for each PROM were programmed within the HIT systems, and access to both PROMs and clinical outcomes was the same.

Interviewee # 1: I don’t think one [way of collecting data] is right or wrong, but there are some pros and cons to each.…By doing it in a specialty-specific way, you guarantee that the data quality is pretty good…The downside is that in many cases you might get incomplete data because people forget or they get busy in clinic, whereas the benefit of having a research person who is…always available is always making sure data is collected.…The downside is if they don’t have a clinical background, you might have some incorrect data in there.

Theme 2: Developing an efficient HIT system

All 5 interviewees and 7 survey respondents agreed that a HIT platform was an important facilitator that will save time and increase ease of data management while reducing risk of data loss as compared to tracking outcomes using pen and paper. The most frequently mentioned system requirements were easy access, allowing concurrent users of the database system, dealing with versioning, and keeping permanent records of changes made, with easy data extraction for use in QI projects. There were no teams that had HIT systems that automatically extracted outcomes from the EMR. Interviewees advised making the HIT system as compatible as possible with other systems to aid in future data exchange. Furthermore, interviewees found it helpful to get advice from someone who has dealt with this process before to prevent mistakes that can later create barriers.

Theme 3: Pilot testing and stepwise implementation

One team started with pilot testing the complete Standard Set for 3 to 4 patients with different ages and cleft diagnoses per clinic day. This enabled them to explore time requirements per visit and gave them the opportunity to adjust workflows accordingly to solve errors early on, before measuring outcomes for all patients. Another team started with the complete set, but scaled implementation up from one patient per week to all patients to ease into it, improving the process of data collection gradually, and working out friction points. The other 2 teams preferred stepwise implementation, starting with implementing PROMs followed by clinical measures. This allowed them to spend more time developing their HIT system.

Interviewee # 3: We decided to go for a pilot phase. I know that different hospitals in the world have chosen different routes, so some have said ‘Okay, we are just going to do only the 5-year-olds’, for example, or ‘We are only going to do the cleft lips for a while’. That’s one approach, a choice you need to make…. We then said, ‘We are going to do the whole set, we want to have all the patients from the beginning’. So, the HIT-system was built for all diagnoses and for all aspects of the set.

Maintenance

Theme 1: Analyzing and utilizing collected data

In order to maintain momentum, most survey respondents and all interviewees felt that it is important to analyze and use locally collected data early in the process (n = 9). For example, QI projects like analyzing data completion or complication rates facilitated opportunities for improvement and sustain motivation. Also, interviewees articulated that research on outcomes data and measurement instruments can provide insights to improve future iterations of the Standard Set. Most importantly, it was found that sharing early wins with the entire team is a good way to maintain engagement, since decreasing commitment levels over time was recognized as a barrier.

Discussion

This study applied qualitative methods and the RE-AIM framework in the evaluation of facilitators and barriers to implementation of the ICHOM Standard Set for the comprehensive appraisal of cleft care. Major themes identified included creating importance and urgency, aligning motivation and priorities through regular meetings, and securing resources. The dimension of implementation was characterized by reorganizing clinical workflows and developing efficient HIT systems, followed by pilot testing and stepwise implementation. Although implementing the Standard Set requires extra time and effort, especially in the beginning, interviewees experienced advancements in patient-centered care as a positive outcome. Analyzing and utilizing the data collected in practice could help sustain implementation over time.

Three methods were identified to reach patients and collect PROM data: paper surveys mailed to the patient’s home; email surveys prior to clinic visit; and real-time data collection using an iPad during clinical visits. Only the 2 electronic approaches are now used by the pilot centers because paper forms were too labor-intensive with higher risk of losing information. Several studies have shown that patients are more receptive toward electronic collection systems compared to pen and paper for the collection of PROMs; however, no clear comparisons have been made between completing surveys at home or during the clinical visit and how this is viewed by the pediatric patient population (Richter et al., 2008; Salaffi et al., 2013; Salaffi et al., 2016; Recinos et al., 2017). Cultural or societal differences might play a role, since the North American institutions chose the in-clinic iPad approach, while the European centers incorporated emailed invites to complete PROMs at home. Other factors that could influence choice of data collection method is the payment model of the health care system and a patient’s travel time to the clinic. Patients will not come to clinic when they do not experience problems or concerns if they have to pay extra for each visit or travel long distances. Missing out on collecting data for these patients could potentially jeopardize a center’s outcomes. These factors should be taken into consideration when deciding on the best way to reach patients. Including patient advocacy groups in this decision could be valuable.

Unfortunately, investing in electronic systems for data collection might not always be feasible, due to limited financial or technological resources, or differing organizational priorities (Bausewein et al., 2011; Malhotra et al., 2016; Foster et al., 2018). Middle- and low-income countries might especially face these challenges. Currently, 2 large initiatives offer support in these circumstances. The European Reference Network for rare and/or complex craniofacial anomalies and ear-nose-throat disorders aims to pool disease-specific expertise, knowledge, and resources from across Europe to improve quality of care. The network is currently developing a registry for the collection of outcome measurement data for cleft care. This registry will be accessible to all participating centers for the primary purpose of quality control, and outcomes research in the future (ERN CRANIO, 2020). Similarly, the ACCQUIREnet collaborative, led by Duke University, makes its REDcap-based implementation available to member institutions that join the network.

Creating importance, and aligning motivation and priorities among team members and leadership is a crucial and universal part of implementing an outcomes measurement framework in clinical practice. This is consistent with recent literature on understanding and overcoming barriers to change, which states that it is important for health care professionals to understand the benefits of changing practices (National Centre for Health Excellence, 2007). Across various health care settings, implementation was boosted when collection of outcome data is supportive of patient-centered care at an individual patient level, instead of at an aggregated level (Boyce et al., 2014; Greenhalgh et al., 2014; Howell et al., 2015).

The current study identified a common belief among cleft professionals that implementation of the Standard Set had a positive effect on their team and on patient-centered care. Previous literature reported that patient–clinician communication, clinician’s awareness of symptoms, and patient satisfaction can be improved by the use of PROMs, and by reviewing the results with the patient (Recinos et al., 2017; Basch et al., 2018). In addition, a recent study showed that over 80% of children completing the CLEFT-Q scales, representing 9 of the 12 PROMs in the Set, liked answering the questions, and felt it made them understand their condition and feelings better (Klassen et al., 2020). The fact that the children get something in return (insight in their own well-being, more individualized care) could be a reason for obtaining relatively high response rates in contrast to the reported email survey response rates of 20% to 40% among adults in literature (Rodriguez et al., 2006; Fowler et al., 2019; Toomey et al., 2019).

Implementation efforts were most constrained by time. Time, as part of resources, was articulated to have an overarching and continuing influence on all dimensions of the RE-AIM framework, especially on adoption and implementation. In general, approximately 5 extra minutes per patient were necessary during clinical visits for the discussion of the PROM results with the patient, registration of clinical outcome data, and in some cases extra speech or audiometry screening. For the latter, planning extra time for speech therapists and audiological consultants might be necessary and coordination with the specific departments regarding other obligations is of considerable importance when implementing the Standard Set. When barriers are not properly addressed due to time constraints, teams might struggle with problems later on, experience setbacks or jeopardize the project. Therefore, intentionally investing time to set the parameters for implementation will benefit every phase and help overcome barriers such as lack of support or motivation.

Limitations and Future Directions

A strength of this study is inclusion of 4 cleft centers with different implementation methods from various countries, representing unique cultures and societal habits. However, all 4 centers are located in high-income countries, limiting the generalizability of these findings to low- and middle-income countries (The World Bank, 2020). It is likely that factors influencing change management will not differ profoundly, while differences in financial and technological resources will be more prominent.

Another important factor limiting the generalizability and interpretability of our findings is the fact that there was a sizeable disparity in the number of people per cleft team approached for participation in this study, and that all interviewees represented one discipline, instead of a variety in stakeholders. The first disbalance is caused by a high turnover of personnel involved in the clinical implementation of the Standard Set, resulting in a limited number of eligible patients at 3 sites for completing the exploratory survey. The loss of continuity in personnel was mentioned by these sites as a barrier in implementation resulting in slowing down the process. The second disbalance of interviewing only surgeons is caused by the fact that the implementation efforts were all led by surgeons as project coordinators, and because a relatively large proportion of clinicians within a cleft team has a surgical expertise. Also, health care management and coordinating tasks are often employed by clinicians, since they are familiar with the clinical workflows.

Furthermore, centers who are currently implementing or have abandoned implementation due to problems were outside the scope of this study. Anecdotally, some of these centers experienced a lack of institutional and financial support. The findings of this study can help teams experiencing challenges in their implementation efforts to move forward, as well as serve as starting point for future research by centers struggling with implementation, and by centers in low- and middle-income countries.

Using an extensive open-ended survey as well as the fact that experts were recruited through purposive sampling could have influenced answers, because participants could assume specific information or opinions are already common knowledge for the researchers. The use of open-ended questions was chosen to gather as many different opinions and feelings as possible, since a qualitative study toward implementation of such a specific outcomes set has not yet been performed. Therefore, it was deemed a preliminary exploratory survey was necessary to explore the main themes and directions for the interviews. A possible lack of in-depth information on the survey was addressed by the follow-up semistructured interviews with clinical leads and coordinators.

Conclusion

The themes identified in this qualitative study may be helpful to other cleft teams that are considering adopting and implementing the Standard Set. Specifically, each team should strive to adequately communicate to all stakeholders the reason for adopting the standard set, seek to align motivation and priorities, and provide frequent communication during the initial phases of implementation. At the organizational level, proper attention must be given to setting up the HIT platform, the implementation effects on workflow and provider burden, and securing resources for sustaining the endeavor. Multisite collaboratives may assist in facilitating implementation.