Abstract
Objective:
Identify facilitators and barriers to implementing standardized outcome measurement in cleft care
Design:
Cross-sectional, qualitative study
Setting/Participants:
Participants included 24 providers and staff from a large, multi-disciplinary cleft team in the southwest United States, five caregivers of children with cleft palate (with or without cleft lip) treated by this team, and three experts involved in implementing a cleft-specific standardized outcome measurement in the United Kingdom.
Interventions:
Semi-structured, qualitative interviews were conducted exploring perceived facilitators and barriers to implementing standardized outcome measurement in cleft care. Interviews were audio-recorded, transcribed, and analyzed for content. The Consolidated Framework for Implementation Research was used to guide the interviews and analysis. The analysis focused on the characteristics of standardized outcome measurement that directly influence its adoption.
Results:
Participants identified both facilitators and barriers to implementing standardized outcome measurement. Facilitators included the strength and quality of evidence supporting improvements in cleft care delivery following implementation of standardized outcome measurement and the relative advantage of standardized outcome measurement over continuing the status quo. Barriers included the difficulty adapting standardized outcome measurement to meet local context and patient-specific needs and the complexity of implementing standardized outcome measurement.
Conclusions:
Providers, staff, and caregivers involved in cleft care perceive multiple benefits from standardized outcome measurement, while also recognizing substantial barriers to its implementation. Results from this study can be used to guide development of an implementation strategy for standardized outcome measurement that builds upon perceived strengths of the intervention and reduces perceived barriers.
Keywords: cleft lip, cleft palate, child, qualitative research, outcome assessment, implementation science
INTRODUCTION
Standardized outcome measurement—measuring treatment outcomes in the same way at the same time for all patients—is a core requirement for evaluating and improving healthcare services. The American Cleft Palate-Craniofacial Association (ACPA) has identified monitoring treatment outcomes as a function of cleft teams for over 20 years (ACPA, 2010). Further, an expert panel convened by the World Health Organization (WHO) cited the standardization of outcome measures in cleft care as necessary to improve surgical outcomes, ensure effective interventions, and deliver evidence-based care (WHO, 2004). Despite these calls for inclusion of standardized outcome measurement as part of routine cleft care, cleft centers in the United States have struggled to achieve consistent collection of standardized measures for aesthetic, dental, and speech outcomes. This difficulty in implementing standardized outcome measurement is not unexpected, however, as implementing new health interventions and new approaches to healthcare delivery in a complex healthcare environment can be exceedingly difficult.
There is an increasingly recognized gap between the interventions and practices that are known to improve health and healthcare delivery and what is actually implement in everyday practice. The field of implementation research seeks to reduce this gap by using evidence-based strategies to increase adoption of health interventions (Mitchell and Chambers, 2017). Central to implementation research are an understanding of end-users’ perceptions of an intervention, the local and national setting where implementation will occur, the people implementing the intervention, and the implementation process. The Consolidated Framework for Implementation Research (CFIR) incorporates each of these domains (Table 1), providing a comprehensive framework for evaluating and planning implementation efforts (Damschroder et al., 2009).
Table 1.
Consolidated Framework for Implementation Research (CFIR).1 The five major domains within the CFIR and the CFIR constructs within the Intervention Characteristics domain addressed by study participants are presented with definitions.
| Domain / Construct | Definition |
|---|---|
| Intervention Characteristics | Key attributes of interventions influence the success of implementation |
| Evidence Strength & Quality | Stakeholders’ perceptions of the quality and validity of evidence supporting the belief that the intervention will have desired outcomes |
| Relative Advantage | Stakeholders’ perception of the advantage of implementing the intervention versus an alternative solution |
| Adaptability | The degree to which an intervention can be adapted, tailored, refined, or reinvented to meet local needs |
| Complexity | Perceived difficulty of implementation, reflected by duration, scope, radicalness, disruptiveness, centrality, and intricacy and number of steps required to implement |
| Outer Setting | Characteristics of the economic, political, and social context within which an organization resides |
| Inner Setting | Features of structural, political, and cultural contexts in which the intervention will be implemented |
| Characteristics of Individuals | Individual knowledge and beliefs toward changing behavior and the level of self-efficacy to make the change |
| Process | Process/approach to pursing implementation |
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science. 4:50, 2009
Using the CFIR to systematically identify facilitators and barriers to implementing an intervention can enable development of implementation strategies that speed adoption of the intervention. The first step in this process is a qualitative assessment of perceived facilitators and barriers by the intervention’s end-users. In the case of cleft care, that means exploring perceptions of standardized outcome measurement among providers of cleft lip and palate care and the families of children with cleft lip and/or cleft palate (CL/P). Once the perceived facilitators and barriers to implementation are identified, appropriate implementation strategies (Powell et al., 2015) can be selected to build upon the perceived facilitators, increasing support and acceptance of the intervention, while at the same time modifying either the intervention or the implementation process to reduce the impact of perceived barriers. These implementation strategies may include having multiple stakeholders involved in the implementation process, pursuing a staged implementation process, employing an implementation advisor, educating clinicians about the intervention through in-person discussion or written materials, developing reminders systems, or identifying and employing implementation champions (Powell et al., 2015). To date, no systematic evaluation has been made of the facilitators and barriers to implementing standardized outcome measurement in cleft care, and this may contribute to limited adoption of this intervention in the United States.
To address this knowledge gap, a qualitative study was conducted with the aim of identifying facilitators and barriers to standardized outcome measurement in cleft care. The study focused on the collection of aesthetic, dental, and speech outcomes of patients at five years of age because these are the core domains of early cleft lip and palate treatment, valid and reliable outcome measures exist for these domains at this age, and prior research shows these measures can be collected at this age (Sandy et al., 2001; Sell et al., 2001; Williams et al., 2001; Al-Ghatam et al., 2015; Ness et al., 2015; Sell et al., 2015; Smallridge et al., 2015). The study used qualitative interviews with providers, staff, and caregivers at a large, multidisciplinary, ACPA-approved cleft team in the United States (U.S.) to identify the characteristics of standardized outcome measurement that could either facilitate its implementation in clinical practice or make it difficult to implement. The study also included experts from the United Kingdom involved in executing that country’s government mandate to collect standardized outcome measurements for all children with CL/P, to identify processes that led to successful implementation of standardized outcome measurement and to identify the challenges that cleft centers in the United Kingdom consistently struggled to overcome. This is the first step in a program of research to implement standardized outcome measurement in cleft care, and results of this study can be used to guide development of an implementation strategy for standardized outcome measurement that builds upon perceived strengths of the intervention and reduces or overcomes the perceived barriers.
METHODS
Participants and Recruitment Procedures
Participants were recruited for three stakeholder categories:
(1) providers and staff of a U.S. cleft team; (2) caregivers of children ages 3–6 years who had a cleft palate with or without a cleft lip (CP±L) and received treatment by the U.S. cleft team; and (3) those involved in the implementation and evaluation of standardized outcome measurement among cleft centers in the United Kingdom. The study protocol was approved by the investigator’s institutional review board prior to commencement of research activities.
Providers were defined as surgeons, orthodontists, speech-language pathologists, or psychologists involved in direct patient care. Staff were defined as nurses, medical assistants, and orthodontic assistants involved in direct patient care, along with cleft team coordinators and office staff involved in care coordination. The cleft team was located in an urban setting in the southwestern U.S. and served families from both rural and urban areas within 300 miles of the healthcare facility. The team provided care from prenatal diagnosis through adulthood, treating approximately 100 new infants with CL/P each year. All providers and staff on the team were invited to participate in the study via an email from the study’s principal investigator, who was also a surgeon affiliated with the center. They were informed of the study’s objectives and were told their participation was optional and would not impact their employment.
Caregiver participants were parents or grandparents who served as a primary caregiver for a child with CP±L. Children were identified by clinic rosters and the caregivers were approached by the center’s coordinator or patient’s provider. They were informed of the study’s objectives and were told their participation was optional and would not impact receipt of healthcare for their children. Purposive sampling was applied to achieve a racially and ethnically diverse sample of caregivers.
Sample size for each stakeholder category was prespecified at the beginning of the study. All providers and staff at the U.S. cleft team were included due to the diverse roles and experience within the team. A sample of five caregivers was chosen to provide perspectives from a diverse sample of caregivers. A sample of three experts from the United Kingdom was chosen to provide a breadth of perspectives, i.e. researcher, clinician, administrator. Inductive thematic saturation was reached, as judged by the failure to identify new themes or codes from the last nine interviews (Saunders et al., 2018).
Demographic characteristics of the participants are shown in Table 2.
Table 2.
Characteristics of the participants interviewed.
| Participant Characteristic | n (%) |
|---|---|
| Total, N | 32 (100) |
| Gender | |
| Male | 7 (22) |
| Female | 25 (78) |
| Race | |
| White | 23 (72) |
| Black/African American | 1 (3) |
| Asian and Pacific Islander | 2 (6) |
| Native American | 2 (6) |
| More than one race | 2 (6) |
| Not reported | 2 (6) |
| Ethnicity | |
| Hispanic or Latino | 10 (31) |
| Not Hispanic or Latino | 22 (69) |
| Role | |
| Local cleft team member | 24 (75) |
| Plastic Surgeon | 3 (9) |
| Oral Surgeon | 1 (3) |
| Orthodontist | 2 (6) |
| Orthodontic Assistant | 4 (13) |
| Speech-Language Pathologist | 2 (6) |
| Psychologist | 1 (3) |
| Nurse or Medical Assistant | 2 (6) |
| Cleft Team Coordinator | 2 (6) |
| Front Office Staff | 7 (22) |
| Caregiver of child with cleft palate ± lip | 5 (16) |
| Involved in implementing standardized outcome measurement in the United Kingdom | 3 (9) |
| Plastic Surgeon | 1 (3) |
| Orthodontist | 1 (3) |
| Researcher | 1 (3) |
Interviews
Semi-structured qualitative individual interviews were conducted by two investigators using a pilot-tested and standardized interview protocol (Supplemental File 1). All investigators had previous training and experience in qualitative research methods. Interviews followed an interview guide developed by the investigators using the CFIR. Time was allowed during the interviews for participants to share stories and explore other ideas related to standardized outcome measurement. Individual interviews with local cleft providers, staff, and caregivers were conducted in a private conference room at the cleft center, while interviews with participants from the United Kingdom were conducted by telephone. Each interview lasted 30–90 minutes and was audio-recorded. Before each interview, participants were reminded of their right to withdraw from the interview or to refuse to answer a particular question. Providers and staff were queried regarding job role and years of experience in cleft care. Caregivers were asked about their child’s age, cleft type, and any family history of CL/P. Additionally, all participants provided their race and ethnicity. Personal identifiers were removed (e.g., names) from the audio recordings. Recordings were transcribed verbatim and checked for accuracy. Transcribed interviews were uploaded in NVivo 12 (QSR International, Victoria, Australia).
Data Analysis
Data analysis followed a ‘menu of constructs’ approach in which constructs from the CFIR were used to systematically and comprehensively explore and identify explanatory themes (Brehaut and Eva, 2012; Damschroder and Lowery, 2013). Investigators read all interviews to familiarize themselves with the data. An initial codebook including all CFIR domains and constructs was created, using definitions from the CFIR (Damschroder et al., 2009). The first four interviews were collaboratively open coded by two members of the research team, using a deductive process of assigning data segments to the CFIR constructs they addressed. The codebook was then modified by creating more specific categories within each CFIR construct; these categories provided organization to the discrete concepts discussed within each construct. Definitions for each of these categories were jointly developed by the research team. The remaining interviews were independently coded by the two team members. Coding was compared after every three to five interviews and differences resolved through discussion. Over time, the codebook was updated as categories within each CFIR construct were created or collapsed. After coding all interviews, investigators reviewed the data within each category. Category definitions were refined as appropriate to fully capture the concepts and shared meaning of the data within the category. Patterns of meaning within each CFIR construct were then identified by examining both the categories within the construct and the data within each category. These patterns were analyzed with respect to participant role, prevalence, agreement among participants, and interaction between constructs. Patterns of meaning were interpreted within the CFIR framework using an inductive approach, leading to the categorization of each CFIR construct as either a potential facilitator or barrier to implementation of standardized outcome measurement.
The CFIR contains five domains: intervention characteristics, outer setting, inner setting, characteristics of individual, and process (see Table 1 for definitions). The present study focused on the intervention characteristics domain. This domain includes the attributes of an intervention that are applicable in every situation or setting in which an intervention would be implemented (Greenhalgh et al., 2004; Rabin et al., 2008). Findings for the CFIR domains of outer setting, inner setting, and characteristics of individuals are not reported as they are highly influenced by context and thus have limited transferability. Constructs with limited coverage by the data are not reported, including constructs within the process domain of the CFIR. Results for these portions of the analysis are available from the authors upon request.
RESULTS
Interviews were conducted with 32 participants: 24 providers (surgeons, orthodontists, speech-language pathologists, or psychologists involved in direct patient care) and clinic staff (nurses, medical assistants, and orthodontic assistants involved in direct patient care, along with cleft team coordinators and office staff involved in care coordination), five caregivers of children with CP±L, and three experts involved in implementing standardized outcome measurement in the United Kingdom. Demographic characteristics of the participants are shown in Table 2. Among the five caregiver participants, four were female, four were Caucasian, and one was Asian. Two of the caregiver participants were of Hispanic ethnicity.
Participants identified both facilitators and barriers to implementing standardized outcome measurement. Facilitators included the strength and quality of evidence supporting benefits from standardized outcome measurement and the relative advantage of standardized outcome measurement over continuing the status quo. Barriers included the difficulty adapting standardized outcome measurement to meet local context and patient-specific needs and the complexity of implementing standardized outcome measurement. Each of these constructs is outlined below using illustrative quotes. Key findings within each construct are presented in Table 3.
Table 3.
Key findings and implications for implementation of standardized outcome measurement in cleft care.
| Construct | Key Findings | Implications for Implementation (proposed implementation intervention1) |
|---|---|---|
|
Evidence Strength & Quality Perceptions of the quality and validity of evidence supporting the belief that the intervention will have the desired outcome |
|
|
|
Relative Advantage Perception of the advantage of implementing the intervention versus an alternative solution |
|
|
|
Adaptability The degree to which an intervention can be adapted, tailored, refined, or reinvented to meet local needs |
|
|
|
Complexity Perceived difficulty of implementation, reflected by duration, scope, radicalness, disruptiveness, centrality, and intricacy and number of steps required to implement |
|
|
Implementation interventions selected from Expert Recommendations for Implementing Change (Powell et al., 2015)
Facilitators
Evidence Strength and Quality
Definition:
Stakeholders’ perceptions of the quality and validity of evidence supporting the belief that the intervention will have desired outcomes (Damschroder et al., 2009)
In general, providers and staff viewed standardized outcome measurement as the gold standard and believed that care should be delivered according to those set standards.
“I think [standardized outcome measurement] would provide a standard by which each patient could be compared and, from that, information could be provided to the family, number one, and from that, understanding of where the patient may fall in the continuum of severity, what the treatment expectations might be.” – Orthodontist
“I think the measures they’ve got are pretty standardized and pretty much accepted… people have stopped challenging whether they mean anything. I think they all accept they do mean things.” – Orthodontist, United Kingdom
Providers also viewed standardized outcome measurement as being the preferred method/approach to collecting the clinical information needed to guide treatment decisions. This was due, in part, to the perception that standardized outcome measurement would be more consistent and more comprehensive than the current processes for care delivery.
“If it [outcome measurement] is standardized we can have more consistency and keep a good track and have something to compare and base our evaluations off of.” – Clinic Nurse
Participants from the United Kingdom confirmed these benefits were realized following implementation of standardized outcome measures within their cleft centers.
“There’s also a wider benefit of having a structured system so we don’t miss any problems…it gives us a benchmark, where we want them to have achieved, ideally, essentially normal speech and a good appearance and not having outstanding issues at that early school age. So, by creating a time scale and having an assessment so we don’t lose anyone, I think that’s very effective for making sure that you provide a good, quality care to all the patients.” – Plastic Surgeon, United Kingdom
The perception that standardized outcome measurement is the gold standard for evaluating treatment outcomes may facilitate implementation by providing face validity for the measurements and engendering support from providers and staff for implementation efforts. Further, the providers’ view that standardized outcome measurement is the preferred method for collecting clinical information may lead providers to replace portions of their current patient evaluations during the implementation of standardized outcome measurement; this would result in a lower time burden than if collecting the standardized outcome measurements in addition to their existing evaluations.
Relative Advantage
Definition:
Stakeholders’ perception of the advantage of implementing the intervention versus an alternative solution (Damschroder et al., 2009)
Many providers and staff preferred using a consistent, systematic approach to patient evaluations.
“To have objective standardized measurements will aid our ability to document and treatment planning.” – Plastic Surgeon
They believed this approach would support their efforts to provide uniform and comprehensive evaluations for all patients.
“It also benefits them to have a standardized timeframe for their speech evaluations and speech therapy…So I think it’s really important that, you know, we take care of them all in the same way at the same time.” – Medical Assistant
Evaluations completed using standardized outcome measurements were viewed as providing more accurate and complete data, and this was perceived as leading to higher-quality treatment decisions.
“You haven’t got a reason not to standardize…standardize and centralize because you’re going to make things so much better. You can’t make things worse.” – Orthodontist, United Kingdom
“Because then they’ll have accurate data.” – Front Office Staff
Providers and staff also believed that standardized measurements and timelines would help patients and families better understand the cleft care process.
“Standardizing it [outcome measurement] and just making sure that they get the same care, the same everything from start to finish and finishing that loop, closing that loop, they [families] feel really good about it.” – Front Office Staff
Several providers and staff members indicated that collecting standardized outcome measurements would be advantageous because it would allow for the comparison of how an individual child is doing relative to other children with CL/P or to an ideal outcome.
“I think [standardized outcome measurement] probably would just help everybody clarify, you know, where the patients are in their cleft rehabilitation and what is done and what is not done and what is OK and what is not OK; what we need to focus on; what we can check off as being done and satisfactory. So, I think it would help everybody on the team to manage the patient’s care as well as just to evaluate how we [the cleft team] did and evaluate if there are problem areas and ways to improve those.” – Plastic Surgeon
“There would be the establishment of a norm for the group of patients who have the same diagnosis to which each individual could be compared…. It would provide a standard by which each patient could be compared.” – Orthodontist
Caregivers shared this view; several commented on their appreciation of a checklist outlining the goals achieved and areas where further treatment would be indicated.
“I think knowledge is power, you know… and it also triggers red flag if all of a sudden you’ve got all of these, you know, you’re noticing one area has got a bunch of deficiencies than, you know, the next kid coming in isn’t going to have a problem because you’ve identified the problem.” – Caregiver
“Kind of like the doctor’s office where they’re in a certain curve, so they’ll say, you know, “Hey, your daughter is right in the curve or where other children should be or they’re low or they’re high, so since they’re lower, we’re going to be doing this to make sure they get up to the curve.” – Caregiver
Providers and staff members also noted that collecting standardized outcome measurements, such as pre- and postoperative photographs, would enable them to show families the effectiveness of an intervention and to identify areas where a child could benefit from further treatment.
“Sometimes I feel like they’re like, ‘Oh, I really don’t see a difference,’ and then when you show him the before picture, they’re like, ‘Oh, wow, like there is a difference.’ So, I think it benefits them in that way.” – Orthodontic Assistant
“I think it’s much more powerful if you have something actually written from the family or numeric or that shows where they’re at, that they actually have concerns, that they want this done, and then the ability to repeat that later and see if there’s actually any change.” – Speech-Language Pathologist
Caregivers confirmed that pre- and post-treatment comparisons would be valuable to them, whether following surgery, speech therapy, or dental treatments.
“Like seeing the progress, that they actually don’t have a speech problem anymore. We’ve gone as far as we can at the time.” – Caregiver
Providers were very interested in knowing if their treatments were beneficial and in comparing results across techniques, protocols, providers, and/or teams. They believed standardized outcome measurement will enable these types of comparisons.
“I think you look at your results and formulate some conclusions that this technique might work best for this situation, this technique is, or problem is, better treated by this surgeon. [You] try to make—put an objective decision-making based upon the results of that…. Knowing that if you got better results from this procedure versus this procedure, it would probably help them a great deal.” – Oral Surgeon
For most participants, the ability to evaluate their treatment outcomes was a strong motivation for adopting standardized outcome measurement. Providers and staff believed that evaluating techniques or interventions would lead to identification of best practices, which would support continuous improvement in care delivery and the tailoring of treatments to each child based on his/her needs.
“I think that [standardized outcome measurement] is going to force us to look at each of these patient’s certain parameters critically and to help us look back and say, how can we potentially alter techniques or alter interventions so that we can get each of these children to what we consider to be a good outcome and not just have 80% of them get to that, like try to get 100% of them to that? I think that that’s how [standardized outcome measurement is] going to benefit us.” – Plastic Surgeon
“If you have a standardized outcome measurement protocol, then you’re able to compare surgical techniques [and] surgeons, and they’re able to see their outcomes in various cases, unilateral, bilateral, and things like that.” – Orthodontist
Because the ability to evaluate treatment outcomes was a strong motivator for adoption of standardized outcome measurements, allowing providers to have access to results in real time and to compare outcomes within and across cleft teams may increase support and engagement for standardized outcome measurement.
Participants involved in standardized outcome measurement at cleft teams in the United Kingdom confirmed the perceptions of local cleft team providers and staff. They indicated that the ongoing, routine collection of standardized outcomes measures allowed providers in the United Kingdom to identify individual patients who could benefit from additional treatment, and it allowed centers to identify the specific treatment domains where care delivery failed to meet established targets.
“There’s quite a lot of things that are picked up in those routine collection of data that may not necessarily get picked up elsewhere, because when you start to pull these things and say, well, actually his growth is not that good. There are problems with his hearing and the speech isn’t so good.” – Orthodontist, United Kingdom
“I know two of the surgeons got together because one couldn’t understand why his results was so different to the other surgeons and -- and he made a change from what he was doing.” – Orthodontist, United Kingdom
Barriers
Adaptability
Definition:
The degree to which an intervention can be adapted, refined, or reinvented to meet local needs (Damschroder et al., 2009)
A minority of study participants commented on the adaptability of standardized outcome measurement. Those who did discuss the topic raised concerns that implementing standardized outcome measurement would force providers to strictly adhere to rigid standards, potentially missing importance nuances or the individuality of each patient.
“When you scrub the interaction – scrub the evaluation with personal characteristics and how a patient holds themselves, different facial movements, when they talk, when they smile – when you scrub all that away and it is a set of numbers and measurements with a face in repose you’re going to miss some asymmetries and you’ll also miss some things that blend.” – Plastic Surgeon
Two providers and one staff member expressly stated the view that standardized outcome measurement was a rigid intervention and that maintaining its fidelity was necessary to generate meaningful data on patients’ outcomes and interventions’ effectiveness.
“Sometimes you’re testing patients on things you know aren’t clinically wrong with them…such as doing a standardized articulation measure when you can hear in the speech sample that they don’t have any errors…. But there’s no way to look at a complete data set and completely understand your practice without doing the pieces.” – Speech-Language Pathologist
These providers expressed concern that the inflexible nature of standardized outcome measurement may make it irrelevant to select children. For example, measures used for speech evaluations in English are not validated in other languages.
“Sometimes, you know, not all kids are developing at the same rate…. So to have the same expectations for every kid…it might be a little frustrating for the child…when, you know, we’re expecting them to sound and speak a certain way and it’s not possible for them.” – Medical Assistant
Others felt that forcing providers to strictly adhere to rigid standards could take away from the nuances of the individual patient evaluation or from connecting with patients.
“I think that…it will just be energy spent trying to get measures rather than trying to connect with the patient.” – Plastic Surgeon
Complexity
Definition:
Perceived difficulty of implementation, reflected by duration, scope, radicalness, disruptiveness, centrality, and intricacy and number of steps required to implement (Damschroder et al., 2009)
Participants viewed standardized outcome measurement as a complex intervention that would require overcoming numerous existing and potential barriers to be successfully implemented. These barriers included achieving consistent patient recall to clinic, coordinating collection of numerous measures across multiple providers, achieving consistency of the measurement process across providers, and difficulties in completing clinical evaluations in some patients.
The most commonly cited barrier to implementing standardized outcome measurement was achieving patient recall to clinic for collection of the measurements.
“Usually the hardest part is getting them here physically.” – Cleft Team Coordinator
Both providers and staff cited the absence of an existing process for achieving consistent patient recall and described a current system that inconsistently and incompletely addressed barriers to clinic attendance. Providers, staff, and caregivers all cited competing life events, such as school, parent work commitments, or lack of an available parent or guardian, as barriers to clinic attendance.
“I think that it has to do with—the dynamics of their family are different. …like if they’re employed, their jobs aren’t as flexible, like they can’t just take off at 2:00 in the afternoon…where other jobs, you might be able to work around that and some of our families, that’s not an option for them.” – Cleft Team Coordinator
“Parents that both work or, you know, that can’t afford to take a day off or a couple of hours off to go and [take] their kids [to clinic appointments].” – Medical Assistant
Other challenges to consistent patient recall were family expectations for providers to initiate recall, lack of a designated person in the clinic to coordinate recall across all specialties, changing patient contact information, and lack of access to reliable transportation.
“They [families] are always waiting for the doctor’s office to lead their care.” – Cleft Team Coordinator
“they move a lot sometimes and they change numbers, phone numbers and so that, you know— that’s a hiccup.” – Orthodontic Assistant
“They use disposable, you know, 90-day phones, so their phone numbers change all the time.” – Orthodontist
“Some of them don’t have transportation…so if they can’t actually get themselves here because they don’t have a car…” – Orthodontic Assistant
Participants from the United Kingdom indicated that achieving consistent patient recall remained a barrier for cleft teams following implementation of standardized outcome measures, noting that each team developed its own strategies and workflows to address its government-mandated recall requirements.
“Children who have social disadvantage…those particular children are quite vulnerable and it can be a challenge to get them to come in.” – Orthodontist, United Kingdom
“I think the transport to the hospital is quite a challenge…So I think part of that is finding a space where the parents can get to and some places around the country, some are better than others.” – Plastic Surgeon, United Kingdom
Participants proposed multiple potential strategies and interventions to improve patient recall to clinic. Providers indicated that having a standard recall and evaluation protocol, a database of patients to identify when a patient is due for evaluation, and a system for reminding patients about their evaluations would all facilitate successfully recalling patients to clinic for evaluation at age 5.
“Well, I don’t think that we have a strong tracking system for—or safety net for—catching patients that we expected to show and really didn’t. That is a system to recognize…this group of patients that are now 5 need to be contacted not just trusted to return as we had asked. And when they don’t, I don’t think we have a good system in place to identify those ones that didn’t come in and then go out and find them, remind them, et cetera.” – Plastic Surgeon
Staff indicated that different methods of communication, such as email or texting, can make it easier to contact parents. Staff also noted that some caregivers find it easier to respond to written messages than to talk on the phone, especially during the workday.
“It's hard to get hold of the families…. I know that a lot of the families work, like both parents work. So, it's like you can leave them a message and then they call you back when they have time. But then if they're not available, then it's like playing phone tag. But another way that I found easy for me to communicate with families is by email…. I'll call the families first and then if I don't—like if I don't hear from them like in a day or two—then I'll just email the families and I found out that they’ll reply faster.” – Front Office Staff
Providers felt that standardized outcome measurement could become very extensive, with numerous items to measure on each child.
“Well, I think the downside is maybe just the burden and the work involved in making everything very systematic.” – Orthodontist
Providers noted it could be difficult to obtain all information on every patient, and this would require coordination of visits across multiple specialties.
“…the burden and the work involved in making everything very systematic.” – Orthodontist
“You need to have a, ideally, a staff member who just gets the families reliably from one bit of the assessment to the next bit and if you need to have the space and you need to have the alignment…Can you reliably get a photography studio right next to a room that’s suitable for a speech sample and are you able to make it—can you do hearing assessment in the same building?” – Plastic Surgeon, United Kingdom.
A small number of providers expressed concern that time spent performing standardized outcome measurement may take away from time available for treatment and counseling.
“It’s very resource intensive. In the UK [United Kingdom], about 25% of the time with—the kind of clinical time of the team—is spent on outcome measurement. So you have to be careful that it doesn’t tip over and adversely affect treatment and care of the patient.” – Plastic Surgeon, United Kingdom
Providers generally preferred a pragmatic approach; limiting standardized outcome measurement to just the most important items was preferable. Participants from the United Kingdom indicated that after they implemented standardized outcome measurement in cleft care, collecting, analyzing, and addressing results of outcome measures became a significant component of each cleft team’s activities. They also indicated that the busier teams, those treating more than 100 new patients each year, were the teams that most consistently completed the outcome evaluations.
“The team with a big volume is actually the easiest team to deal with because they have all the systems in place…. It’s an organizational thing.” – Orthodontist, United Kingdom
Providers and staff agreed that implementing standardized outcome measurement would require a consensus within the disciplines both on what to measure and the process for measuring it.
“I think the other problem is that everybody does everything their own way, you know. So…I think if you had different locations, you would have to identify a lead at each of these areas…and bring them together and say, like, ‘This is the standard for pictures, like all pictures will look’…. I think everything would just have to be spelled out and there would have to be standards.” – Cleft Team Coordinator
Further, each clinic would need to consistently adhere to the chosen process(es).
“Between the four to five of us [speech-language pathologists], there would need to be an agreement on what that protocol was, how it would be administered, you know, what would happen in that to be able to make that a standardized procedure across the institutions.” – Speech-Language Pathologist
Providers perceived that this consistency was not occurring in the current system, with different clinics measuring different outcomes in different ways at different times.
“I think the other problem is that everybody does everything their own way” – Cleft Team Coordinator
Providers and staff indicated that selection of standardized outcome measurements should be done by content experts, and that context experts should be responsible for developing specific protocols to achieve consistency across locations.
“You’ve got to have a protocol in place from the beginning and everybody has got to be trained and follow it.” – Orthodontist
“We have different speech pathologist on our team…having those two providers on the same page as far as what they’re measuring the same thing, so it’s not too totally different speech assessments.” – Cleft Team Coordinator
Many providers, staff, and caregivers mentioned that some patients may have difficulty in completing the evaluations. Patient anxiety, mood, or behavior can limit participation with examination.
“Some of them are afraid from past experiences, from surgeries and all of the treatments that they’ve had.” – Orthodontic Assistant
Most providers and staff believed that these were important barriers, with almost every study participant commenting on at least one of them. Anxiety was frequently attributed to a patient’s awareness of prior painful experiences, a patient’s distress over what the examination would involve, and a patient’s discomfort with being put on display.
“They’re going to have more anxiety if they’ve been traumatized by medical procedures.” – Psychologist
“Some of them are afraid from past experiences, from surgeries and all of the treatments that they’ve had.” – Orthodontic Assistant
“Some children are very, very shy. They don’t want to be photographed.” – Plastic Surgeon
A patient’s behavior—e.g., failure to remain still, hesitancy to open mouth, repeat sentences, etc.—was cited as a common limiter to performing a complete examination. Many providers and caregivers had developed methods for attempting to reduce patient anxiety and address negative behaviors.
“If they’re very apprehensive, we bring them back for behavior modification and kind of just play with them, brush their teeth and floss and, you know, just get them comfortable with being in the area, in the building, and you’d be surprised, most of them are happy to come in and look forward to it.” – Orthodontic Assistant
Caregivers in particular noted that creating a relaxed atmosphere that was friendly and engaging was often effective at improving patient cooperation.
“You know, have fun with them…more of a relaxed environment, almost like a school setting or fun setting.” – Caregiver
“…making it something maybe more of fun atmosphere and not so clinical.” – Caregiver
Providers and staff noted that some of these behaviors were normal for children age 5 and were addressed by adapting their technique for evaluation or having the patient’s parents get involved through positive reinforcement.
“If the child is scared and you can talk to the kid and the parents and let them know, ‘Look, it’s just a camera. We’re just taking pictures,’ then, you know, they’re OK.” – Orthodontic Assistant
“I think sometimes some parents are like firm but gentle with the kid, ‘Nope, we’re going to do this and we’re not going to leave until it’s done,’ and those kids are like, ‘Fine, I’ll do it.’ This is the expectation.” – Speech-Language Pathologist
DISCUSSION
Standardized measurement of treatment outcomes is essential for evaluation of treatment interventions and creation of a continuous learning health system (WHO, 2004; Medicine, 2006; Allori et al., 2017). Results of this study indicate that providers and staff involved in cleft lip and palate care are supportive of standardized outcome measurement: they believe it will improve their care delivery and they are enthusiastic about the opportunity that standardized outcome measurement provides for comparing outcomes from different techniques, treatment protocols, providers, and cleft teams. This support for standardized outcome measurement among cleft team members is tempered by the perception that implementation would require addressing several known challenges in cleft care delivery, including patient recall to clinic, coordination across specialties, standardization among providers, and patients’ fears and anxieties about medical evaluation. These results provide a valuable description of the barriers that must be addressed for successful implementation of standardized outcome measurement in cleft care, while also suggesting that providers, staff, and caregivers would be supportive collaborators in the implementation process.
Facilitators of Implementing Standardized Outcome Measurement
These findings of strong support for standardized outcome measurement are consistent with and extend those from prior research in cleft care. Research from the United Kingdom, which implemented standardized outcome measurement as part of their cleft care centralization effort in the 1990s and 2000s, has found consistent support among caregivers and providers for standardized outcome measurement efforts (Waylen et al., 2015). Similarly, cleft and craniofacial patients and caregivers report a preference for consistent evaluations as part of team care (Stock et al., 2018; Myhre et al., 2019). The current study builds upon this prior work by identifying the specific advantages of standardized outcome measurement important to providers, staff, and caregivers involved in cleft care, namely a consistent evaluation process, a systematic evaluation of each child, and the ability to evaluate treatment outcomes. This study also confirms, using the experiences of providers and administrators involved in cleft-specific outcome measurement efforts in the United Kingdom, that these advantages can be realized through adoption of standardized outcome measurement. Providers and administrators pursing implementation of standardized outcome measurement in cleft care can use this knowledge to strengthen internal support for implementation efforts.
Barriers to Implementing Standardized Outcome Measurement
This study’s identification of barriers to standardized outcome measurement extends prior work by describing the perceived barriers among not only providers and staff from each clinical discipline, but also the barriers anticipated by clinic front-office staff and caregivers who are directly involved in care coordination and measure collection. The finding that some providers are concerned about the inflexibility of the outcome measures is consistent with findings from similar efforts to standardize outcome measurement in surgical care (Yi et al., 2013), but the finding that a small minority of providers expressed this concern suggests that efforts to implement standardized outcome measurement in cleft care may not be delayed by widespread concern over applicability or value of the measures collected. The identification of patient recall, coordination across specialties, and standardization of measurement among providers as perceived barriers fits with experiences at other cleft centers in the United States (Bittar et al., 2018; Cooper et al., 2019) and suggests that successful efforts to overcome these barriers in other health conditions may be directly applicable to implementation efforts in cleft care (Guy et al., 2012; Miriovsky et al., 2012; Tricco et al., 2014). Efforts to implement standardized outcome measurement at cleft centers in the United States will need to address each of these barriers if they are to be successful.
Implications
The present study is the first step in a program of research to achieve consistent use of standardized outcome measures in cleft care in the United States. The facilitators and barriers to standardized outcome measurement identified in this study can be targeted by selecting specific evidence-based interventions to address each item (Grol and Wensing, 2005b; Powell et al., 2015; Kirk et al., 2016). The selected interventions for change would be combined in a multiple-component implementation strategy package that could then be tested in the clinic setting (Grol and Wensing, 2005a). Examples of proposed implementation strategies addressing key findings in this study are shown in Table 3.
Building internal support for implementation efforts is essential for achieving changes in healthcare delivery (Bradley et al., 2004; Damschroder et al., 2009). Based on the present study’s finding that standardized outcome measurement is viewed as both the gold standard and the preferred approach for providers and staff to perform outcome assessments, opening implementation team meetings with a brief review of this preference for standardized outcome measurement may motivate team members to tackle the difficult aspects of implementation (Gilbody et al., 2003). Given this study’s finding of strong enthusiasm for standardized outcome measurement as a tool for evaluating treatment outcomes, internal support for implementation may be strengthened by providing outcome measurement results on individual patients to providers in real time so they can see the benefits of using a standardized approach to perform pre–post assessments (Bradley et al., 2004; Likosky, 2012; Powell et al., 2015). If possible, devoting implementation resources to perform outcome comparisons for techniques or protocols of interest to providers may also help engender the support necessary for sustaining implementation—providing feedback on their current practice activities to providers has been effective in supporting ongoing outcome measurement programs in both general and cardiac surgery (O'Connor et al., 1996; Hall et al., 2009). Incorporating these interventions into a comprehensive implementation strategy may be useful for building on existing support standardized outcome measurement.
Application of targeted implementation strategies may also be useful in addressing the perceived complexity of standardized outcome measurement. This study identified patient recall to clinic and coordination across specialties as complexities of achieving outcome measurement and chronic challenges throughout cleft care. To address these challenges, it may be beneficial to facilitate learning from other teams by organizing visits for opinion leaders and discipline-specific implementation leaders to other sites where they can directly visualize standardized outcome measurement as part of routine care (O'Connor et al., 1996; Likosky, 2012; Powell et al., 2015). In addition, conducting cyclical small tests of change, particularly for processes such as patient recall where no uniform approach will work in every setting, may lower the energy necessary for major changes and support development of a customized approach that address local context. This process of cyclical small tests has been used successfully at other cleft teams to support implementation of standardized outcome measures (Bittar et al., 2018), and it fits with this study’s finding that successful approaches to patient recall at centers in the United Kingdom were frequently team dependent. While no implementation strategy will work for all cleft teams, employing these interventions as part of a focused approach to implementation of standardized outcome measurement in cleft care may lead to a strategy that addresses the perceived barriers of end users while potentially reducing waste or poor fit from a less targeted implementation strategy (Brownson et al., 2013; Lobb and Colditz, 2013; Powell et al., 2015; Knoepke et al., 2019).
Limitations
The results of this study must be interpreted in the context of the study design. While interviews were conducted with providers and staff across multiple disciplines, these providers and staff were from a single cleft center. The perceived facilitators and barriers identified in this study may not be transferable to providers and staff at other cleft centers, and future implementation work at other cleft centers should begin by exploring perceptions of providers and staff at that center. The study was also limited by including only five patient caregivers, potentially leading to limited insights on the patient experience of outcome measurement. Future work to explore the patient and caregiver perspective may identify effective approaches to reducing patients’ discomfort, anxiety, and/or fear related to their evaluation. Finally, facilitators and barriers identified in this study were those perceived by interviewees, most of whom had not participated in a concerted effort to standardized outcome measurement in cleft care. Actual facilitators and barriers experienced during implementation may vary from those that are initially perceived. The limitations reflect the difficulties in planning implementation efforts for complex interventions, and highlight opportunities for continued research to expand our knowledge on implementation methods.
CONCLUSIONS
Standardized outcome measurement in children with CL/P offers an opportunity to facilitate comprehensive, consistent evaluation and treatment for patients and to continuously evaluate treatment interventions. Results of this study demonstrate that providers, staff, and caregivers involved in cleft care identify multiple benefits from standardized outcome measurement, and are likely to support implementation efforts. However, the study also identified several barriers to implementation of standardized outcome measurement, many of which are related to known challenges with care coordination and reducing patient fear and anxiety. Results from this study can be used to guide development of an implementation strategy for standardized outcome measurement that builds upon perceived strengths of the intervention and reduces or overcomes the perceived barriers. While the strategy will need to be customized for each context, identification of common themes and effective components may reduce the challenges and ultimately speed implementation of standardized outcome measurement in cleft care.
Supplementary Material
Acknowledgments
Financial Disclosure Statement: Dr. Sitzman received support from the National Institute of Dental and Craniofacial Research (K23 DE025023). No other external funding was provided for this manuscript. The authors have no other financial relationships relevant to this article to disclose.
Abbreviations:
- ACPA
American Cleft Palate-Craniofacial Association
- WHO
World Health Organization
- CFIR
Consolidated Framework for Implementation Research
- CL/P
cleft lip and/or cleft palate
- U.S.
United States
- CP±L
cleft palate with or without cleft lip
References
- Al-Ghatam R, Jones TE, Ireland AJ, Atack NE, Chawla O, Deacon S, Albery L, Cobb AR, Cadogan J, Leary S, et al. Structural outcomes in the Cleft Care UK study. Part 2: dento-facial outcomes. Orthod Craniofac Res. 2015;18 Suppl 2:14–24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Allori AC, Kelley T, Meara JG, Albert A, Bonanthaya K, Chapman K, Cunningham M, Daskalogiannakis J, de Gier H, Heggie AA, et al. A Standard Set of Outcome Measures for the Comprehensive Appraisal of Cleft Care. Cleft Palate Craniofac J. 2017;54:540–554. [DOI] [PubMed] [Google Scholar]
- American Cleft Palate - Craniofacial Association. (2010, March/31/2010). "Standards for Approval of Cleft Palate and Craniofacial Teams." American Cleft Palate - Craniofacial Association Retrieved 06/21/2013, from http://www.acpa-cpf.org/team_care/standards. [Google Scholar]
- Bittar PG, Carlson AR, Mabie-DeRuyter A, Marcus JR, Allori AC. Implementation of a Standardized Data-Collection System for Comprehensive Appraisal of Cleft Care. Cleft Palate Craniofac J. 2018;55:1382–1390. [DOI] [PubMed] [Google Scholar]
- Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM. Data feedback efforts in quality improvement: lessons learned from US hospitals. Qual Saf Health Care. 2004;13:26–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Brehaut JC, Eva KW. Building theories of knowledge translation interventions: use the entire menu of constructs. Implement Sci. 2012;7:114. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Brownson RC, Jacobs JA, Tabak RG, Hoehner CM, Stamatakis KA. Designing for dissemination among public health researchers: findings from a national survey in the United States. Am J Public Health. 2013;103:1693–1699. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cooper DC, Peterson EC, Grellner CG, Naidoo SD, Skolnick GB, Pfeifauf KD, Smyth MD, Snyder-Warwick AK, Patel KB. Cleft and Craniofacial Multidisciplinary Team Clinic: A Look at Attrition Rates for Patients With Complete Cleft Lip and Palate and Nonsyndromic Single-Suture Craniosynostosis. Cleft Palate Craniofac J. 2019;56:1287–1294. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Damschroder LJ, Lowery JC. Evaluation of a large-scale weight management program using the consolidated framework for implementation research (CFIR). Implement Sci. 2013;8:51. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gilbody S, Whitty P, Grimshaw J, Thomas R. Educational and organizational interventions to improve the management of depression in primary care: a systematic review. JAMA. 2003;289:3145–3151. [DOI] [PubMed] [Google Scholar]
- Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82:581–629. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Grol R, Wensing M (2005a). Effective implementation: A model. Improving patient care: The implementation of change in clinical practice. Grol R, Wensing M and Eccles M. Edinburgh, England, Elsevier: 41–57. [Google Scholar]
- Grol R, Wensing M (2005b). Selection of strategies. Improving patient care: The implementation of change in clinical practice. Grol R, Wensing M and Eccles M. Edinburgh, England, Elsevier: 122–134. [Google Scholar]
- Guy R, Hocking J, Wand H, Stott S, Ali H, Kaldor J. How effective are short message service reminders at increasing clinic attendance? A meta-analysis and systematic review. Health Serv Res. 2012;47:614–632. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hall BL, Richards K, Ingraham A, Ko CY. New approaches to the National Surgical Quality Improvement Program: the American College of Surgeons experience. Am J Surg. 2009;198:S56–62. [DOI] [PubMed] [Google Scholar]
- Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L. A systematic review of the use of the Consolidated Framework for Implementation Research. Implement Sci. 2016;11:72. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Knoepke CE, Ingle MP, Matlock DD, Brownson RC, Glasgow RE. Dissemination and stakeholder engagement practices among dissemination & implementation scientists: Results from an online survey. PLoS One. 2019;14:e0216971. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Likosky DS. Lessons learned from the northern New England Cardiovascular Disease Study Group. Prog Pediatr Cardiol. 2012;33:53–56. [Google Scholar]
- Lobb R, Colditz GA. Implementation science and its application to population health. Annu Rev Public Health. 2013;34:235–251. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Medicine Io. Performance Measurement: accelerating improvement. Washington, D.C.: The National Academies Press; 2006. [Google Scholar]
- Miriovsky BJ, Shulman LN, Abernethy AP. Importance of health information technology, electronic health records, and continuously aggregating data to comparative effectiveness research and learning health care. J Clin Oncol. 2012;30:4243–4248. [DOI] [PubMed] [Google Scholar]
- Mitchell SA, Chambers DA. Leveraging Implementation Science to Improve Cancer Care Delivery and Patient Outcomes. J Oncol Pract. 2017;13:523–529. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Myhre A, Agai M, Dundas I, Feragen KB. "All Eyes on Me": A Qualitative Study of Parent and Patient Experiences of Multidisciplinary Care in Craniofacial Conditions. Cleft Palate Craniofac J. 2019;56:1187–1194. [DOI] [PubMed] [Google Scholar]
- Ness AR, Wills AK, Waylen A, Al-Ghatam R, Jones TE, Preston R, Ireland AJ, Persson M, Smallridge J, Hall AJ, et al. Centralization of cleft care in the UK. Part 6: a tale of two studies. Orthod Craniofac Res. 2015;18 Suppl 2:56–62. [DOI] [PMC free article] [PubMed] [Google Scholar]
- O'Connor GT, Plume SK, Olmstead EM, Morton JR, Maloney CT, Nugent WC, Hernandez F Jr., Clough R, Leavitt BJ, Coffin LH, et al. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group. JAMA. 1996;275:841–846. [PubMed] [Google Scholar]
- Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, Kirchner JE. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14:117–123. [DOI] [PubMed] [Google Scholar]
- Sandy JR, Williams AC, Bearn D, Mildinhall S, Murphy T, Sell D, Murray JJ, Shaw WC. Cleft lip and palate care in the United Kingdom--the Clinical Standards Advisory Group (CSAG) Study. Part 1: background and methodology. Cleft Palate Craniofac J. 2001;38:20–23. [DOI] [PubMed] [Google Scholar]
- Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, Burroughs H, Jinks C. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant. 2018;52:1893–1907. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sell D, Grunwell P, Mildinhall S, Murphy T, Cornish TA, Bearn D, Shaw WC, Murray JJ, Williams AC, Sandy JR. Cleft lip and palate care in the United Kingdom--the Clinical Standards Advisory Group (CSAG) Study. Part 3: speech outcomes. Cleft Palate Craniofac J. 2001;38:30–37. [DOI] [PubMed] [Google Scholar]
- Sell D, Mildinhall S, Albery L, Wills AK, Sandy JR, Ness AR. The Cleft Care UK study. Part 4: perceptual speech outcomes. Orthod Craniofac Res. 2015;18 Suppl 2:36–46. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Smallridge J, Hall AJ, Chorbachi R, Parfect V, Persson M, Ireland AJ, Wills AK, Ness AR, Sandy JR. Functional outcomes in the Cleft Care UK study--Part 3: oral health and audiology. Orthod Craniofac Res. 2015;18 Suppl 2:25–35. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stock NM, Anwar H, Sandy JR, Rumsey N. Centralization of Cleft Lip and Palate Services in the United Kingdom: The Views of Adult "Returners". Cleft Palate Craniofac J. 2018;55:676–681. [DOI] [PubMed] [Google Scholar]
- Tricco AC, Antony J, Ivers NM, Ashoor HM, Khan PA, Blondal E, Ghassemi M, MacDonald H, Chen MH, Ezer LK, et al. Effectiveness of quality improvement strategies for coordination of care to reduce use of health care services: a systematic review and meta-analysis. CMAJ. 2014;186:E568–578. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Waylen A, Ness AR, Wills AK, Persson M, Rumsey N, Sandy JR. Cleft Care UK study. Part 5: child psychosocial outcomes and satisfaction with cleft services. Orthod Craniofac Res. 2015;18 Suppl 2:47–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- WHO. Global strategies to reduce the health care burden of craniofacial anomalies: report of WHO meetings on international collaborative research on craniofacial anomalies. Cleft Palate Craniofac J. 2004;41:238–243. [DOI] [PubMed] [Google Scholar]
- Williams AC, Bearn D, Mildinhall S, Murphy T, Sell D, Shaw WC, Murray JJ, Sandy JR. Cleft lip and palate care in the United Kingdom--the Clinical Standards Advisory Group (CSAG) Study. Part 2: dentofacial outcomes and patient satisfaction. Cleft Palate Craniofac J. 2001;38:24–29. [DOI] [PubMed] [Google Scholar]
- Yi SG, Wray NP, Jones SL, Bass BL, Nishioka J, Brann S, Ashton CM. Surgeon-specific performance reports in general surgery: an observational study of initial implementation and adoption. J Am Coll Surg. 2013;217:636–647 e631. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.