Abstract
Complementary approaches to problem solving in healthcare and public health: implementation science and human-centered design”: Combining implementation science and human-centered design approaches is novel and these complementary approaches can be applied together to optimize the integration of evidence-based practices within clinical and public health settings.
Keywords: Implementation science, Evidence-based strategies, Human-centered design, Design thinking, User-centered design, Translation, Health interventions, Health policy
Implications.
Practice: Human-centered design (HCD) methods can be used to consistently operationalize implementation strategies.
Policy: HCD and implementation science (IS), when used together, can provide an avenue for developing stakeholder engaged policy interventions and implementation strategies.
Research: Integrating HCD and IS is a novel approach and future research should be aimed at understanding which HCD strategies are most effective for operationalizing implementation strategies and how IS can be used to inform and evaluate HCD research.
INTRODUCTION
The timely, effective adoption and implementation of evidence-based practices, interventions, tools, programs, and policies (hereafter referenced as evidence-based practices) is important to improve health care delivery and patient outcomes. Implementation science (IS), user-centered design (UCD), and human-centered design (HCD) are three research approaches that focus on translating research evidence into the real world. In their recent article titled “A glossary of user-centered design strategies for implementation experts,” Dopp et al. [1] established a precedent for combining IS and UCD approaches and offered a new glossary of UCD strategies IS experts could use. In this commentary, we build upon this work by combining IS and HCD approaches and offering a how-to guide for IS experts to operationalize implementation strategies using HCD methods. Combining IS and HCD approaches is novel to health care research and practice, and we believe that these complementary approaches can be applied together to optimize the integration of evidence-based practices within clinical and public health settings.
IMPLEMENTATION SCIENCE
IS explores methods to effectively translate evidence-based care, interventions, and policies into practice to improve health [2]. By accounting for context and multilevel determinants, researchers and practitioners may better address implementation challenges for evidence-based practices and maximize their potential benefits on population health. The field leverages dozens of frameworks, theories, and conceptual models [3] to inform IS and uses a variety of measures and study designs [4] to understand implementation processes and develop and test implementation strategies [5]. More specifically, IS theories and frameworks can help (a) identify factors that may influence implementation processes or outcomes, (b) provide guidance for conceptualizing an implementation challenge and inform study hypotheses, including how to overcome barriers to implementation, and (c) select and tailor implementation strategies to address delivery gaps.
Implementation strategies promote the integration of evidence-based practices into public health and health care settings. Powell et al. [6] identified 73 implementation strategies in their Expert Recommendations for Implementing Change study, of which many involve stakeholder engagement, such as conducting educational meetings, clinical reminders, and conducting local needs assessments to improve implementation outcomes, such as acceptability, adoption, appropriateness, costs, feasibility, fidelity, penetration, and sustainability [7]. These strategies can be selected to address specific multilevel barriers to implementation and improve implementation outcomes, which, in turn, strengthens the health impact of evidence-based practices [7]. For example, if a needs assessment uncovers low provider awareness of an evidence-based practice to improve asthma inhaler adherence, then educational meetings with providers may be an effective implementation strategy for increasing adoption of this practice. Methods for selecting and refining implementation strategies for a given context are continuing to be developed. Some recommended approaches for selecting strategies include conjoint analysis, simulation modeling, intervention mapping, and concept mapping, among others [8–10].
In addition to implementation frameworks, outcomes, and strategies, a broad variety of study designs can be used to study implementation, including effectiveness-implementation hybrid designs (which includes effectiveness and implementation research aims and data collection); mixed methods (integrating qualitative and quantitative methods); factorial designs (e.g., sequential multiple assignment randomized implementation trial); two-level nested randomized designs; cluster randomized control trials; crossover designs; and simulation models among others [5,11,12]. Taken together, the field has utilized a set of research methods to rigorously study and evaluate the implementation of evidence-based practices into public health and clinical settings.
HUMAN-CENTERED DESIGN
HCD is a repeatable, creative approach to problem-solving that brings together what is desirable to humans with what is technologically feasible and economically viable [13]. Dopp et al. [1] offer a glossary of USD strategies for IS experts, which focuses on the individual for which a solution is designed (e.g., patient or practitioner), whereas HCD focuses on the individual, those who are around them, and the systems in which the individual is a part. Dopp et al. [1] offered this when comparing HCD to UCD:
The closely related approach of human-centered design more explicitly seeks to integrate an innovation into human activities and systems by considering individuals beyond primary users (including those who interact indirectly with the innovation, such as clinic leaders who oversee implementation, as well as those who are unintentionally affected by it, such as family members of patients) in the design process.
Given the multiple levels of influence (e.g., patient, provider, clinic, organization, and system) that can impact successful implementation, IS experts could benefit from combining a multilevel, HCD approach to operationalizing implementation strategies.
Over the past 30 years, HCD has evolved from diverse disciplines, including computer science, visual design, and architecture, and has been primarily embraced in the private sector [14,15]. However, the public sector has started to embrace HCD [16]. Recently, public health researchers have started to apply HCD approaches and methodologies to community-based participatory research projects as a way to better understand the experiences of end users (i.e., intended beneficiaries) and to codevelop health interventions with them [17,18]. For this commentary, the authors rely on the HCD process as defined by IDEO, a leading global design company, which has successfully used HCD to create groundbreaking products like Palm pilots and Oral-B toothbrushes [19].
IDEO’s HCD process for problem-solving consists of three distinct phases: the inspiration phase, the ideation phase, and the implementation phase [13,19]. After identifying a particular problem for which a solution is desired, designers’ (i.e., those engaged in HCD) first aim is to build empathy toward and draw inspiration from individual users (e.g., patients, patients’ families, clinicians, and staff) through in-depth conversations and experiences in Phase 1 of HCD [18]. The purpose of this first phase is not to arrive at a solution; instead, the goals are to more completely understand the intended users, the barriers (i.e., “pain points” in HCD) they have experienced given the problem, and the solutions (i.e., “workarounds”) they have found [13]. Second, in the ideation phase, designers generate numerous ideas for how to solve the problem, informed by the users’ thoughts, feelings, and experiences. Third, in the implementation phase, designers quickly prototype (i.e., test) the different ideas with users to solicit immediate feedback. This is achieved through designing short experiments with low-fidelity prototypes. Low-fidelity prototypes are simple versions of a solution, often paper based, that are quickly produced to test broad concepts [13]. Prototyping allows for the recombination and refinement of these concepts into a solution that is desirable, feasible, and viable for a specific set of users. These short iteration cycles help to secure buy-in by repeatedly engaging collaborators, which also allows for a smoother, broader implementation of the product or service at the conclusion of the project [13,18]. To make HCD more accessible to the general public, IDEO’s nonprofit arm, IDEO.org, published The Field Guide to Human-Centered Design in 2015 [13]. This field guide includes HCD mindsets, methods, case studies, and resources.
COMBINING HCD AND IS APPROACHES
We typically consider IS when there is an evidence-based practice with proven efficacy that has not yet been effectively implemented in health care or community settings. Through IS, researchers can develop and test strategies to improve care delivery of evidence-based practices [5]. We might consider HCD when developing a new intervention. Both fields acknowledge the importance of multiple stakeholder perspectives, iterative study cycles to optimize outcomes of interest, and consideration of the end users to improve implementation in real-world settings. Based on these complementary strengths, we believe that IS and HCD can be combined to provide “client-centered” approaches for implementing health care and public health practices, and we offer two ways to conceptualize how to integrate the two approaches.
First, we could view HCD as a process that occurs toward the beginning of the translational research pathway (i.e., discovery), and IS on the distal end of the pathway. Indeed, the final phase of HCD includes an implementation phase, so there are opportunities to integrate these two fields in the effort to develop patient-, provider-, and system-centered implementation strategies across the research continuum. IS frameworks, measures, and study designs could play a key role in strengthening the rigor of HCD research projects in the implementation phase.
Second, we could view HCD as a practical method for operationalizing implementation strategies. As previously outlined, IS leverages strategies to optimize the delivery of interventions and stakeholder engagement is paramount. HCD offers IS a set of methods (i.e., activities) to engage with intended beneficiaries [13,18,19]. Therefore, HCD may provide a new approach for selecting, optimizing, and operationalizing implementation strategies.
HCD methods may be particularly useful for operationalizing implementation strategies [6] within four of the nine broader implementation strategy categories identified by Waltz et al. [20]: use evaluative strategies, adapt and tailor to context, develop stakeholder interrelationships, and engage consumers. Publications have provided guidance on how to select, tailor, and specify the 73 implementations strategies [6,9], but there is still little guidance for how to execute specific implementation strategies; that is—how do researchers actually apply these implementation strategies in the field? For example, if researchers want to employ “involving patients/consumers and family members” as an implementation strategy in their research, how do they operationalize this implementation strategy? Operationalizing implementation strategies through the use of low-cost, accessible HCD methods could help researchers and practitioners assess which implementation strategies are most acceptable and feasible, as well as how these strategies should be executed. Using HCD methods to operationalize implementation strategies will also provide implementation scientists a shared language with those who practice HCD and vice versa. Fig. 1 below summarizes the interrelationship between HCD and IS and illustrates how combining these approaches can impact population health. In order to further illustrate how HCD can enhance IS and how IS can enhance HCD, we present the case study below.
CASE STUDY: THE REAL TALK APP
This case study reports the development of a new mobile app where the first author and her team used HCD as the approach for intervention development and implementation. We will report the activities completed by the team in the development and implementation of the Real Talk app and note (a) where HCD methods offered ways to operationalize IS strategies and (b) where IS could have enhanced this HCD project in identifying determinants of implementation and offering ways to evaluate both effectiveness and implementation outcomes.
About the Real Talk app
In 2017, the first author and the two other cofounders of the technology nonprofit MyHealthEd, Inc., applied HCD to build and launch the first version of their Real Talk app for teenage users aged 13–15 [24]. To date, the app has more than 15,000 users in all 50 states and in more than 125 countries. The purpose of the app is to build a community for teens around taboo health issues, such as sexual health and mental health, and let users know that they are not alone. In the app, users can browse, share, and react to stories on a variety of topics, as well as connect with high-quality online resources from organizations like amaze.org and TeensHealth.
How HCD can enhance IS
While the MyHealthEd, Inc., team did not apply an explicit IS framework through their design work, they did apply several implementation strategies, including: (a) involve patients/consumers and family; (b) conduct cyclical small tests of change; and (c) intervene with patients/consumers to enhance uptake/adherence (Table 1). The team applied these implementation strategies by using the Inspiration, Ideation, and Implementation methods from IDEO.org’s field guide [13] as described below. The examples below illustrate how HCD methods could be used to operationalize IS strategies.
Table 1.
Implementation strategy [6] | Implementation strategy description [6] | Human-centered design method [13] | Real Talk case study example |
---|---|---|---|
Involve patients/consumers and family | Engage or include patients/consumers and families in the implementation effort | Card sort Co-creation session Conversation starters Guided tour Resource flow |
Card sort: Teenagers sorted cards that presented options related to where they were most comfortable talking about sex and/or relationships, who they were most comfortable talking with, and what topics interested them the most. Results of these card sorts informed the content and mode of delivery for the intervention. |
Conduct cyclical small tests of change | Implement changes in a cyclical fashion using small tests of change before taking changes systemwide. Tests of change benefit from systematic measurement, and results of the tests of change are studied for insights on how to do better. This process continues serially over time, and refinement is added with each cycle. | Rapid prototyping Integrate and iterate Define success Measure and evaluation |
Rapid prototyping: Team members used InVision to design multiple versions of a feature, like the story submission experience, and tested the viability of these options (e.g., form vs. chatbot) with potential users. Data from these tests were used to make decisions. |
Intervene with patients/consumers to enhance update/ adherence | Develop strategies with patients to encourage and problem solve around adherence | Co-creation session Live prototyping |
Co-creation: Team members hosted co-creation sessions with potential users where they were given paper with blank iPhone screens and were asked to (a) draw their ideal app to learn about sex education and (b) pitch their app to the other potential users for feedback. Team members listened to the pitches and used similar language for the iTunes App Store description. |
Involving patients/consumers and family
In order to operationalize “involving patients/consumers and family” as an implementation strategy, the MyHealthEd, Inc., team involved teenagers aged 13–15 (intended users) early in the HCD process. IDEO.org’s field guide [13] offers a number of specific HCD methods (i.e., activities) to involve end users that include activities like Card Sorts, Conversation Starters, a Guided Tour, or a Resource Flow. The team used the field guide’s Card Sort method to answer questions regarding where teenagers felt most comfortable talking about sex and/or relationships. In order to do this, the team created cards with the following options: school, home, bus, church, friend’s house, and other. Then, the team asked the teenagers to rank the cards in terms of comfort level. After meeting with teenagers across the country and completing the same activity, the team quickly realized that teenagers did not want to talk about sex and/or relationships in school, so they moved away from thinking that they might implement their intervention in schools. This, along with other insights gained through the formative research process, led to a direct-to-consumer approach via a native smartphone app rather than a school-based approach.
Conduct cyclical small tests of change
As part of the ideation phase, the team then conducted dozens of small cyclical tests of change to get feedback from teenagers and other stakeholders (e.g., parents, health teachers, school administrators, and faith leaders) on different features and design elements in the app. This HCD phase directly relates to the implementation strategy for conducting cyclical small tests of change but adds a more specific methodology. The Real Talk app’s user interface and user experience designers used the software InVision to create clickable prototypes of the different app screens. Then, the team used the IDEO.org’s field guide [13] Rapid Prototyping method to share the InVision prototypes with intended users, collect reactions and data, and make adjustments. For example, the team heard from intended users that they would prefer to interact with sexual health content via stories rather than facts or statistics. Teens also wanted the ability to share their own stories through the app, so the MyHealthEd, Inc., team rapidly tested different versions of the story submission experience. One major test compared a form-based study submission experience (e.g., users submit their entire story by typing it into a box) with a chatbot experience (e.g., users respond to prompts from a chatbot to share their stories piece by piece). After testing these two options, the team found that the majority of their intended users preferred the more interactive chatbot because it was as easy and familiar as text messaging a friend. This resulted in building the interactive story submission feature rather than the form-based feature.
Intervene with patients/consumers to enhance uptake/adherence
Prior to implementation and dissemination, the MyHealthEd, Inc., team was also very intentional about engaging with teenagers to develop strategies together to increase uptake (i.e., app downloads) and adherence (app usage). Through using the Co-Creation Session method from IDEO.org’s field guide [13], the team convened a group of teenagers to design alongside them by empowering them to jointly create and brand the solution. Specifically, the team worked with teenagers to name the health app. Teenagers came up with the name “Real Talk” because it captured the raw or “cringey” nature of the stories submitted by other teenage users, but it did not overtly signal that the app covers sexual health education topics. Teenagers wanted a resource like this to be discreet and this insight informed the app logo (two generic white chat bubbles without signals to sexual health content). Lastly, the team held multiple Co-Creation Sessions for teenagers to design and pitch their own sexual health apps. Both the drawings and language that the teenagers used to pitch their app concepts to other teenagers shaped the language and images shown for the app as it is advertised in the iTunes App Store. The description reads:
Real Talk is a community for teens packed with real stories about cringey moments. Browse through stories, search for topics that matter most to you, and use emojis to share your reactions. You can also share your own story directly in the app - it’s as easy as texting with a friend. Join thousands of teens who already use and love Real Talk. With totally relatable stories, you won’t feel as alone as you go through the struggles of growing up [24].
Additional language offered by teenagers in Co-Creation sessions was used in other marketing and outreach materials. Applying this HCD Co-Creation Session method led to an app description that was more teen-friendly than what the MyHealthEd, Inc., team initially envisioned before collaborating with the teens. The use of this specific HCD method provided a protocol to inform the language used to attract new users of the app. In the first year of launching the app, Real Talk was downloaded more than 10,000 times by teenagers across the globe.
How IS can enhance HCD
While the MyHealthEd, Inc., team considered implementation from the start, they did not employ an IS framework or study design. As mentioned earlier, IS can enhance HCD by identifying multilevel determinants of implementation and offering more rigorous evaluation of an evidence-based practice.
IS frameworks that focus on multilevel determinants of implementation can provide structure to studying the implementation of an evidence-based practice. HCD largely considers determinants for implementation on the individual level in the case of Real Talk app from the perspective of teens and their families. However, when considering the implementation of an evidence-based practice, it is essential to consider the multilevel determinants that impact an individual’s use of that practice. Investigating multilevel determinants iteratively throughout the development and evaluation of the Real Talk app could help hone in on the appropriate implementation strategies, as well as provide a more holistic view of the effectiveness of the practice. For example, exploration of multilevel determinants for implementing Real Talk outside of patients and families could prevent disconnects between the patient and their providers, clinics, retail stores, and pharmacists, who also play a role in their sexual health. As individuals act within systems, it is important to study, act upon, and evaluate across multiple levels rather than within a vacuum on the individual level. A number of IS frameworks do well in systematically providing a multilevel perspective on implementation determinants and processes.
Next, IS frameworks, measures, and study designs could provide structure for evaluating the effectiveness and implementation of practices, particularly throughout the rapid prototyping and cyclical experiments. In addition to assessing determinants of implementation, as mentioned above, IS frameworks are available to provide structure to the evaluation of the implementation of evidence-based practices and newly developed innovative solutions to improve health [7,23]. Implementation outcomes have been outlined by the field and include measures such as acceptability, appropriateness, feasibility, and costs, among others [7]. Assessing these implementation outcomes, as well as effectiveness outcomes, is essential for understanding the total impact of Real Talk on adolescent sexual health outcomes. Hybrid effectiveness-implementation designs allow for more rigorous testing and documentation of rapid prototyping cycles by incorporating the exploration of not only effectiveness outcomes but also implementation outcomes. For example, teenagers could have been randomized to view one of three sets of marketing materials each with different content. Then, the team could assess the implementation outcomes (e.g., acceptability, appropriateness, and download rates) and the effectiveness outcomes (e.g., sexual health knowledge) of the teenagers and determine which of these three sets of marketing materials leads to the strongest outcomes. These data on implementation outcomes are key for optimizing, scaling-up, and implementing the intervention in different settings (i.e., scale out) if found to be effective and, if not effective, may point to reasons why the intervention failed to have the intended impact on health.
CONCLUSION
Overall, HCD offers specific methods that can readily operationalize implementation strategies to improve the translation of health innovations into practice. Using HCD to execute implementation strategies provides a set of tools for implementation researchers to develop and test implementation strategies associated with health interventions. Additionally, IS offers specific approaches to identifying and analyzing multilevel systems and barriers to implementation, as well as rigorous study designs that would enhance HCD research by providing guidance for how to document and evaluate the iterative, cyclical experiments [22,23]. By combining the processes and tools from HCD and IS, we believe that health care and public health researchers can develop a common language to improve implementation outcomes and health outcomes for patients and communities.
Funding: Megan Roberts is funded through the National Center for Advancing Translational Sciences, National Institutes of Health through Grant KL2TR002490.
Compliance with Ethical Standards
Conflicts of Interest: E.C. was a cofounder and employee of MyHealthEd, Inc., the technology 501 (c)3 nonprofit that created the Real Talk app mentioned in this manuscript, from August 2016 through June 2019. G.N. and M.C.R. declare that they have no conflicts of interest.
Authors’ Contributions: E.C. and M.C.R. conceptualized this study; E.C., M.C.R, and G.N. analyzed the data; and E.C., M.C.R. and G.N. co-authored this manuscript.
Ethical Approval: This article does not contain any studies with human participants performed by the authors. This article does not contain any studies with animals performed by any of the authors.
Informed Consent: This study does not involve human participants and informed consent was, therefore, not required.
References
- 1. Dopp AR, Parisi KE, Munson SA, Lyon AR. A glossary of user-centered design strategies for implementation experts. Transl Behav Med. 2019;9(6):1057–1064. [DOI] [PubMed] [Google Scholar]
- 2. Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non-specialist. BMC Psychol. 2015;3(1) : 1– 12. doi: 10.1186/S40359-015-0089-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337–350. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Lewis C, Mettert K, Dorset C, Weiner B. Measures and outcomes in implementation science. In: Chambers D, Vinson C, Norton W, eds. Advancing the Sciences of Implementation Across the Cancer Continuum. New York, NY: Oxford University Press; 2018: 57– 77. [Google Scholar]
- 5. Brown CH, Curran G, Palinkas LA, et al. . An overview of research and evaluation designs for dissemination and implementation. Annu Rev Public Health. 2017;38(1):1–22. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Powell BJ, Waltz TJ, Chinman MJ, et al. . A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Proctor E, Silmere H, Raghavan R, et al. . Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Powell BJ, Beidas RS, Lewis CC, et al. . Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. 2017;44(2):177–194. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Leeman J, Birken SA, Powell BJ, Rohweder C, Shea CM. Beyond “implementation strategies”: Classifying the full range of strategies used in implementation science and practice. Implement Sci. 2017;12(1):125. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Fernandez ME, ten Hoor GA, van Lieshout S, et al. . Implementation mapping: using intervention mapping to develop implementation strategies. Front Public Health. 2019;7(Jun):158. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–226. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health. 2011;38(1):44–53. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. IDEO.org. 2015. The field guide to human-centered design. Available at http://www.designkit.org/resources/1. Accessibility verified November 19, 2019.
- 14. McKim RH Experiences in Visual Thinking. 2nd ed. Monterey, CA: Cengage Learning; 1980. [Google Scholar]
- 15. Rowe PG Design Thinking. Cambridge, MA: The MIT Press; 1987. [Google Scholar]
- 16. Nanos, J. 2016. Designers bring privatesector ideas to public policy. Available at https://www.bostonglobe.com/business/2016/10/28/designers-bring-private-sector-ideas-public-policy/oQS7HSvcSmEhUpN9gJ6QqM/story.html. Accessibility verified October 28, 2019.
- 17. Kia-Keating M, Santacrose DE, Liu SR, Adams J. Using community-based participatory research and human-centered design to address violence-related health disparities among Latino/a youth. Fam Community Health. 2017;40(2):160–169. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Chen E, Leos C, Kowitt SD, Moracco KE. Enhancing community-based participatory research through human-centered design strategies. Health Promot Pract. 2020;21(1):37–48. [DOI] [PubMed] [Google Scholar]
- 19. Brown T, Wyatt J. 2010. Stanford social innovation review design thinking for social innovation. Available at https://ojs.unbc.ca/index.php/design/article/viewFile/1272/1089. Accessibility verified November 19, 2019.
- 20. Waltz TJ, Powell BJ, Matthieu MM, et al. . Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: Results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:109. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Moen RD, Norman CL. 2010. Circling back. Quality progress. Available at https://search-proquest-com.libproxy.lib.unc.edu/docview/816914443?pq-origsite=360link. Accessibility verified August 7, 2020.
- 23. Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Qual Saf. 2014;23(4): 290–298. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. MyHealthEd, Inc. 2017. Real Talk: Stories by teens. Available at https://apps.apple.com/us/app/real-talk-stories-by-teens/id1274703738. Accessibility verified November 19, 2019.