Abstract
Introduction
Engaging patients and frontline clinicians in re-designing clinical care is essential for improving care delivery in a complex clinical environment. This study sought to assess an innovative user-centered design approach to improving clinical care quality, focusing on the use cases of de-intensifying non-beneficial care within the following areas: (1) de-intensifying diabetes treatment in high-risk patients; (2) stopping screening for carotid artery stenosis in asymptomatic patients; and (3) stopping colorectal cancer screening in average-risk, older adults.
Methods
The user-centered design approach, consisting of patient and patient-clinician charrettes (defined as intensive workshops where key stakeholders collaborate to develop creative solutions to a specific problem) and participant surveys, has been described previously. Following the charrettes, we used inductive coding to identify and categorize themes emerging from the de-intensification ideas prioritized by participants as well as facilitator notes and audio recordings from the charrettes.
Results
Thirty-five patients participated in the patient design charrettes, generating 134 unique de-intensification ideas and prioritizing 32, which were then distilled into six patient-generated principles of de-intensification by the study team. These principles provided a starting point for a subsequent patient-clinician charrette. In this follow-up charrette, 9 patients who had participated in an earlier patient design charrette collaborated with 7 clinicians to generate 63 potential de-intensification solutions. Six of these potential solutions were developed into multi-faceted, fully operationalized de-intensification strategies.
Discussion
The de-intensification strategies that patients and clinicians prioritized and operationalized during the co-design charrette process were detailed and multi-faceted. Each component of a strategy had a rationale based on feasibility, practical considerations, and ways of overcoming barriers. The charrette-based process may be a useful way to engage clinicians and patients in developing the complex and multi-faceted strategies needed to improve care delivery.
Supplementary Information
The online version contains supplementary material available at 10.1007/s11606-021-07124-6.
KEY WORDS: user-centered design, Veterans, overuse, de-intensification
INTRODUCTION
Guideline groups and programs such as Choosing Wisely1 are starting to address the need for de-intensification: Stopping or scaling back services routinely delivered in primary care. However, there are no definitive models for, or clear guidance about, who needs de-intensification.2 Further, there is recognition that de-implementing a common practice involves unique challenges, calling for unique approaches3–5. While there is broad agreement that efforts to curtail overuse will need to be multi-faceted (addressing barriers at the patient, clinician, and organizational levels) and tailored (fitting well within a particular health system context)3–5, there remains insufficient understanding about how to systematically develop such interventions.
Multi-faceted and tailored interventions may also be important for effectively addressing many other clinical care problems beyond overuse6, 7. Thus, there is a great need to develop and test well-specified, complex interventions. Such specification requires making multiple decisions about each key element of the intervention. For example, a multi-component audit and feedback intervention that has five modifiable elements (content, intensity, method of delivery, duration, and context) results in hundreds of potential design variations, each of which requires decisions that can enhance or diminish its effectiveness. The task is even more daunting for complex interventions designed to overcome barriers at multiple health system levels.
Moreover, it is prudent to engage the types of clinicians who are responsible for, and the types of patients who are affected by, any treatment or testing changes. Patients and clinicians are well-positioned to re-design care processes to tackle important clinical problems efficiently. These stakeholders have the most direct experience and detailed knowledge about local care processes, the clinical context driving decision-making, and the barriers inhibiting the de-implementation necessary to address overuse8. They can also minimize unintended consequences by re-designing care in ways that avoid undermining patient trust or exacerbating clinician burnout.
Our goal in this study was to assess the ability of a co-design method, described previously9, to enable those involved in frontline clinical encounters to produce well-specified strategies for re-designing clinical care—to meet the complex and important challenge of overuse. To do this, we focused on the challenge of de-intensifying unnecessary medical services that are routinely observed in primary care. We engaged patients and clinicians in design charrettes, a user-centered design (UCD) approach that involves intensive workshops bringing together key stakeholders to build on each other’s best ideas and develop creative solutions to a problem9–12. The specific aim of these charrettes was to generate detailed strategies for de-intensification in three high-priority areas9: (1) de-intensifying diabetes treatment in high-risk patients; (2) stopping screening for carotid artery stenosis in asymptomatic patients; and (3) stopping colorectal cancer screening in average-risk, older adults.
METHODS
This study was part of a larger project, the first stage of which was to conduct a systematic review of guidelines to identify high-priority areas of overuse in routine adult primary care.13 In the second stage, we completed analyses to estimate the opportunity for improvement around each overuse topic within the VA nationally, to inform an expert panel tasked with identifying and validating high-priority recommendations using modified RAND/UCLA appropriateness methods9–14. In the final stage (the study described herein), we selected three of the validated, high-priority de-intensification recommendations for further discussion during patient and patient-clinician design charrettes, aiming to develop detailed de-intensification strategies. Methods for conducting the patient and patient-clinician user-centered design charrettes have been described previously.9 The selected recommendations were as follows: (1) stopping or decreasing the dose of diabetes medications in patients age > 65 who have a low hemoglobin A1c (A1c <6.5); (2) not doing screening colonoscopies in average-risk adults age > 80, and not conducting screening colonoscopies more than every 10 years; and (3) not doing screening for carotid artery stenosis in asymptomatic adult patients without a history of cerebrovascular disease. Details about the VA and non-VA “opportunities for improvement” specific to these three recommendations (from stage 2) can be found in Table S.1. In addition to being rated high priority, these three recommendations were selected because we felt they identified a set of topics that are not only applicable to both genders, but which might also elicit a diversity of concerns from participants across topics (e.g., cancer screening vs screening for cardiovascular prevention vs treatment of diabetes). Patients first focused on only one recommendation at a time when generating initial de-intensification ideas during the patient charrettes. However, in generating more detailed and operationalized de-intensification strategies during the subsequent patient-clinician charrette, participants developed strategies that could be applied to any of the three topics.
We conducted two patient design charrettes in July 2019 and one patient-clinician design charrette in November 2019. These involved expert presentations and facilitated small group activities. Pre- and post-surveys collected patient demographics, assessed changes in opinions before vs. after the session, and assessed participant’s views on the quality of the session. Portions of the charrettes were audiotaped. The local Veterans Affairs (VA) Healthcare System Institutional Review Board approved the study.
Patient Charrettes
A random sample of 60 patients meeting inclusion criteria were recruited, stratifying by gender and race to enhance the diversity of perspectives9. We limited our sampling to those age >65 due to the increasing need among older populations for de-intensification in the three study areas. We targeted recruitment to those with diabetes on a hypoglycemic agent (relevant to de-intensifying diabetes treatment), those with at least one colorectal cancer screening test in the past, and no prior diagnosis of colon cancer (relevant to de-intensifying colorectal cancer screening), or those without a history of a stroke (relevant to de-intensifying carotid artery screening). Patient charrettes lasted 6.5 h. During the facilitated small groups, patients worked through multiple user-center design activities intended to foster an understanding of the clinical problem within a clinical context—and generate a list of creative ideas to address the issue of overuse.
Patient-Clinician Charrettes
All primary care clinicians from one academic VA facility with at least 2 days of clinic per week were invited to participate. Also, a subset of patients from the patient charrettes, those who indicated enthusiasm on the post-session survey to continue working on the topic and those who were highly recommended by the facilitators based on their constructive participation, were invited to participate in a follow-up session with clinicians. The patient-clinician charrette lasted 4 h. Patients and clinicians worked in small collaborative groups to further develop the ideas (or principles) generated during the patient charrette (Figure 1). This step aimed to develop concrete, multi-faceted strategies that specific stakeholders can implement to support de-intensification efforts.
Figure 1.
Development process for charrettes 1 and 2: ideas, principles, potential solutions, fully operationalized strategies, and final team synthesis of all ideas. *See Table 1; †see Table 2; ‡see Table 3.
ANALYSIS
After the patient charrettes, project staff conducted a rapid review of all prioritized ideas, facilitator notes, and audio recordings. The study team used a consensus process to group similar prioritized ideas together, resulting in six distinct de-intensification principles (Figure 1). These principles were presented at the beginning of the patient-clinician charrette to frame the discussion as participants developed more fully operationalized de-intensification strategies.
Following the conclusion of all charrettes, the analytic team (J. S., L. D., M. K., S. S., and T. C.) used inductive coding to categorize all ideas (prioritized and non-prioritized) into 5 groups representing who would be responsible for initiating the de-intensification strategy (doctor, patient, other staff, health system, or national-level office). The team reviewed every idea within each category and considered the following questions: What are the similarities across the ideas? What is the common thread in all these ideas? Do the commonalities lie in who participants think should be involved, or how this process should happen? Does anything stand out about how the ideas are framed or structured? Themes were developed to summarize what we heard from the participants for each of the five categories.
RESULTS
For the patient charrettes, letters were mailed to 316 patients, 278 of whom were successfully contacted. Of those successfully contacted, 59 (19%) verbally agreed to participate, of which 35 (59%) attended, providing written informed consent. Participants had an average age of 71 years, a majority (91%) had at least some college education, 31% were female, and 26% were African American. Two-thirds (66%) of all patients identified themselves as “maximizers” (those predisposed to take medical actions) on the pre-survey, and 86% preferred an active decision-making role while considering the clinician’s opinion or sharing responsibility with their clinician (full survey responses and demographics can be found in Table S.2).
Of the 35 patients participating in the patient charrettes, most (88%) reported being very or somewhat enthusiastic to continue working on the topic in a follow-up session. Among the very enthusiastic patients (n=26), 18 were rated as highly engaged and constructive by facilitators and were invited to participate in the follow-up charrette with clinicians. Nine agreed and attended the session (Table S.2)14
All twenty-nine eligible clinicians were emailed a recruitment letter, and seven provided written informed consent (24%). One participant was a nurse practitioner, and the rest were general internists. Four (57%) of the clinicians were female, two (29%) were Asian, and the average age was 46 years (Table S.2). Two clinicians completed clinical training within the past 10 years, and the remaining 5 completed training over 10 years ago.
Patient Charrettes
Participants created 134 unique ideas for de-intensifying care, 32 of which were prioritized by participants (See ideas in Table S.3). By design, the de-intensification ideas generated during these charrettes were high-level and unrefined (e.g., “use better people skills” or “create mandatory training”) and needed more development to become practical implementation strategies. Patients were encouraged to use their creativity and set aside feasibility considerations as they brainstormed ideas (instruction included, “every idea is a good idea”). The study team synthesized the 32 prioritized ideas into 6 “principles of de-intensification” (Figure 1, Table 1)
Table 1.
De-intensification Principles Identified in the First Team Synthesis
| Focus | Description | N | |
|---|---|---|---|
| 1 | Provide patient-centered communication | Emphasized treating a patient as a person rather than a number and that how the clinician conveys information to the patient is key (“it’s not what you say, but how you say it”) | 8 |
| 2 | Educate individual patients during the clinician-patient encounter | Clinicians needing to take the time to educate their patients about de-intensification and use multiple channels of communication (e.g., written, videos) | 5 |
| 3 | Provide alternatives when de-intensifying | Address situations where the clinician must first earn a hesitant patient’s trust and consider doing more now to enable appropriate de-intensification at a future time | 3 |
| 4 | Provide patient education through targeted outreach | Encourage doctors and other staff to offer resources to their patients outside of the clinic (e.g., offering a telehealth program) | 2 |
| 5 | Educate broader populations of patients through mass/social media | Messaging via billboards or public service announcements | 3 |
| 6 | Educate clinicians | Educating clinicians and staff about current recommendations for de-intensification though mandatory trainings, staff meetings, and audit & feedback | 3 |
Patient-Clinician Charrette
The study team presented the 6 de-intensification principles derived from the patient charrette to participants in the patient-clinician charrette. Using these principles as a guide, they developed 63 potential de-intensification solutions, prioritizing nine. The participants were asked to further develop the prioritized potential solutions into concrete, operationalized strategies by identifying the following: (1) the specific actions required to make the strategy work, (2) the individual(s) in the healthcare system who should perform each action, (3) the barriers to implementing each action (patient/clinician level, health-system level, and national VA level), and (4) the actions necessary to overcome those barriers (see published methods14 for a complete description of this activity). Due to time constraints, only six of the nine prioritized solutions were developed into operationalized strategies (Figure 1, Table 2).
Table 2.
Strategies Operationalized During the Patient and Clinician Charrette
| Strategy | |
|---|---|
| 1 | Purchase heath information tablets that display age-appropriate de-intensification recommendations for patients to use in waiting areas |
| 2 | Develop videos for waiting areas that highlight current de-intensification recommendations |
| 3 | Give patients a ‘top 10’ list of personalized recommendations |
| 4 | Hire a patient health educator (nurse (RN) or public health professional (MPH)) to discuss de-intensification recommendations with patients |
| 5 | Send patients a letter, 1 week before their appointment, asking them to document issues for discussion with their clinician |
| 6 | Offer academic detailing to educate clinicians about current de-intensification recommendations |
The study team analyzed audiotapes and notes from the charrette, creating summary statements and logic maps, describing each group’s process for developing the strategy, and depicting rationales for each decision (Figure S.1). As one example, Figure 2 provides the logic map related to the strategy for having the national VA purchase health information tablets. The rationale for this strategy in the words of one clinician: “I would rather my patients get information from a reliable source (i.e., a tablet programmed with vetted, evidence-based content) than just anywhere on the internet.” The initial conception of this health information tablet idea began with using stationary kiosks located within waiting rooms. The group identified 6 potential barriers to accomplishing this, prioritizing the most impactful barrier: the high quantity of kiosks needed. They first solved this barrier by implementing a time limitation on the kiosks, but ultimately decided kiosks were not optimal as patient-participants indicated they would prefer viewing recommendations while seated. The group then decided to propose handheld tablets instead of kiosks, arguing this strategy is more feasible while still retaining the original intent of providing age-appropriate information. Participants then discussed how tablets could be supported at the local level, specifically examining how to increase patient awareness of the tablets. They described a 5-pronged reinforcing approach to ensure integrated communications about the tablets (Figure 2). Clerks and other clinic staff would be encouraged to distribute tablets and remind patients to use them. These efforts would be reinforced by messages on monitors throughout the medical center, waiting rooms, and online through the patient (My HealtheVet) portal.
Figure 2.
Logic model for a fully operationalized de-intensification strategy: purchase health information tablets.
Finally, participants identified barriers to their 5-pronged approach, leading to even more specification about the nature of messages (large and eye-catching), and reinforcement by a wide array of staff, using multiple channels and communication modes. The full narrative of the development process for this strategy and each of the other prioritized strategies (with in-depth rationale) are provided in Figure S.1.
Characterization of Ideas Generated in All Charrettes
To further characterize the de-intensification ideas and strategies, the study team examined all the ideas (prioritized and non-prioritized) produced during all charrettes, identifying five agents/entities responsible for implementing the de-intensification ideas: (1) patient, (2) clinician, (3) other staff, (4) local health system, (5) national health system (Figure 1; detailed descriptions of each category in Table 3). In the patient charrettes, most ideas were at the clinician level (51%). In the patient-clinician charrette, the largest percentage of ideas were at the local health system level (43%). The team inductively coded all session content related to each idea, creating a summary description of the concepts and themes observed at the five levels (Figure 1, Table 3).
Table 3.
No. of De-intensification Ideas Developed in the Design Charrettes, by Responsible Agent; N (%)
| Actor/entity responsible for the strategy | Description of de-intensification ideas for each actor/entity | Patient-only charrettes |
Patient-clinician charrette |
Total |
|---|---|---|---|---|
| Patient | Patients taking control of their own healthcare | 29 (21) | 3 (5) | 32 (16) |
| Clinician | Improved communication between clinician and patient | 70 (51) | 7 (11) | 77 (39) |
| Other staff (e.g., nurses) | Using multiple levels and touchpoints to communicate with the patient about de-intensification | 3 (2) | 9 (14) | 12 (6) |
| Local health system (e.g., hospital, clinic) | Increase and improve clinician education about de-intensification | 27 (20) | 27 (43) | 54 (27) |
| National health system | Improve dissemination of information to health systems, clinicians and patients | 3 (2) | 17 (27) | 20 (10) |
| Uncategorized | Outliers, did not organically fit into another category | 6 (4) | 0 (0) | 6 (3) |
Participant Satisfaction
Both patients and clinicians were highly satisfied with the forum. They felt it was a good use of their time and were happy with their final products (Table S.4). However, clinicians felt less comfortable with the idea of disagreeing with the group’s decision (50% felt they could disagree) than patients (89% felt they could disagree).
DISCUSSION
The de-intensification strategies developed through the patient-clinician charrette were detailed and multi-faceted. Each idea was robustly deliberated as participants considered the feasibility and how to operationalize de-intensification in current practice. The final strategies appeared responsive to the need to fit within current clinical workflows and deal with competing concerns for both patients and clinicians. By design, barriers and facilitators to successful de-intensification were identified based on the first-hand experiences of participating clinicians and patients. Each strategy was designed to explicitly consider how patients, clinicians, and health systems can all play a role and all benefit. These are the types of multi-faceted strategies that evidence overwhelmingly indicates is often needed to improve care15.
The top-level ideas generated during the patient-clinician charrettes were not necessarily innovative (e.g., creating top 10 lists for patients; academic detailing for clinicians). However, the failure of sustained implementation can often be pinpointed to lack of fully operationalized strategies that reflect what those most affected think is achievable. Our participants produced detailed, practical, and fully operationalized strategies using a novel approach within a health system that others can build on. Nonetheless, specific aspects of the operationalized strategies were surprising. For instance, five of the six strategies that were prioritized and fully operationalized by patients and clinicians were aimed at patients—rather than, more commonly in the context of overuse, the clinicians, or the healthcare system.15 Three of the six strategies help prepare patients to be active participants in their care by providing personalized information and recommendations. These strategies reflect the dominant themes voiced by patients about the importance of empowering patients, believing patients themselves could become agents of change to motivate clinicians to de-intensify. Only one strategy was aimed explicitly at clinicians: providing education through academic detailing. This strong emphasis on patient-facing interventions and patient involvement in addressing clinical care quality issues was unexpected. The Expert Recommendations for Implementing Change (ERIC), a list of 73 implementation strategies widely used within implementation science, lists only five strategies aimed specifically at “consumers” (patients)16, 17. In our study, both patients and clinicians acknowledged that changing clinician behavior is more challenging for many reasons, including competing clinical demands and lack of time.
We observed that the charrette process generated commitment and enthusiasm by patient and clinician participants, with nearly all participants reporting high satisfaction levels (Table S.4). There is an increasing desire to involve patients in care design—to create a more patient-centered experience and to build trust between patients, clinicians, and systems18. One recent study highlighted patients’ concern about their lack of voice within medical care19. Likewise, clinicians are experiencing increasingly high levels of burnout, and a potential solution is involving them in re-designing care processes20, 21.
Notably, clinicians felt much less comfortable disagreeing with the groups’ ideas than patients. This finding was not surprising in retrospect and was likely a result of our intentional efforts to level the playing field between patients and clinicians regarding authority and expertise. Specifically, introductory remarks during the patient-clinician charrette were intended to empower patients, emphasizing them as experts on their own experiences.
The specific de-intensification strategies generated in this study are not necessarily generalizable, as these results are based on a small sample of patients (n=35) and clinicians (n=7) from a single health system within the VHA. However, our goal in this study was not to produce generalizable solutions but rather to demonstrate how a user-design process that closely involves patients and clinicians can lead to practical strategies. While even these user-developed strategies would likely need to be further adapted for specific clinical contexts, repeating these methods with larger numbers of patients and clinicians from different health systems may produce findings that are more generalizable across settings. Our published protocol9 and these findings, together, provide a pathway for others to apply, evaluate, and improve on our effort in the context of overuse or for other quality improvements. We have demonstrated the feasibility of this user-centered process and its value in producing detailed, richly justified, multi-component strategies in the context of de-intensification.
A limitation of our study is that leaders at the facility (or regional and national levels) were not included; doing so may have revealed important system-level considerations such as approaches to ensure strategies are aligned with leadership goals22. While our patients and clinicians provided valuable insight into barriers and feasible solutions based on their first-hand experiences, they did not necessarily consider leadership perspectives related to financial or bureaucratic constraints. For example, hiring new personnel (a patient health educator) may be effective, but this strategy may not be financially viable because of other priorities at the health system level. Thus, as described in our prior report (see Figure 1)9, the strategies developed in the patient-clinician charrettes are prototypes that need further refinement and testing. Nonetheless, this user-design approach allows the end-users to generate ideas first, allowing leaders to then reflect on what is possible, rather than have leaders develop possible solutions which then may (or may not) be vetted by the end-users. As a next step, we suggest identifying champions among the leadership to help prioritize and structure the system-level strategies developed by patients and clinicians.
A strength of the charrette-based process is the participant sampling strategy. Organizations deploying this process should attempt to recruit the relevant clinician and patient populations for a clinical topic and optimize the diversity of viewpoints represented during the intensive workshops23, 24. Diversity of input is essential and allows the results to be communicated to other patients and clinicians as being developed “by people like you.” That being said, diversity across all potentially important dimensions is impossible with small sample sizes, and there will almost always be a potential for selection bias to affect results with UCD methods. For example, most patient participants in this study rated themselves as medical maximizers and preferred an active decision-making role. These participants may have been more predisposed to generating strategies to improve patient empowerment than the general population.
In conclusion, we found that an in-depth user-centered design process has the potential for developing holistic solutions. Those on the frontline of clinical care can help inform decisions about the complex, multi-faceted, and highly tailored interventions that are needed to successfully meet complex challenges like reducing overuse.
Supplementary Information
(DOCX 730 kb)
Author Contribution
N/A
Funding
This study was funded by the Department of Veterans Affairs Health Services Research and Development IIR 15-131. Dr. Caverly is supported by a career development award from the VA HSR&D (CDA 16-151).
Declarations
Conflict of Interest
Dr. Caverly, Ms. Skurla, Ms. Klamerus, and Ms. Damschroder have nothing to disclose; Ms. Sparks and Mr. Reed report grants from Department of Veterans Affairs, during the conduct of the study; Dr. Hofer reports grants from VA HSR&D, during the conduct of the study; Dr. Kerr reports grants from VA HSR&D, during the conduct of the study; past personal fees from BIND as a member of the Clinical Advisory Board, outside the submitted work; and Dr. Kerr serves on the Advisory Board for the NIH-funded US Deprescribing Network.
Footnotes
Prior Presentation:
This data was previously presented at the 2019 Academy Health Annual Research Meeting.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.The ABIM Foundation. Choosing Wisely website. Updated 2020. Accessed on December 14, 2020. https://www.choosingwisely.org/
- 2.Grimshaw JM, Patey AM, Kirkham KR, et al. De-Implementing wisely: developing the evidence base to reduce low-value care. BMJ Quality and Safety. 2020;0: 1-9. [DOI] [PMC free article] [PubMed]
- 3.Morgan DJ, Brownlee S, Leppin AL, et al. Setting a research agenda for medical overuse. BMJ. 2015;351:h4534. doi: 10.1136/bmj.h4534. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Prusaczyk B, Swindle T, Curran G. Defining and conceptualizing outcomes for de-implementation: key distinctions for implementation outcomes. Implement Sci Commun. 2020;1(43). [DOI] [PMC free article] [PubMed]
- 5.Norton WE, Chambers DA. Unpacking the complexities of de-implementing inappropriate health interventions. Implement Sci. 2020;15(2). [DOI] [PMC free article] [PubMed]
- 6.Rogal SS, Yakovchenko V, Waltz TJ, et al. Longitudinal assessment of the association between implementation strategy use and the uptake of hepatitis C treatment: Year 2. Implement Sci. 2019;14(36). [DOI] [PMC free article] [PubMed]
- 7.Chapman E, Haby MM, Toma TS, et al. Knowledge translation strategies for dissemination with a focus on healthcare recipients: and overview of systematic reviews. Implement Sci. 2020;15(14). [DOI] [PMC free article] [PubMed]
- 8.Altman M, Huang TT, Breland JY. Design thinking in health care. Prev Chronic Dis. 2018;15:E117. doi: 10.5888/pcd15.180128. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Klamerus ML, Damschroder LJ, Sparks JB, et al. Developing Strategies to Reduce Unnecessary Services in Primary Care: Protocol for User-Centered Design Charrettes. JMIR Res Protoc. 2019;8(11):e15618. doi: 10.2196/15618. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Dopp AR, Parisi KE, Munson SA, Lyon AR. A glossary of user-centered design strategies for implementation experts. Transl Behav Med. 2018. 10.1093/tbm/iby119 [Medline: 30535343] [DOI] [PubMed]
- 11.Martin B, Hanington BM. Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions. Beverly, MA: Rockport Publishers; 2012.
- 12.Lennertz B, Lutzenhiser A. The Charrette Handbook: The Essential Guide for Accelerated, Collaborative Community Planning. Chicago, IL: APA/Planners Press; 2006.
- 13.Kerr EA, Klamerus ML, Markovitz AA, et al. Identifying Recommendations for Stopping or Scaling Back Unnecessary Routine Services in Primary Care. JAMA Intern Med. 2020;180(11):1500–1508. doi: 10.1001/jamainternmed.2020.4001. [DOI] [PubMed] [Google Scholar]
- 14.Fitch K, Bernstein S, Aguillar M, et al. The RAND/UCLA Appropriateness Method User's Manual. Santa Monica, CA: RAND; 2001.
- 15.Colla CH, Mainor AJ, Hargreaves C, Sequist T, Morden N. Interventions aimed at reducing use of low-value health services: a systematic review. Med Care Res Rev. 2017;74(5):507–50. doi: 10.1177/1077558716656970. [DOI] [PubMed] [Google Scholar]
- 16.Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10(21). 10.1186/s13012-015-0209-1 [DOI] [PMC free article] [PubMed]
- 17.Waltz TJ, Powell BJ, Chinman MJ, et al. Expert Recommendations for Implementing Change (ERIC): protocol for a mixed methods study. Implement Sci. 2014;9:1–12. doi: 10.1186/1748-5908-9-39. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Montori V. Why we revolt: A patient revolution for careful and kind care. The Patient Revolution. 2017.
- 19.Liu J, Hou S, Evans R, Xia C, Xia W, Ma J. What Do Patients Complain About Online: A Systematic Review and Taxonomy Framework Based on Patient Centeredness. J Med Internet Res. 2019;21(8):e14634. doi: 10.2196/14634. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Rotenstein LS, Torre M, Ramos MA, et al. Prevalence of burnout among physicians: a systematic review. JAMA. 2018;320(11):1131–1150. doi: 10.1001/jama.2018.12777. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Rotenstein LS, Johnson AK. Taking Back Control – Can Quality Improvement Enhance the Physician Experience? Health Affairs Blog. 2020; doi.10.1377/HBLOG20200110.543513
- 22.Damschroder LJ, Knighton AJ, Griese E, et al. Recommendations for strengthening the role of embedded researchers to accelerate implementation in health systems: Findings from a state-of-the-art (SOTA) conference workgroup. Healthc (Amst). 2021;8 Suppl 1(Supple 1):100455. [DOI] [PMC free article] [PubMed]
- 23.Ryfe DM. Does Deliberative Democracy Work? Annu Rev Polit Sci. 2005;8:49–71. doi: 10.1146/annurev.polisci.8.032904.154633. [DOI] [Google Scholar]
- 24.Safaei J. Deliberative democracy in health care: current challenges and future prospects. J Healthc Leadersh. 2015;7:123–136. doi: 10.2147/JHL.S70021. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
(DOCX 730 kb)