Brainwriting Premortem

Supplemental Digital Content is Available in the Text.

Abstract

Background:

Many health care interventions encounter implementation challenges because of inadequate stakeholder engagement and identification of barriers. The brainwriting premortem technique is the silent sharing of written ideas about why an intervention failed. The method can engage stakeholders and identify barriers more efficiently than traditional brainstorming focus groups.

Purpose:

We evaluated the method during a transition of care intervention in the Veterans Health Administration (VA). Clinicians from 10 VA facilities participated in 10 brainwriting premortem sessions.

Methods:

Using descriptive and content analytic methods, we assessed the quantity and quality of ideas generated, facilitator experience, and participant psychological safety.

Results:

In total, 217 unique ideas were generated. Many were deemed high quality. The written data were immediately available for analysis, allowing rapid feedback and real-time decision making. Participants reported high satisfaction and psychological safety.

Conclusion:

The brainwriting premortem approach is a novel, efficient alternative to brainstorming focus groups that can rapidly inform program implementation at minimal cost.

Stakeholder engagement and adapting an intervention to the local context are critical factors for the effective implementation of health care interventions.1 Researchers, nurses, and quality experts typically interview key stakeholders and conduct preimplementation focus groups with staff to help prepare for implementation. To identify potential barriers to program success, risk analysis brainstorming sessions are commonly used. Brainstorming sessions are 1- to 2-hour meetings where 6 to 12 people share opinions and ideas.2 This approach is supported by a body of literature that describes how brainstorming can capture complex high-quality information from local voices.2–4 Criticisms of the approach include the necessity for a trained facilitator team to moderate discussion, record the session, take notes, and monitor for individuals who dominate the conversation or take the group off topic.5 Furthermore, to ensure that all participants feel safe to present their ideas, facilitators must create an environment of psychological safety, where participants feel safe to speak up.3,6 Finally, the preparation and analysis of the data require input from qualitative methodologists.7

The brainwriting method, which was designed for the marketing industry, addresses many of these criticisms.8 Brainwriting is defined as the silent sharing of written ideas in a structured group setting.5 During the 10- to 30-minute sessions, handwritten ideas are passed between participants.9 The written approach can result in the generation of large numbers of quality ideas, while eliminating the need for a trained facilitator to keep the group on track, recording devices to capture conversations, and transcription services to write out audio recordings. This allows the data to be rapidly analyzed and used for real-time decision making.10–12 Finally, the method is designed to foster an environment where everyone feels safe to share, making it an ideal setting for a project premortem.11

A project premortem11 is a novel approach to identify potential barriers to program implementation from clinicians before the program has started. Participants are asked to imagine that a project has already been implemented in their organization or community. They then are told that the program has failed, in that the program did not meet the stated outcomes and/or caused adverse events. Participants identify what happened that caused the program to fail. The premortem is unlike a typical critiquing session that focuses on what might go wrong.8 Instead, a premortem encourages the use of prospective hindsight by asking teams what did go wrong. A participant's task is to generate plausible reasons for the project's failure. This approach allows people who have past experiences with similar projects or are worried about weaknesses to speak up to improve a project's chance of success. Although there are many potential advantages of a brainwriting premortem approach to quality improvement and implementation work, the method has never been formally evaluated.

The aim of this project was to describe and evaluate the experience of conducting brainwriting premortem sessions in the Veterans Health Administration (VA) during the nationwide implementation of a quality improvement project. We designed a mixed-methods evaluation to understand the feasibility and effectiveness of the method on identification of barriers to program implementation. We measured this through evaluating the quantity and quality of ideas generated, participant satisfaction, psychological safety, and facilitator experiences.

METHODS

Population and evaluation design

The population included clinicians and administrators at 10 VA hospitals and primary care clinics involved in the rural transitions nurse program (TNP). This 5-year mixed-methods national quality improvement project is funded by the VA Office of Rural Health with support from the VA Office of Nursing Services. The project is designed to improve transitions of care for rural Veterans following an inpatient stay at an urban VA hospital.13 Five preimplementation site visits that included 5 hospitals and 9 primary care clinics were conducted during fall 2016 to evaluate how TNP would fit within the local context of each organization. In addition to the brainwriting premortem sessions, the implementation team conducted key informant interviews, ethnographic observations, and group sessions that included process mapping at each site. The study design, implementation, and evaluation were guided by the Practical, Robust Implementation and Sustainability Model.14 The design and reporting of the qualitative data from this evaluation were performed per the Consolidated Criteria for Reporting Qualitative Research checklist.15

Description of brainwriting premortem protocol

The brainwriting premortem protocol (see Supplemental Digital Content Protocol, available at: http://links.lww.com/JNCQ/A467) was created with guidance from existing literature.5,9,16–18 Protocol development was iterative and included feedback from the TNP implementation team. A standardized introduction was used to begin the brainwriting session to ensure consistency across groups. Semistructured prompts were included in the protocol to allow for some flexibility in accordance with issues raised and level of participation within the groups. The prompts were primarily aimed at keeping a group focused. The protocol was pilot tested at 1 hospital and 2 allied primary care clinics. Adaptations were made on the basis of facilitator feedback. Ongoing testing of the protocol during subsequent site visits allowed for the refinement of the protocol, the data analysis strategy, and the reporting methods.

In the brainwriting premortem sessions, 4 to 10 participants sit around a single table. A facilitator introduces the program to be implemented. To help participants contemplate how the program failed, the participants are asked to think about what challenges they have experienced implementing and participating in similar programs. Each participant is given a pen and a piece of paper. Once participants have written their initial ideas, they place their paper in the center of the table. They then choose another sheet of paper from the center (that contains someone else's initial thoughts) and read the idea(s), adding new ideas or expanding on another idea already listed on that sheet before returning it to the center of the table. If the participants cannot come up with an idea in a reasonable time period, they can return the paper without writing anything. After approximately 10 minutes, the facilitator collects the papers and allows participants to reflect on the ideas generated and elaborate on them if desired. The written results are not returned to the participants for comment or correction.

Description of TNP brainwriting premortem sessions

The TNP brainwriting premortem sessions were conducted in a conference room within a VA hospital or clinic. Local site liaisons booked the room and invited leadership, administrators, and clinicians who would be impacted by TNP. Facilitators of the focus groups were members of the TNP implementation team. Six facilitators received training in the brainwriting premortem protocol (see Supplemental Digital Content Protocol, available at: http://links.lww.com/JNCQ/A467). Although facilitators were asked to adhere to the protocol outline and script, they were allowed to make contextually sensitive adaptations as needed. Examples included adding site-specific details to the script, allowing participants to write for longer or shorter periods, and breaking up large groups into multiple smaller groups. The brainwriting premortem session was scheduled as part of a 60- to 90-minute group activity that included sharing information regarding the TNP and a process-mapping activity. The facilitators were introduced to participants as members of the implementation team. Their professional and educational backgrounds were shared with participants, along with the purpose of the project. A postbrainwriting premortem survey was distributed after each session to assess participant satisfaction with the group's productivity, satisfaction with work processes of the group, fear of giving ideas to the group, and worry that their ideas would be criticized by the group. The surveys were scored on a Likert scale ranging from 1 to 5 (strongly disagree to strongly agree).

Statistical analysis

Descriptive statistics for participant demographics, postbrainwriting surveys, and ideas generated are reported in this analysis. The number of unique ideas were quantified as those that occurred only once in the coding. The number of expanded ideas were quantified as an idea that had additional content added to the original idea. This was detected by a change in handwriting. The number of ideas that received agreement were quantified by ideas with check marks next to them. Rapid inductive, team-based content analysis of the brainwriting results was conducted in Excel 16.3 (Microsoft, Redmond, Washington) by 3 coders to identify emergent themes and high-quality ideas. Ideas were deemed high-quality if they identified barriers that the research team could address through education, awareness, or adaptations to the intervention. The data were used to inform transitions nurse training and site-specific adaptations to the TNP, as needed. Examples of high-quality ideas generated from 1 hospital and associated primary care clinic are reported in this article. Facilitator experiences with hosting the brainwriting premortem sessions were collected using semistructured interviews and analyzed using content analytic methods.

RESULTS

Brainwriting premortem sessions were conducted at 5 TNP implementation hospitals and 5 associated primary care clinics. Four primary care clinics did not conduct sessions because of challenges in scheduling or a requirement to pay overtime for staff to attend during their lunch hour. Attendance at the sessions ranged from 2 to 26 participants (n = 116; mean = 11.6) (Table). Registered nurses (n = 38) and licensed practical nurses (n = 11) were the dominant group in attendance, followed by physicians (n = 17), social workers (n = 10), pharmacists (n = 9), administrators (n = 4), medical residents (n = 3), medical support assistants (n = 2), and other health care support staff (n = 18). No other persons were present during the brainwriting premortem sessions other than the participants and members of the TNP implementation team. Lack of participation by local staff was due to scheduling conflicts, absenteeism, or potentially, refusal to participate on the day of the site visits.

Table. Hospital and Clinic Characteristics and Performance Results.

Site	Participants, N	Total Ideas Generated, N	Unique Ideas, N	Ideas With Agreement, N	Expanded Ideas, N
Hospital A	12	24	19	0	0
Clinic A	18	59	29	34	7
Hospital B	26	50	39	0	0
Hospital C	15	50	30	23	4
Clinic C	11	100	25	34	2
Hospital D	9	10	14	0	0
Clinic D	2	7	7	0	0
Hospital E	13	29	25	0	2
Clinic E-1	6	27	21	2	0
Clinic E-2	4	8	8	0	0
Total	116	364	217	93	15

Postbrainwriting premortem surveys were administered after 6 of the 10 sessions, with an average response rate of 73% across sites. Surveys were not distributed at 4 sites because of time constraints of clinical staff. Overall, the participants reported satisfaction with their group's productivity (median: 4 [agree]) and work processes (median: 4 [agree]). The participants agreed (median: 4) that they were able to give ideas to the group and agreed (median: 4) that they had no concerns about criticism from the group, suggesting high levels of psychological safety (see Supplemental Digital Content Table 1, available at: http://links.lww.com/JNCQ/A468).

Across the 10 sites, the brainwriting premortem activity generated a total of 364 ideas regarding potential barriers to program implementation. Of those, 217 were unique ideas, 93 ideas were supported by other members of the group, and 15 ideas were expanded (ie, other members of the group added content to a previous idea). Individual sites' performance ranged from 7 unique barriers identified for a group of 2 health care providers to 100 unique barriers for group of 11 health care providers (Table). Examples of high-quality ideas are listed in Supplemental Digital Content Table 2, available at: http://links.lww.com/JNCQ/A469. The data created by participants were delivered in written format and were available for analysis immediately after the session (see Supplemental Digital Content Figure, available at: http://links.lww.com/JNCQ/A470).

High-quality ideas from 1 hospital and an associated primary care clinic are presented in Supplemental Digital Content Table 2, available at: http://links.lww.com/JNCQ/A469. The participants indicated concerns about the lack of role definition for the transitions nurse, the lack of clarity regarding Veteran eligibility information, and anticipated high volume for the TNP. Veteran-centric challenges, such as Veterans with disabilities who struggle to communicate via phone or have difficulty comprehending discharge instructions, homelessness, lack of transportation, and concerns of poor adherence to appointments and follow-up instructions, were identified. Veteran-related communication challenges, due to poor documentation in the electronic medical record, lack of phone, or incorrect number for the Veteran, were additional potential barriers. Concerns in the primary care clinic context included lack of appointment times due to full schedules, lack of specialty services in rural areas, and lack of current working relationships with primary care teams (see Supplemental Digital Content Table 2, available at: http://links.lww.com/JNCQ/A469).

The participants at the example sites were highly engaged and offered solutions to remedy program barriers identified during the brainwriting premortem sessions (see Supplemental Digital Content Table 2, available at: http://links.lww.com/JNCQ/A469). The participants shared that hiring more transitions nurses would address anticipated high volume. Veteran-centric challenges could be addressed by providing education and materials at the Veteran's health literacy level and stressing the importance of follow-up appointments with their primary care provider. Veteran communication barriers could be addressed through beginning discharge education early and confirming address and telephone numbers prior to discharge. Primary care clinic barriers could be addressed by hiring more staff, opening more appointment slots, and effective scheduling. Finally, the lack of relationships with primary care teams could be addressed through transitions nurses traveling to clinics to meet face-to-face with primary care colleagues.

The 10 brainwriting premortem sessions were hosted by 6 female (5 white, 1 African American) members of the TNP team. Their educational backgrounds ranged from a bachelor's degree in nursing, a master's degree in public health, to a PhD degree in anthropology. Three of the team members had previous experience hosting focus groups, while 3 had none. Fidelity to the brainwriting premortem protocol was reported by all facilitators for the setup and introductory script. Adaptations to the protocol included the facilitator providing additional prompts. Some facilitators allowed participants to spend less than the 10-minute time allotment on brainwriting for the group preferred to verbally discuss failure points and potential solutions. The verbally shared data were not included in these analyses.

The facilitators reported positive experiences running the sessions and the method was easy to implement. A noted benefit was the ability to collect a large number of targeted ideas, in writing, from a lot of people in a short amount of time. Although participants reported that the activity was fun, some did not necessarily want to write. However, the facilitators reported that for those who engaged in the process, the writing appeared to make participants think and provided themes for others to comment and expand on. The facilitators indicated that the participants appeared flattered that they were being asked their opinions, “almost as if no one had ever asked for their opinions before.” Although scheduling and recruiting participants were a challenge at all sites, the facilitators reported that once people were in the room, there was a high level of engagement and excitement when sharing ideas with each other.

The facilitators shared tips to running a successful brainwriting premortem session. This included practicing with a friendly group before going on-site, ensuring that the participants understood the program being implemented, and introducing sessions by stating, “We don't have the answers for you. This is about getting your feedback to inform this project at your site.” An unintended consequence of the premortem approach was that some participants used the session to vent about their organizations.

DISCUSSION

Multiple methods are available to nurses to engage stakeholders and collect data pertinent to program implementation. The approach selected depends on the objective of the study, issues related to the availability of participants, budget, and desired richness of the data.19 In this nationwide project, we successfully conducted 10 brainwriting premortem sessions led by facilitators with varying levels of education and group facilitation experience. The 10-minute writing sessions engaged 116 health care providers. The participants reported high satisfaction and psychological safety with the method.

The relationship between quality and quantity in idea generation is strongly linked, though the measurement of quality is subjective and often rated on novelty or usefulness.20 Across the 10 sites, 217 unique barriers to program implementation were generated. Numerous ideas were deemed high quality in that they were pertinent and actionable. The data were immediately available for analysis, allowing for rapid feedback to participants. The barriers and solutions enhanced the understanding of each site's local context, structures, and culture, resulting in targeted adaptations to the TNP intervention. The method was deemed user-friendly by facilitators and generally well received by participants. Facilitators with no experience hosting a focus group and those with extensive experience reported the method easy to implement. Taken together, our results suggest that the brainwriting premortem method is a rapid, moderate-skill, inclusive approach to engage stakeholders and identify large numbers of high-quality barriers to the implementation of an intervention. This approach could be used prior to implementation of new electronic medical record systems, new staffing models, or in diverse community-based health projects.

Our study adds to an existing body of literature on the importance of stakeholder participation and the assessment of barriers for improving the quality, relevance, dissemination, and implementation of research.1,21,22 The results of this study identified multiple key strengths of the brainwriting premortem method over other group-based elicitation approaches. The method is efficient and accessible. It can be successfully conducted in 10 minutes, compared with traditional focus groups that require up to 120 minutes.19 The approach does not require a highly trained facilitator, opening up this method to groups outside of academic research circles. The method provides high-quality and actionable data at minimal cost that can inform real-time decision making. The method can be used in qualitative or mixed-methods studies, as well as evaluations of programs or policies. Finally, the method can be taught to implementers and community members to continue to solicit stakeholder input to sustain programs or adapt interventions along the way.

An important finding of this study was the high productivity of participants, as witnessed by the large number of ideas written by each group. We propose that this is because brainwriting facilitates fluid thought processes8,9,16 and feedback. The participants were witnessed filling an entire page with ideas in minutes. As opposed to verbal elicitation approaches, the participants could review others' ideas, did not have to wait for others to stop talking, nor did they experience competing cognitive demands when attending to ideas of others while attempting to generate their own idea.8 Furthermore, the risk of forgetting an idea or determining that it is no longer relevant while waiting for a chance to speak was eliminated.5 The method encouraged participation since individuals could not hide in the crowd and created a climate of psychological safety. Finally, the method facilitated the sharing of ideas by clinicians that was focused on a single program and tied to implementation efforts.

The facilitators reported that the method appeared socially and cognitively stimulating. We believe that this is because the premortem approach invites those who will be doing the work to provide input before the project begins. This builds engagement and an expectation that the project will not be perfect from the start.11 In addition, the sharing of plausible failure points between people with multiple skills, roles, and backgrounds allowed for the cross-pollination of ideas. Although approaches such as Healthcare Failure Event Mode Analysis23 could stimulate the sharing of potential risks to program implementation, the premortem's prospective hindsight approach addresses a range of cognitive biases. This is reported to minimize overconfidence, the planning fallacy, optimistic biases, and groupthink that affect many teams once a program has been approved.10,24

Limitations

This study should be interpreted in the context from which it was derived. This study has some limitations. First, we examined only a premortem approach. Future research is required to more closely examine the premortem approach versus a preimplementation risk analysis–type focus group. Such work is important, given that the 2 approaches attempt to collect barriers to program implementation. The premortem approach may have been foreign to some and more challenging to explain; however, the minimization of cognitive biases is a significant advantage of the method.24

The generalizability of this study is limited by the intervention and characteristics of the study participants. We evaluated only brainwriting premortem focus groups. Comparison of this method with a brainstorming premortem approach or one-on-one interviews using the premortem question will reveal which method generates the greatest number and quality of ideas at the lowest cost and with the least expertise. Individual interviews may generate more ideas and richer data19; however, they take more time (interview, transcription, analysis), would not have engaged as many participants, and would not have allowed group interaction. The brainstorming approach requires skilled facilitation to ensure full participation and detailed transcription of recordings.

Study participants were largely health care providers recruited from hospitals and clinics participating in the TNP. The participants were college-educated, English-speaking, literate members of society. The success of strategies for adapting this method to low-literacy populations requires further investigation. All sessions were held in person. Adaptation of the protocol to a virtual (ie, web-based)17 or asynchronous format16 is plausible but would require an innovative electronic platform and highly engaged participants. Furthermore, the difference in number and type of participants in each group challenged our ability to interpret productivity across and between groups. Finally, postbrainwriting premortem surveys were not distributed at all sites.

CONCLUSION

The brainwriting premortem approach is a novel, efficient alternative to brainstorming focus groups that can rapidly inform program implementation at minimal cost. The method engages diverse members of a team or community and allows for rapid collection of handwritten data that can be analyzed in real time. This can lead to immediate program adaptations that will facilitate successful adoption, implementation, and sustainability of diverse health care or community-based programs.