You Don’t Know What You Don’t Know: Using Nominal Group Technique to Identify and Prioritize Education Topics for Regional Hospitals

Abstract

BACKGROUND:

Our 121-bed children’s hospital is a quaternary care referral center for a 33-county region. Referring hospitals asked for Pediatric Acute Care Education Sessions (PACES). To determine which topics to prioritize for these sessions, nominal group technique (NGT) methods were used to obtain stakeholder-prioritized consensus on education topics.

METHODS:

Five NGT sessions were conducted over 6 weeks at referring hospitals throughout central and northern California. Each session lasted ∼90 minutes and engaged a diverse multidisciplinary group of stakeholders. At these sessions, stakeholders answered the question “What are your top 5 clinical topics that should be prioritized by PACES?” NGT numeric ranking methods were used to determine prioritized topics. A thematic analysis was performed on the session transcripts.

RESULTS:

The 5 sessions had 43 total participants, including nurses, respiratory therapists, physicians, and administrators. The top 4 prioritized topics were sepsis, diabetic ketoacidosis, respiratory failure or support, and bronchiolitis and/or respiratory syncytial virus. Unique education needs for each hospital were also uncovered in the NGT discussion. Three qualitative themes emerged from the discussion: diverse educational needs, the need for guidelines on telemedicine and transfer, and relationship building.

CONCLUSIONS:

The use of NGT to engage multisite, multidisciplinary stakeholders helped to inform an education program. The use of NGT methods provided rich information that would not have been obtained through surveys alone and helped facilitate relationship building. The PACES group was able to identify and prioritize education topics of interest to referring community hospitals.

Pediatric patients account for 20% of all visits to emergency departments (EDs)¹; most are seen in community hospital EDs.² Many community EDs, located in primarily adult-focused institutions, face unique challenges in caring for pediatric patients because of lack of specialized equipment and access to trained pediatric emergency medicine physicians.^2,3 Collaboration between university-affiliated hospitals and community hospitals has been shown to improve outcomes,^4,5 educate and develop robust workforces,^6–8 and build quality improvement capacity.⁸

Our 121-bed children’s hospital, a quaternary care center located in northern California, is the referral center for many children across a 33-county region covering 65 000 mi² and serving ∼6 million people.⁹ The children’s hospital receives transfers from >130 EDs and hospitals in the region and accepted >2500 patients as transfers in 2017. The children’s hospital also has a robust on-demand telemedicine program for pediatric and neonatal critical care.

In early 2018, the Pediatric Acute Care Education Sessions (PACES) workgroup was formed. The group’s objective was to determine how best to address education requests from regional EDs and hospitals. The team administered a preliminary needs assessment via electronic survey with 9 preselected acute care topics sent to physician and nurse contacts at the regional hospitals. The assessment had a low response rate and limited useful data; almost every participant rated that they were “extremely interested” in all 9 topics. After reviewing the survey data, the workgroup determined that more feedback and representation from the referring hospitals was needed in the planning and prioritization of topics. Therefore, in this study, nominal group technique (NGT) was used to obtain stakeholder-prioritized consensus on education topics.

Methods

Sessions were scheduled with 5 interested hospitals, and NGT was used to engage stakeholders in each session. NGT is a method previously used in other medical settings to develop health care guidelines,^10–13 inform practice change,^14,15 develop research priorities,^16,17 and adapt educational policies.^18,19 NGT is used to facilitate effective group decision-making to obtain stakeholder input in response to a specific question.^20,21 NGT allows for all voices to be heard, removes power dynamics that can often occur in group discussions, and increases stakeholders’ ownership of the ensuing project.²² NGT consists of 4 phases: silent generation of ideas, sharing ideas in a round-robin style, discussion (clarification) of the list of ideas, and ranking of ideas from the list.²³

Five NGT sessions were conducted over 6 weeks at referring hospitals throughout central and northern California (Table 1). Each session lasted ∼90 minutes and was attended by a range of health care professionals (Table 1). Recruitment efforts were aimed for ∼10 participants per session, a preferred group size for NGT to maximize participation and minimize dissatisfaction.²⁴ The sessions were moderated by a pediatric hospitalist who has training in NGT methods and a project manager. Each session included robust participant discussion, which was recorded and then transcribed for review. This study was determined to be an exempt quality improvement project by the institutional review board.

TABLE 1.

NGT Hospitals and Session Attendees

Hospital	Hospital Characteristics			NGT Session Attendees
	Annual ED Pediatric Volumea	Pediatric Admissions?	NICU Admissions?	Total Attendees	Administrators and/or Educators	ED or Flight: Physician, Nurse, or Respiratory Therapist	Inpatient or Neonatal: Physician, Nurse, or Respiratory Therapist
1	3902	No	No	8	6	1	1
2	15 155	No	Yes	7	5	2	0
3	11 998	Yes	Yes	8	1	3	4
4	12 375	Yes	Yes	12	5	4	3
5	12 632	Yes	Yes	8	6	2	0

^aData are from the California Office of Statewide Health Planning and Development, obtained September 2018.

After introductions, the moderators introduced the concept and phases of NGT to the participants. Participants were given ∼5 minutes to write their answers to the question “What are the clinical topics you think should be prioritized by PACES?” (silent generation phase). Next, each participant took turns sharing 1 answer at a time, until each participant shared all of their answers (round-robin phase). Responses were written on a flip chart visible to all. After the round-robin sharing, the floor was open for discussion. Participants were able to ask each other to elaborate on responses, combine or split topics, or add topics (discussion phase). Once the list was finalized, participants independently and anonymously answered the question “What are the top 5 clinical topics you think should be prioritized by PACES?” (ranking phase). After the session, 1 moderator assigned point values to each response (5 points for the first choice, 4 points for the second…1 point for the fifth) and compiled the results in a spreadsheet.

Thematic analysis of NGT discussions was used to gain deeper understanding of stakeholder needs PACES could address.^23,25 The 2 moderators independently generated initial codes for any interesting features of the transcriptions. They met to compare and discuss codes, collate codes into potential themes, and refine and finalize the themes. The list of prioritized topics and the themes were presented by e-mail to the study participants to solicit feedback as a form of member checking.²⁶ Respondents confirmed that participants agreed with the description of the themes.

Results

The 5 sessions had 43 total participants (Table 1), including physicians, nurses, respiratory therapists, and administrators and/or educators, such as chief nursing officers, chief medical officers, and nurse educators. Across the 5 sessions, 46 topics had at least 1 point, and the top 50% of topics had at least 10 points (Table 2). The top 4 topics prioritized across the groups were sepsis (65 points), diabetic ketoacidosis (61 points), respiratory failure or support (46 points), and bronchiolitis and/or respiratory syncytial virus (RSV) (45 points).

TABLE 2.

Cumulative Results From NGT Sessions for Priority Topics

Topic	Description	Total Points
Sepsis	IV fluids, management, and guidelines (nonneonatal population)	65
Diabetic ketoacidosis	Fluid and medication management	61
Respiratory failure or support	Which support to use; indications for intubation	46
Bronchiolitis and/or RSV	Scoring system; indications for high-flow nasal cannula	45
Oxygen modalities	High-flow nasal cannula versus noninvasive positive-pressure ventilation	34
Asthma	Scoring, choice of systemic steroid	31
Pediatric assessment (scoring)	Implementation and use of a scoring system	31
Pediatric trauma	Head trauma, appropriate imaging, motor vehicle collisions	27
Transfer guidelines	How to prepare a patient for interfacility transfer	22
Intubation	Indications for intubation, managing intubated patients, vasopressors, continuous infusions	20
Neonatal abstinence syndrome	Exposure-dependent medication choices and doses	18
Cardiac issues and/or complications	Identifying disorders, treatment, transfer, and/or secondary cardiac issues	18
Febrile illness	Evaluation in newborn and pediatric patients, indications for lumbar puncture	17
Overdose or ingestion	Exposure-dependent treatment, including antidotes and dosing	16
Skills	Medical and/or procedural, family and/or parent interaction, distraction, role of child life	16
Gastroenteritis or dehydration	Hydration techniques; when to order Clostridium difficile testing	14
ED stabilization	Stabilizing critically ill patients before transfer	14
Mental health	Stabilization procedures, community resources	13
Real-time videos	Symptom recognition; desire for skills training	13
Opioid exposure	Intentional and accidental; medication choices, including antidotes and dosing	12
Shock	Assessment, medications, management	10

Three primary themes were identified from the qualitative analysis: (1) diverse educational needs, (2) guidelines for telemedicine and transfer, and (3) relationship building. Each referring hospital had education needs unique to their hospital (Table 3). Prioritized acute care topics included not only various diagnoses across different patient populations (such as neonatal or oncologic) but also procedures, skills, and policies. Nurses and respiratory therapists at all of the sites shared the desire for additional training and resources such as just-in-time videos. Second, expectations surrounding the use of telemedicine and transfers are unclear. Almost every participant expressed a desire to have guidelines on when to use telemedicine and how to appropriately prepare a patient for interfacility transfer. Participants felt such guidelines would streamline the process and reduce inefficiencies. Third, the NGT process was a valuable experience in building relationships; participants expressed a strong desire to engage in ongoing collaboration with the PACES workgroup.

TABLE 3.

Representative Quotes for Each Theme

Theme	Quotes
Diverse educational needs	“The 2 criteria I think are most important are one, things that are common, and then 2, things that have a lot of practice variation. I think that would be the most helpful from a doc perspective.”
	“So, I just want to say, I feel like there will be kind of 2 different camps. Like, you know? I’m not as concerned about all the neonatal things and the management because I work in the ED, where, you know, I think our topics might differ as far as our top 5, based on just…the clientele we’re dealing with.”
	“So, I’m going to [go] off topic a little bit. I think whatever the topics are, we need real-time online videos. Because we can send all these people to these classes, but if they don’t see that [sic] our admitted patients for 2 years, they’re not going to remember it. So, for instance, if we look at respiratory distress, we need something they can click on and actually use. This is what respiratory distress looks like in a 2-year-old, and this is what you need to do, because I think we’ve all done in small places, send people to classes, they don’t remember it when they need it.”
	“Pediatric assessment and skills kind of go hand in hand. How do you listen? How do you start an IV? How do you speak to the different age groups of children and deal with some…some of the moms that are really, you know, stressed, making the child anxious, and you know, the mom that seems like she doesn’t really care? And that might be frustrating too. And how do we suction safely and do venipuncture? And so, a lot of it comes from an inpatient standpoint of kids that aren’t really that sick. They’re not really any sicker than your kids at home, but there’s still a huge, uncomfortable feeling when the nurses upstairs are caring for these pediatric patients. So, kind of, just empowering and like, as us adults every day, it’s the same, but here’s the few differences that you look for.”
	“And, I just would like to just ask that you really do look at your process around maybe asking the hospitals about videos because, you know, not…not just here but in lots of small hospitals that at least I’ve interacted with. And because we don’t have them, our nurses resort to YouTube, and they look at YouTube pediatric assessments and…like, they want something.”
Guidelines for telemedicine and transfer	“And then how to we best prepare the patients who are being transferred, because I think we can really do a better job with that.”
	“But, interventions that absolutely have to be done at [our ED] versus things that can wait to be done at [the admitting hospital] to not delay transport.”
	“We transport a lot of kids out here, so it would be helpful to know if there’s any kind of algorithms or prework that should lead up to the transport, versus do we have everything? I don’t know if we have everything. What do you mean we don’t have everything? What’s going on? Which typically can happen.”
Relationship building	“Are there things that the medical center wished that we knew or did that we don’t do, maybe we should do? They’ve been getting the ideas from us, but how about the reverse?”
	“We’re so lucky that [the hospital’s] like mission, part of their mission is in rural outreach; we couldn’t be luckier.”
	“If people are already working on sepsis at [the hospital], we’re not going to do something separate. Like, we’ll be collaborative.”

Discussion

With the use of NGT methods, our study team was able to engage multisite, multidisciplinary stakeholders to inform a regional pediatric acute care education program. We found that regional hospital partners expressed interest in education on a variety of topics, with sepsis, diabetic ketoacidosis, respiratory failure or support, and bronchiolitis and/or RSV receiving the most votes across the 5 sites. By using NGT methods, we captured responses from a wide variety of stakeholders; these topics were developed by the stakeholders and therefore relevant to their specific needs.

The importance of stakeholder engagement in research and quality improvement has been gaining prominence.^27,28 NGT is recognized as a deliberative process used to encourage stakeholder participation such that all members are able to contribute to discussions and decision-making.^27,29 NGT is a valuable method for engaging stakeholders, particularly in groups in which there are concerns about power dynamics. In our sessions, we had a wide range of stakeholders, from the chief medical officer of a hospital to a bedside ED nurse. NGT allowed for all of their voices to be heard. Also, because the attendees from the children’s hospital simply served as facilitators, community participants defined their own goals, which strengthened existing partnerships and fostered development of new relationships. Since the NGT sessions, participants have collaborated with the children’s hospital on additional quality improvement projects.

The discussion during each session also provided the workgroup with rich information not obtained through survey methods alone. Among the 9 preassigned survey topics, 6 topics (pneumonia, urinary tract infection, minor head injuries, intravenous (IV) fluids, croup, brief resolved unexplained episode) were not in the top half of the NGT results. Although there was some overlap in topics between the survey and the NGT sessions, content analysis of the discussion provided us with specific details about what should be included in each PACES session; for example, in the future bronchiolitis and/or RSV session, the presenters will include information about respiratory scoring systems and parameters for high-flow nasal cannula use. Also, on the basis of thematic analysis of the discussion, we will include guidelines for telemedicine use and guidelines for transfer in each PACES education session. We also plan on creating just-in-time videos and providing site-specific skills training sessions for respiratory therapists and nurses; this need would not have been identified without the NGT sessions and subsequent discussion.

This study has some limitations. We only included 5 referral hospitals. However, 5 sessions are within the range of the commonly used number of sessions for NGT.³⁰ It is possible that not all perspectives were included at each hospital because the moderators limited the sessions to ∼10 participants to maximize participation and satisfaction.²⁴ Finally, because only hospitals in California were included, the priority topics we identified may not be generalizable nationwide. However, the value realized from using the NGT process to engage stakeholders and inform an education program should be generalizable.

Conclusions

Using NGT methods across 5 community hospitals, the PACES workgroup at our university-affiliated hospital was able to identify and prioritize education topics of interest to community hospitals to be discussed during upcoming education sessions.

Members of the PACES Working Group

Julie B. Clayton, RN, MSN, NEA-BC, Barton Healthcare, South Lake Tahoe, California

Kara L. Davis, BSN, CCRN, MJC, Enloe Medical Center, Chico, California

Mark R. Eyre, MBA-HCM, RRT, RCP, Adventist Health-Rideout, Marysville, California

Bill Foo, MD, Mercy Medical Center-Redding, Redding, California

Sharon Joo, DO, MPH, Mercy Medical Center-Redding, Redding, California

J. Eileen Morley, MD, Adventist Health-Rideout, Marysville, California

Mary Rhonda Sneeringer, MD, FAAP, Barton Healthcare, South Lake Tahoe, California