Engaging End-Users in Intervention Research Study Design

Abstract

Objective:

Coordinated Healthcare Interventions for Childhood Asthma Gaps in Outcomes (CHICAGO Plan) is a 3-arm multicenter pragmatic trial to evaluate asthma interventions in high-risk Chicago children presenting to Emergency Departments (ED) with asthma. A formative evaluation with end-users to provide input into the trial design and outcome instruments was conducted prior to trial initiation.

Methods:

A multi-level data gathering framework from the field of design and standard qualitative methods were employed. This included one focus groups with asthma Community Health Workers (CHWs, N=8), two focus groups with caregivers of children with asthma (N=9), in-home interviews with caregivers (N=9), key informant interviews at six EDs and outpatient clinical sites (N=19), and ED tours and observations (N=6). Data were presented, discussed, and organized into themes.

Results:

Data indicated that changes to the study design and discharge tool were warranted. A key insight was that ED discharge protocols typically place patient education at a single inopportune time, as families are preparing to leave the ED. At this point in time, families are less receptive to education due to fatigue and a desire to expedite the discharge process. The trial design was modified to reposition the discharge asthma plan to occur at earlier “teachable moments.” Delivery of the asthma discharge plan was assigned to study-employed ED coordinators instead of ED providers and staff. Other potential challenges to study recruitment and implementation were raised and addressed.

Conclusions:

Engagement of end-users in the design phase of implementation research is critical to improve research feasibility and relevance.

Introduction:

Asthma is one of the most common chronic condition of childhood, affecting 6.2 million children nationally and leading to more school days missed than any other disease[1–2]. Considerable and widening health disparities can be seen in pediatric asthma[1–3]. Minority and low-income children are more likely to visit the emergency department (ED), require hospitalization, and die from asthma than non-Hispanic White children[2–3]. ED rates for non-Hispanic black children are more than four times higher than for non-Hispanic white children[4]. Some ED use for asthma is a result of severe asthma, but often children present to the ED with uncontrolled asthma due to improper management at home, in the ambulatory setting, or both[5–6]. Self-management education and care coordination support both during and after ED visits should improve ambulatory follow-up and asthma control for these children, however studies testing these types of interventions have not shown the expected results[7–9]. Perhaps for ED asthma interventions to be successful, they need to target all areas of a child’s asthma care which includes healthcare settings, the home, and community[9].

In Chicago, the prevalence, morbidity, and disparities related to asthma are among the worst in the country[10–11]. In certain low-income minority Chicago communities, the pediatric asthma prevalence exceed 25%[10–11]. Furthermore, children in underserved predominantly African-American and Latino communities demonstrate 5–7 fold higher asthma ED rates than those seen in non-minority Chicago communities[12]. To address these Chicago-based disparities, the Coordinated Healthcare Interventions for Childhood Asthma Gaps in Outcomes (CHICAGO) Plan (www.chicagotrial.uic.edu) formed as a multi-center comparative effectiveness trial funded by the Patient-Centered Outcomes Research Institute (PCORI)[13]. The CHICAGO Plan aims to test proven strategies to improve the care and outcomes of African-American and Latino children with uncontrolled asthma presenting to EDs in Chicago[14]. Successful strategies included an enhanced ED-discharge process using an asthma discharge tool [15] and home visits from community health workers (CHWs)[16–17]. The CHICAGO Plan was designed as a 3-arm randomized controlled trial with a usual care arm, enhanced ED-discharge arm (using the asthma discharge tool), and enhanced ED-discharge with CHW home visitation arm[14].

Because of the challenges faced by previous ED-based interventions [9], the first year of the CHICAGO Plan was spent conducting formative work with caregivers, providers, administrators and CHWs to inform the primary study outcomes, the usual care arm feasibility and acceptability, the content and process of the ED intervention, recruitment plan logistics, and the content of the CHW intervention. Our hypothesis was that this process would highlight challenges in the original study design and suggest alternative pathways for study implementation which would improve the likelihood of the study’s success. This manuscript describes the process, results, and subsequent changes made in the trial’s design as a result of this formative process.

Methods

The formative work was conducted by the CHICAGO Plan Patient and Stakeholder Work Group which is led by the Chicago Asthma Consortium (CAC) and Respiratory Health Association (RHA), with support from investigators and the Illinois Institute of Technology (IIT) Institute of Design. Human subjects approval was granted by the University of Illinois at Chicago, Illinois Institute of Technology, Lurie Children’s Hospital, Mount Sinai Hospital, University of Chicago Medicine, Cook County Hospital, and Rush University Medical Center; written informed consent was obtained for focus groups and interviews.

The formative research design utilized a multi-level data gathering framework (Figure 1), in which relevant tools (e.g., the asthma discharge tool), stakeholder needs, and the context in which stakeholders work and live are targeted for inquiry. This approach comes from the field of design and is useful because it provides a broad perspective on the desired intervention while also revealing specific requirements that can guide implementation[18–19].

Figure 1:

Formative Research Components

Preparing for the Formative Research:

To prepare stakeholders for the research process and vet the formative research plan, the CAC hosted a public stakeholder meeting in June of 2014 on the Emergency Management of Asthma. Over 75 people attended, including physicians, researchers, nurses, respiratory therapists, CHWs, program managers and representatives from pharmaceutical and medical device companies. This meeting included presentations from Chicago Public Schools about school-based asthma policies, an allergy/immunology specialist about asthma action plans, and an ED physician sharing Illinois asthma surveillance data. CHICAGO Plan researchers also spoke about the study aims and design. Presentations were followed by an hour-long panel discussion which highlighted challenges related to ED-based asthma education, communication with primary care providers about the ED visit, and emergency management of asthma in schools. The formative research plan was reviewed and approved by CAC’s Community Advisory Board (http://cac.websitetesturl.com/about/community-advisory-board/) who provided suggestions for the content of the formative interview guides separately.

Focus Groups (FG):

FGs were conducted with caregivers and CHWs. Eligibility for caregiver FGs included: English or Spanish-speaking adult caregiver of a 5–11 year old child with an asthma-related ED visit at one of the study sites in the past 12 months. CHWs had to have more than 12-months experience conducting home-based asthma interventions with the target population. Participants were recruited from participating clinical centers using flyers, personal outreach, and email blasts from the CAC, RHA and other community-based organizations. In total, nine caregivers participated in two caregiver FGs and eight CHWs participated in the CHW FG. Participant demographics are shown in Table 1; the caregivers and CHWs were mainly African American or Hispanic which represented the target demographics of the CHICAGO Plan. FGs were held at RHA. The sessions were conducted in English (a Spanish-language group was offered but all spoke English), lasted about two hours each, and were audio recorded. Participants were compensated $50. See Table 2 for FG question summary.

Table 1:

Participant Demographics

	Caregivers (N=9)	CHWs (N=8)	Key Informant Interviews (N=19)	User-centered Interviews* (N=9)
Female	7	8	13	9
Age, mean (range)	34 years (24–46 years)	48 years (32–64 years)
Education level
High school/GED	3	-
Associate degree	3	3
Bachelor degree	1	4
Masters degree	2	1
Hispanic	2	2	4	1
African-American	7	6	3	8
Has a child with asthma	9	2
Has asthma themselves	4	3
Last time in ED
In last 2 weeks	1
Last 2–4 weeks	1
More than 4 weeks ago	7
Child age, mean (range)	7 years (5–10 years)
Child female	4
Child public insurance	8
Caregiver marital status
Single	4
Live with partner/spouse	4
Live separate from partner	1
Number of people living in home, mean (range)	4.6 (2–9)
Children	1.9 (1–5)
Children with asthma	1.2 (1–2)
Years as a CHW, mean (range)		7 (3–15)
Years working in asthma, mean (range)		10 (3–34)
Employed
By hospital system		6
By community agency		2
Job description
ED physician			6
ED nurse			4
ED administrator			4
Outpatient provider			5

^*3 caregivers participated in both the focus group and user-centered interview

Table 2:

Focus Group, Key Informant, and User-Centered Interview Question Domains

Caregiver Focus Groups	CHW Focus Groups
I. Experiences as a caregiver of child with asthma:  – Most recent experience in ED  – Experiences and beliefs regarding different kinds of asthma medications, follow up visits for asthma, and education in the ED II. Feedback on study design:  – Which outcomes instruments preferred and why: Childhood Asthma Control Test (cACT), Asthma Control Questionnaire (ACQ), Pediatric Asthma Caregiver Quality of Life Questionnaire (PACQLQ), PROMIS® Asthma Impact scale[20–25]  – Perceptions of a CHW visiting their home to provide asthma self-management education	I. Experiences of their clients in the ED  – Medications, follow up visits, receipt of educational services in the ED II. Discharge tools  – What their clients were need  – Feedback on a paper discharge tool III. Study design  – Opinions on the recruitment and implementation plan after hearing about the study design  – Which outcomes instruments they thought would work better with their clients and why: Childhood Asthma Control Test (cACT), Asthma Control Questionnaire (ACQ), Pediatric Asthma Caregiver Quality of Life Questionnaire (PACQLQ), PROMIS® Asthma Impact scale[20–25]
Key Informant Interviews	User-Centered Interviews
I. Discharge process  – Describe and demonstrate the discharge process from start to finish, including showing how they used any tools, paperwork or supports with patients and caregivers.  – List aspects of the discharge experience that are most difficult for caregivers and highlight those most impactful on patient outcomes.  – Give feedback on the discharge tool II. Medications  – Describe ED procedures regarding inhaled corticosteroid and quick-relief medication prescriptions, education III. Follow up  – Describe ED procedure for follow up appointments	– Caregivers were asked about immediate family and health concerns, recent asthma events, ED experiences, outpatient visits and healthcare access.  – Caregivers were asked to sketch their journey learning to manage asthma and their last ED experience, noting timing, tension points and handoffs.  – Caregivers sketched home floor plans and highlighted perceived safe and unsafe zones to reveal caregiver knowledge of asthma triggers.  – A home tour captured images of home environmental conditions, storage for asthma medications and devices, and storage for discharge documents which prompted discussion about how they are used.  – Caregivers gave feedback on the discharge tool.

Key Informant Interviews:

Key informant interviews were conducted with 6 ED physicians, 4 ED nurses (in charge of staffing), 4 ED administrators, and 5 outpatient providers from the six clinical sites. At two sites, both a nurse and an administrator were interviewed to further explore issues regarding staffing and discharge processes. Interviews lasted one hour, were conducted in the ED or clinic and were audio recorded and transcribed. Participants were compensated $50. See Table 2 for interview question summary.

User-Centered Interviews:

User-centered interviews (N=9) were conducted in the home and designed to help understand caregiver behaviors as they occur naturally, shedding light on discrepancies between what participants say they do versus what they actually do. Eligibility for caregivers was the same as for the FGs. Recruitment was done in conjunction with FG recruitment. In-home interviews lasted 2–3 hours and engaged several data collection techniques, including interview questions, drawing and collaborative documentation of the home environment using photos and narrative (Table 2). Participants were compensated $100.

ED Observations:

ED observations documented the real-world context in which patients, caregivers and clinical staff engage in asthma emergency care. Direct observation embeds researchers directly into the activity context to document the interplay of people, their activities and tasks, objects and tools, and the role and influence of the built environment on these interactions. This reveals needs, barriers and workarounds to which participants have adapted and therefore no longer notice or think to share but should be accounted for in intervention design. Inclusion criteria were EDs participating in the CHICAGO Plan. Researchers sketched the floor plans of the EDs, noted hot zones of activity, engaged in rough behavioral mapping of the site and observed the flow of patients and clinical staff. Observations of 5 EDs and one call center were conducted.

Analysis:

Digital recordings from all interviews, FGs, and observations were transcribed. Analysis used the POEMS framework [26] which is an inductive, bottom-up process from the field of design where data from a range of stakeholders in a variety of settings are combined to understand better the context and implication of participant responses. Results from all data sources were visually displayed in a process of contextual inquiry[18–19] allowing the CHICAGO Plan investigators to view and discuss as a group. A smaller group from the Patient and Stakeholder Work Group then reviewed these results and transcripts in detail, drawing final themes to be applied to the study design and materials.

Results

Resulting Themes

Themes from all the data are summarized in Table 3. FG caregivers described their ED usage is an automatic response when their child is having an asthma exacerbation: “Basically with him, I’m the emergency room queen. Anything happens, I’m going to the emergency room.” ED usage is reinforced by personal experiences: “I’m very familiar with it because I’m an asthmatic myself. I’m 33, I’ve been through it a lot, through the ER, so I do what’s been taught to me…” EDs are also a reliable source of medication. As most insurances will only fill one canister of albuterol per month, many caregivers said they were forced to take their child to the ED when they ran out of this medicine. “One asthma pump is not going to last you a whole month. This is what the doctors don’t realize.” The effects of medication shortages can be exacerbated when families share albuterol inhalers to save money. “There are times I had nothing, I don’t have an income. I’m not paying $200 for this every month. I’ve got bills to pay…all of us have asthma, we can feed off of each other with an asthma pump.”

Table 3:

Formative Research Results by Theme and Domain of Intervention and Relationship to CHICAGO Plan Study Design

	Patient	Provider	System ED	System outside ED
MEDICINES
Pts go to ED for medicines (albuterol doesn’t last a month and insurance won’t cover more so go to ED)	Intervention targets			Unable to address
Medicines expensive	Intervention targets			Unable to address
Pharmacy not always open				Unable to address
Pts can’t always get transportation to pharmacy	Intervention targets			Unable to address
Pts don’t always understand electronic prescriptions	Added to intervention	Added to intervention
Assumptions about what pts have and know		Added to intervention
Pts don’t always have equipment (spacers)	Added to intervention			Unable to address
Pts don’t use devices properly	Intervention targets
Pts sometimes get albuterol in ED, sometimes not		Intervention targets	Intervention targets
Pts sometimes get prednisone and ICS, sometimes not		Intervention targets	Intervention targets
Some pts afraid of addiction and side effects so don’t use	Intervention targets	Intervention targets
Pts believe some brands work better than others	Intervention targets
ED FOLLOW-UP
Appointment dates too far in future				Intervention targets
Can’t get through on phone				Intervention targets
Office hours bad				Unable to address
Don’t get their doctor				Unable to address
Long waits				Unable to address
Don’t know how to use walk-in/acute care	Intervention targets			Unable to address
Insurance issues				Intervention targets
Intimidated by doctor so avoid	Intervention targets	Intervention targets
Don’t go if feeling ok	Intervention targets
EDUCATION
Paper-based education not helpful	Intervention targets	Intervention targets	Intervention targets
Pts want education and action plans	Intervention targets
Not always shown how to use equipment		Added to intervention	Added to intervention
ED physicians don’t feel education their job, refer to outpatient provider		Added to intervention	Added to intervention	Unable to address
ED assumes incorrect level of knowledge		Intervention targets
Physicians don’t all know guidelines or equipment		Intervention targets
Language barriers			Intervention targets
Literacy barriers			Added to intervention
Timing of discharge bad			Added to intervention
OTHER
Pt fear of asthma and providers	Intervention targets	Unable to address	Unable to address	Unable to address
Go to ED because child sick				Unable to address
Go to ED because it is what pts know	Intervention targets			Unable to address
Long wait times in ED			Unable to address
ED makes pts feel stupid, inferior		Added to intervention	Added to intervention
Pts feel get poor quality care		Added to intervention	Added to intervention
Asthma not priority in ED		Added to intervention	Added to intervention
Very stressful caring for child with asthma	Intervention targets

ED=emergency department

Pts=patients/caregivers of patients

ICS=inhaled corticosteroids

Caregivers also report that follow-up care is difficult to get and incongruent with their needs. They described long wait times on the phone, being given appointments too far into the future, or seeing a different doctor each time. Caregivers often do not see the purpose for the follow-up visit either. “I know what it is. I’m not going to make an appointment for two, three weeks later, and everything’s good by the time it gets here. Of course you’re not going to hear her wheezing...” These findings—both the centrality of the ED and the struggle with follow-up care—underscored how caregivers can reasonably conclude that asthma is an event best treated by doctors in an ED, rather than a long-term condition best managed at home.

Caregivers also gave feedback on data collection tools, expressing preference for the Childhood Asthma Control Test (cACT). “Because if it’s for my son, he knows the faces, that’s what they ask him when he goes to the ER. How do you feel? Instead of him expressing, he points to the picture. They’ll say ‘oh, you feel like this?’ He’ll tell them yes.” Caregivers said the overall study design was fine, but expressed concern about recruitment in the ED. Some said they would be more likely to participate if they could sign up a few days after the ED visit, once the stress of the asthma exacerbation had resolved.

The CHWs’ reflections on their clients’ beliefs and behaviors were similar to those reported by caregivers. The CHWs described issues with medications, including parents using the ED to get medication, mainly albuterol. When patients overuse albuterol or it is shared among family members, they run out before the pharmacy will refill. Often parents can’t afford the medicine or they do not have access to a 24-hour pharmacy upon discharge in the middle of the night. Families struggle to make and keep follow up appointments. When families call, the appointments they are given are frequently months away. Families do not know about walk-in or acute care clinics, and call centers do not tell them. When the acute asthma episode is over, other competing demands take over and families miss follow up appointments. “Or they use the ED because they have so many problems at home, they only deal with the asthma when it flares up. ‘Okay, he is doing fine, I got some medication. Now I need to go back home and put out more fires.’” Finally, the CHWs said families often receive ED-based education but cannot use it effectively, especially non-English speakers. “In my experience, I’ve learned that more of a vocal approach, an explanation of something scrubbed in a piece of paper works a lot better. They went home with some information but not really what they needed to prevent another ED visit.”

The key informant interviews, user-centered interviews, and site observations opened another window for understanding the ED process and flow. The discharge experience is fragmented, often delivered by multiple ED staff. Discharge protocols position patient education when caregivers are least prepared to take advantage of it—in the last 20 minutes of a typically 3–6 hour ED visit. ED nurses expressed frustration with caregivers’ unwillingness or inability to focus on patient education at discharge. Caregivers also reported frustration around lengthy discharge processes, repetition of information they say they already know, and the need to manage work, other children, and household obligations. Collectively, these conditions reduce receptivity to education at discharge. Caregivers often received large amounts of information at an ED visit. ED informants across sites reported numerous challenges in communicating that information to caregivers in the ED. ED informants said they do not review discharge documents with patients, because time is tight and the documents/tools available to them are long, complicated, and hard to use with caregivers. Informants report that discharge documents are “killing trees” because they are often left on the counter by caregivers and never make it home.

In the home, discharge information was stored out of sight. Families showed researchers bags and drawers filled with papers from the ED. Medications were stored in many different locations. Caregivers demonstrated an incomplete understanding of asthma, medication confusion, and workarounds to compensate for end of month shortages, such as designating one family inhaler, meaning it could not travel to school or daycare because other family members might need it. Caregivers expressed the need for a discharge tool that facilitates communication, education, and coordination of care with others (child care providers, extended family, school, and camp personnel) who share in the care of their children.

ED informants across the six sites expressed dissatisfaction with current discharge documents that were too long and complex for their patient population. Outpatient providers consistently noted that discharge paperwork was not helpful at follow-up visits because the information lacked detail. Caregivers who retained discharge documents viewed them as evidence of an event for others to see, not as relevant instructions for themselves. One caregiver commented that if the document wasn’t important enough for the ED staff to review, it couldn’t be very valuable.

Implications for the Planned Trial

The data brought to light a range of issues in participating EDs, many of which could directly impact intervention delivery. As shown in Table 3, some of the issues raised require changes in systems that operate outside the ED, such as pharmacy locations and hours, insurance coverage and policies, and clinic accessibility and processes, and so were not addressed. Issues within the control of the CHICAGO Plan affirmed some of the initial study design plans regarding the interventions and their implementation, but suggested modifications in some areas.

Specifically, the original intent of the study was to provide better discharge supports for asthma patients. The formative work confirmed that the existing tools and systems in the target EDs do not properly promote effective communication strategies that fit both the operational realities of the ED environment (time-constraints, multiple handoffs) and the abilities of caregivers to absorb unfamiliar and complex content. These data validated our plan to implement a redesigned asthma management discharge tool that provides education, follow-up information, and supports adherence by providers and caregivers to recommended asthma management guidelines[27]. Because of barriers with the quality and compatibility of the electronic medical record systems, we realized this tool would have to be hand written using premade color templates.

It also became clear that the existing ED physicians and nurses did not have time to deliver this tool properly. In response, the CHICAGO Plan created ED coordinators to recruit participants and deliver the asthma discharge tool. These coordinators could spend more time with families in discussion which would help with literacy issues and relieve some of the education burden on providers. Additionally, it was observed that ED discharge teaching typically occurred at a single point in time, usually as families were preparing to leave the ED and least ready to take full advantage of interactions with ED staff. In response, the CHICAGO Plan study design was modified to reposition the self-management education and other information in this discharge process so that it occurred earlier in the ED process, during the treatment and observation period.

The home-based CHW intervention was acceptable to caregivers and did not require modification from the original plan. We had concerns regarding the control condition in the ED because having a usual care arm in high-risk populations where intervention is needed is sometimes poorly received. All ED informants expressed that usual care was acceptable because “usual care” in their ED varied depending on conditions and staffing. Our formative work suggested however that the need for asthma equipment was tremendous. Therefore we decided our ED coordinators would provide spacers and inhaler technique training for all participants, regardless of arm in order to improve the ability of EDs to prioritize asthma, and allow patients to feel valued and supported.

Not all recommendations of the end-users were implemented. For example, we decided to use the PROMIS® Asthma Impact scale as the primary outcome instead of the caregiver-preferred cACT because the PROMIS instrument could be administered by phone or in-person (cACT needs to be administered in-person) and we intended to implement a combination of phone and in-person follow-ups. This instrument choice minimized the need for in-person follow-up visits and reduced the participation burden. However, we included the cACT at in-person visits. Regarding recruitment, caregivers, CHWs, and CAC’s Community Advisory Board all raised concerns about the ability to adequately recruit families in the ED due to the high stress of the ED environment. Because the goal of the CHICAGO Plan was to test an ED-based intervention for this specific population, we acknowledged the challenges of our recruitment plan and used the recommendations to optimize our plan (modifying our approach and language), but did not change our recruitment location.

Discussion

Interventions that have demonstrated success in one population or disease area do not necessarily work the same in others, and this becomes very apparent when attempting to translate research interventions into diverse real-world situations[28–29]. While the interventions proposed in the CHICAGO Plan had established efficacy, formative work was necessary to determine if modifications were needed for implementation in Chicago EDs. The results shown here confirmed many of the original assumptions but also suggested several important changes to the study design. Instead of trying to implement our discharge intervention using existing personnel, we created a new position (the ED coordinator) in order to effectively deliver the intervention. We modified the content and design of the ED discharge tool using our study results[27]. We added universal spacer provision and inhaler technique teaching to address the realities of the ED systems and families.

This process of engaging end-users in research design and implementation is central to the mission of PCORI[30], and has been employed successfully by others in our PCORI-funded asthma consortium[31]. Unique aspects of the CHICAGO Plan’s formative work include the use of strategies from the field of design which expand the traditional qualitative research framework to incorporate the influences of context and physical environment. For example, key informants reported how they perceived things but caregivers experienced these events differently and observations (huge piles of discharge papers in homes, lack of privacy in EDs, etc.) verified that good intentions frequently did not translate into the desired outputs.

Formative work is intended to generate hypothesis, not test them. Our small sample size is a limitation and the results cannot be applied to any other population or setting. Data that emerge from a specific social situation like a focus group or interview are highly contextualized and prone to social desirability[32], which is why our team used multiple sources of data to determine the final themes. We did not use a qualitative data analysis software and analyzed data using methodologies from the design industry[18–19,26], which vary somewhat from standard qualitative research methodologies[33–34].

Conclusion

The ED remains the place where many high-risk children with asthma seek care[5], making the development of effective ED-based asthma interventions critical. While implementation research necessitates the application of evidence-based theories and strategies to the intervention study design[36], the engagement of end-users in the design phase before launching the interventions is a critical step to optimize research feasibility and relevance. This aligns with the increasing emphasis on research approaches and interventions that are “patient-centered[37].” Our results show how complex the drivers of asthma care and care seeking behaviors are; the input from and engagement of all end-users, not just patients and families, is important for optimal study implementation. Finally, the field of medicine should recognize and accept the limitations of medical providers and clinical systems, and formally engage design and communication experts in planning, testing, and delivery of clinical care interventions.