Wonderings to research questions: Engaging patients in long COVID research prioritization within a learning health system

Abstract

Background

An integral component of research within a learning health system is patient engagement at all stages of the research process. While there are well‐defined best practices for engaging with patients on predetermined research questions, there is little specific methodology for engaging patients at the stage of research question formation and prioritization. Further, with an emerging disease such as Long COVID, population‐specific strategies for meaningful engagement have not been characterized.

Methods

The COVID‐19 Focused Virtual Patient Engagement Studio (CoVIP studio) was a virtual panel created to facilitate patient‐centered studies surrounding the effects of long‐term COVID (“Long COVID”) also known as post‐acute SARS‐CoV‐2 syndrome (PASC). A diverse group of panelists was recruited and trained in several different areas of knowledge, competencies, and abilities regarding research and Long COVID. A three‐step approach was developed that consisted of recording panelists' broad wonderings to generate patient‐specific research questions.

Results

The “wonderings” discussed in panelists' training sessions were analyzed to identify specific populations, interventions, comparators, outcomes, and timeframes (PICOT) elements, which were then used to create a survey to identify the elements of greatest importance to the panel. Based on the findings, 10 research questions were formulated using the PICOT format. The panelists then ranked the questions on perceived order of importance and distributed one million fictional grant dollars between the five chosen questions in the second survey. Through this stepwise prioritization process, the project team successfully translated panelists' research wonderings into investigable research questions.

Conclusion

This methodology has implications for the advancement of patient‐engaged prioritization both within the scope of Long COVID research and in research on other rare or emerging diseases.

1. INTRODUCTION

The systematic gathering and generating of data to drive change and improve care within Learning Health Systems (LHS) still face barriers to implementation to ensure continuous patient‐centered, evidence‐based care, particularly in low‐resource settings. ¹ , ² Although the LHS model advocates for patient‐centered care, there is limited evidence of health systems effectively and genuinely involving patients and community partners in the “learning” process. ³ To meaningfully advance the health and quality of care for patients, address health disparities, and truly provide “patient‐centered” care experiences, patients must be engaged in the research and innovation process as much as in the care delivery process. Patient engagement increases the research's relevance to patient needs, improves the quality of a project, and impacts healthcare policy and practice. ⁴ , ⁵ , ⁶ , ⁷ Additionally, by including patients and stakeholders in research prioritization, LHS can gain patient‐centric insights, influence patient‐reported outcomes and quality of life, facilitate patient‐centered care, and promote equity and inclusivity by incorporating diverse voices. ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ The experiences of patients with Long COVID, combined with the perspectives of relevant collaborators such as family members, caregivers, clinicians, researchers, and policymakers is an essential element in a learning health system. ¹¹ , ¹²

The Patient Engagement Studio (PES) at the University of South Carolina is a pivotal component within the Health Research Collaborative, a framework of Learning Health System, that unites Prisma Health System (the largest nonprofit health organization in South Carolina) with three academic partners (Clemson University, Furman University, and the University of South Carolina). ¹³ , ¹⁴ The PES is a dynamic hub where patients, caregivers, community groups, health system innovators, clinicians, and academic researchers converge to collaboratively produce impactful research and drive innovations that advance both health outcomes and research methodologies. ¹³ , ¹⁴ Much of the work done by the PES began as an effort to involve patients in the development of research and health system innovations and serves as the central locus for the innovative methodology employed in this study.

1.1. Long COVID

Long COVID‐19 Syndrome, known colloquially as “Long COVID,” consists of severe post‐acute sequelae that occur in a subset of patients following infection with Severe Acute Respiratory Syndrome Coronavirus 2 (SARS‐CoV‐2), the causative agent of COVID‐19. These sequelae may occur in the absence of severe acute infection or preexisting comorbidities. ¹⁵ Long COVID‐19 Syndrome is defined as the persistence of symptoms >12 weeks after acute infection ¹⁶ ; however, the constellation of symptoms that make up Long COVID are not clearly defined and may be highly variable between patients. The most commonly reported symptom manifestations include mild cognitive impairment (“brain fog”), fatigue, headache, sleep disturbance, dizziness, and dyspnea. ¹⁵ , ¹⁷ Long COVID is also known to have a significant impact on patient quality of life, with a reduced ability to provide self‐care, increased anxiety and depression, and disruption of activities of daily living commonly reported by those affected. ¹⁵ With an estimated 200 million suffering from Long COVID worldwide, along with the potential for countless new cases, as COVID infections continue to occur, Long COVID poses a significant threat of a public health emergency that will persist long after the COVID pandemic has ended. ¹⁵ , ¹⁸

While the full underlying pathology of Long COVID has not yet been fully elucidated, ¹⁹ millions of patients are searching for relief from their symptoms with few, if any standardized treatments available. ²⁰ Current medical treatment guidelines are primarily focused on the management of symptoms and require patients to interact with already overwhelmed healthcare systems. ²⁰ , ²¹ In the absence of evidence‐based treatment options or access to care, people experiencing Long COVID often rely on self‐prescribed or off‐label treatments, such as penicillin, vitamin C, and antiretrovirals. ²² , ²³ , ²⁴ Self‐prescribing presents risks for harmful drug‐to‐drug interactions, unsafe doses, and unregulated purchases from overseas. ²⁵ These challenges call attention to the need for patient‐focused research on Long COVID management practices, factors influencing uptake, benefits/harms, and costs.

1.2. Inclusion of patient voices and research prioritization

During the onset of the COVID‐19 pandemic, policymakers responded to the public health emergency rapidly using existing infrastructures which made it challenging to include individuals with lived experience such as patients, families, and frontline health and social care professionals in the solution. ²⁶ To fill this gap, patient and civil service advocacy groups, who typically lack a seat on expert committees, took the lead in providing information, advice, and support within their communities. ²⁶ These groups capitalized on online communication to reach larger groups and unite patient groups. ²⁶

To enhance the impact of research, early engagement of individuals with lived experience (1) helps in building a shared research vision, (2) allows for better prioritization by aligning the priorities of the stakeholders and the patients, (3) brings forward unanticipated research topics, and (4) ensures efficient use of resources as research targeted toward the questions important to patients and community. ⁵ , ²⁷ , ²⁸ Research prioritization can be utilized to ensure a project addresses questions most pertinent to the target population and available resources are used efficiently. While the inclusion of researchers and other project partners (such as funding agencies) in the process of research prioritization has long been the norm, the inclusion of patients is becoming more common. ²⁹ Patient‐focused prioritization of the research process can begin by engaging with collaborators to identify topics of interest or questions that need to be addressed by the project being conducted, followed by ranking these based on urgency and importance. ²⁹ Early‐stage patient involvement in the prioritization process guarantees the project priorities truly align with the concerns of the target population, as well as results in the discovery of novel research questions. ³⁰ While some researchers who are experiencing Long Covid symptoms created a research collaborative facilitating patient‐led research, ³¹ to date, we could find no literature focused specifically on patient‐centered prioritization of Long COVID research.

1.3. Purpose

The overarching project aimed to (1) build a diverse nationwide network of CoVIP Studio collaborators (panelists) including patients with the lived experiences of and clinical/researchers with expertise with Long COVID; (2) train the panelists to build their knowledge, competencies, abilities to be meaningful partners throughout the research process, and identify the panelists areas of research interest; and (3) implement the CoVIP studio activities by engaging the panelists in ongoing and new research opportunities focused on COVID‐19 and its long‐term effects, and evaluate the virtual engagement experience. Herein, we focus on one portion of this project, the research question prioritization which is grounded in the J Lind Alliance Framework. ³² Specifically, we aim to report how we worked with patients and collaborators to refine their interests and curiosities surrounding their health conditions through a stepwise process to produce realistic and investigable research questions. Therefore, the purpose of this paper is to describe this methodology used to prioritize research questions with a group of patients and other collaborators within the Long COVID community.

2. METHODS

2.1. CoVIP studio project

In 2021, the PES received a Eugene Washington Patient‐Centered Outcomes Research Institute (PCORI) Engagement Award to create a COVID‐19‐Focused Virtual Patient Engagement Studio (CoVIP Studio). The CoVIP Studio provided a virtual platform where researchers can develop, enhance, and disseminate patient‐centered studies on the long‐term effects of COVID‐19 and its disproportionate impacts on certain populations. While research question prioritization was only one portion of the larger CoVIP project, the elements of effective and meaningful engagement built into the context of the prioritization methodology were essential for the broader project.

2.2. Participant population and training

A diverse group of individuals, who have experience with Long COVID as a patient, caregivers, and/or clinicians, were recruited. Outreach was conducted through social media platforms, listservs, and word‐of‐mouth to recruit individuals from across the United States. Out of the 18 panelists who were recruited for the CoVIP Studio, most of the panelists were women 13 (72%), white 14 (78%), with a mean age of 46.39 years (sd 11.31), with an age range of 28 to 66 years, and highly educated with 72% (n = 13) having a bachelor's degree or higher. Table 1 provides a further description of the panelists' demographics. Figure 1 provides the geographical distribution of the locations in the United States where panelists lived.

TABLE 1.

Demographic information of 18 selected CoVIP studio panelists.

Demographic	All panelists N = 18 n (%)	PIO survey respondents N = 10 n (%)	Priorities survey respondents N = 10 n (%)
Gender
Woman	13 (72%)	9 (90%)	8 (80%)
Man	5 (28%)	1 (10%)	2 (20%)
Race/ethnicity
White, Non‐Hispanic	14 (78%)	9 (90%)	8 (80%)
Black or AA, Non‐Hispanic	2 (11%)	1 (10%)	1 (10%)
Bi‐racial	2 (11%)	0	1 (10%)
Educational attainment
Some college	4 (22%)	1 (10%)	2 (20%)
Associates degree	1 (6%)	0	0
Bach. or some Grad.	5 (28%)	5 (50%)	4 (40%)
Masters degree or more	8 (44%)	4 (40%)	4 (40%)
Age
18–29 years	1 (6%)	1 (10%)	1 (10%)
30–49 years	10 (56%)	5 (50%)	4 (40%)
50–64 years	5 (28%)	3 (30%)	4 (40%)
65+ years	2 (11%)	1 (10%)	1 (10%)
Geographic region
Northeast (Massachusetts, New York, Pennsylvania)	3 (17%)	0	1 (10%)
Mid‐Atlantic (Virginia, Maryland)	2 (11%)	2 (20%)	2 (20%)
South (Alabama, Florida, North Carolina, South Carolina)	3 (17%)	2 (20%)	2 (20%)
Midwest (Michigan, Ohio, Wisconsin)	4 (22%)	4 (40%)	2 (20%)
Southwest (Texas)	2 (11%)	1 (10%)	2 (20%)
West (California, Colorado, Washington)	4 (22%)	1 (10%)	1 (10$)

FIGURE 1.

Geographical location of CoVIP panelists, each darker gray state represents the location of at least one of the panelists.

Eight training sessions took place over the course of 5 months (Table 2) to build panelists' knowledge, competencies, and abilities. These training sessions prepared the panelists to act as meaningful partners throughout the PCOR/CER process and communicate directly with researchers regarding research opportunities focused on COVID‐19 and its long‐term effects. Furthermore, the training sought to build trust and provide team building through sharing panelists' personal experiences of Long COVID with other panelists and the PES staff.

TABLE 2.

Teambuilding and training sessions series.

Sessions	Homework assignments (asynchronous)	Team building (synchronous—30 min)	Learning topics (synchronous—30 min)
1		30‐s introductions Find your word	Intro to CoVIP/ground rules Technology overview (orient CoVIP panelists to various platforms used in the project) PCORI RF Workshop Overview—The Navigator (Orient CoVIP Panelists to our PCORI Online Workshop)
2	PCORI RF Workshop a PCORI Approach to PCOR + Engaging in Stakeholder‐Driven Research	Understanding what it means to be a COVID‐19 Long Hauler: Sharing my COVID‐19 story	Latest research on COVID (Guest speaker: two researchers currently working on Long COVID)
3	PCORI Module 1: Developing Research Questions (optional) PCORI Module 2: Designing the Research Study	The Big Picture—why did you join the group?	Reflections on PCORI RF Engagement: + Q&A
4	What is CER/PCOR? + 10‐step framework (recorded)	Think‐pair‐share—Magic Wand exercise (fixing something)	Wonderings to research questions: what are some broad (or specific) research areas that you like more research (related to COVID and Long COVID)?
5	Communicating with researchers and research group presentation framework and explanation (recorded)	Think‐pair‐share—Trusting your healthcare team and COVID‐19	Communication with Researchers Q&A—what are your concerns? Fears? Challenges?
6	PCORI Module 3: Planning Patient‐Centered Consent and Study Protocols (optional) PCORI Module 4: Sampling, Recruiting, and Retaining Study Participants	Practice research study review—Session I
7	PCORI Module 5: Understanding and Sharing Research Findings	Photovoice—Give us 3–5 pictures on how you can imagine yourself to be a Patient Scientist to inspire research questions to help address how COVID impacted your life or surroundings?	Research question creation and exploring emerging themes (based on PICOT)
8		Practice research study review—Session II

^a PCORI Research Fundamentals: https://www.pcori.org/engagement/research‐fundamentals.

2.3. Research prioritization process

This exploratory sequential mixed‐methods study utilized a three‐phase approach for research question prioritization (Figure 2). ³³ While the general phases were predetermined, the flexible nature of the process allowed for modifications to the procedure based on feedback from the panelists. This methodology refined the panelists' broad “research wonderings” to specific, investigable, and prioritized research questions.

FIGURE 2.

Graphic description of the flow of the research prioritization process.

2.4. Feasibility and distribution

Both process and summative evaluations assessed the feasibility and benefits of using a virtual studio to engage patients, clinicians, and researchers in nationwide, multi‐site research opportunities. Lessons learned, patient concerns, and outcome measures meaningful to patients were disseminated using several forms of media including videos, infographics, publications, and publicly available online sources (PCORI Website). The final evaluation of the project will be presented in a forthcoming manuscript.

3. RESULTS

The prioritization process consisted of three main steps (Figure 2) which refined broad research “wonderings” into investigable research questions.

3.1. Phase one

Wonderings are defined as questions people have about a health condition they are experiencing. For example, questions may include inquiries into why they got sick, how their condition might progress, and what treatments might work for someone like them. If wonderings included questions surrounding treatments that were harmful/or showed non‐significant effects (e.g., hydroxychloroquine) we would provide the patients with the supporting evidence and not include those wonderings in our list. While the main purpose of the training was to build trust among the panelists and Project Team during the initial phase of the project (February–June 2022), training also laid the groundwork for the prioritization process.

Our approach to eliciting these “wonderings” from patients was both organic and iterative, shaped by the spontaneous and natural interactions within the panel. As patients shared their experiences and posed questions about their conditions, or expressed curiosity about treatments they or others were trying, such as hyperbaric oxygen therapy, we recorded these instances.

The dynamic process unfolded both during live training sessions, where facilitators were present to document the discussions, and within our dedicated Slack group where patients continued their conversations asynchronously. Specifically, during one of the synchronous training sessions, project facilitators led discussions among the panel on their “wonderings” about their condition. We used the MentiMeter application to enhance engagement, collect information, and help facilitate our discussion with panelists.

To translate these “wonderings” into potential research questions, we used a multi‐step thematic analysis. Initially, project team members reviewed the aggregated wondering to identify common themes and patterns following the synchronous training sessions and discourse on Slack. These common themes were then mapped onto the PICOT framework to identify specific populations, interventions, comparators, outcomes, and timeframes that were of interest to the panelists. These elements of the PICOT format for clinical research questions were then used to create a survey, which was designed to identify PICOT elements of greatest importance to the panel.

3.2. Phase two

The PICOT survey was administered between June and July 2022 using Qualtrics survey software (Appendix S1) and completed by 10 of the panelists (Table 1). Upon the survey's closure, the project team analyzed the responses to extract patterns in the panelists' preferences for types of research questions, as detailed in Table 3. Additionally, we determined the top five populations, interventions, and outcomes that were of utmost importance to the panelists, as presented in Table 4. Based on these priority topics, members of the project team adapted 10 research questions using the Population, Intervention, Comparator, Outcomes, and Timeframe (PICOT) format. ³⁴ Here, we integrated the panelists' preferred topics and the nuances of their conditions into 10 meticulously formulated research questions. This entailed an iterative process where each element of the PICOT—populations, intervention, comparator, outcomes, and timeframe—was tailored to align with the lived experiences of Long COVID patients. The team was particularly attentive to ensuring that the questions were not only reflective of patient priorities but also adhered to the methodological rigor necessary for comparative effectiveness research.

TABLE 3.

Survey 1—Type of Research Question (Rank the types of research questions you are most interested in being investigated according to your order of preference; 1 being your favorite and 6 being your least favorite).

Type of research questions	Mean order	Median order	Mode rank order
Questions about how well a treatment works to improve outcomes for patients. (May include medications, lifestyle interventions, education initiatives, etc.)	1.8	1.5	1
Questions about how well an intervention (policy, medication, behavior, etc.) is at preventing disease or death	3.2	3.5	4
Questions about the ability of a test or procedure to determine who has a condition/disease and who does not have a condition or disease	3.6	4	5
Questions about the negative impact or harmful effect of an intervention or other exposure	4.1	5	5
Questions about the probable cause of a disease or the likelihood that a person will develop an illness	3.6	3.5	6
Questions about how patients' experiences and concerns	4.7	5	6

TABLE 4.

Survey 1—Pick up to 5 Population, Intervention, Outcome (PIO) that you are most interested in to be part of a research study.

Population	Count	Frequency
Adults with Long COVID	6	60%
People with Long COVID who experience brain fog/cognitive changes	6	60%
People who experienced mild COVID symptoms	5	50%
Females with Long COVID	4	40%
People with Long COVID who experience lasting physical changes	4	40%
People with Long COVID and other chronic health conditions	4	40%

Intervention	Count	Frequency
Hyperbaric chambers	7	70%
Cardiovascular/heart rehabilitation	5	50%
Education to improve patient–doctor relationships	5	50%
Neurological/brain rehabilitation	5	50%
Covid variants	4	40%
Education on Long COVID health effects	4	40%
Paxlovid	4	40%

Outcomes	Count	Frequency
Complications/progression of Long COVID	10	100%
Cognitive outcomes	6	60%
Access to care	4	40%
Quality of life	4	40%
Activities of daily living	3	30%
Developing Long COVID symptoms	3	30%
Mitochondrial dysfunction	3	30%
Physical activity levels	3	30%
Risk factors for contracting Long COVID	3	30%

The PICOT survey's role was pivotal, serving as a bridge between patient‐led inquiry and the structured demands of scientific research. The validity of this approach is underscored by its patient‐centric nature, ensuring that the research questions are deeply rooted in the lived experiences of the panelists. This methodological choice is supported by the fact that patient experiences are heterogeneous, and a one‐size‐fits‐all PICOT structure may not capture the nuances of every patient's journey. Through this approach, we can construct research questions that are both grounded in patient priorities and flexible enough to explore a range of clinical interventions, outcomes, and patient subgroups.

3.3. Phase three

Panelists were then emailed a second survey (Appendix S1) which asked them to select their top five research questions out of the 10 produced in Phase Two by the study team. Panelists were then instructed to rank their top five selections based on perceived order of importance. Finally, the panelists were asked to distribute one million fictional grant dollars between their five chosen questions. The second survey was initially administered electronically, but due to a low response rate, a member of the project team followed up by phone. Phone calls assessed any barriers that might have hindered survey completion and facilitated survey completion by engaging in discourse about question prioritization. Demographics of those who completed the second survey are provided in Table 1.

Within the survey and the analysis, a rank of one denoted the highest priority and a rank of five denoted the lowest priority. Additionally, higher dollar amounts given to a research question indicate a higher level of priority for the panelists. The results of the survey were first analyzed by assigning weights to a ranking number: a rank of one was awarded 100 points, a rank of two was awarded 80 points, a rank of three was awarded 60 points, a rank of four was awarded 40 points, and a rank of five was awarded 20 points. The number of panelists who ranked a specific PICOT question and where they ranked that question was listed. The assigned weight for each rank was multiplied by the number of people who ranked a specific PICOT question in each rank. The weighted scores were then added up for a total ranking score for each PICOT question.

For analysis of funding dollar allocation, the dollars that panelists assigned to a ranked question were totaled for each question. The total allocated dollar amount for each question was then multiplied by the number of patients who selected that question. This approach serves to amplify the priority level of each research question based on both its perceived importance based on the allocated dollars and the prevalence of its selection among the panelists. Based on this product, questions were rank‐ordered based on the total weighted dollars allocated. In the final step, the ranking from prioritization of the top five PICOT questions by panelists was added to the ranking of questions by total weighted dollar for an overall composite ranking of questions. Results of the second survey (Table 5) revealed the top three research questions identified by panelists to be:

1. In people with Long COVID who experience brain fog/cognitive changes, is Hyperbaric Oxygen therapy more effective than neurological/brain rehabilitation at improving cognitive outcomes and quality of life for patients, after 6 to 12 months of treatment?

2. Which complementary and integrative health approach is most effective for improving Long‐COVID symptoms, natural remedies, mind/body therapies, or a combination of both?

3. Are adults with multiple SARS‐CoV‐2 infections more likely to develop Long‐haul COVID compared to those who only experienced one SARS‐CoV‐2 infection?

TABLE 5.

Ranking of identified patient‐centered research questions.

Subject	Research question	Ranking score	Total weighted funds allocated ($1000)	Order from Rank	Order from Grant	Sum Order	Final Rank Order
A. Hyperbaric oxygen	In people with Long COVID who experience brain fog/cognitive changes, is hyperbaric oxygen therapy more effective than neurological/brain rehabilitation at improving cognitive outcomes and quality of life for patients, after 6–12 months of treatment?	520	$12 000	1	2	3	1
G. Natural remedies, mind/body therapies	Which complementary and integrative health approach is most effective for improving Long COVID symptoms, natural remedies (vitamins and herbal supplements), mind/body therapies (yoga, chiropractic, massage therapy, etc.), or a combination of natural remedies and mind/body therapies?	480	$7840	2	4	6	2
E. Multiple infections	Are adults with multiple SARS‐CoV‐2 infections more likely to develop Long‐haul COVID compared to those who only experienced one SARS‐CoV‐2 infection?	220	$12 560	7	1	8	3
B. Provider education	Do patients with Long‐term COVID symptoms see improvements in knowledge, beliefs, and attitudes and symptomology about Long ‐COVID after providers participate in education on ways to improve patient–provider relationships compared to education about Long‐haul COVID etiology?	460	$4000	3	6	9	4 (tie)
C. Patient education	Do patients with Long‐term COVID symptoms see improvements in knowledge, beliefs, and attitudes and symptomology about Long COVID after they participate in education on ways to improve patient–provider relationships compared to education about Long‐haul COVID etiology?	300	$8610	6	3	9	4 (tie)
F. Access to care	What are patients' ideas for improving access to care for those experiencing brain fog due to Long COVID?	340	$2400	5	7	12	6
H. Clinics and support groups	How does access to care (ie, Proximity to Long COVID care clinic) compared to participation in support groups impact condition prognosis for patients with Long Covid?	460	$260	4	8	12	7
D. Spiritual practice vs physical activity	In people who experienced mild COVID symptoms, how does either a daily spiritual practice compared to meeting the 150 min of physical activity a week guidelines influence the development of Long COVID symptoms and/or progression of Long COVID symptoms?	180	$7200	8	5	13	8
I. Pre‐menopause vs post‐menopause	How do symptoms of Long‐COVID vary between pre‐menopausal vs post‐menopausal patients?	40	$150	9	9	18	9
J. Menstrual cycle	Are patients who are infected with COVID during the follicular phase of their menstrual cycle (first day of period to ovulation) less likely to develop severe symptoms than patients infected during the luteal phase (second half of cycle)?	0	$0	10	10	20	10

4. DISCUSSION

The results of this prioritization project offer valuable insights into the process of engaging patients in the prioritization of research questions within the framework of a Learning Health System (LHS), particularly in the context of Long COVID. By combining qualitative interactive virtual live sessions, electronic surveys, and individual audio‐visual interviews with composite quantitative analyses of rank scoring and funding dollars allocations, we crafted a methodology that includes multiple data collection points and aligns with the principles of patient‐centered research and collaborative learning. Furthermore, our analysis allowed us to adequately weigh and prioritize the resultant research questions. This system incorporated three weighted variables to calculate a final overall ranking: the number of panelists who selected a specific research question as their top choice, the ranking assigned to each question by the panelists, and the allocation of fictional grant dollars to fund each respective research question. The consideration of both rank and assigned grant dollars balanced the overall importance of a particular question, as two patients may give a question the same ranking but differ greatly in the grant dollars they allocate to it. This nuanced approach to prioritization aligns closely with the principles of data‐driven decision‐making within LHS.

In the broader context of the Lind framework, ³² which underscores the integration of patient perspectives into the continuous learning cycle of LHS, our methodology showcases a pragmatic and effective way to achieve this integration. By actively involving patients and considering their insights in the research prioritization process, we not only enhance the relevance of the research but also promote the principles of equity and inclusivity within LHS. The top three research questions identified by the panelists offer a glimpse into their priorities and curiosities regarding Long COVID, providing valuable directions for future research. These questions span a spectrum of critical areas within Long COVID research, from treatment effectiveness and quality of life improvements to the evaluation of different health approaches and the potential impact of multiple SARS‐CoV‐2 infections on Long COVID development.

The timeline of the COVID‐19 pandemic has highlighted the urgency of collaborative research and patient engagement within the healthcare ecosystem. The medical, scientific, and government entities were able to rapidly mobilize resources to prevent and treat SARS‐CoV‐2 infection and acute COVID‐19, resulting in multiple widely available vaccines and therapeutic strategies to reduce disease severity only 1 year after the disease had been identified. ³⁵ , ³⁶ , ³⁷ , ³⁸ However, the response to the growing Long COVID epidemic has been much slower, due in part to a lack of clear information. Since Long COVID was first identified, understanding the disease progression and symptomatology has been an ongoing struggle. ¹⁶ , ³⁹ Our project's ability to adapt and innovate amidst challenges provides a means of prioritization of possible Long COVID research directions and reflects the resilience and adaptability of LHS. Long COVID‐related issues are vital to patients and the greater community and should be properly studied to more readily contribute to the growing knowledge of COVID‐19's prolonged symptoms and effects.

4.1. Lessons learned

Key takeaways from panelists reinforced previous literature focused on meaningful vs. tokenistic patient engagement. ⁸ , ⁹ , ⁴⁰ For example, panelists noted the purposeful acknowledgment of decreasing the power differential between the PES staff as researchers with “lots of letters after their names”; this was accomplished by the policy within the PES of referring to all within the sessions by first names. In addition, the panelists have highlighted other key factors that led to meaningful engagement in this project including the right of patients to be included in all steps of the research process, fostering a culture of respect among the patients and project team, and the preference for patients to choose the format for feedback.

When involving patients in research prioritization, it is critical to adapt to the needs of that specific patient population, as highlighted by the project's second survey. ⁴ The second survey initially received a low response rate, prompting the addition of a project team member to administer the survey. The language in the first survey was deliberately crafted to be easily understood by a broad audience. This approach aimed to solicit general feedback on the populations of interest, interventions, and outcomes. In contrast to the first survey, the second survey aimed to identify scientific, patient‐centered Comparative Effectiveness Research (CER) questions using the PICOT model. Given the nature of this stage, the survey intentionally included elements that are inherent to the PICOT framework, making the language more complex. The readability analyses of the second survey revealed a higher level of difficulty, equivalent to college graduate reading, reflecting the scientific nature of the questions compared to the grade 5 language in the first survey. The complexity in the language of the second survey was a limitation but was also driven by the need to formulate questions that align with the rigorous standards of scientific research, especially to be shared with PCORI researchers. The goal was to produce questions that could contribute meaningfully to the scientific discourse in the field of CER and PCOR.

To address this barrier, a team member emailed panelists and then conducted the survey in individual sessions with patients over the phone. Several panelists reported that when their wonderings were presented as formalized PICOT questions, the complexity made them difficult to comprehend. The addition of an administrator made it easier for the patients to complete the survey because they were able to engage in discourse and ask for clarification, which increased their understanding of the more complex presentation of the questions. In previous projects, this final prioritization of questions was done either in person or in a “live” virtual session so the panelists could ask questions if they did not understand the process. When prioritization of research questions cannot be accomplished in a “live” format, it could be important for research teams to consider ways to provide additional information and explanations for panelists who are not familiar with the scientific language used in research questions.

With the growing literature surrounding engaging with patients using remote/virtual methods, ⁴¹ , ⁴² , ⁴³ we also learned patient expert diversity is enhanced by the use of remote sessions. Offering meetings using video conferencing applications enabled participation by people who may be too ill or busy to travel, who may live far from any research university, or whose caregiving responsibilities or economic limitations would make travel a hardship. These people would be excluded from traditional research panels where colocation is a requirement. Patients generally do not have the institutional connections that members of the corporate and university communities do, and some are experiencing economic hardship caused by illness. This is true for COVID‐19 patients and could apply to those experiencing other illnesses as well.

Complexities in data collection also arose due to the panelist's experiences with symptoms of Long COVID infection. On occasion, symptoms limited the panelist's ability to attend training and complete surveys in a given time frame, as many individuals with Long COVID experience increased difficulty with cognitive tasks (“brain fog”). ¹⁷ , ⁴⁰ This study acknowledges the range of neurocognitive challenges faced by individuals with Long COVID and incorporates a flexible, patient‐centered approach to program design. This allows patients with varying degrees and types of cognitive deficits to contribute meaningfully, ensuring that the project remains grounded in the lived experience of those it aims to serve. Furthermore, accommodations must be made to ensure that the patient's health and quality of life are taken into account during all steps of the research process and a project plan should be created in conjunction with the patient. ¹⁷ , ⁴⁴ , ⁴⁵ During this project, two training sessions were conducted per week which provided flexibility for panelists. Phone calls were also used during the administration of the second survey to help with concentration and clarity which is often affected by Long COVID symptoms.

Finally, in the PES's previous research prioritization work with patients with diabetes, the research questions were primarily centered around different treatments/interventions rather than investigations into etiology and diagnosis. However, based upon the panelists' wonderings, and the complexity and recent identification of Long COVID, this led us to need to include questions about the types of research the panelists were interested in investigating in survey 2 (Table 3).

4.2. Limitations

Several limitations warrant consideration in the interpretation of our findings. First, using the PICOT model to translate the “research wonderings” brought forth by the panelists into more specific scientific research questions created challenges. While this step was essential to frame the research questions comprehensively, the results were at times difficult for patient experts to interpret due to the transition from lay terms to technical terms.

Moreover, the internal validity of the developed research questions remains a concern. While efforts were made to align the questions with the original intentions of the patients, discrepancies may still exist. Patient perspectives are inherently diverse and translating them into specific research queries can introduce potential misalignments. Specifically, we note the limitation of educational diversity in our panel and recognize that it may impact the generalizability of our findings. However, given the primary focus of our paper to explain and elaborate on the methodology used for research prioritization, intentionally placed less focus on the actual research questions generated. We aimed to provide a detailed account of the process, not to draw conclusions about the outcomes or results of the research questions generated. Furthermore, as the science around long Covid progresses, a new panel of patients and stakeholders may come up with different wonderings and research priorities. Another internal concern was that the sample size for both surveys was relatively small. This limited the statistical power of our prioritization exercise and subsequently the generalizability of our findings.

The external validity of the project is a concern mainly due to the condition itself. Long COVID symptoms, severity, timing, and disease impact on quality of life are not consistent across all patients. ¹⁵ , ¹⁷ , ²⁰ As a result, there is a chance that symptoms given high priority by one patient may not be experienced at all by another patient. To address the challenges of diverse patient engagement highlighted by our project, we propose a set of scalable strategies for LHS. These include leveraging digital platforms for broader reach, utilizing community partnerships to tap into diverse patient networks, and applying targeted outreach to underrepresented patient groups. Recognizing the resource constraints typical of many LHSs, we suggest a tiered approach to patient engagement that prioritizes key activities based on available resources, with the flexibility to expand as additional resources become available. This pragmatic approach allows for the gradual integration of diverse patient engagement into research prioritization, making it more feasible for LHS with varying levels of resources.

In this project, a principal objective was to ensure geographic diversity, aiming to capture a wide array of experiences from patients and caregivers across different regions of the country. We believe that regional representation is crucial, especially given the variability in healthcare systems and resources, cultural perspectives, and policy environments that can influence the experience of Long COVID patients. To this end, our recruitment strategy was successful in engaging participants from broad geographic locations, which allowed us to gather a diverse set of wondering and perspectives on a national scale. However, we acknowledge that geographic diversity alone does not encompass all aspects of heterogeneity. Our participant group was less varied regarding other important demographics. This lack of broader diversity may influence the generalizability of the findings.

Although we believe that our patient‐centered methodology can serve as a template for similar endeavors in other healthcare conditions and settings, the nuances of different health conditions and LHS structures may introduce variations in the prioritization process. Furthermore, while we acknowledge potential concerns regarding the practicality and methodological application of the research questions generated by patients, it is important to note that these questions provide a foundational perspective for the researcher. The questions can be further refined and adapted in collaboration with patients to ensure both feasibility and alignment with patient priorities.

4.3. Future directions

Looking ahead, these findings pave the way for future research initiatives within the LHS framework. Exploring the effectiveness of specific interventions for Long COVID, evaluating different health approaches, and understanding the implications of multiple SARS‐CoV‐2 infections are all critical areas that warrant further investigation. Additionally, the methodologies employed in this study can serve as a blueprint for engaging patients in research prioritization in various healthcare domains, further strengthening the role of LHS in shaping the future of healthcare research and practice. In this first execution of this methodology, a small sample size was needed as both the condition itself and the virtual delivery were new to the practice of patient‐led prioritization for patients and caregivers with Long COVID. While the current project contained elements of a Delphi Panel, ⁴⁶ other patient‐led prioritization projects have employed formal Delphi Panels composed of patients, researchers, and other stakeholders in the learning health system with larger numbers of participants. ¹¹ The next steps may include querying a larger panel ¹⁷ to increase statistical power and generalizability.

5. CONCLUSION

With so many unanswered questions related to this debilitating, life‐altering condition, there is an urgent need for prioritization of research related to Long COVID. This methodology is an effective way to expand the field of COVID research in a manner that centers the needs of individuals experiencing its effects. Elements of trust and team building, and flexibility for procedural changes responsive to patient needs were integral to the process. This project's strength lies in the successful execution of a process that allows lay individuals to communicate their priorities to researchers in a way that is specific enough to inform meaningful scientific inquiry.

While it is critical to design projects that consider the needs of specific patient populations when involving patients in research, this project provides insight into considerations for inclusion that can be adapted for other conditions. ⁴ In fact, post‐viral fatigue was first observed during the 1918 Flu ¹⁷ and has been observed for other viral infections including SARS, as well as a number of health conditions including Parkinson's, stroke, and traumatic brain injury. ⁴⁷ Regardless of the etiology, disease‐related exhaustion can negatively impact patient quality of life, and patients can provide insight that is not currently available in these areas. ⁴⁷ Thus, the results of this work will have impacts beyond patients experiencing Long COVID. The COVID‐19 pandemic has changed research on a broad scale, and this novel methodology for meaningful engagement of Long COVID patients in research prioritization provides a framework for the inclusion of patients in research prioritization related to other rare or emerging conditions.

FUNDING INFORMATION

This project was funded through a Patient‐Centered Outcomes Research Institute (PCORI) Eugene Washington PCORI Engagement Award (EASC‐COVID‐00293).

CONFLICT OF INTEREST STATEMENT

Ann Blair Kennedy and Nabil Natafgi have been funded jointly by PCORI for one additional Engagement Award from 2020 to 2021. Ann Blair Kennedy, Nabil Natafgi, Katherine Parris, and Evan Katzman are members of AcademyHealth. No other authors report any conflicts of interest.

Supporting information

Appendix S1. Supporting information.

Kennedy AB, Mitcham A, Parris K, et al. Wonderings to research questions: Engaging patients in long COVID research prioritization within a learning health system. Learn Health Sys. 2024;8(Suppl. 1):e10410. doi: 10.1002/lrh2.10410