Evaluation and lessons learned regarding Public Involvement: a panel advising on an Alzheimer’s disease and related dementia cohort study

Background

Public involvement (PI) in research is increasingly requested, yet systematic evaluation is limited. This study evaluates whether a panel advising an Alzheimer’s Disease and Related Dementia’s (ADRD) Cohort Study fulfills PI quality criteria and leads to perceived Cohort Study improvement and describe lessons learned.

Methods

A panel of 49 members of the public advised the Cohort Study through questionnaires and meetings. Panelists and Cohort Study researchers completed questionnaires based on PI quality criteria as formulated in the Patient Engagement Quality Guidance Framework, specifically developed to capture insights on the quality of PI. Findings were supplemented by reflections of the panel organizers and three panelists. Additionally, we outlined the panel’s aim and design to provide a practical example of PI in ADRD research.

Results

Of 49 panelists, 43 consented for use of their data for evaluation purposes (72% female (31/43), mean age 62 years, 82% (35/43) in good health). Response rates to the evaluation questionnaire were 37% for panelists (16/43) and 67% for researchers (4/6). Most panelists reported that the panel’s purpose (95%; 15/16) and communication were (very) clear (94%; 15/16), certainly adequate resources to provide input (75%; 12/16), and sufficient transparency (87%; 14/16). All felt respected and found it easy to participate in online panel questionnaires. However, 56% (9/16) found attending in-person meetings challenging due to time or travel. Researchers reported positive impact on Cohort Study quality, e.g., improved participant information letter, yet also highlighted the required time and effort to involve a panel and suggested to improve management of panelists’ expectations. Among the lessons learned are the value of offering multiple ways to participate, and the need for more communication between all parties involved.

Conclusions

Engaging a PI panel was perceived to improve an ADRD Cohort Study and fulfilled most quality criteria from the Patient Engagement Quality Guidance Framework. However, results must be interpreted with caution due to potential consent bias.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40900-026-00844-1.

Plain language summary

Dementia is a growing health concern worldwide. Today, around 50 million people are living with the condition, and this number is expected to rise to 152 million by 2050. At the moment, there are only limited treatment options. In this study, researchers worked with 49 members of the public, known as a panel, to support a long-term dementia research project. The panel gave advice through online surveys and in-person meetings. Afterwards both the researchers from the long-term dementia research project and panel members reflected on how the process worked. Most panel members felt that the purpose of the panel was clear, that communication was handled well, and that they were treated with respect. Panel members said it was easy to take part in online meetings and surveys, although a few said attending in-person meetings was more difficult. Researchers thought that the panel was very useful, for example, by helping to make written information for study participants easier to understand. However, they also pointed out that involving the public takes time and effort, and that it is important to set realistic expectations. The lessons learned suggest that offering different ways for people to take part, along with clear and open communication towards researchers and panel members, makes these panels more successful. Overall, the panel improved the dementia research study, though the conclusions should be interpreted carefully, as not every panel member gave feedback.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40900-026-00844-1.

Introduction

The involvement of members of the public in health research, often referred to as public involvement (PI), is gaining momentum. PI is typically defined as “doing research with or by the public, rather than to, about, or for them” [1]. Arguments for PI can be normative or instrumental/substantive [2–4]. Normative arguments entail that people have the right to be involved in research that affects them [5, 6], or that they pay for via taxes and labor [7]. It conveys values ranging from human rights, (epistemic in-) justice, to fairness and democracy [4]. The instrumental or substantive argument focuses on the consequences of participation, e.g., enhancing the design, fostering trust and acceptance of research in society, and ensuring that research focuses on delivering tangible benefits to the community [8]. People are seen as experts in their experiences and thus provide valuable knowledge [4, 7, 9], which may complement the focus of professionals in the set-up and execution of studies [10]. PI thus seems to be the right thing to do when it comes to health research, and is also gaining in recognition and support.

The increasing recognition and support for PI also applies to the field of Alzheimer’s Disease and Related Dementias (ADRD). Dementia is a major public health problem [11]. The number of people with dementia worldwide is predicted to increase from 50 to 152 million by 2050 due to ageing populations and the existing lack of widely accessible effective treatment options [12]. Recent shifts in the ADRD field further increase the relevance of PI. Biomarkers, such as tau or amyloid-β, can identify ADRD related pathology years before symptoms appear, enabling early diagnosis, prediction, and potentially prevention [13–15]. This shift in focus towards earlier disease stages requires including cognitively healthy participants as well as those with clinical signs and symptoms in cohort studies and registries, to study participants throughout the entire disease trajectory. Since dementia onset may take years, long-term participation is necessary, yet difficult, which may for instance be enhanced via PI [16]. Consequently, dementia registries and cohort studies, such as the Irish national dementia registry, increasingly engage in PI during set-up and execution [17].

Multiple reviews describe the impact, strengths and limitations, and lessons learned of PI (e.g., references [8, 18, 19]). However, many PI activities and/or initiatives lack methodological or formal impact evaluation [18]. This also pertains to the field of dementia, although ample PI activities are taking place [20–22]. Where impact is evaluated, it tends to focus on easy to measure aspects, e.g. the number of people involved, resulting in ignoring the impact of the PI on the course of research itself and the values and norms that govern research, which are less quantifiable [3, 23]. Furthermore, the experiences of people living with dementia, care partners, the general public, and health professionals are insufficiently evaluated and documented [21]. Comprehensive evaluation of PI initiatives is important to increase the value of PI for health research.

This study aims to evaluate a panel with members of the public, who were asked to advise on an ADRD Cohort Study. The evaluation examines whether the panel’s input is perceived to improve the ADRD Cohort Study and whether the panel aligns with important PI principles. To address this aim, the study comprises three evaluation components, using a mixed-methods design. First, an evaluation questionnaire among panelists. Second, an evaluation questionnaire among Cohort Study researchers. Both questionnaires are systematically developed using the Patient Engagement Quality Guidance Framework (PEQG), an assessment tool specifically developed to capture insights on the quality and lessons learned of PI [24]. Third, the panel organizers and three panelists who co-designed the current study and co-author this article reflected on the panel and questionnaire results to formulate the most important lessons learned for future PI initiatives in ADRD research. Additionally, we provide a detailed description of the panel’s aim, formation, and working procedures in the method section. This third component provides a practical example and contextualization of the lessons learned and demonstrates how PI can be operationalized in ADRD research.

Methods

ABOARD project, ABOARD Cohort Study & ABOARD Cohort Panel

In 2022, a panel was set up to advice on various aspects of the ABOARD Cohort Study, a Dutch nationwide ongoing data collection within the ABOARD project. Box 1 provides an overview of the context.

Box 1.

Overview of the context of the panel comprising the ABOARD project, ABOARD Cohort Study, and the ABOARD Cohort Panel

ABOARD project

ABOARD stands for A Personalized Medicine Approach for Alzheimer’s Disease and aims for a future with personalized, patient-orchestrated diagnosis, prediction and prevention of AD [24]. This large-scale research project is carried out by a public-private consortium in the Netherlands, consisting of 30 + partners of relevant backgrounds, i.e., healthcare, societal organizations (including patient organizations), industry, and academia and education.

ABOARD Cohort Study

Part of ABOARD is the ABOARD Cohort Study: a Dutch national data collection infrastructure aiming to (1) study the Alzheimer’s disease trajectory using both self-reported outcomes and medical data, (2) link to available registries, and (3) serve as a central platform to initiate future additional study [25]. Anyone at risk of AD (defined as age above 45) can participate. More information on the ABOARD Cohort Study can be found in the articles by Dreves, et al. and De Boer, et al. [25, 26]. The ABOARD Cohort Study aims to further develop into a registry for ADRD.

ABOARD Cohort Panel

The ABOARD Cohort Panel (hereafter referred to as ‘panel’) was initiated to provide advice on the set-up and execution of the Cohort Study. The panel is meant to facilitate alignment between the Cohort Study and the ABOARD Cohort Study participants’ expressed wishes and needs, with the goal to improve the data collection of and research done with(in) the Cohort Study. Requirements were being able to speak and write Dutch, at least 18 years of age, and willing to think about how to improve the Cohort Study. Panelists may be eligible and willing to participate in the Cohort Study, but do not necessarily have to be participants. Not all panelists are thus participants of the Cohort Study. This set-up was consciously designed to allow for critical voices to become part of the panel and to be inclusive to people who, for instance, do not meet the inclusion criteria due to their age. The panel comprises cognitively unimpaired people and those with subjective or mild cognitive complaints, family members and/or caregivers of someone with ADRD, and/or have affinity with ADRD and the Cohort Study because of other reasons (see result section Table 1). An initial call for participation in the panel in spring 2022 resulted in 351 interested persons. Hereafter, an online, interactive webinar was conducted with information on ABOARD, the Cohort Study, and the panel. This resulted in 63 panelists signing up in July 2022. Over time, 14 panelists dropped out due to personal reasons or time limitations. Characteristics of the panelists are displayed in Table 1.

Panel set-up, organization, and activities

We recruited panelists in April 2022 via newsletters and social media of the Brain Foundation Netherlands (a health fund and patient organization) and distributing flyers in supermarkets, community centers, and public libraries. The organization of the panel is conducted by ABOARD project members from three organizations: the Brain Foundation Netherlands, Health-RI, and Amsterdam University Medical Centers (hereafter referred to as Amsterdam UMC) (see Fig. 1). These ‘panel organizers’ meet monthly to discuss and plan panel activities. Cohort Study researchers are affiliated to the Amsterdam UMC as well, but are working mostly at another department, therefore allowing for a clear division of roles, except for one researcher also being part of the panel organization to enhance alignment between Cohort Study and panel.

Fig. 1.

Coordinated efforts in panel organization

We investigated the panel’s preferences regarding frequency, form, and modality of providing input during an initial panel questionnaire. This resulted in a panel set-up of a maximum of four meetings (in-person or online) and four online questionnaires per year. In the initial questionnaire, panelists were also invited to indicate the topics they wished to discuss. Taking into account these suggested topics, along with the needs of the Cohort Study, panel organizers and Cohort Study researchers jointly selected the subjects to be addressed by the panel. Questionnaires are distributed to all panelists and may be completed online at a time and place of their choosing. The meetings are structured to facilitate in-depth discussion of specific topics in smaller groups, determined by participants’ interests and the availability of those able to attend at the scheduled date and time. For the meetings, attending panelists receive a reimbursement of 15 euros and compensation for travel expenses. Figure 2 provides an overview of all questionnaires and meetings up to the end of 2024.

Fig. 2.

Panel activity overview

Evaluation activities

The current evaluation comprises three constituent elements (see Fig. 3): (1) an evaluation questionnaire among panelists, (2) an evaluation questionnaire among Cohort Study researchers, and (3) an overall reflection of the panel organizers and three panelists, leading to a set of lessons learned.

Fig. 3.

Panel and study flow

We sought to conduct the evaluation in partnership with panelists by inviting all panelists to become a co-researcher and co-author for the current study. Three panelists responded positively (MS, LK, MaB). The process began with discussions about expectations regarding how, to what extent, and when the panelists wanted to be involved. Regular meetings were held with the team including the panelists (MiB, MS, LK, MaB, TS, LV, TR) to discuss all aspects of the current study, for instance comprising the aim, methodology, the selection of the assessment tool, the interpretation of results, the reflection session to extract lessons learned, the selection of quotes for inclusion in the manuscript, and the provision of feedback on the manuscript.

Evaluation questionnaire for panelists and evaluation questionnaire for Cohort Study researchers

The team (MiB, MS, LK, MaB, TS, LV, TR) developed two questionnaires: one for panelists (24 questions) and one for Cohort Study researchers (19 questions). The questionnaire started with sociodemographic questions (see additional file 1). The core of the questionnaire was based on the Patient Engagement Quality Guidance Framework (PEQG), which is a structured approach to plan and evaluate meaningful and effective patient involvement and can be used across the entire research stages [24]. PEQG was originally co-developed with key stakeholders, including patients, in the field of medicine development, but can be applied to broader health-related research involving the public as well. The framework centers around seven main quality criteria: (1) shared purpose, (2) respect and accessibility, (3) representativeness of stakeholders, (4) roles and responsibilities, (5) capacity and capability for engagement, (6) transparency in communication and documentation, and (7) continuity and sustainability [24]. The quality criteria were operationalized in the questionnaires by asking participants to score the fulfillment of that criterion on a 4-point Likert scale and using an open ended question to ask how this criterion could be improved. All PEQG criteria were captured in the questionnaires, except ‘representativeness of stakeholders’, since it was already known from demographic information that the panel is not representative of the Cohort Study target group (e.g. regarding country of birth and gender). Additionally, the questionnaire for researchers of the ABOARD Cohort Study did not contain questions regarding ‘capacity and capability for engagement’ since the Cohort Study researchers do not have a view on the capability of engagement of panelists. The panelist questionnaire did contain questions regarding ‘capacity and capability for engagement’. In addition to the quality criteria, we added a section comprising results and perceived impact, with questions such as perceived positive/negative impact, surprising insights, lessons learned, and feedback on the way of working and organization. This section was incorporated to evaluate the study’s central aim: whether the involvement of this PI panel is perceived to enhance the ADRD Cohort Study and whether the panel aligns with key PI principles. This framing enables the extraction of lessons learned for other researchers. Furthermore, the items comprising ‘results and perceived impact’ were considered important components of an evaluation of a PI initiative according to the team, although these items were not part of the PEQG tool. Both questionnaires can be found in the additional files 1 and 2. The questionnaire for panelists was written in accessible (B1) language and pilot tested in November 2024 with a person who was eligible to be a panelist. During pilot testing, we asked feedback on clarity and comprehensiveness and made changes accordingly. The questionnaires and two reminders were sent by email over a period of three weeks at the end of 2024. The questionnaire was administered online by using the generic survey distribution software Survalyzer.

Reflection of the panel organizers and three panelists

In September 2023, a preliminary internal evaluation took place, during which panel organizers discussed what worked well and what could be improved regarding panel organization. In January 2025, a second evaluation meeting took place with the panel organizers and three panelists who are co-researchers in the current study. This second evaluation incorporated the results from the evaluation questionnaire; reflection on these findings by the panel organizers and the three panelists led to a set of lessons learned intended to inform future PI-initiatives. The lessons learned are described at the end of the results section.

Ethics and reporting guidelines

At the start of the panel, panelist were asked consent for the use of their data in a scientific paper, and 43 out of the 49 panelist consented. Panelist could still participate in the panel if they did not sign consent. However, for the current publication, this means that we can only report characteristics of 43 out of the 49 of panelists, and the evaluation questionnaire was only sent to the panelists who provided consent.

The Amsterdam UMC’s Medical Ethics Committee, located at AMC, declared that this study (W22_135 #22.178) is not subject to the Dutch Medical Research Involving Human Subjects Act (WMO). This study complies with the General Data Protection Regulation and the ethical standards of the Helsinki Declaration [27]. Questionnaire data were pseudonymized and stored in a restricted data access location at the Amsterdam UMC, location AMC. For reporting, we used the Checklist for Reporting of Survey Studies (CROSS) guidelines [28] and the GRIPP2 Checklist (see additional file 3).

Data analysis

Descriptive analyses were conducted using IBM SPSS version 28. Answers to open questions were analyzed in Excel by both KK and TR independently to reduce subjectivity using deductive coding (see Figs. 4 and 5), with a coding scheme inspired by the PEQG [24]. This was followed by thematic content analysis to identify patterns of meaning (themes) in the text [29]. Coding was done independently after which KK and TR (both trained in qualitative research) discussed results with each other, followed by discussions with MiB, MS, MaB, LK, and TS until consensus was reached.

Fig. 4.

Coding scheme for responses of panelists

Fig. 5.

Coding scheme for responses of researchers

Results

Evaluation questionnaire panelists

Out of the 43 panelists who provided informed consent and received the evaluation questionnaire, 16 responded (37% response rate). These panelists were on average 61 years, the majority was female, highly educated, they lived in several regions of the Netherlands, and have family or friends with dementia. This is in line with characteristics from the entire panel (see Table 1).

Table 1.

Panel & study sample characteristics

	Characteristics of entire panel (n = 43) at the panel kick-off in 2022	Characteristics of panelists (n = 16) who completed the evaluation questionnaire in 2024
Mean age	61.9 ± 11.3	61.2 ± 14.9
Gender	72.1% (31/43) female	68.8% (11/16) female
Educational attainment [30]	Secondary school (VSO, VMBO, HAVO, VWO, ULO, MULO, HBS): 12% (5/43) Secondary vocational education (MBO): 16% (7/43) Higher vocational education (HBO): 28% (12/43) University (WO): 40% (17/43) Other: 5% (2/43)	Secondary school (VSO, VMBO, HAVO, VWO, ULO, MULO, HBS): 6% (1/16) Secondary vocational education (MBO): 13% (2/16) Higher vocational education (HBO): 25% (4/16) University (WO): 57% (9/16)
Employment situation (multiple choice)	Fulltime work: 14% (6/43) Parttime work: 35% (15/43) Retired: 40% (17/43) Voluntary work: 26% (11/43) Looking for work: 5% (2/43) Unable to work: 5% (2/43)	Fulltime work: 19% (3/16) Parttime work: 25% (4/16) Retired: 38% (6/16) Voluntary work: 6% (1/16) Looking for work: 6% (1/16) Unable to work: 6%(1/16)
Self-perceived health	Excellent: 5% (2/43) Very good: 35% (15/43) Good: 42% (18/43) Fair: 19% (8/43)	Excellent: 6% (1/16) Very good: 31% (5/16) Good: 56% (9/16) Fair: 6% (1/16)
Country of birth	Netherlands: 95% (41/43) Belgium: 2% (1/43) Surinam: 2% (1/43)	Netherlands: 100% (16/16)
Link with dementia (multiple answers possible)	Friends/family with dementia: 65% (28/43) Personal interest in the topic: 67% (29/43) Dementia is a societal problem: 54% (23/43) Work-related link: 19% (8/43) I have (an increased risk of) dementia myself: 2% (1/43) Other: 12% (5/43)	Friends/family with dementia: 76% (12/16) Personal interest in the topic: 50% (8/16) Dementia is a societal problem: 25% (4/16) Work-related link: 19% (3/16)

Evaluation questionnaire panelists: clarity

Most panelists indicated that that the goal of the panel was very clear (71%; 11/16), whereas 24% (4/16) found it clear and 6 (1/16) found it a bit unclear. When asking them to elaborate upon their answer using open-ended questions, however, some panelists indicated to have forgotten the exact goal, or referred to the goal of the Cohort Study or the ABOARD project instead of the panel, indicating limited understanding. When asked about how the goal of the panel could be clarified more, participants indicated briefly repeating the goal of the panel in every email to the panel. Some indicated that they could also look up the goal of the panel themselves.

Panelists generally found the communication to be very clear (81%; 13/16), or clear 13% (2/16) whereas 6% (1/16) found it a bit unclear. E-mails, questionnaires, and other communication was indicated as clear by many panelists. Most participants said that no changes should be made regarding communication. Others suggested using more visuals and writing e-mails more to the point.

There is often a lot of time between calls on the panel and sometimes I could not participate because of other activities. In practice, I then forget a bit [about the exact goal and purpose of the panel]. (Panelist 11)

The emails I receive are clear and give good explanations of what will be done with the results. (Panelist 6)

Evaluation questionnaire panelists: approach

The transparency towards panelists (defined in the questionnaire as ‘open, honest and clear communication, this ensures that you know what goes on in the panel and what happens to it’) was considered very good (81%; 13/16), just enough (6%; 1/16), or just not enough (12.5%; 2/16). When asked to elaborate, the majority indicated that they found the communication open, appreciated the feedback, and developed trust in the panel. Others said that panel organizers could provide more overall transparency, for instance, via an archive on previous activities and results. One participant referred to a specific e-mail regarding the panel’s input on the social media campaign for the Cohort Study and said that they missed an update of what is done with the results hereafter.

81% of panelists (13/16) indicated that they felt fully respected, whereas 19% (3/16) said that they felt a little bit respected, both during online questionnaires and meetings. In participants’ responses to the question what can be done to make them feel more respected, panelists encouraged the continuation of current practices. According to panelists, respect could be achieved via receiving results and updates about ADRD research, and not by means of financial compensation, since they are intrinsically motivated, not financially motivated.

It [transparency] is fine, I think, but I find it harder to judge. I don’t know what you could be more transparent about, because I don’t always know if there is information that could be shared. I don’t really have insight into the timeline and progression of the process besides the input I provide myself and the emails I see coming along. (Panelist 9)

The contact is always nice, and I am not forced into anything. I am also taken seriously. My input is too. (Panelist 9)

I am treated with care and although I do not use them, I get invitations to meetings where I can speak to project staff. This appeals to me and induces trust. (Panelist 2)

Evaluation questionnaire panelists: ability to participate

All panelists indicated that it is easy to participate in online questionnaires. The responses to online meetings and in-person meetings showed mixed results (online meetings: 50% (8/16) easy, in-person meetings: 44% (7/16) easy). Participants added that time constraints, work obligations, location, and, for some, financial constraints, limited digital skills, limited energy, and older age could make it difficult to participate in these meetings. Suggestions for improvement comprised continuing to offer online and in-person meetings and questionnaires. When panelists are not able to join a meeting, they indicated that they would like to receive a summary of the meeting, and the possibility to provide a written contribution afterwards.

It is an issue of prioritizing. Again, what do I add to the panel? Am I relevant? I am working full-time, plus volunteering, then you just have to make choices about where to spend your time. Filling in a questionnaire is easy then, an online panel meeting is sometimes difficult to schedule. (Panelist 5)

Participants indicated that they certainly have enough resources and knowledge to be a panelist (75%; 12/16) or just enough resources and knowledge (25%; 4/16). Panelists doubted their knowledge more than their resources yet indicated that they do not need any additional training. Nevertheless, a few suggestions for additional training comprised public speaking and videoconferencing. Panelists wanted to continue to receive results and updates, as well as invitations to events, such as online webinars.

Evaluation questionnaire ABOARD Cohort Study researchers

Four out of 6 invited Cohort Study researchers (67% response rate) completed the evaluation questionnaire. Participating researchers (consisting of a project lead, geriatrician & senior researcher, research assistant and PhD researcher) were all female, age range 25–49, two were involved since the beginning, one involved after 0.5 years and one after 2 years from the start.

Evaluation questionnaire researchers: organization

The researchers indicated that the goal of the panel is clear and that the goals of the Cohort Study and panel are related to each other. The division of tasks between the Cohort Study researchers and the panel organizers was indicated to be a bit clear (50%; 2/4) and a bit unclear (50%; 2/4). When asked about what could have been done to increase the connection between the panel organizers with the researchers, suggestions for improvement comprised increased mutual visibility, having a shared folder for sharing documents, and providing information about the panel organization to new researchers. One researcher also joined the panel organization team, which seemed to work well for increasing connection and flow between the teams.

Although I keep needing clear material/one liners/A4 sheets of what the cohort [panel] stands for and what it has delivered. Which is then clearly shareable with the outside world. (Researcher 2)

I think that I was an important bridge between the two teams, where at the beginning I was not quite sure what exactly my role (in both teams) was going to be. I think, especially in the beginning, it might have been more efficient to have joint meetings more regularly. (Researcher 1)

I am also not really familiar with how you (the panel team) plan and/or compile the content of your questionnaires/panel sessions. But a biennial brainstorming/streamlining/evaluation session would certainly be useful. (Researcher 3)

Evaluation questionnaire researchers: approach

Three researchers said that the panel is fully transparent (75%), whereas one indicated that the panel is a bit transparent (25%). To exemplify, researchers mentioned that there was open communication during panel sessions, that panelists always receive a summary, and that they as researchers felt respected.

I think this is done very neatly, with both panelists and researchers being informed each time of activities (questionnaires/meetings), but also afterwards of their main outcomes (Researcher 1)

Good exchange of views and pleasant collaboration. Feels like one team. (Researcher 4)

Evaluation questionnaire researchers: impact

Even though most researchers hope that the panel is long-lasting and will continue after the overarching ABOARD project ends, the feasibility hereof was doubted due to a lack of funding and termination of contracts of panel organizers.

When asked about examples of perceived positive impact due to the panel, researchers mentioned multiple improvements made to the Cohort Study, due to input from the panel, such as rewriting the informed consent letter and improvements made to an online cognitive test.

At the top of the list are the improvements to the consent letter and Cohort Study statement. I still often cite this as a lesson for other studies, where we simply saw that this resulted in many more participants answering, ‘yes’ to the separate consent questions. For me personally, the think-aloud session of the digital memory test was very valuable. We were also able to make very concrete adjustments as a result of this session (often simple but effective such as adding more whitespace to the test instructions). (Researcher 1)

Developing documents together is very important for participant participation and adherence. (Researcher 4)

When asked for examples of any negative impact or results, researchers reported no negative results. Still, one researcher did stress that being involved in a Cohort Study that is advised by a panel takes time and effort, and can be a challenge for researchers (25%). She also highlighted the panel’s relevance/impact should be communicated better or more often to the outside world.

No idea, except that it takes time and challenges me as a researcher to clearly articulate what we think (but of course that is really only positive). (Researcher 4)

As for unexpected results, researchers included the increase in the proportion of Cohort Study participants consenting to data linkage after adaptation of participant information and informed consent based on panel feedback (50%; 2/4), as well as general tips and tricks that were learned through working with a panel (25%; 1/4). Lessons that researchers derived from working with the panel comprised the importance of transparency, clear communication, and the value of asking input from a variety of people.

The main lesson for me is that it is always valuable to ask for input from different people. As a researcher, you often tend to think that it all ‘goes well’ but we have seen that this way of working really leads to very concrete areas for improvement, successfully! (Researcher 1)

What researchers viewed as elements that worked well and that they would do again, is the mix of several types of panel activities, such as questionnaires and meetings (50%; 2/4). Others mentioned the opportunity for researchers to pick topics to discuss with the panel (50%; 2/4). One researcher had a suggestion for improvement (25%; 1/4): to better manage expectations of panelists, for instance, regarding how many panelist were expected at an in-person meeting to prevent disappointment among panelists that expected to meet a lot of other panelists.

Smart set-up, that we have a combination of surveys and focus groups, and that people can get involved more or less intensively. (Researcher 2)

Sometimes, maybe do a bit more expectation management. For instance, you saw that, especially at the in-person meetings, people sometimes expected a higher number of panelists or also had a bit more of a social objective themselves. We could have easily avoided this by saying how many panelist we expected. (Researcher 1)

Lessons learned formulated by panel organizers and three panelists

In September 2023, panel organizers conducted a preliminary internal evaluation in their team, and in January 2025, a second evaluation meeting took place among panel organizers and three panelists. These activities led to identification of lessons learned: several elements that seem to work well and directions for improvement see Box 2).

Box 2.

Organizational lessons learned & recommendations - from internal evaluation

What works well

• Offer multiple forms of participation: The panel’s design, including a mix of questionnaires and meetings (in-person/online), works well and enhances accessibility, as panelists come from all over the country. • Take sufficient time for setting-up: A couple of months were taken to discuss and decide on the exact setup, which were necessary to reach consensus among all involved organizations. Clarifying the panel’s goals and scope within the ABOARD project also took time and multiple conversations. • Strive for impact: Measuring the actual improvements made based on the panel’s input is difficult. However, all involved organizations are satisfied with the panel’s input and how this leads to improvements, even in the small panel meetings. An example of improvement was the adaptation of participant information and informed consent. The initial version of the text was complex, and after close review of the panel with specific suggestions regarding what sentences to change and how, the text was simplified. Another example is an online cognitive test, which was tested by the panel in a think-aloud session, leading to specific suggestions on where and how to improve the online cognitive test.

What could be improved

• Enhance diversity: If we look at the demographics of panelists critically, we lack diversity. The panel is not representative of the Dutch general population regarding educational attainment, digital literacy, cultural background, and lack involvement of people who experience cognitive complaints themselves. This may be partly due to the way of recruitment and organization of the panel, which were mainly conducted online (see discussion). • Facilitate panel understanding: The structure of the broader ABOARD project, Cohort Study, and panel was unclear to some panelists. Panel organizers mentioned some panelists did not know the exact differences, or whether they were only a panelist or Cohort Study participant as well. However, by asking specific and clear questions during panel activities, valuable input was harvested, despite panelists not always fully grasping the overall structure of the consortium. • Communicate transparently and carefully: The collaboration between panel organizers (AMC, Brain Foundation Netherlands, and Health-RI) works well, generating oversight that focuses on whether panel input is actually followed-up on. However, it can lead to challenges regarding transparency or communication, as the coordinator may not be fully updated on all developments within the Cohort Study or by accident may not fully update all involved parties themselves. The latter, for instance, occurred once when feedback of a panelist (a suggestion for different answer categories in a questionnaire) was not taken into account. The feedback was discussed in a meeting with panel organizers and panelists, but had to be followed up by someone who was not present in the meeting. This person was only passively notified of the change in answer categories (via a comment in a word document), leading to the wrong version being used. This could have been prevented with clearer communication. When discussing this situation with the panelist, the panelist mentioned that “such experiences can undermine the intrinsic motivation of panelists. Intrinsic motivation among those affected by dementia is a fragile and precious factor, and the loss of a panel participant is a loss to the study”. Therefore, transparency, ensuring extra checks, and sufficient communication with all organizations, is highly important when working so closely with panelists, researchers, and panel organizers. In 2022, one Cohort Study researcher joined the panel organizing team, which enhanced collaboration and quick communication between the teams.

Discussion

This study aimed to investigate whether Public Involvement (PI) in the form of a panel comprising members of the public leads to perceived improvement of an Alzheimer’s Disease and Related Dementias (ADRD) Cohort Study, while fulfilling important PI quality criteria as mentioned in the Patient Engagement Quality Guidance Framework (PEQG). We outlined the panel’s set-up and design to provide a practical example of how PI can be operationalized in ADRD research and evaluated the panel systematically through questionnaires administered to both panelists and researchers. The findings demonstrate that engaging a panel with members from the public leads to perceived improvement of an ADRD Cohort Study, while fulfilling important PI quality criteria: shared purpose, respect and accessibility, capacity and capability for engagement, transparency, and continuity and sustainability; yet not representativeness.

Panelists and researchers were not asked to score the representativeness of the panel. We made this decision mainly since it was already known from demographic information that the panel is not representative of the Cohort Study target group; more diversity in gender and ethnicity should be strived for. It is likely that the panel is also not representative regarding literacy or language levels, since we expect the panel to be relatively high-literate while older people in the Netherlands are more likely to have lower literacy levels [31]. Additionally, very few panelists reported having cognitive complaints or being at risk for it and a relatively high proportion (20%) of the panelists have a work-related link to dementia and might therefore be more knowledgeable of dementia and research. Due to time and budget constraints, no targeted recruitment was undertaken to improve the panel’s representativeness. Panel organizers acknowledged that the quality of input could be enhanced by diversifying the panel. In our reflection, we therefore identified ‘enhancing diversity’ as the first factor to work on in order to improve the panel. Ideally, the voices of individuals with a more diverse cultural or ethnic background, low (health/digital) literacy, lower educational attainment and/or with cognitive complaints would be included, as the Cohort Study aims to meet their needs and preferences as well.

An important lesson learned is that having questionnaires as well as online and in-person meetings is positive according to both panelists and researchers. This multi-modal approach allows panelist to choose when and how to provide input, enhancing accessibility. Flexibility on how to engage is identified as important by previous research as well [32–34], especially for maintaining engagement [21]. Despite offering multiple engagement options, the number of active panelists declined over time from 63 at the start to 49 panelists currently. Difficulties with ongoing commitment is a known barrier [22, 35]. Nevertheless, both panel organizers and Cohort Study researchers did not consider this decline problematic, as the quality and volume of input remained strong and valuable according to panel organizers and Cohort Study researchers. For a sustainable panel, considering limited funding in the future and termination of contracts of the current involved panel organizers, changes regarding the panel’s organization are needed, potentially resulting in limiting the size of the panel or the number of activities that take place.

Other lessons learned centered around organizational aspects, such as taking sufficient time to set-up the panel to discuss the objective(s), organization/roles, procedures and requirements, and arrange necessary resources (in time, money and staff). This not only limits the likelihood of running into (preventable) problems throughout the execution, but also may limit the chance of tokenism, where the panel would only have a symbolic role but no impact. A related lesson learned is connected to the governance structure for the current panel. During the evaluation, some researchers noted lack of clarity regarding the responsibility and role division between the Cohort Study researchers and the panel organizers. Individuals working for three different organizations - not the researchers - are responsible for organizing the panel. Among the panel organizers, coordination is assigned to the organization that has no other task or stake in the Cohort Study except for coordinating the panel. According to the panel organizers, this triangular approach (panelists – researchers – organizers) has the benefit that researchers are less able to cherry pick where to ask input from the panel, and what to do with the feedback. However, panelists expressed a need for more information and results on the Cohort Study, which only Cohort Study researchers are able to provide, resulting in a dependency position for the panel organizers and panelists. Communication is crucial in PI [21], and it may be argued that with more parties involved in the governance, communication may become even more important yet also more complex. In our panel, we learned that we should focus even more on continuous and clear communication and alignment to ensure clarity regarding roles, responsibilities and tasks, including the communication hereof to panelists. For instance, we noticed over the years that people sometimes forgot what the panel was, what the Cohort Study was, and who was involved, and that the ABOARD project, the Cohort Study and the panel were sometimes mixed up. We tried to clarify this as much as possible by repeating the goals of the panel and Cohort Study shortly in every meeting or questionnaire, however, this distinction remained a challenge. In order to maintain a high level of commitment from the members of the panel, it is essential to ensure that communication and feedback on the impact of their input is provided in a careful and transparent manner. This may also demonstrate to panelists that their input is valued and that they are being taken seriously.

This study has several strengths. First, previous research indicated that impact of PI is often reported anecdotally [18]. Burton, et al. suggests that future research should explicitly assess the (dis)advantages of PI from the viewpoints of all relevant stakeholders, as well as directly soliciting the experiences of PI advocates [21]. To our knowledge, our study is one of the firsts to assess PI systematically by gathering insights from both researchers and panelists themselves in the context of dementia research. Second, previous evaluations of PI initiatives have largely focused on easily quantifiable metrics while neglecting less measurable factors [3, 23]. In contrast, our study used a framework that considers a diverse range of pre-defined quality criteria for meaningful involvement, including more qualitative aspects such as respect and transparency. A third strength of our study is the active involvement of three panelists in this study and the reporting thereof. One of the authoring panelists indicated co-writing the manuscript as one of the positive experiences that stem from the panel, in addition to the regular panel activities. She emphasizes the importance of having positive experiences as a panelist, since this is an incentive to continue contributing as a volunteer, to continue giving time and attention to the various opportunities for participation. Other examples of positive experiences (in addition to the regular panel activities) that she mentioned are the recruitment of panelists in other activities within the ABOARD project, including being a member of the Cohort Study Data Access Committee, participating in presentations to share experiences as a panelist, and contributing to this study. It is important to create sufficient opportunities for those positive experiences, and we highly recommend involving panelists in multiple activities.

This study also has several limitations. The panelist response rate of 37% (16/43) is low. Combined with the observed decline in panelist engagement over time and the assumption that less satisfied participants are possibly more likely to leave or become less active, this might suggest a bias in the responses to the questionnaire towards more positive feedback. It is likely some more critical, unspoken reflections were not captured. To better capture dissatisfaction or negative impact, future PI activities could incorporate an optional, short exit evaluation for panelist who choose to discontinue. In this panel, we did not ask a reason from people who withdrew, which may have limited our understanding of negative experiences. Another possible contributor to the potential positive skew in responses is the nature of the evaluation methodology itself, using online questionnaires. We conducted an additional in-depth reflection with the panel organizers and three panelists who volunteered to be actively involved as co-researchers in the current study; involving more panelists might have been useful to avoid a blind spot being left out and would have provided more room for critical or in-depth reflections regarding topics not covered in the questionnaire. Furthermore, the use of the PEQG tool, which is primarily designed for self-assessment [24], may not have effectively prompted consideration of complex issues such as unmet expectations of panelists, power imbalances, or positionality. In hindsight, we could have more explicitly invited negative or critical feedback on certain topics, such as unintentional negative impact of the panel, instead of using open-ended questions. Previous research by Muntinga, et al. [36], also add that using an intersectional perspective (taking into account e.g. intersections of age, gender, and professional status) during analysis may help to uncover power dynamics and other ethical issues that occurred along the way.

Future research is thus recommended to further develop evaluation tools and methodologies that can capture the ethical complexities, intersectionality, and power dynamics inherent in public involvement processes, ideally suitable for panelists to use instead of just professionals. Such tools would better support critical reflection and contribute to more balanced and meaningful evaluations of PI practices. A second recommendation for future research is to continue conducting systematic evaluations on PI-initiatives to allow for a broad overview of evaluation insights, minimizing the chance of blind spots during evaluations, and enhancing overall quality of PI initiatives when following-up with concrete actions based on the outcome of evaluations. From a broader perspective, engaging in systematic evaluations and publishing the results hereof may guide the set-up of future PI-initiatives and may showcase the value and benefits of PI when meaningfully supported [37, 38]. Such support extends beyond logistical resources like time, funding, planning, and staffing; it also encompasses institutional backing, opportunities for increasing participant knowledge and attitudes, access to suitable evaluation tools, opportunities for skill-building, and appropriate attention to critical ethical considerations [38, 39].

Conclusion

Panelists and researchers indicated that our set-up of a panel in the context of an ADRD-related Cohort Study fulfilled important PI quality criteria, including transparency, respect, and accessibility. Several ways in which the panel positively impacted the Cohort Study were identified by Cohort Study researchers. Based on these findings, we conclude that the panel has successfully contributed to enhancing the Cohort Study. However, our sample is biased and results should be interpreted with caution. Lessons learned comprise elements that work well: offering multiple forms of participation, taking sufficient time for setting-up, striving for impact; and elements that could be improved: enhancing diversity, facilitating panel understanding regarding the project structure and the goal of the panel, and communicating transparently and carefully with all parties involved. Future research is recommended to conduct systematic evaluations on other PI-initiatives and report on the results hereof as well as further develop evaluation tools and methods so that these can better capture the complex reality of PI in health research.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

ABOARD

A Personalized Medicine Approach for Alzheimer’s Disease

ADRD

Alzheimer’s Disease and Related Dementia’s

AUMC

Amsterdam University Medical Centre

PI

Public Involvement

PEQG

Patient Engagement Quality Guidance

Biographies

Miriam Beusink

is PhD student at the Netherlands Cancer Institute and ELSI Program Officer at Health-RI. As a sociologist, she focuses on ethical, legal and societal issues in responsible use of health data in research. Involving the citizen and patients‘ perspective is important to her, which is why she is involved with many citizen engagement activities.

Tanja de Rijke

is a health scientist and PhD student on the topic of person-centered dementia care and inclusive (digital) tools in memory clinics and public health.

Tieneke Schaaij-Visser, PhD

is Partnership Manager at Lygature with a background in biomedical sciences and a PhD in oncology. She has over a decade of research experience and has since specialized in public-private partnership management, data infrastructure and the ethical, legal, and societal implications of research, with an emphasis on patient and public involvement. Tieneke contributed to the initiation and coordination of the ABOARD Cohort panel.

Marja Berkhout

contributes voluntarily to ABOARD as a member of the ABOARD Cohort panel. Her mother was diagnosed with Alzheimer’s disease, and Marja is the only caregiver. Marja is interested in the risk factors for AD, the method of diagnosis, multimorbidity, prevention, and the complexity for the caregiver as a ‘case manager’. Additionally, she is member of the Data Access Committee of the Cohort Study. Previously, she was also involved in research into ‘Returning to work after cancer’ and published on this topic.

Lidwien Kroon

contributes voluntarily to ABOARD as a member of the ABOARD Cohort panel. This is important to her, because she has experienced from her work in psychogeriatrics how important it is to know at an early stage when dementia is present. This way, you can properly connect with the person with dementia and can provide good guidance to them and their environment. Additionally, she is member of the Data Access Committee of the Cohort Study.

Marlon Smeitink

contributes voluntarily to ABOARD as a member of the ABOARD Cohort panel and member of the ABOARD End User Committee. This is important to her, because she believes that experts by experience (informal caregivers, patients, etc.) should be included into research from the earliest stage possible. Their input is needed and contributes to the implementation process of the results from the start of a project. Marlon has a mother with early-onset Alzheimer’s and a grandmother with Alzheimer’s.

Sophie van der Landen

is a neuropsychologist and PhD candidate at the Alzheimer Center Amsterdam. Her research focuses on the use of digital tools to enhance diagnostic accuracy and care in dementia. With a background in clinical neuropsychology, she aims to bridge the gap between technological innovation and clinical practice to support timely, personalized dementia care.

Kyra Kaijser

holds a bachelor’s degree in Psychobiology and a master’s in Biomedical Sciences with a specialization in Neuroscience. She has experience in academic research and medical communication, including work at the Alzheimer Center Amsterdam. Her interests focus on neurodegenerative diseases, public involvement, and promoting inclusive research practices. Kyra is committed to bridging the gap between science and society, with a strong emphasis on generating meaningful social impact.

Dirk-Jan Saaltink

is Program Manager Science at the Brain Foundation Netherlands. He has a background in biology with broad interest in the brain, behavior, science and biology.

Els van der Rhee

is Program Manager Patient Involvement at the Brain Foundation Netherlands and coordinator of a advisory council with people with lived experience.

Susanne Rebers

is an ELSI researcher and advisor at the Netherlands Cancer Institute. Topics of interest include consent for secondary use, patient information, patient engagement and other ELSI topics. Furthermore, she is member of the board at COREON (www.coreon.org) and works at Health-RI as program manager ELSI.

Casper de Boer

is a senior researcher at the Alzheimer Centre Amsterdam. He holds a PhD in clinical neurosciences. His primary research interest is remote data capture in Alzheimer’s disease. In this context, Casper has been the local clinical lead of two pan-European academic-industry consortium studies. Currently, he is the national coordinator of the Dutch ABOARD Cohort Study initiative for real-world data collection in Alzheimer’s disease. In this role, he is involved in the scientific advisory group of the InRAD initiative.

Hanneke Rhodius-Meester

is a clinical geriatrician. She works both at the Alzheimer Center Amsterdam, and at the Memory Clinic at Ullevål, Oslo University Hospital, combining patient care and research. Her research focus is on digital tools in daily clinical practice, aiming to improve diagnosis and care.

Wiesje van der Flier

is full professor and scientific director of Alzheimer center Amsterdam at Amsterdam UMC, the Netherlands. In addition, she is clinical epidemiologist. Van der Fliers main research areas are looking for the origin of AD, diagnosis & prognosis, and intervention & prevention. She leads the Amsterdam Dementia Cohort Study and ABOARD. Together with colleague Philip Scheltens, she has written the book ‘Het Alzheimermysterie’.

Leonie (N.C.) Visser, PhD

currently works as an assistant professor at UMC Utrecht and Amsterdam UMC in the Netherlands. With a background in Medical Psychology, she focusses on investigating and stimulating active patient (and public) involvement in healthcare and biomedical research, often by improving and supporting medical communication. She has been involved in many projects conducted in the context of Alzheimer’s disease and dementia, yet has broadened her scope to other clinical domains.

Author contributions

MiB: conceptualization; formal analysis; investigation; methodology; project administration; visualization; writing – original draft. TR: conceptualization; data curation; formal analysis; investigation; methodology; project administration; visualization; writing – original draft. TS: conceptualization; methodology; writing – review & editing. MaB: conceptualization; methodology; writing – original draft; writing – review & editing. LK: conceptualization; methodology; writing – review & editing. MS: conceptualization; methodology; writing – review & editing. SL: writing – review & editing. KK: data curation; formal analysis; investigation; project administration; writing – review & editing. DS: conceptualization; methodology; writing – review & editing. ER: data curation; writing – review & editing. SR: conceptualization; methodology; supervision; writing – review & editing. CB: writing – review & editing. HR: writing – review & editing. WF: funding acquisition; writing – review & editing. LV: conceptualization; methodology; supervision; funding acquisition; writing – review & editing.

Funding

WF and LNCV are recipients of ABOARD, which is a public–private partnership receiving funding from ZonMW (#73305095007) and Health∼Holland, Topsector Life Sciences & Health (PPP-allowance; #LSHM20106). More than thirty partners participate in ABOARD. ABOARD also receives funding from Edwin Bouw Fonds and Gieskes-Strijbisfonds. www.aboard-project.nl. The salary of TR is paid by ABOARD. HR is recipient of the Memorabel Dementia Fellowship 2021 (ZonMw projectnumber 10510022110004). The salary of MB is paid by the Antoni van Leeuwenhoek hospital, an institutional grant of the Dutch Cancer Society and of the Dutch Ministry of Health, Welfare and Sport (grant number not applicable). The sponsors were not involved in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the manuscript for publication.

Data availability

The datasets generated and/or analyzed during the current study are not publicly available due to potential identifiability. Data are in controlled data access storage at the Amsterdam UMC, location AMC.

Declarations

Ethical approval and consent to participate

The Medical Ethics Committee of the Amsterdam UMC, location AMC, declared that this study (W22_135 #22.178) is not subject to the Dutch Medical Research Involving Human Subjects Act (WMO). This study complies with ethical standards of the Helsinki Declaration. Participants signed a written informed consent form prior to the study.

Consent for publication

Not applicable.

AI statement

During the preparation of this manuscript, the authors used DeepL Write, Quillbot, and Perplexity to assist with grammar checking and language improvement purposes only. We also employed Napkin.ai for the creation of images. The authors reviewed and edited all AI-generated grammar and language suggestions and take full responsibility for the final version of the manuscript.

Competing interests

HR performs contract research for Combinostics, all funding is paid to her institution. WF performs contract research for Biogen. Research programs of WF have been funded by ZonMW, NWO, EU-FP7, EU-JPND, Alzheimer Nederland, CardioVascular Onderzoek Nederland, Health~Holland, Topsector Life Sciences & Health, stichting Dioraphte, Gieskes-Strijbis fonds, stichting Equilibrio, Pasman stichting, stichting Alzheimer & NeuroPsychiatry Foundation, Philips, Biogen MA Inc, Novartis-NL, Life-MI, AVID, Roche BV, Fujifilm, Combinostics. WF has performed contract research for Biogen MA Inc, and Boehringer Ingelheim. WF has been an invited speaker at Boehringer Ingelheim, Biogen MA Inc, Danone, Eisai, WebMD Neurology (Medscape), Springer Healthcare. WF is consultant to Oxford Health Policy Forum CIC, Roche, and Biogen MA Inc. WF participated in advisory boards of Biogen MA Inc and Roche. All funding is paid to her institution. WF was associate editor of Alzheimer, Research & Therapy in 2020/2021. WF is associate editor at Brain Research. Research (projects) by LV has been funded by ZonMW, Health~Holland, Topsector Life Sciences & Health, Alzheimer Nederland, EISAI, and Amsterdam Public Health research institute. The other authors (MB, TR, TS, MB, LK, MS, SL, KK, DS, ER, SR, CB) declare no competing interests.