Development of ICF-based patient-reported outcome and experience measures to study social participation among people with chronic diseases: a mixed-methods protocol

Abstract

Introduction

Living with a chronic disease impacts many aspects of life, including the ability to participate in activities that enable interactions with others in society, that is, social participation (SP). Despite efforts to monitor the quality of care and life of chronically ill people in Belgium, no disease-specific patient-reported measures (PRMs) have been used. These tools are essential to understand SP and to develop evidence-based recommendations to support its improvement. This protocol presents the phases for the disease-specific development of patient-reported outcome and experience measures to assess SP and its potential determinants among people living in Belgium with cancer, cystic fibrosis, diabetes, HIV or a neuromuscular disease.

Methods and analysis

This protocol applies the PROMIS Instrument Development and Validation Scientific Standards and COnsensus-based Standards for the selection of health Measurement INstruments to develop PRMs in a disease-specific manner to quantify the components of the International Classification of Functioning, Disability and Health (ICF). A mixed-method approach is used to create broad initial item pools based on patient (focus groups) and literature perspectives which are compared within ICF-standardised language by applying the refined ICF linking rules. An item set is first created based on this cross-matching exercise and then validated by multidisciplinary expert panels. Cognitive assessment and pilot testing are followed by the dissemination of the survey to a representative sample in Belgium. Advanced psychometric testing (classical test theory and item response theory) is applied to inform an item reduction strategy for the final measures and to develop scales for the ICF components.

Ethics and dissemination

Ethical approval was granted by the Ethics Committee of the Ghent University Hospital on 20 February 2023 to organise the patient focus groups (ONZ-2022-0470). Ethical approval for dissemination of the PRMs and psychometric testing will be sought at the Ghent University Hospital Ethics Committee at the start of Phase 6. Results will be disseminated through peer-reviewed journals and professional conferences.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• The resulting measurements are based on the International Classification of Functioning, Disability and Health (ICF), enhancing their comparability both nationally and internationally across different types of health information.

• Patient and expert involvement is central to the development of the measures, ensuring relevance and adequate content and face validity.

• Recognised guidelines and frameworks are integrated to overcome common challenges in patient-reported measures development, including limited standardisation, lack of transparency and lack of patient involvement early in the development.

• The study phases enable an evaluation of the suitability of the ICF model to assess social participation and determinants and of the necessity of having disease- and/or Belgium-specific measures.

• The number of focus groups and participants for item generation and cognitive assessment is limited.

Introduction

The proportion of people living with a chronic disease in Belgium increased by 17% between 2001 and 2018.¹ In Europe, chronic diseases affect one-third of the working-age population.² They are the leading cause of mortality and morbidity³ and are responsible for many functional limitations⁴ that impact social activities such as obtaining and maintaining employment,5,8 education,⁹ social leisure or engaging in the community.¹⁰ Maintaining these activities has shown a positive impact on quality of life.^{5 11} The involvement in social activities that allow interactions with others in the society or community is most often used to define the concept of social participation (SP).12,14 With an emphasis on community life and public spaces, SP thus broadly refers to the involvement in community-based activities and interpersonal interactions within the context of, for example, (un)paid work or recreational activities.¹³

It is crucial to measure and monitor SP based on reliable data to inform policymakers and (healthcare) professionals who have to organise and provide adequate support to people living with chronic diseases. However, this is challenging due to the scarcity of population-level data that cover the social aspects of people’s quality of life. In recognition of SP as a vital aspect of quality of life and health,¹⁵ the Belgian Minister of health launched an Inter-federal Plan for Integrated Care to bridge well-being and healthcare services.¹⁶ In preparation for this plan, a stakeholders’ consultation was performed which stressed the need for population-level data that monitors the health and well-being of ill people.¹⁶

To measure the impact of health conditions, one can use the International Classification of Functioning, Disability and Health (ICF).¹⁷ The ICF biopsychosocial model provides a standardised language and systematic coding scheme for describing and classifying functioning and disability in a comprehensive way. The ICF framework allows for the selection of appropriate ICF categories (accompanied by alphanumeric codes) to assess four components: body functions and structures, activities, participation and environmental factors¹⁷ (figure 1). These are conceptualised as dynamically interacting with each other and with the ICF component of personal factors, for which no ICF codes exist.¹⁷ The use of the ICF offers several advantages. First, the ICF has been widely recognised and used for its accuracy in describing and measuring disability^{18 19} and for enhancing the comparability of health information.18,20 Its application in policy development and social policy is widespread,¹⁹ as is the use of the ICF component of participation to assess SP.^{13 14} Furthermore, the multidimensionality of the ICF model enables research into SP and its relationship with other ICF components. Importantly, the ICF model has been criticised for its emphasis on medical and physical functioning, while insufficiently addressing psychological functioning and global health status.^{18 21 22}

Figure 1. ICF model and its components. ICF, International Classification of Functioning, Disability and Health.

The assessment of SP can be explored in multiple ways, such as through the collection and analysis of registry-based information on, for example, labour market participation. However, when measuring SP and functioning, patient-reported outcome and experience measures (PROMs and PREMs) are considered highly informative and essential²³ and have been gaining popularity.^{24 25} PROMs are questionnaires that enable patients to directly report outcomes regarding their health status or (health-related) quality of life, as well as perceived levels of impairment or disability.²⁴ PREMs, on the other hand, collect information on patients’ experiences or satisfaction with processes of care or support.^{24 26} Presently, in the context of social participation, these experiences may also relate to support from, for example, the social security system or within the work environment. Patient-reported measures (PRMs), encompassing both PROMs and PREMs, can be either generic or disease-specific.^{24 27 28} While impairments of body functions and structures or activities are often assessed in a disease-specific manner, (social) participation is most commonly measured generically through the frequency of participating in social activities (eg, volunteer activities, sport/social club activities and paid work).29,33 Despite the ICF model’s usefulness in classifying environmental factors, it falls short in capturing patients’ qualitative experiences and satisfaction, which are essential for understanding their perspectives on the impact of these factors. This conceptual mismatch represents a significant limitation for the development of PREMs that warrant consideration.

Over the past few decades, the use of PRMs has significantly grown, together with the recognition that their effectiveness hinges on meticulous development, relevance to patients and thorough validation in the target population.^{25 34} A 2021 review on generic and condition-specific PROMs²⁵ identified recent trends among measures to address the limited standardisation and increasing heterogeneity that characterises the evolving field of PROMs.²⁷ These trends include, among others, the use of the Instrument Development and Validation Scientific Standards³⁵ and item banks developed by the Patient-Reported Outcomes Measurement Information System (PROMIS).^{25 35} PROMIS standards describe development phases that involve patients, experts and a literature review, ensuring content and face validity.³⁵ PROMIS standards on validation describe the application of advanced psychometric testing (item response theory (IRT)) that enables Computerized Adaptive Testing (CAT),^{25 36} a sophisticated method of delivering assessments.³⁷ This method applies item selection algorithms based on IRT-calibrated item banks only to present items that offer the most accurate estimates for the individual under assessment.³⁷ In recent years, CAT has emerged as a powerful mode of delivery for PRMs²⁵ that enables more flexible measures,³⁸ validity across groups³⁹ and more timely and potentially clinically meaningful results.2140 40,42

Another trend relates to the use of the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) publications^{34 43 44} to assess the methodological quality or validity of existing PROMs.²⁵ Although PROMIS and COSMIN standards were designed explicitly for PROMs, they have previously been used to develop and evaluate PREMs as well.^{28 45} This suggests that following these standards results in PRMs (consisting of both PRO and PRE items) of sound methodological quality.

This protocol describes study phases performed within the framework of the SPADIS project (Measuring Social Participation Among People Living With Chronic Diseases——Webpage) by the Belgian Scientific Institute of Public Health (Sciensano) that aims to explore ways of measuring and monitoring the social participation of people living with chronic conditions in Belgium and its determinants. For this project, PRMs will be disseminated which comprise a generic module as well as disease-specific modules.

The present protocol describes the development and validation of the disease-specific modules of the PRMs to measure social participation and its potential determinants among people living in Belgium with one of five chronic diseases for which health surveillance is organised by Sciensano: cancer, cystic fibrosis, diabetes, HIV and neuromuscular diseases. A multiphase approach is applied using the above-mentioned frameworks (PROMIS Standards,³⁵ COSMIN rating scales⁴³) to address common issues and challenges that are associated with the validity and reliability of PRMs. The refined ICF linking rules²⁰ are applied throughout to ensure compliance with the ICF model, while a thematic inductive analysis⁴⁶ approach enables the identification and integration of items on SP or its potential determinants that go beyond the ICF model’s emphasis on physical functioning²¹ and addresses the model’s inability to classify patients’ experiences and satisfaction. The widespread application of the ICF model and ambitions from Belgian social security institutions to standardise disability assessments based on the ICF model primarily informed our decision for the ICF model. This approach ensures that our findings are policy-relevant while the twofold coding strategy facilitates a critical evaluation of the model’s capacity to comprehensively capture social participation and patient experiences.

The generic module that will accompany the developed disease-specific modules during their first dissemination will provide patient responses to, among others, validated generic item sets on (social) participation and potential determinants, enabling the identification of topics that require disease- and/or Belgium-specific assessments versus topics for which generic items or item sets suffice based on final psychometric testing.

Methods and analysis

Patient and public involvement

The inclusion of patients and experts is key in the development of disease-specific PRMs.^{35 43} In disease-specific focus groups, patients are asked to identify the most relevant and prominent aspects of functioning and disability, including social participation (concept elicitation). Expert panels will strengthen content and confirm face validity. Patient input further informs the design and format of the PRMs and surveys (cognitive interviews and pilot test). In the present protocol, we build on previously established PRMs of high content and face validity. By mainly adopting items from such PRMs, the described study phases require a smaller amount of patients to reach sufficient content validity. The involvement of patients up until the pilot testing in this protocol is covered by ethical approval from the ethical committee of Ghent University Hospital. A second approval will be requested in advance of the dissemination of the developed PRMs or electronic surveys. The patient responses collected in this dissemination will be used to psychometrically validate the PRMs.

Phases of the PRM development and validation

An overview of the methodological steps and study phases is provided in figure 2. Phases 1 and 2 describe the collection of results from the literature and patient perspectives to ensure content and face validity of a broad initial item (ie, questions and related scales) pool. To secure alignment with the ICF model and its components, the refined ICF linking rules²⁰ are applied to categorise both patient and literature perspectives within the ICF-standardised language (Phases 1 and 2). This facilitates the cross-matching of the two perspectives to deduct an initial set (Phase 3). Subsequent phases concern expert validation (Phase 4), cognitive assessment and pilot testing (Phase 5) and dissemination for psychometric testing in a representative sample (Phase 6).

Figure 2. An overview of the methodological steps for the development of disease-specific patient-reported measures to measure social participation and its determinants, in alignment with the ICF model. ICF, International Classification of Functioning, Disability and Health.

In addition to ICF linking, thematic inductive coding^{46 47} will be applied across all phases to facilitate cross-matching and to identify relevant aspects of social participation and its potential determinants, particularly those of psychological nature or related to global health.^{18 21 22} This approach ensures a broader perspective that goes beyond the ICF model’s focus on physical functioning and rather general classifications of environmental factors,^{18 21} and that aligns with patients’ perspectives and preferences.^{18 48} It is also important to highlight that the Phase 6 psychometric testing will evaluate items from both the disease-specific modules and the generic module, which is not detailed in this protocol, facilitating the identification of topics that necessitate disease- or Belgium-specific assessments versus those adequately addressed by generic items.

Phase 1: Developing an initial ICF-standardised item pool

Phase 1 consists of an initial broad collection of potential items (item pool) that can be used to measure SP and its potential determinants. In conformity with the PROMIS guidelines,³⁵ a literature review is conducted to identify already developed disease-specific PRMs that operationalise—or measure constructs that can be reconciled with—the ICF components.^{35 49} When deemed eligible (criteria in table 1), these PRMs and their items contribute to an initial item pool in terms of content, structure and composition.³⁵ This disease-specific approach mainly takes into account that items that measure constructs related to the ICF components of body functions, structures and activities may vary across health conditions.

Table 1. Eligibility criteria considered for the inclusion of PRMs in the study (based on PROMIS and COSMIN Standards).

	Eligibility criterion	Interpretation
a	Conceptual reconciliation	PRMs based on a clear theoretical and conceptual framework are included in this study if their constructs or (HRQoL) subdomains can be theoretically and conceptually reconciled with the ICF components under investigation.
b	Comprehensiveness	Closely related to the first criterion (a), a higher amount of ICF components that are covered by a PRM’s constructs or (HRQoL) subdomains results in a higher eligibility for its inclusion in the study as the goal is to define a broad initial item pool. In the presence of existing ICF core sets, comprehensiveness refers to the amount of ICF codes from the ICF comprehensive core set which are covered by the PRM (Core Set Representation).55
c	Methodological quality	Only PRMs that were developed with sufficient methodological quality in terms of general study design, content validity, cross-cultural validity and reliability and measurement error at the item level are included in this study (based on COSMIN standards explained below). A ranking in terms of methodological quality is made for the PRMs identified in the literature.
d	Popularity in scientific literature	Items from gold standard or more ‘popular’ disease-specific (HRQoL) PRMs are preferred for inclusion in the final item sets although this criterion is not of primary importance. Popularity is assessed by examining the frequency with which the PRMs occur in the literature review and the citation counts of the articles describing their development, validity and/or reliability in their most current form.

COSMINCOnsensus-based Standards for the selection of health Measurement INstrumentsHRQoLhealth-related quality of lifeICFInternational Classification of Functioning, Disability and HealthPRMspatient-reported measuresPROMISPatient-Reported Outcomes Measurement Information System

For diabetes,⁵⁰ breast cancer,⁵¹ head and neck cancer⁵² and neuromuscular diseases,⁵³ ICF comprehensive core sets have been developed by the ICF research branch and others50,53 which illustrate the heterogeneity across health conditions. These are subsets of ICF codes selected and validated by experts, patients and literature for a specific condition to allow for a comprehensive assessment of the ICF components.⁵⁴ Hence, for these conditions, the literature review identifies potential PRMs to inform the structure and composition of items rather than the content.⁵⁵

It could be that ICF core sets have already been operationalised into PRMs that have been extensively validated in appropriate target populations. In that case, one can adopt eligible PRMs (criterion c, table 1) in their entirety. In the absence of existing PRMs that assess ICF core sets (ie, HIV, cystic fibrosis and haematological malignancies),³⁵ disease-specific health-related quality of life (HRQoL) instruments can be searched for the item pool generation.⁵⁶ These instruments conceptualise health and well-being as multidimensional, similarly to the ICF.⁴⁴ However, few existing HRQoL instruments incorporate items related to the environmental factors (contextual factors, eg, attitudes),⁵⁷ leading to a gap in measures that can inform PRE items in relation to contextual barriers and facilitators. The emphasis on HRQoL instruments permits the researchers to identify potential health-related determinants of SP that transcend the ICF model’s focus on physical functioning.^{18 21 22}

The search for this literature review is conducted in Embase and Medline and involved different combinations of the following search words and related MeSH or Emtree terms Name of chronic disease (eg, Diabetes Mellitus) AND (2) PROM, (3) PREM, (4) ICF, (5) Social participation, (6) health-related quality of life, (7) outcome assessment and (7) systematic review. The detailed search strategies will be reported in future condition-specific publications that describe the application of the current protocol. This literature review is performed independently by two researchers who select disease-specific PRMs based on an evaluation of eligibility criteria a and b (table 1). A subsequent discussion by the two researchers leads to a consensus on the PRMs that are to be evaluated (criterion c, table 1) for potential inclusion in the following study phases.

The 2019 COSMIN Study Design checklist is used to evaluate measurement properties of existing PRMs.⁴³ For each property, COSMIN standards are provided and accompanied by 4-point rating scales (V—Very good, A—Adequate, D—Doubtful and I—Inadequate) that are illustrative of the methodological quality of study design choices.⁴³ Importantly, one can adapt the checklist and its use to fit specific study objectives⁴⁴ by considering the intended use of the instruments.⁵⁸ For the inclusion of existing PRMs in this study, the COSMIN standards related to general study design and content validity as well as measurement properties that offer information at the item level (cross-cultural validity, inter- and intra-rater reliability and measurement error) will be evaluated with the 4-point rating scales⁴³ (online supplemental file a). The rating is performed by two independent researchers.^{34 59} A subsequent discussion to reach consensus leads to an overall ranking of eligibility for each PRM.^{34 59} Additionally, the rating and ranking of PRMs can be guided by systematic reviews that evaluate disease-specific PRMs and match the present research objectives both in terms of theory and (COSMIN) properties under review.

To facilitate the comparison or cross-matching of PRM items, the refined ICF linking rules²⁰ are applied to standardise the items. ICF standardisation refers to the categorisation of items within the ICF language through the stepwise application of ICF linking rules.²⁰ More specifically, items of PRMs are categorised by ‘ICF code’, by ‘perspectives’ adopted by the items and by ‘response options’ used by the items (table 2).²⁰

Table 2. Item categorisation and ICF standardisation as described in ‘Refinements of the ICF linking rules to strengthen their potential for establishing comparability of health information’ (Cieza et al [20]).

Classifier	Categories	Definitions and/or examples20
ICF code (or concept)		Refers the ICF-standardised answer to the question What is this piece of information about? or What is this item about? Both the main concept as well as additional concepts of a PRM item can be represented by an alphanumeric ICF code belonging to the ICF components
Perspectives		Describes the viewpoint of how the ICF concept or code was operationalised in the PRM
	Performance	Describes what an individual does in his/her given environment. It includes all aspects of the environment (physical, social, attitudinal) and can also be referred to as the person’s lived experience
	Capacity	Describes an individual’s ability to execute a task or an action and represents the highest probable level of a person in a given moment
	Appraisal	Appraisal refers to the extent to which a person’s expectations have been satisfied
	Need or dependency	The kind and levels of needs an individual requires as a consequence of the problem or difficulty
Response options		Refers to the response categories or options provided by the PROM for the item that is considered
	Intensity	eg, How much are you currently limited in getting around?Response options: Not at all—a little—some—quite a lot—completely
	Frequency	eg, In the last 3 months, how often have you undertaken preparing main meals?Response options: never—less than once a week—1–2 times per week—most days
	Duration	eg, Speak with your neighboursResponse option: How long?
	Confirmation or agreement	eg, I’m waking up in the early hours of the morning.”Response options: Yes—No
	Qualitative attributes	eg, What does your pain feel like?Response options: (1) flickering, quivering, pulsing, (2) jumping, flashing, boring, (…), (20), nagging, nauseating, agonising

ICFInternational Classification of Functioning, Disability and HealthPRMspatient-reported measuresPROMpatient-reported outcome measures

ICF-standardised items are stored in a table that facilitates cross-matching with patient perspectives (Phase 3) to strengthen the content validity of the PRMs under development,^{20 35 43 55} further referred to as ‘ICF Table’. Two researchers perform this exercise independently and discuss the results to reach consensus.²⁰ In parallel, items from the eligible PRMs are classified based on the thematic inductive analysis^{46 47} to facilitate the cross-matching exercise, especially for items that address topics for which the ICF offers no codes or insufficient detail. The output of Phase 1 consists of an initial literature-based (ICF-)standardised item pool.

Phase 2: Developing a patient-validated final ICF-standardised item pool

Disease-specific focus groups are organised for each of the included diseases until saturation is reached⁶⁰ with a minimum of two focus groups (one in Dutch and the other in French).^{35 43} Focus groups are group interviews on topics raised by an interviewer where the essential collected data refers to what participants say.⁶¹ For the purpose of PRM content development, participants of focus groups belong to the intended target population of the PRM.³⁵ In each focus group, using an interview guide (online supplemental file b), five to eight patients are asked how they would define and assess the ICF components (excluding personal factors). Adults (18 years and older) living in Belgium with cancer, cystic fibrosis, diabetes, HIV or a neuromuscular disease are recruited with invitational material drafted by the research team and disseminated by Belgian patient organisations and reference centres. The selection of representative and heterogeneous groups of participants is informed by a registration form that comprises diagnostic (including multimorbidity) as well as sociodemographic (age, gender and region) and socioeconomic questions (educational level, working status).

A tape-based analysis⁶¹ of the audio recordings followed by the application of the refined ICF linking rules as a means of deductive thematic analysis²⁰ is performed to add the patient perspectives or patient-defined items to the ICF Table. In parallel, the thematic inductive analysis is applied by two independent researchers to ensure that the inclusion of patient perspectives and preferences is not hampered by the ICF model’s more physical functioning-oriented focus.^{18 21} This twofold coding strategy supports the cross-matching with items of Phase 1²⁰ and strengthens content validity of the subsequently deducted item set (Phase 3).³⁵ In this phase, topics may emerge which are not yet covered by the eligible PRMs identified in Phase 1.

At the end of Phase 2, the output consists of the ICF Table that describes a final (ICF-)standardised item pool based on the collected patient perspectives and the PRMs identified in Phase 1.

Phase 3: Developing an initial item set

The third phase consists of cross-matching patient and literature perspectives from the ICF Table^{20 35 55} to deduct an (ICF-)standardised initial item set. A corresponding non-ICF-standardised initial item set is then developed by adopting the original (items from) eligible PRMs or by designing items based on PROMIS item banks and/or patient perspectives (figure 3).

Figure 3. Operationalising the ICF-standardised initial item set by adopting or designing items. ICF, International Classification of Functioning, Disability and Health.

Based on the ICF Table that describes the (ICF-)standardised item pool, SPADIS researchers follow a stepwise approach to build an (ICF-)standardised initial item set for measuring the ICF components (excl. personal factors):

1. Items that are defined by the collected patient perspectives in the (ICF-)standardised initial item set are included.³⁵

2. Items that were not addressed in the focus groups but that are covered by the majority of ICF Table PRMs which best met the eligibility criteria (comprehensive, adequate conceptual and theoretical reconciliation and high content validity) are also included.

3. ICF codes not reported during the focus groups but found in existing ICF comprehensive core sets are similarly included and represented by corresponding items from the most eligible PRMs in the ICF Table.

This ICF-standardised initial item set is now to be operationalised by adopting or developing items to create a corresponding non-ICF-standardised initial item set. For items or topics that were not classifiable within the ICF, a similar stepwise approach is applied based on the thematic inductive coding.⁴⁶ A PRM of Phase 1 can be adopted in its entirety⁴³ if the PRM is favourably evaluated in terms of methodological quality (COSMIN) and target population, and if its original item set aligns well with the current ICF-standardised initial item set and item set based on the thematic inductive approach.

In all other cases, individual PRM items (content, structure and composition) can be operationalised³⁵ based on the (ICF-)standardised initial item set (figure 3). To enhance face validity,⁴⁹ (ICF-)standardised items from PRMs in the ICF Table that match items from the deducted item set in terms of ICF code, perspective and response options, can be adopted in their unstandardised form.⁶² That is, an original item from PRM that is ranked highest in terms of eligibility in the ICF Table (criteria a and c) can be adopted (options 1 and 2, figure 3) to operationalise a corresponding item from the (ICF-)standardised initial item set. In the event that two or more PRMs offer appropriate items and received the same eligibility rating based on criteria a and c, criterion d ‘popularity in scientific literature’ informs the decision on which item to adopt as this will increase the comparability of results from the dissemination and analyses.

Items from the (ICF-)standardised initial item set for which PRMs in the ICF Table do not offer matching items can be operationalised in two different ways. Language-specific PROMIS item banks⁶³ are examined for generic items that align with items from the (ICF-)standardised initial item set and with the present research objectives (option 3, figure 3). In the absence of appropriate items in PROMIS item banks, the PROMIS standards³⁵ for Item Structure and Composition are followed to design items based on the patient perspectives from the focus groups (option 4, figure 3).

This final approach can also be used to operationalise items related to environmental factors in the absence of matching items in the ICF Table. Taking into account the high intercountry variability in terms of environmental factors,²⁴ distinct country-specific PREMs²⁴ will likely be developed. They will enable the assessment of patient experiences and satisfaction in regard to, for example, specific health and security services, systems and policies in Belgium.

Phase 3 results in a patient- and literature-validated initial disease-specific PRM item set to measure SP and its potential determinants, in alignment with the ICF model’s components (excluding personal factors).

Phase 4: Expert validation

Between 5 and 10⁶³ field experts or health professionals are to be invited for a presentation of the final Phase 3 initial item set. It provides the opportunity to actively discuss the taken steps and perform an expert validation exercise.³⁵ The selection criteria for the expert panels are related to heterogeneity or multidisciplinarity (eg, specialist physician, social worker, psychologists, general practitioner and nurse), training, experience and qualifications.⁶⁴ In spite of the subjective nature of expert judgements, content validity can be achieved in an expert panel by using Lawshe’s content validity ratio (CVR) at the item level and content validity index (CVI) at the ICF component level.⁶⁵ Lawshe’s suggested 50% agreement threshold is applied to both the CVR and CVI.⁶⁶ Besides establishing an appropriate degree of content validity,⁶⁶ the expert panel enables the researchers to collect feedback on the phrasing of the questions and the PRMs as a whole.

Phase 4 results in a (literature-, patient- and) now expert-validated PRM item set for the assessment of SP and its potential determinants in alignment with the ICF model and its components (excluding personal factors).

Phase 5: Cognitive interviewing and pilot testing

If items from the item set have not yet been validated in French and/or Dutch prior to their inclusion in the described study phases, translated items are to be created by the research team via a documented forward and backward translation³⁵ and reconciliation⁶⁷ approach.

Cognitive assessment

Patients are invited to participate in one-on-one interviews with one of the researchers. A cognitive assessment will be performed with an emphasis on readability and comprehensibility of the item set.³⁵ As the current methodological approach adopts items from eligible PRMs which have undergone rigorous cognitive interviewing, the interviews will be limited in number yet in conformity with PROMIS specifications,³⁵ that is, 10 in total. Furthermore, a minimum of four interviews may already be sufficient to identify troublesome questions.⁶⁸

For items that were designed by the research team rather than adopted (option 4, figure 3), the item set will be expanded by negatively phrased questions in juxtaposition to positively phrased similar questions, or vice versa, to check for reckless or inconsistent participant responses.⁶⁹ Based on the cognitive assessment, items from the item set are to be adjusted to enhance readability and comprehensibility in the target population.

Pilot testing

An electronic version of the item set is developed and disseminated as one survey to representative samples.³⁵ The dissemination channels and strategies may depend on the target population, disease- and country-specific organisation of care (eg, existence of national disease-specific reference centres or not) and specific research objectives. After its launch, the survey is accessible by URL until 36 individuals have participated.^{65 67} The data collected from the pilot testing will be analysed to determine and resolve any apparent issues. In addition, the duration for filling out the survey will be measured to inform the response likelihood of the target population.⁷⁰

The results from Phase 5 consist of a literature-, patient- and expert-validated item set and first electronic survey that has been assessed for feasibility, usability and face validity.⁷¹

Phase 6: Dissemination and psychometric testing

Phase 6 involves the dissemination of the resulting electronic surveys, comprising the (novel) disease-specific PRMs as well as the generic PRM module, and a detailed analysis of the psychometric properties. This will enable an efficient item reduction strategy⁷² to develop final, validated disease-specific PRMs to measure the SP and its determinants in accordance with the ICF model and its components.

The objective of this phase is to determine the degree to and reliability by which the item sets operationalise the ICF components of body structures and functions, activities and participation as single unidimensional constructs.³⁵ Importantly, one needs to recognise that the ICF components of activities and participation are often addressed in an integrated manner. To fit specific research goals, the researchers will address both components in a separate manner and assign items to the respective constructs based on conformity to the established, distinct definitions.¹⁷ For the ICF component of environmental factors, this study will limit psychometric testing to the individual item level and will not seek confirmation of (satisfaction with) environmental factors as a valid unidimensional construct. The insights produced in this phase will further inform an evaluation of the suitability of the ICF model to comprehensively measure SP and its determinants for people living with chronic diseases in Belgium.

Generic module of the PRM

The generic PRM module that accompanies the developed disease-specific modules will include sociodemographic and socioeconomic items to operationalise the personal factors ICF component. The existence of sociodemographic^{73 74} and socioeconomic⁷⁵ inequalities⁷⁶ in terms of functioning and participation for patients living with a chronic disease has long been recognised. These items are therefore included to ensure that items are relevant for the measurement of SP and its determinants across different sociodemographic and socioeconomic subgroups. The generic module will also include generic items or item sets on (social) participation (eg, PROMIS Ability to Participate in Social Roles and Activities⁷⁷), present and past labour market positions, (satisfaction with) job or work environment, self-efficacy and work ability (eg, work ability index) and the EuroQoL 5D (EQ-5D). By applying the psychometric analyses on the respective disease-specific PRMs in combination with the generic PRM module, this phase can inform an evaluation on which outcomes of SP and their related determinants require disease-specific assessments or whether generic items or item sets suffice.

Dissemination

Considering the chronic diseases under investigation and the research goals, different dissemination strategies are foreseen. They include collaborations with national reference centres and patient organisations. The decision on the dissemination period should take into account the required sample sizes for the psychometric tests that are described below.

Psychometric testing

Phase 6 consists of the application of both classical test theory (CTT) and IRT on the survey responses to investigate the appropriate psychometric properties of the item sets, to inform an item reduction strategy for the final PRMs and to determine suitable scales for the ICF components of body structures and functions, activities and participation.

The CTT tests include basic statistics such as inter-item correlation, item-scale correlations and internal consistency reliability (omega coefficient) as well as confirmatory factor analysis. These tests provide information on reliability, local dependence and unidimensionality at the construct or scale level.⁷⁸ CTT, however, is less apt to describe the contributions of items separately and fails to inform on whether items improve the measurement or not.⁷⁸

IRT, on the other hand, models the relationship of individual items to the construct that is being assessed and provides information on the performance of each item in a more comprehensive manner by evaluating different item parameters (item location, discrimination, guessing and trait score).⁷⁹ For example, through the examination of information function curves, IRT can evaluate how item- and scale-level precision vary at different levels of the construct. Furthermore, IRT-developed instruments offer the advantage of ability invariance, meaning that the IRT estimate of the construct is independent of the measure being used because it considers the difficulty of items during scoring.⁷³ Potentially confirming item invariance, a property where item parameters are equal across different samples, defines an additional advantage of IRT.^{73 78}

Several IRT models can be used for the analysis of items with polytomous, ordinal response options (eg, Likert scales) as is typically seen in PROMs and PREMs.⁷³ For the estimation of item-specific discrimination parameters and category response parameters (guessing parameters not applicable for most PROMs),⁷³ the Graded Response Model (GRM)^{73 80} and the Generalized Partial Credit model^{73 80} (GCPM) can be fitted. This flexibility makes the GRM and GCPM more likely to fit PRO data.⁷³ The sample size requirement of at least 300 and minimum of 5 items per construct is similar for both models.⁶⁴ However, a choice between the two models can be made by comparing item-level fit results from fitting the construct to the GRM versus the GPCM^{73 81} and by examining the rate of missing data.⁸⁰ Following the PROMIS guidelines,³⁵ these IRT tests provide information on local independence of the items for each construct as well as item-specific reliability across the latent construct continuum and differential item functioning (DIF). DIF refers to the presence of differences in how items perform across different groups, even when these groups have the same underlying level of the construct being measured.⁸²

A cautious evaluation of the above-mentioned test results and subsequent application of an item reduction strategy, based on established thresholds, will inform the final item sets for each of the constructs or ICF components.

The output of Phase 6 consists of surveys with item sets to measure SP and its related determinants, in alignment with the ICF components. These psychometrically sound PROM and PREM tools may be disease- and country-specific and are based on literature, patient and expert perspectives.

Significance of the study

For people living with chronic diseases in Belgium, there are currently no validated disease-specific PRMs for a comprehensive assessment of SP and its determinants reflecting the ICF model and its components. The multiphase mixed-methods design of this protocol places an emphasis on patient involvement by collecting patient perspectives early on in the development and building on established PRMs with high content validity. The PROMIS and COSMIN Standards and refined ICF linking rules²⁰ were applied to address the limited standardisation and transparency for PRM development in current literature.²⁵ Advanced psychometric testing (IRT) is used to enable more valid, reliable and flexible measures.³⁸ Furthermore, it facilitates the potential CAT implementation of the PRMs in clinical practice, which can be used for monitoring at a national level. By applying a thematic inductive analysis on the collected perspectives from patients and established instruments in parallel to the ICF-refined linking rules, the present methodology addresses recognised limitations of the ICF model to comprehensively assess functioning. The developed insights may aid policy-makers and federal agencies in considering alternative models and frameworks to inform healthcare and social security policies. The dissemination of a generic PRM module alongside the disease- and Belgium-specific PRMs enables an evaluation of whether measuring SP and its determinants necessitates disease- and/or Belgium-specific instruments.

The developed instruments will enable valuable research into social participation and its determinants for the target populations in Belgium. Moreover, the described methodological steps can be applied to other chronic diseases to bolster our general understanding of social participation among people living with chronic conditions in Belgium.

Current study phase

Since November 2023, patient focus groups (Phase 2) are being organised and literature reviews (Phase 1) on eligible disease-specific PROMs are performed. The end of Phase 1 and Phase 2 of the present protocol is currently foreseen for April 2025. The subsequent phases, including psychometric testing and fine-tuning of the PRMs, are set to be completed by January 2026.

Ethics and dissemination

Personal (health) information is collected in several of the described study phases. Diagnostic and personal identifying information are provided through the registration form to inform heterogeneous focus groups and for scheduling. Written consent is requested for the patient registration form as well as for the focus groups. Collected data are encrypted and stored securely on Sciensano servers in password-protected folders accessible only to SPADIS researchers. Ethical approval was granted by the Ethics Committee of the Ghent University Hospital on 20 February 2023 to use the registration form and organise the patient focus groups (ONZ-2022-0470). Ethical approval for dissemination of the PRMs and psychometric testing will be sought at the start of Phase 6.

The results of the current project will be distributed through peer-reviewed publications and conferences. Furthermore, the current protocol describes work package 3 of the SPADIS project, for which all results will be bundled and communicated externally by a SPADIS monography or report.

Data availability statement

No data are available.