Skip to main content
NCBI home page
Clinical Medicine logoLink to Clinical Medicine
. 2026 May 27;26(4):100592. doi: 10.1016/j.clinme.2026.100592

A FeAsibility items Checklist for assessing implementation characTeristics of patient-reported Outcome measures in Research, Regulation and Routine clinical care (FACTOR3): Development and evaluation

Asad Bhatty a,b,c,⁎, Chris Wilkinson d,e, Visvesh Jeyalan e, Ali Wahab a,b,c, Jeremy Dwight f, Marcin Ruciński f, Edwin de Beurs g, Melanie Calvert h,i,j,k, Richard P Gale d,l, Mike Horton m, Andrea Manca n, Tom Melvin o, Massimo Di Maio p, Amar Rangan q, Julie Sanders r, Tonya Winders s, Adam B Smith a,b, Chris P Gale a,b,c; , on behalf of the FACTOR3 Consortium
PMCID: PMC13315201  PMID: 42208661

Abstract

Study objective

Patient-reported outcome measures (PROMs) provide valuable data to inform regulatory decision making, health technology assessment and routine clinical care. We aimed to develop a feasibility item checklist for PROMs and their selection, beyond their psychometric properties (FACTOR3).

Methods

We followed a five-stage method to select parameters for PROM evaluation. 1) A scoping literature review identified candidate items for consideration; 2) round 1 modified Delphi was used to select items for inclusion or exclusion, conducted by the design group (n = 14); 3) feedback on the checklist was provided by representatives from the European Medicines Agency and National Institute for Health and Care Excellence; 4) round 2 modified Delphi was used to finalise item selection, conducted by the clinical domains group (n = 21) and 5) evaluation of the feasibility item checklist using a selection of PROMS (EQ5D-5L, HeartQOL, the Oxford Hip Score, The EORTC Core Questionnaire QLQ-C30, Re-QOL-10 and NEI-VFQ-25) (Fig. 1).

Results

The scoping review identified 13 items relating to the intrinsic feasibility of using PROMs which were considered in the modified Delphi. The final FACTOR3 checklist included eight unique candidate items: price; licensing, comprehensibility, duration, coverage, translations, electronic device compatibility; and minimal important difference. Of the six PROMS evaluated, the intraclass correlation coefficient was 0.81, suggesting good reliability.

Conclusions

FACTOR3 is a feasibility item checklist to assess the implementation characteristics of PROMs in research, regulation and routine clinical care. It may be used in conjunction with existing psychometric evaluation and user guides for PROMS to facilitate their use in health care.

Keywords: Patient-reported outcome measures, PROMs, Feasibility, Implementation

Highlights

  • •

    There are additional feasibility factors intrinsic to a PROM, beyond its psychometric properties, that impede its routine use

  • •

    Eight variables were selected through consensus of interdisciplinary stakeholders that was agreed upon by European regulators

  • •

    The FACTOR3 checklist evaluates a PROM's feasibility characteristics with broad applicability in clinical care and trials.

  • •

    It may be used with psychometric evaluation thus providing a comprehensive selection process for contemporary use of PROMs

Introduction

Patient-reported outcome measures (PROMs) provide valuable information on the impact of disease and treatment on a patient’s symptoms, emotional wellbeing, functioning and quality of life.1, 2, 3 Trial data on PROMs provide evidence of the efficacy and tolerability of treatments from the patient perspective and are valued by regulators.4, 5 PROMs are increasingly used in routine clinical practice and may be used to support patient communication, shared decision making6 and early detection of treatment side effects.7, 8

The European Medicines Agency (EMA) and the US Food and Drug Authority recommend the use of PROMs for medical device or drug evaluation, recommend that such PROMs be validated in the target population and disease, and the results appropriately communicated to healthcare providers and patients when supporting pharmaceutical labelling claims.4, 5 In the UK NHS, PROMs are routinely used to evaluate the outcomes of elective hip and knee surgery, with results informing an assessment of healthcare provider performance, as well as supporting clinical decision making9 and benchmarking quality of care.10, 11, 12

There are over 7,000 validated PROMs spanning the range of medical diseases and interventions,13 but limited guidance exists in formally assessing their intrinsic feasibility of implementation. The Patient-Reported Outcomes Tools: Engaging Users and Stakeholders (PROTEUS) consortium recommends a qualitative assessment of the reliability and validity of a PROM prior to its selection,14 such as The ‘COnsensus-based Standards for the selection of health Measurement INstruments’ (COSMIN) evaluation.15 Assessment of some feasibility items is recommended, such as the cost of using a PROM, licensing16, 17, 18 and the ability of a PROM to be understood by the user.19 However, there are additional factors such as operational costs, including searching for more information on a PROM through subscription websites,13 regulatory approval for the use of a PROM in a randomised clinical trial20, 21 and metrics that interpret change in PROM scores over time, such as the minimal important difference (MID) that determines the ability of a PROM to be used in research, regulatory affairs and routine clinical care,22, 23 but are not included in recommended assessment guides.

We aimed to develop a feasibility item checklist for assessing the implementation characteristics of PROMs in research, regulatory affairs and routine clinical care, and evaluate it against commonly used PROMs. This could complement user guides and address an implementation gap (Fig. 2).

Fig. 2.

Fig. 2

Open in a new tab

Visual graphic on how the FACTOR3 checklist fits into the selection process.

Material and methods

The development of the FACTOR3 checklist consisted of five stages: 1) scoping review to identify candidate items for consideration, conducted by the steering committee; 2) first stage modified Delphi to include or exclude items, conducted by the design group; 3) feedback on the checklist by the regulator group; 4) second stage modified Delphi to provide feedback on items, conducted by the clinical domain group; and 5) evaluation of the feasibility item checklist using a selection of PROMS.

Steering committee

The steering committee comprised clinical experts (A.B., C.W., C.P.G), a PROMs methodologist (A.B.S.) and a data manager (S.C.). They designed the study, formulated a list of candidate variables from the scoping review, identified and invited individuals to the other three working groups (design, clinical domain and regulator group), and collated and presented the results of the polls to the groups.

Scoping review

A scoping review was performed by the steering committee to identify potential factors in the published literature and existing user guides. All prospective and retrospective articles and grey literature2, 14, 24 relating to feasibility of implementation of PROMs in adults from inception to 6 March 2024 in English was included. PubMed was searched using a pre-defined search strategy (Supplementary Section) and was conducted by A.B. These were assimilated and formed the basis of the questions for voting.

Modified Delphi

The steering committee identified potential members for each of the working groups by selecting co-authors of PROM research manuscripts, those holding senior positions in regulatory affairs, healthcare professionals with experience of PROMs, and from personal references and recommendations by other Delphi participants. The aim was to achieve wide representation across multiple geographies (Europe, North America and Oceania) and clinical and academic disciplines. We therefore contacted PROM methodologists, developers of PROMs, health economists, biostatisticians, patients, patient representatives and multidisciplinary healthcare providers. No minimum sample size was placed on the number of working group members and Delphi participation. Each working group member’s expertise was confirmed by the steering committee (apart from patients and patient representatives) before being invited to participate via email. Once accepted, the steering committee allocated each panellist either the design or clinical domain group detailed below, sent them an information pack via email and invited them to attend a virtual presentation about the project.

The first stage of modified Delphi meetings consisted of 14 experts as part of the design group who voted in the online survey. Participants were invited to provide feedback via free text, during voting and discussion was encouraged around the rationale for including/excluding items distilled from the scoping review and the scope of the checklist. Feedback to provide additional items was welcomed.

Once the list of variables and their definitions for the checklist were agreed, they were reviewed by the regulators for their feedback and agreement. The group comprised of representatives from the European Medicines Agency (EMA) and the National Institute for Health and Care Excellence (NICE) who were either PROM methodologists, patient representatives or trialists. They did not participate in the voting process.

The second stage of modified Delphi meetings consisted of 21 experts as part of the clinical domain group who voted in the online survey from the results garnered from the first stage Delphi. Participants were encouraged to include/exclude items distilled from the design group. Members of the design group and steering committee only voted once in the first stage to prevent unequal weighting of votes.

The virtual presentation was delivered and chaired by A.B. and supported by C.W. and C.P.G. and sought to summarise the current PROM selection process as per user guides, highlight aspects of feasibility that are not addressed in the literature, and the candidate feasibility items extracted from the scoping review for discussion. Following the presentation, panellists were emailed an online survey. All panellists received the same presentation and options for voting.

The survey was in English using the ‘Jisc’ platform (https://www.jisc.ac.uk/). Participants were asked to provide their name and email address to protect against duplicate voting. As part of the survey, participants were asked to review the candidate feasibility item checklist variables that had been generated from the scoping review and to vote to ‘include’ or ‘exclude’ each variable. Judgement was based upon the panellist’s perception of the feasibility and applicability of each variable. Written comments were invited for each item, which were collated and thematically categorised by the steering committee.

The threshold for inclusion of a candidate feasibility item checklist variable was at least 75% of panellists voting on an item to include and was decided a priori.25, 26 The threshold for exclusion was at least 75% of participants selecting for the variable to be excluded or achieving less than 75% to include. No financial or in-kind incentive was offered to panellists for completing their survey or virtual meeting. The overall process is shown in Fig. 1.

Fig. 1.

Fig. 1

Open in a new tab

Visual graphic on the formation of the FACTOR3 checklist.

Evaluation of checklist

Three clinician reviewers who were independent of the modified Delphi method for the development of the feasibility item checklist, were invited to evaluate six PROMs against the final checklist. Specific PROMs (EQ5D-5L, HeartQOL, the Oxford Hip Score, The EORTC Core Questionnaire QLQ-C30, Re-QOL-10 and NEI Visual Function Questionnaire-25) were selected because they were routinely used in clinical practice (for example within the NHS of England), endorsed by international organisations, widely cited in the literature and to cover different clinical specialties. Each reviewer was given a sample of the questionnaires and URL links to each organisation’s website, as some checklist items may be difficult to answer on a questionnaire alone. For example, licensing and cost information will not be available by reviewing the questions in a PROM.

The presence of each item of the feasibility item checklist was assigned a score of 1 and the overall score was recorded. A PROM that scored highly against the feasibility item checklist was deemed feasible or easy to implement, with feasibility being defined as the intrinsic characteristics of a PROM that allows ease of implementation. This differs to alternative definitions of feasibility that assess the extent to which a PROM may be successfully used or carried out within a particular setting.27 The intraclass correlation coefficient score (ICC) was performed to evaluate reviewers’ inter-rater reliability, which was collated and stored in MS Excel.

Patient and public involvement

Two patient representatives from the European Society of Cardiology Patient Forum (J.D. and M.R.) were involved in the first stage modified Delphi meetings and one patient representative (T.W.) was involved in the second stage modified Delphi meetings. They provided feedback on all candidate items extracted from the steering committee and scoping review and emphasised the inclusion of patient comprehension and comprehensiveness of a PROM into the checklist. These two items achieved consensus.

Results

Steering committee

Following the scoping review, an initial list of 13 items was drafted by A.B. (see Supplementary Table 1). The steering group reviewed these and merged some items together to form a list of nine unique candidates. This list was used in the modified Delphi exercise.

Design group

This consisted of five PROM methodologists, one biostatistician, two health economists, four healthcare providers and two patient representatives. All participants were from Europe (n = 14, mostly the UK), and there were nine men and five women. A series of engagement and feedback meetings was held, with subsequent voting on the nine items. The results were then presented and discussed among the design group during an online meeting, and the design group voted to include all nine items. The distribution of votes is shown in Fig. 3. There was detailed discussion about the ideal scope of the checklist, whether additional variables were required and whether there should be weighting of items to give more importance to some items over others.

Fig. 3.

Fig. 3

Open in a new tab

Voting distribution of first stage modified Delphi with the design group.

Clinical domain group

The group included individuals with expertise in cardiology (n = 4), ophthalmology (n = 3), dermatology (n = 3), orthopaedics and rheumatology (n = 3), medical oncology (n = 2), mental health (n = 1), neurology (n = 1), clinical epidemiology (n = 1), urology (n = 1), medical device regulatory affairs (n = 1), and a patient representative (n = 1). The experts were from Europe (n = 12), North America (n = 7) and Oceania (n = 2) and comprised 14 men and seven women. Sequential meetings were held amongst the 21 experts from the clinical domain group. From the nine-candidate list of items derived from the scoping review and design group consensus, eight items were voted on to include into the FACTOR3 checklist. The distribution of votes is shown in Fig. 4. There was consistent feedback over the scope of the checklist, refinement of the names of items and how to utilise the checklist in both trials and clinical practice, which is reported below.

Fig. 4.

Fig. 4

Open in a new tab

Voting distribution of second stage modified Delphi with clinical domains group.

FACTOR3 checklist

The final feasibility items checklist (FACTOR3) includes eight items and is to be used in addition to the psychometric evaluation of a PROM. The checklist is provided in the Supplementary Section.

  • 1)

    Licensing structure that aligns with the goal

    Often validated PROMs are subject to licensing arrangements that may vary according to intended use, and obtaining appropriate permissions can be time consuming and possibly subject to prohibitive terms and conditions involving access to data. Panellists felt that these were important considerations during a PROM selection process.17 Some panellists also cautioned against the use of unlicensed PROMs because it raised uncertainty over the origin of a questionnaire and if any results obtained would be valid.

  • 2)

    Cost

    This domain refers to both using a PROM and its operational cost. Linked to the licensing structure of a PROM is the cost of administering and managing it, which may impede its routine use.

    Operational costs that may be incurred involve implementing an electronic PROM into existing software of electronic health records. Also, obtaining a valid and culturally appropriate translation could incur costs; the panellists felt that this may be a barrier for some organisations and should therefore be considered when selecting a PROM.28, 29

  • 3)

    Comprehensibility

    Results from PROM data may be unreliable if patients refuse to complete the questionnaires because they do not understand the questions (leading to increased missingness of data) or, worse, answer while not fully comprehending the question.30 This problem is compounded if the PROM was developed with no or limited patient involvement.31 Allowing the content of a PROM to be at a grade six or age 10 reading level can help ensure patient comprehension as literacy levels vary across the population, and increases the likelihood of patients completing the questionnaire.29 Although comprehension overlaps with content validity, and therefore assessed in qualitative assessments,15 our patient representatives emphasised its inclusion over others. Similar to other items, our panellists recommended the use of a patient group reviewing the comprehensibility of potential PROMs prior to their implementation.

  • 4)

    Translated and validated into the language of choice

    Another way that PROM data could be unreliable is if there is no validated translation that isw culturally adapted to provide equivalent meaning to the original PROM for a diverse patient population.29 This was agreed to and is recommended by regulators.32

  • 5)

    Duration

    Longer PROMs are associated with low respondent rates in comparison to shorter ones, especially for patients who are ill or fatigue easily.21 However, PROMS that are long and/or take longer to complete can be important, depending on the target population and objective of data collection objectives. Participants may be willing to complete lengthier measures if they understand why the data are being collected and how they will be used.33 The panellists agreed that the choice of PROM should be considered against the duration of time it takes for completion, as well as the organisation’s goal in using the PROM. Some panellists advised for a local patient and public group to be involved early in a project to determine if the PROM has a high respondent burden or not.

  • 6)

    Comprehensiveness

    This item had two considerations. The first was whether the PROM had regulatory approval to be used as an outcome measure in the targeted area of research. The panellists felt that regulatory approval was equally as important as having robust psychometric properties.

    The second consideration was how well the PROM encapsulates the elements of the target disease. Striking the balance between shorter questionnaires and adequate coverage that doesn’t contribute to respondent burden was also raised. Comprehensive PROM coverage improves the reliability of responses by ensuring PROM data reflects the patient perspective entirely.34 The experts stressed its inclusion despite some overlap with a questionnaires content validity35 and therefore psychometric evaluation. The experts emphasised that the PROM should be validated in the target population.

  • 7)

    Electronic device compatibility

    Many healthcare organisations use electronic health records, and the ability to incorporate a PROM with its software or complete it by smartphone use is clearly advantageous. This increases response rates and allows questionnaires to be completed beyond the hospital visit.36 Subsequent interpretation and analysis of data is less burdensome than using paper records. The ability to implement an electronic questionnaire streamlined with existing healthcare systems that is appropriate and acceptable to licensed owners of the PROM was also a consideration. Some panellists felt that in some circumstances a PROM may be compatible with a smartphone app but not incorporated with healthcare records, which may conversely increase administrative burden.

  • 8)

    Minimal important difference

    The MID provides a clinical context to the results of a PROM and is therefore meaningful to clinicians. It provides a measure of the smallest change in the PROM of interest that patients perceive as important, either beneficial or harmful, and that may potentially warrant a change in management.22 A barrier to PROM implementation is negative clinician perspectives,8 and the panellists felt that providing appropriate context to PROM results was, therefore, important.

Factors not included in the final checklist

One item that was not included in the final checklist was the availability of the PROM to be provided in an alternative format. Alternative PROM formats give a method of improving the reliability of responses. These include visual analogue scales of PROMs37 to improve comprehension and reduce respondent burden. Furthermore, a concern from focus group members was to ensure adequate support for all patient populations to complete an assessment such as, but not limited to, patients with visual or hearing impairment and neurodivergent patients.

Checklist evaluation

The results of each reviewer’s assessment of the PROMs are in Supplementary Table 2. The calculated ICC was 0.81 suggesting the checklist has good inter-rater reliability.

How to use the FACTOR3 checklist

The FACTOR3 checklist evaluates the intrinsic feasibility characteristics of a questionnaire and can streamline the PROM selection process prior to implementation in research, regulatory affairs and routine clinical care. The PROTEUS guide remains an important document to consult that elaborates the current selection process; identifying all available questionnaires on a disease domain and evaluating their psychometric properties.2, 14 We recommend completing the FACTOR3 checklist early in the PROM selection process, such as after the psychometric evaluation stage. This ensures that all important feasibility concerns are comprehensively addressed prior to implementation that aligns with the organisation’s goals.

These goals could differ depending on available resources, target population characteristics and context of care. For example, in research, comprehensiveness in terms of regulatory approval could matter more to an organisation than other items and help to decide which PROM to choose further on top of its qualitative assessment. Whereas other items such as price and licensing structure of a questionnaire for routine clinical care in a publicly funded institution may be paramount. Therefore, the Delphi panellists voted against weighting each item which allows the checklist to be flexible with any context.

Discussion

Using a modified Delphi approach that was inclusive of a scoping review, expert opinion and stakeholder involvement, we developed and evaluated the FACTOR3 feasibility item checklist for assessing the intrinsic implementation characteristics of PROMs in research, regulation and routine clinical care. FACTOR3 may be used in conjunction with existing psychometric evaluation,15 regulatory guidance,4, 38 ethical39 and protocol guidance,2 PROTEUS trials and practice resources14, 24, 40 and other user guides for PROMs – providing a comprehensive and complementary selection process for the contemporary selection of PROMs. FACTOR3 has eight components – price, licensing, comprehensibility, duration, translation, coverage, electronic device compatibility and minimal important difference, which were informed by the literature and international experts and stakeholders in PROMs. Our evaluation of six PROMs (frequently employed in research and or clinical settings) against FACTOR3 criteria by three independent assessors found that the feasibility item checklist provided consistency in results.

Generating high-quality patient-reported outcome data informs clinical care and the generalisability of trial results yet is affected by respondent burden of the target population.3, 19, 21, 41 High levels of missing data or unreliable responses due to poor comprehension can lead to misinterpretation of results.2 This can deepen health inequalities in poorly served populations, partially due to feasibility concerns.42 The lack of available valid translations, and poor comprehension and digital literacy, impact underserved communities that are traditionally underrepresented in research and trials the most.41, 43 Therefore the Delphi panellists felt that addressing respondent burden in more depth in a feasibility checklist was needed, which differs from the PROTEUS guide.33 Maximising the response rates and reliability needs to be catered to the diverse patient population, which differs by socioeconomic status and literacy.41, 44 PROMs that are available in electronic formats, as well as the availability of pen and paper options, may suit patients with poorer digital literacy who do not own an electronic device.45 However, the use of electronic health records is widespread and the ability of a PROM to align with existing software may improve compliance,36 provides a source for data collection and storage46 and is some patients’ preferred mode of completing questionnaires.41 All these factors could be implemented without bias, as a meta-analysis into mixed modes of administration of PROMs has shown.47

To our knowledge, this checklist is the first to evaluate the intrinsic feasibility of implementing a PROM with broad applicability for clinical practice and trials. It provides a practical supplement to existing psychometric and quality assessments in the evaluation of prospective PROMs depending on the organisation’s goals that is in line with regulatory requirements.4, 38 We identified a list of intrinsic feasibility items necessary for consideration prior to the selection and adoption of PROMs distilled from expert opinion using the modified Delphi method. However, we recognise limitations of this work. We used a focused definition of feasibility, whereas other more broad definitions exist such that evaluate the extent to which a PROM may be successfully carried out within a particular healthcare or research setting.27 Furthermore, to ascertain some aspects of feasibility (such as licensing and cost) requires users to explore a diverse range of materials to clarify which may increase administrative burden initially.17 Although the items were filtered from a scoping review, the final checklist was agreed upon by consensus of experts and is therefore subject to potential selection bias. Furthermore, the group of experts and stakeholders were chosen to be a representative sample, but the sample size was small and completed in English only. However, the experts had broad expertise in PROMs use in clinical practice and trials, and the stakeholder group consisted of major European regulators and patient representatives that shaped the discussion and provided further strength to our findings.

Conclusion

Informed by scoping review, and modified Delphi methods including PROM methodologists, clinicians, patients and regulators, we have developed a checklist of implementation feasibility items that should be considered when selecting a PROM. We anticipate that the keys users and beneficiaries of FACTOR3 will be researchers conducting studies and writing papers, the healthcare sector when considering a PROM for routine use and assessment of clinical practice, and regulators for the evaluation the success of clinical outcomes in regulatory decision making. The checklist could have broad applicability and serves to act as a decision-making tool, in conjunction with existing psychometric assessments and user guides, for the selection of a PROM.

CRediT authorship contribution statement

Tom Melvin: Writing – review & editing, Methodology, Investigation. Asad Bhatty: Writing – review & editing, Writing – original draft, Visualization, Investigation, Formal analysis, Data curation. Massimo Di Maio: Writing – review & editing, Methodology, Investigation. Chris Wilkinson: Writing – review & editing, Supervision, Methodology, Investigation, Formal analysis. Mike Horton: Writing – review & editing, Methodology, Investigation. Andrea Manca: Writing – review & editing, Investigation. Tonya Winders: Writing – review & editing, Methodology, Investigation. Jeremy Dwight: Writing – review & editing, Project administration, Methodology, Investigation. Adam B. Smith: Writing – review & editing, Supervision, Software, Resources, Formal analysis, Data curation. Marcin Ruciński: Project administration, Methodology, Investigation. Amar Rangan: Writing – review & editing, Methodology, Investigation. Visvesh Jeyalan: Writing – review & editing, Validation, Project administration. Julie Sanders: Writing – review & editing, Validation, Resources, Methodology, Investigation, Formal analysis. Ali Wahab: Writing – review & editing, Validation, Data curation. Richard P. Gale: Writing – review & editing, Resources, Investigation. Chris P. Gale: Writing – review & editing, Supervision, Resources, Project administration, Funding acquisition, Formal analysis, Conceptualization. Edwin de Beurs: Writing – review & editing, Validation, Methodology, Investigation. Melanie Calvert: Writing – review & editing, Supervision, Methodology, Formal analysis.

Ethics approval and consent to participate

No ethical approval was required for this scoping review and modified Delphi exercise.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:

All authors have completed the ICMJE uniform disclosure form at https://www.icmje.org/disclosure-of-interest/ and declare:

A.B., A.B.S., V.J., A.W., A.M. declare no conflicts of interest.

C.W. reports grants from British Heart Foundation and National Institute for Health and Care Research and has unpaid roles with EHJ Quality of Care and Clinical Outcomes (associate editor), and NICE Indicator Advisory Committee.

M.H. reports grants from NIHR, MRC, European Huntington’s Disease Network, UKRI, and Proctor & Gamble.

T.Me is a member of the European Society of Cardiology – Regulatory Affairs Committee, European Medicines Agency – medical device expert panel and is chair of the Biomedical Alliance in Europe, Regulatory Affairs Committee.

T.W. is the president of the Global Allergy and Airways Patient Platform.

J.D. and M.R. are members of the European Society of Cardiology Patient Forum.

C.P.G. has received grants from Alan Turing Institute, British Heart Foundation, National Institute for Health and Care Research, Horizon 2020, Abbott Diabetes, Bristol Myers Squibb and European Society of Cardiology and reports consulting fees from AI Nexus, AstraZeneca, Amgen, Bayer, Bristol Myers Squibb, Boehrinher-Ingleheim, CardioMatics, Chiesi, Daiichi Sankyo, GPRI Research B.V., Menarini, Novartis, iRhythm, Organon and The Phoenix Group and speaker’s fees from AstraZeneca, Boston Scientific, Menarini, Novartis, Raisio Group, Wondr Medical, Zydus. C.P.G. is also a deputy editor: EHJ Quality of Care and Clinical Outcomes, NICE Indicator Advisory Committee member, chair ESC Quality Indicator Committee member and participated on a Data Safety Monitoring Board or Advisory Board for DANBLCOK trial and TARGET CTCA trial.

M.J.C. is director of the Centre for the Centre for Patient Reported Outcomes Research, deputy director of the Birmingham Health Partners Centre for Regulatory Science and Innovation and is a National Institute for Health and Care Research (NIHR) Senior Investigator. M.J.C. receives funding from the National Institute for Health and Care Research (NIHR), UK Research and Innovation (UKRI), NIHR Birmingham Biomedical Research Centre (BRC), NIHR ARC West Midlands, LifeArc, European Regional Development Fund, Innovate UK (part of UKRI), Merck, GSK and Gilead. M.J.C. has received personal fees from Aparito Ltd, Boehringer Ingelheim, CIS Oncology, Halfloop, ICON, Merck, Pfizer and Vertex outside the submitted work. In addition, a family member owns shares in GSK. She is a member of the PROTEUS Consortium. All other authors declare no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.

MDM reports honoraria from AstraZeneca, Janssen, Merck Sharp & Dohme (MSD), Novartis, Pfizer, Roche, GlaxoSmithKline, Amgen, Merck, Takeda, Ipsen, Viatris for consultancy or participation to advisory boards; direct research funding from Tesaro/GlaxoSmithKline; institutional funding for work in clinical trials/contracted research from Beigene, Exelixis, MSD, Pfizer and Roche.

AR has received research grants from NIHR, CIHR, Horizon 2020, AO UK&I, DePuty J&J Ltd, and Stryker Ltd. He is a member of the NIHR i4i funding board.

Acknowledgements

None.

Footnotes

Appendix A

Supplementary data associated with this article can be found in the online version at doi:10.1016/j.clinme.2026.100592.

Appendix A. Supplementary material

Supplementary material

mmc1.docx (29.9KB, docx)

.

Data availability

All data requests should be submitted to c.p.gale@leeds.ac.uk for consideration.

References

  • 1.Black N. Patient reported outcome measures could help transform healthcare. Br Med J. 2013;346:f167. doi: 10.1136/bmj.f167. [DOI] [PubMed] [Google Scholar]
  • 2.Calvert M., Kyte D., Mercieca-Bebber R., et al. Guidelines for inclusion of patient-reported outcomes in clinical trial protocols: the SPIRIT-PRO extension. JAMA. 2018;319(5):483–494. doi: 10.1001/jama.2017.21903. [DOI] [PubMed] [Google Scholar]
  • 3.Rivera S.C., Kyte D.G., Aiyegbusi O.L., Slade A.L., McMullan C., Calvert M.J. The impact of patient-reported outcome (PRO) data from clinical trials: a systematic review and critical analysis. Health Qual Life Outcomes. 2019;17(1):156. doi: 10.1186/s12955-019-1220-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.European Medicines Agency. Committee for Medicinal Products for Human Use. Reflection paper on the regulatory guidance for the use of health-related quality of life (HRQL) measures in the evaluation of medicinal products; 2005.
  • 5.U.S. Food and Drug Authority. Principles for selecting, developing, modifying, and adapting patient reported outcome instruments for use in medical device evaluation. Guidance for industry and food and drug administration staff, and other stakeholders. 〈https://wwwfdagov/media/141565/download2020〉 Last accessed November 2024.
  • 6.Velikova G., Booth L., Smith A.B., et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol. 2004;22(4):714–724. doi: 10.1200/JCO.2004.06.078. [DOI] [PubMed] [Google Scholar]
  • 7.Calvert M., Kyte D., Price G., Valderas J.M., Hjollund N.H. Maximising the impact of patient reported outcome assessment for patients and society. BMJ. 2019;364:k5267. doi: 10.1136/bmj.k5267. [DOI] [PubMed] [Google Scholar]
  • 8.Snyder C.F., Aaronson N.K., Choucair A.K., et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012;21(8):1305–1314. doi: 10.1007/s11136-011-0054-x. [DOI] [PubMed] [Google Scholar]
  • 9.NHS England. Provisional Patient Reported Outcome Measures (PROMs) in England for hip and knee replacement procedures (April 2022 to March 2023). In: Secondary Care Analysis (Ed.). 〈https://digital.nhs.uk/data-and-information/publications/statistical/patient-reported-outcome-measures-proms/hip-and-knee-replacement-procedures-april-2022-to-march-2023#2023〉.
  • 10.Black N. Patient reported outcome measures could help transform healthcare. BMJ. 2013;346:f167. doi: 10.1136/bmj.f167. [DOI] [PubMed] [Google Scholar]
  • 11.Chen J., Ou L., Hollis S.J. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res. 2013;13(1):211. doi: 10.1186/1472-6963-13-211. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Wilkinson C., Bhatty A., Smith A.B., Dwight J., Sanders J., Gale C.P. Embracing the promise of patient reported outcome measures in cardiology. Eur Heart J Qual Care Clin Outcomes. 2024;10(8):651–652. doi: 10.1093/ehjqcco/qcae073. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Online support for clinical outcome assessments [Internet], ePROVIDE™; 2024 〈https://eprovide.mapi-trust.org/advanced-search〉.
  • 14.The PROTEUS guide to implementing patient-reported outcomes in clinical practice: a synthesis of resources [Internet]; 2023.
  • 15.Mokkink L.B., Elsman E.B.M., Terwee C.B. COSMIN guideline for systematic reviews of patient-reported outcome measures version 2.0. Qual Life Res. 2024;33(11):2929–2939. doi: 10.1007/s11136-024-03761-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Bull C., Teede H., Watson D., Callander E.J., editors. Selecting and Implementing Patient-reported Outcome and Experience Measures to Assess Health System Performance. American Medical Association; 2022. (JAMA Health Forum). [DOI] [PubMed] [Google Scholar]
  • 17.Glenwright B.G., Simmich J., Cottrell M., et al. Facilitators and barriers to implementing electronic patient-reported outcome and experience measures in a health care setting: a systematic review. J Patient-Rep Outcomes. 2023;7(1):13. doi: 10.1186/s41687-023-00554-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Marcus A. Pay up or retract? Survey creator's demands for money rile some health researchers. Science. 2017;15(3):169–182. [Google Scholar]
  • 19.Nguyen H., Butow P., Dhillon H., Sundaresan P. A review of the barriers to using patient‐reported outcomes (PROs) and patient‐reported outcome measures (PROMs) in routine cancer care. J Med Radiat Sci. 2021;68(2):186–195. doi: 10.1002/jmrs.421. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Jung A., Challoumas D., Pagels L., Armijo-Olivo S., Braun T., Luedtke K. Guidelines for the development and validation of patient-reported outcome measures: a scoping review. BMJ Evid-Based Med. 2024;29(6):363–373. doi: 10.1136/bmjebm-2023-112681. [DOI] [PubMed] [Google Scholar]
  • 21.Turner R.R., Quittner A.L., Parasuraman B.M., Kallich J.D., Cleeland C.S. Group MFP-ROCM. Patient-reported outcomes: instrument development and selection issues. Value Health. 2007;10:S86–S93. doi: 10.1111/j.1524-4733.2007.00271.x. [DOI] [PubMed] [Google Scholar]
  • 22.Schünemann H.J., Guyatt G.H. Commentary—goodbye M (C) ID! Hello MID, where do you come from? Health Serv Res. 2005;40(2):593–597. doi: 10.1111/j.1475-6773.2005.00374.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Johnston B.C., Thorlund K., Schünemann H.J., et al. Improving the interpretation of quality of life evidence in meta-analyses: the application of minimal important difference units. Health Qual Life Outcomes. 2010;8:116. doi: 10.1186/1477-7525-8-116. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Calvert M., Blazeby J., Altman D.G., Revicki D.A., Moher D., Brundage M.D. Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. JAMA. 2013;309(8):814–822. doi: 10.1001/jama.2013.879. [DOI] [PubMed] [Google Scholar]
  • 25.Diamond I.R., Grant R.C., Feldman B.M., et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–409. doi: 10.1016/j.jclinepi.2013.12.002. [DOI] [PubMed] [Google Scholar]
  • 26.Batra G., Aktaa S., Wallentin L., et al. Methodology for the development of international clinical data standards for common cardiovascular conditions: European Unified Registries for Heart Care Evaluation and Randomised Trials (EuroHeart) Eur Heart J Qual Care Clin Outcomes. 2021;9(2):161–168. doi: 10.1093/ehjqcco/qcab052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Proctor E., Silmere H., Raghavan R., et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health Ment Health Serv Res. 2011;38:65–76. doi: 10.1007/s10488-010-0319-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Ganz P.A., Gotay C.C. Use of patient-reported outcomes in phase III cancer treatment trials: lessons learned and future directions. J Clin Oncol. 2007;25(32):5063–5069. doi: 10.1200/JCO.2007.11.0197. [DOI] [PubMed] [Google Scholar]
  • 29.Wild D., Grove A., Martin M., et al. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: report of the ISPOR task force for translation and cultural adaptation. Value Health. 2005;8(2):94–104. doi: 10.1111/j.1524-4733.2005.04054.x. [DOI] [PubMed] [Google Scholar]
  • 30.Bowling A. Mode of questionnaire administration can have serious effects on data quality. J Public Health. 2005;27(3):281–291. doi: 10.1093/pubmed/fdi031. [DOI] [PubMed] [Google Scholar]
  • 31.Wiering B., de Boer D., Delnoij D. Patient involvement in the development of patient-reported outcome measures: a scoping review. Health Expect. 2017;20(1):11–23. doi: 10.1111/hex.12442. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.US Food and Drug Administration Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims. Fed Regist. 2009;1:65132. [Google Scholar]
  • 33.Aiyegbusi O.L., Cruz Rivera S., Roydhouse J., et al. Recommendations to address respondent burden associated with patient-reported outcome assessment. Nat Med. 2024;30(3):650–659. doi: 10.1038/s41591-024-02827-9. [DOI] [PubMed] [Google Scholar]
  • 34.Reeve B.B., Wyrwich K.W., Wu A.W., et al. ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Qual Life Res. 2013;22(8):1889–1905. doi: 10.1007/s11136-012-0344-y. [DOI] [PubMed] [Google Scholar]
  • 35.Mokkink L.B., Terwee C.B., Patrick D.L., et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737–745. doi: 10.1016/j.jclinepi.2010.02.006. [DOI] [PubMed] [Google Scholar]
  • 36.Stukenborg G., Blackhall L., Harrison J., et al. Cancer patient-reported outcomes assessment using wireless touch screen tablet computers. Qual Life Res. 2014;23:1603–1607. doi: 10.1007/s11136-013-0595-2. [DOI] [PubMed] [Google Scholar]
  • 37.Reading Turchioe M., Grossman L.V., Myers A.C., Baik D., Goyal P., Masterson Creber R.M. Visual analogies, not graphs, increase patients' comprehension of changes in their health status. J Am Med Inform Assoc. 2020;27(5):677–689. doi: 10.1093/jamia/ocz217. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.U.S. The Food and Drug Authority (FDA) Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims: draft guidance. Health Qual Life Outcomes. 2006;4(1):79. doi: 10.1186/1477-7525-4-79. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Cruz Rivera S., Aiyegbusi O.L., Ives J., et al. Ethical considerations for the inclusion of patient-reported outcomes in clinical research: the PRO ethics guidelines. JAMA. 2022;327(19):1910–1919. doi: 10.1001/jama.2022.6421. [DOI] [PubMed] [Google Scholar]
  • 40.Brundage M.D., Crossnohere N.L., O’Donnell J., et al. Listening to the patient voice adds value to cancer clinical trials. J Natl Cancer Inst. 2022;114(10):1323–1332. doi: 10.1093/jnci/djac128. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Aiyegbusi O.L., Roydhouse J., Rivera S.C., et al. Key considerations to reduce or address respondent burden in patient-reported outcome (PRO) data collection. Nat Commun. 2022;13(1):6026. doi: 10.1038/s41467-022-33826-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Calvert M.J., Cruz Rivera S., Retzer A., et al. Patient reported outcome assessment must be inclusive and equitable. Nat Med. 2022;28(6):1120–1124. doi: 10.1038/s41591-022-01781-8. [DOI] [PubMed] [Google Scholar]
  • 43.Ibrahim H., Liu X., Zariffa N., Morris A.D., Denniston A.K. Health data poverty: an assailable barrier to equitable digital health care. Lancet Digit Health. 2021;3(4):e260–e265. doi: 10.1016/S2589-7500(20)30317-4. [DOI] [PubMed] [Google Scholar]
  • 44.Jonas N., Thorn W. Literacy Skills and Family Configurations, OECD Education Working Papers, No. 192. OECD Publishing; Paris: 2018. [DOI] [Google Scholar]
  • 45.Retzer A., Calvert M., Ahmed K., et al. International perspectives on suboptimal patient‐reported outcome trial design and reporting in cancer clinical trials: a qualitative study. Cancer Med. 2021;10(16):5475–5487. doi: 10.1002/cam4.4111. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Dawson J., Doll H., Fitzpatrick R., Jenkinson C., Carr A.J. The routine use of patient reported outcome measures in healthcare settings. BMJ. 2010;340:c186. doi: 10.1136/bmj.c186. [DOI] [PubMed] [Google Scholar]
  • 47.Rutherford C., Costa D., Mercieca-Bebber R., Rice H., Gabb L., King M. Mode of administration does not cause bias in patient-reported outcome results: a meta-analysis. Qual Life Res. 2016;25(3):559–574. doi: 10.1007/s11136-015-1110-8. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary material

mmc1.docx (29.9KB, docx)

Data Availability Statement

All data requests should be submitted to c.p.gale@leeds.ac.uk for consideration.

Articles from Clinical Medicine are provided here courtesy of Royal College of Physicians

RESOURCES