Quality in qualitative evidence: new best practice principles from NICE’s real-world evidence framework

Health technology assessment (HTA) and regulatory frameworks are evolving to prioritize patient-centeredness and real-world impact [1,2]. Recent publications have also highlighted the importance of more fully incorporating stakeholder views, societal and ethical context into the decision-making process [3–5]. In the assessment of new health technologies as well as the creation of clinical guidelines, standard practices for incorporating stakeholder views typically include direct involvement of stakeholders in the deliberation process, such as, on decision-making committees, and/or presentation of evidence specifically generated to better understand stakeholder perspectives [6].

Qualitative data are a key source of this information, capturing people’s beliefs, experiences, attitudes, behavior and interactions. It asks questions about how and why, providing context to decisions and richer information on stakeholder perspectives which are otherwise inaccessible through even the most robust quantitative assessments of clinical and cost–effectiveness.

The delivery of healthcare in the UK has been undergoing a significant transformation, driven by the arrival of innovative medicines such as gene therapies (potentially disruptive to existing treatment pathways), complex digital health technologies including AI-powered applications, and a growing emphasis on personalized care and shared decision-making to navigate available options [7,8]. In such a landscape, qualitative information takes on additional importance to explore how patients and clinicians adapt to, perceive and interact with innovations. Traditional quantitative approaches alone cannot capture the nuances of user experience and acceptance, which are critical to successful adoption of new medicines and technologies.

In this editorial we reflect on the evolving role of qualitative evidence at the National Institute for Health and Care Excellence (NICE), challenges in its current use, and how we might support more effective integration of stakeholder voices and experience into the HTA process.

How is qualitative evidence incorporated at NICE?

People and communities’ involvement and engagement in the development of NICE technology appraisals is a well-established process, outlined in NICE’s public and patient involvement policy [9]. Alongside this approach, patient-based evidence derived from qualitative research methods are typically included from reviews of published qualitative studies, company-generated qualitative data or accompanying submissions from patient organizations. At the level of the primary study, qualitative evidence evaluated at NICE typically includes data collected through interviews, focus groups, or as free-text responses embedded within surveys.

In NICE’s guidelines program, methodologies like qualitative evidence synthesis have an established role for evidence reviews with a qualitative or mixed-methods focus. The appropriate conduct of these reviews is outlined in NICE’s methods manual [10], for example, recommending specific review question formulation frameworks like SPIDER [11] and PerSPEcTiF [12] and quality assessment frameworks like GRADE-CERQual [13] and the Critical Appraisal Skills Programme [14] checklist.

In the context of NICE technology appraisals, qualitative data may support a range of decision-critical elements, including disease burden; acceptability of interventions; feasibility of implementation; positioning of technologies in clinical pathways; subpopulations needing special consideration; patient preferences for treatment delivery; outcomes that matter most and additional perspectives, for example, those of carers and healthcare providers [15].

The role of qualitative evidence for medical devices & digital technologies

For medical devices and digital health technologies, qualitative evidence is valuable to inform how patients engage with a technology and the confidence of healthcare professionals in using these technologies. For example, NICE committees considered the ease of use of the PeritX system (a catheter drainage system) and nurses’ perceived efficiency and confidence using SecurAcath (a catheter securing device) in MTG9 [16] and MTG34 [17], respectively. The PeritX system for managing treatment resistant malignant peritoneal ascites was recommended by NICE, enabling community-based symptom management. In this evaluation, the committee considered semi-structured interviews that explored patient’s experience using the device. This evidence provided valuable insights, ranging from experience of improved symptoms and convenience of use to some negative experiences around visible ascitic fluid and the device making people feel ‘like a patient’.

In accordance with increasing lifecycle focus of assessments of technologies at NICE, ‘early use’ recommendations (a feature of NICE’s ‘Early Value Assessments’) assess health technologies early in their lifecycle enabling quicker access to promising technologies in areas of unmet need, while further evidence is generated. Alongside a conditional recommendation for use, NICE develops accompanying evidence generation plans which prioritize areas of uncertainty highlighted in the first NICE assessment, provide guidance on study designs and data to be collected during the period of use prior to reappraisal. In these evaluations, qualitative evidence gaps have been frequently highlighted as an area needing further evidence. For example, in the Early Value Assessment of digital technologies for managing nonspecific low back pain [18], the committee highlighted a significant evidence gap for how patients perceive treatment effectiveness and technology acceptability as well as the preferred positioning of the technology in the clinical pathway. These considerations have a direct relevance to the successful adoption of a technology, as well as informing the future structure of economic models. In this evaluation, qualitative evidence was considered among the ‘essential’ evidence gaps highlighted at first appraisal, meaning the committee would not be able to make a positive recommendation at reappraisal without the evidence gaps being addressed.

The increasing need to assess artificial intelligence (AI) enhanced technologies such as those recently considered to diagnose fracture on x-ray in urgent care [19] or to triage patients into dermatology services [20] are likely to highlight the importance of such ‘contextual’ considerations. AI-based clinical prediction models that classify risk or diagnose conditions, are often emerging technologies at an immature stage of evidence generation. These come with numerous uncertainties ideally addressed using robust qualitative study designs. For example, compatibility with clinical pathways, explainability (the extent to which patients or user understand what the technology predicts and how it works) and acceptability of adoption for users and patients, including fears around the ‘fairness’ of prediction. While quantitative evidence on pathway efficiency, resource requirement and frequency of engagement are also indicative of effective pathway integration, qualitative approaches can help to explore the ‘why’ behind patient and clinician views, particularly in complex adoption scenarios [21]. Table 1 outlines qualitative evidence gaps highlighted in NICE EVA evaluations, to date.

Table 1. . Qualitative evidence gaps highlighted in published Early Value Assessment topics .

HTE9 (digital therapies for anxiety) HTE8 (digital therapies for depression)	• How people with anxiety interact with the technologies and if they find them acceptable to use • Challenges in using the technologies, such as problems with access, or their effect on waiting times
HTE3 (digital CBT, children and young people)	• Levels of user engagement • Acceptance by children and young people
HTE18 (digital rehab for COPD)	• Patient preference, acceptability and uptake
HTE17 (digital technologies for psychosis)	• Patient and staff experiences of using the technologies • How use varies in particular groups
HTE16 (digital technologies for low back pain)	• Perceptions from people with back pain about whether the technologies help alleviate pain and help them to return to normal daily activity • Satisfaction, engagement and accessibility of the technology, including barriers encountered by people using and continuing to use the technologies
HTE15 (digital technologies for agoraphobia)	• Level of engagement, acceptance and adherence
HTE19 (digital self-management of COPD)	• Engagement and adherence • Patient preference and acceptability
HTE13 (virtual wards)	• Acceptability of care and the carer burden • Experience of healthcare professionals delivering care on virtual wards.
HTE12 (AI for suspected lung cancer)	• Ease of use and acceptability of the software by clinicians is needed • Experiences around implementing the technology and any improvements in the delivery of diagnostic services, particularly around accuracy of the technology in identifying abnormalities, appropriateness of image triage and the impact on speed of review and reporting.
HTE11 (AI to aid contouring)	• Experiences and opinions of healthcare professionals on clinical acceptability of contours provided by the technology.

AI: Artificial intelligence; COPD: Chronic obstructive pulmonary disease.

The role of qualitative evidence for innovative medicines

In the context of rare and ultra-rare diseases, inherent limitations in evidence due to heterogenous patient groups of limited size, often result in significant uncertainties in the clinical and cost–effectiveness of a technology. Therefore, qualitative evidence can provide information on aspects not fully captured by available quantitative data [22,23]. For example, in NICE technology appraisal HTS19 of elosulfase alfa for treating mucopolysaccharidosis type 4A (MPS4A) [24], qualitative evidence, submitted by a patient organization, was a source of evidence used to support claims of clinical effectiveness. MPS4A is a rare, progressive condition affecting multiple organ systems. Qualitative evidence highlighted the experienced benefits of improved endurance and increased energy for most patients taking this treatment. It also highlighted other outcomes such as improved life expectations, better sleep, disease stability and ability to plan for the future.

Patient input and qualitative information can also reveal uncertainties in the pivotal trial evidence by highlighting aspects that were not captured in those trials, as seen in the NICE technology appraisal HST27 of afamelanotide for treating erythropoietic protoporphyria [25]. Erythropoietic protoporphyria is a rare genetic condition causing painful skin reactions that are triggered on exposure to sunlight or artificial light, with profound effects on quality of life. Qualitative information from patient testimonies provided important insights into factors influencing light exposure time and conditioned light avoidance behavior which impacted interpretation of trial evidence. Given the challenges in quantifying health benefits associated with afamelnotide due to lack of scientifically robust instruments which measure change in quality of life and limitations in trial evidence, patient and expert testimonies provided further information to support decision making. However, the need for qualitative evidence to be collected systematically and analyzed using standard techniques was highlighted as potentially limiting the influence of qualitative data in this case.

Robust qualitative methods can support more influential uses

As described above, a common concern when evaluating qualitative patient-based evidence is lack of systematic evidence generation or inconsistent adherence to quality standards or frameworks for qualitative data collection methods and analysis [26,27]. A review of the qualitative evidence in NICE’s technology appraisal program submissions from October 2019 to September 2021 highlighted very limited reporting of formal methods of qualitative data collection and analysis despite patient-based evidence being common in submissions [27]. In this review, out of 83 submissions which included patient-based evidence, 14 described systematic (methods-based) data collection and only six employed formal qualitative data analysis methods.

Due to increasing complexity in the healthcare system, there is a likely need for further triangulation of evidence from different sources, both quantitative and qualitative. However, for evidence to be considered robust and trustworthy for decision making, adherence to quality standards is critical to ensure robust data collection and analysis, as well as transparent reporting.

NICE’s real-world evidence (RWE) framework [28] was published in 2022 and aims to signal ‘what good looks like’ to encourage greater and better use of decision-grade RWE, as well as more consistent appraisal of that evidence. Although qualitative data were initially included within the RWE framework’s definition of ‘real-world data’, stakeholders highlighted the lack of guidance on developing good quality qualitative evidence during consultation.

As a living document, the RWE framework is intended to evolve to ensure that it remains relevant and up to date. Therefore, in January 2025, the framework was updated to include recommendations for the conduct and reporting of qualitative research. These recommendations focus on qualitative studies that collect data through interviews, focus groups or as free text response in surveys, which have been the most common types of qualitative data collection methods evaluated in NICE submissions. Recommendations support a structured and principled approach to qualitative research through: development of a clear research question and study protocol; ensuring ethics and data protection; choosing an appropriate approach to sampling, data collection and analysis; clear reporting and robust critical appraisal. Key recommendations in the new appendix are described in Table 2.

Table 2. . Recommendations from NICE’s real-world evidence framework (Appendix 4) .

Developing the research question and study protocol	• Provide a rationale for why a qualitative approach is considered the best way to meet the research aims. • Use available question formulation frameworks to clearly specify key components of the research question (e.g., PICo, SPIDER, and SPICE). • Design a detailed study protocol that covers: the research question and aims; rationale for a qualitative approach; the theoretical framework underpinning the qualitative approach; study design; data collection methods; analytical approach and ethical and regulatory compliance.
Study ethics and data protection	• Provide information on attained ethical approvals and patient consent, including any adaptations for vulnerable populations. • Collect, store, process and delete collected data in accordance with current data protection laws, with safeguards and sufficient transparency information provided to participants.
Sampling strategy	• Tailor the sampling strategy to the outlined research aims, e.g., purposive sampling, homogenous or maximum variation sampling. • Clearly describe:   ○ Study recruitment strategy, including any incentives provided for participation   ○ Number of participants in the final study and reasons for stopping recruitment into the study (e.g., data saturation or other)   ○ All relevant characteristics of the included participants   ○ Where available, information on the characteristics of those who chose not to participate in the study and reasons for nonparticipation.
Study design	• Using surveys to collect qualitative data may not allow for structured and in-depth exploration of a given topic but offers a feasible way to collect information from a broad range of individuals. • For in-depth exploration, focus groups and interviews are the preferred methods.   ○ Focus groups can effectively explore collective perspectives on a topic, such as, exploring consensus and disagreement within or across groups of people.   ○ Individual interviews are particularly suited when exploring personal or sensitive themes.
Data collection	• Clearly describe the data collection method used (for example, semi-structured interviews, focus groups and open-ended questions in surveys) and the setting. Justify the selected method in relation to its ability to support the research aims. • Specify the mode of data collection, such as whether collected as audio or video recording or any other method and if repeat interviews were undertaken. • Consider the content, tone, order and phrasing of questions asked. Reflexivity (that is, considering how a researcher’s own values and preferences may impact the findings of the research) should be maintained throughout. • Include members of the target population (for example, patients or clinicians) and subject matter experts in the design of interview questions, focus group guide or survey. • Pilot questions within a subset of the target sample to help ascertain the questions’ validity in meeting research aims and anticipate possible miscommunication. • Provide materials relating to the data collection methods as part of the study appendix or on request, e.g., the interview questions, interview schedule and interview guide or full survey.
Data analysis	• Provide a full description of the approach used to develop code lists, perform thematic analysis and assess research reflexivity in the study report. • Substantiate reported themes with the data collected, for example with verbatim extracts, text excerpts or field notes. • Enhance the credibility of thematic analysis through use of multiple researchers or independent coders. • Investigate ‘negative’ cases that challenge the views of the majority, or conflict with previous findings, these may be insightful or support further targeted data collection. • Statements of researcher reflexivity are generally recommended as good practice.
Reporting and assessment of qualitative studies	• The COREQ reporting tool provides guidance for reporting primary qualitative studies collected through interviews and focus groups. Reporting standards can be similarly applied for qualitative data that is collected as free text embedded within a quantitative survey. • Triangulate findings with other data sources e.g., from quantitative studies. • The Critical Appraisal Skills Programme (CASP) checklist is recommended to assess the quality of previously conducted primary qualitative studies.

Qualitative research in the context of NICE’s People & Communities Strategy

There are several challenges that will need to be navigated for successful implementation of such best practice recommendations by patient organizations and other stakeholders. For example, while qualitative evidence might be well placed for understanding rare diseases (as in the example above), in reality, rare disease organizations in the UK are often small with a limited number of staff and resources. Furthermore, organizations need to have confidence that any qualitative evidence produced will be taken into consideration by NICE committees and impact the decision-making processes; otherwise, any efforts to conduct timely, and sometimes costly, qualitative research may not be justified. Consequently, NICE will aim to provide support and guidance to organizations to ensure they are clear on when qualitative evidence is most needed and to be notified in advance to ensure successful implementation of the qualitative evidence generation.

One successful model of working has been collaborative research led by NICE with other agencies and qualitative experts, such as the qualitative work on improving implementation of NICE recommendations on end-of-life care [29,30]. This provided patient, family, and healthcare professional perspectives to understand implementation barriers and quality indicators to support recommendations for effective end of life care delivery. Another example includes the ongoing Early Value Assessment of digital therapy for chronic tic disorders and Tourette syndrome [31] where a patient survey developed in partnership with Tourettes Action explored many aspects of experience of care. This survey received over 1500 responses and the summarized themes provided patient perspective on treatment options and acceptance of digital technologies in the appraisal process.

In addition, NICE’s working alongside people and communities strategy has been co-produced with people and communities who have lived experience of working with NICE across our different programs as well as key stakeholders from voluntary and community sector [32]. The strategy outlines NICE’s vision “To have a best-practice approach to involvement and engagement, to improve the impact of our guidance and ensure the best care for people and communities” and outlines an evolving approach to the more effective incorporation of the stakeholder voice into NICE’s decision-making processes (Box 1).

Box 1. The five aims of NICE’s People and Communities Strategy.

• Impactful involvement and engagement: involve the right people, at the right time, in the right way.

• Tailored approaches: tailor the way in which people and our communities can engage with NICE.

• Foster an innovative culture: test with, and learn from, new and innovative ways to work alongside people and communities.

• Productive partnerships: transform our approach and ways of working with people and communities.

• Focus on people first: embed an ethos of curiosity for involvement and engagement across NICE.

Conclusion

The ability of qualitative evidence to provide nuanced understanding of contextual issues, including patient and carers lived experience and perspectives, has valuably supplemented quantitative measures of clinical and cost effectiveness in NICE guidance. The importance of this approach is set to increase with greater emphasis on patient-centeredness in HTA decision making and the use of multidimensional approaches to evaluate new innovative health technologies. While concerns around the quality and methods used to generate qualitative evidence may hamper the influence of such evidence, the most recent RWE framework update aims to provide guidance to encourage well developed qualitative studies which can provide rich and credible insights. NICE’s vision is to be a leader in best-practice approaches to involvement and engagement of stakeholders, in order to improve the impact of our guidance and ensure the best care for people and communities. This will be supported by the development of a three-year strategy for involvement and engagement co-developed with people and communities who have lived experience of working with NICE as well as key stakeholders from voluntary and community sector [32].