Abstract
Background
Patient safety is a critical aspect of healthcare, with patients often being the first to notice safety concerns. However, traditional reporting mechanisms have limitations, and many patients may not report safety issues due to fear of repercussions or lack of clarity in existing systems. There is a growing need for tools that enable patients to report safety concerns easily and effectively. This study aimed to undertake preliminary development and cognitive testing of a Patient Reported Safety Concern Tool, designed to capture a broad range of patient safety issues across various healthcare settings that could enhance quality of care and foster continuous safety improvement.
Methods
A two-phase, qualitative study was conducted in Wales, virtually through online platforms (Zoom or MS Teams) between January and September 2023. In Phase 1, 26 adults (aged 25–54, 23.1% female) participated in three online focus groups recruited through purposive sampling. In Phase 2, 10 additional participants (aged 25–84, 70% female) were purposively sampled for online cognitive interviews. Participants were eligible if they were 18+ and had accessed healthcare within the past 6 months. Individuals with professional expertise in patient safety were excluded. Data were analysed using qualitative content analysis in NVivo 12. A coding framework was developed inductively and iteratively refined.
Results
Focus group participants preferred the term ‘safety concern’ over terms like ‘incident’ or ‘event,’ as it was more relatable and inclusive of both physical and emotional harm. Feedback led to refinements in item clarity, such as extending the recall period to 6 months and rewording prompts for detail. Cognitive interviews confirmed that version 2.0 was easy to understand and relevant. Minor adjustments were made, including extending the recall period to 12 months and adding ‘ambulance services’ as a setting. The final version, 3.0, demonstrated high content and face validity, with participants expressing a strong willingness to complete the tool if distributed routinely.
Conclusion
The Patient Reported Safety Concern Tool was co-developed with public participants and refined through cognitive testing, demonstrating strong content and face validity. Participants felt confident the tool would help identify safety concerns not captured through conventional systems. Future work will focus on validating the tool in wider populations, understanding barriers to completion, and integrating it into existing patient safety learning systems to inform actionable safety improvements.
Keywords: patient safety, patient-reported concerns, tool development, focus groups, cognitive interviews, healthcare improvement
Introduction
Identifying and analysing patient safety incidents is critical for continuous learning and improvement in healthcare systems, as emphasized by the World Health Organization’s (WHO) Global Patient Safety Action Plan 2021–30 [1]. Many countries have mandated the routine capture of such incidents through national quality and safety delivery plans, ensuring data analysis drives meaningful organizational learning [2]. While patients’ roles in improving safety are increasingly recognized [3, 4], their insights are often not integrated into healthcare systems as effectively as organizationally reported data. To address this gap, it is essential to develop mechanisms that make patient-reported data both accessible for patients and actionable for organizations, moving beyond mere incident metrics to meaningful safety insights.
Patients provide unique perspectives on safety incidents, including near-misses or concerns not captured in organizational reporting systems or medical records [5–8]. This complementary role is vital for a holistic understanding of harm in healthcare, as neither organizational reports nor patient accounts alone represent a ‘gold standard’ [8]. Patient-reported data can serve multiple purposes, including enhancing accountability, highlighting systemic risks, and fostering bilateral learning between patients and healthcare organizations [3]. However, achieving these goals requires tools that effectively capture and translate patient-reported safety concerns into actionable insights.
The COVID-19 pandemic provided an unprecedented opportunity to explore patient-reported safety concerns across diverse healthcare settings. During the pandemic, a seven-item Patient Reported Safety Concern Tool (version 1.0) was developed by a working group comprising patients, health services researchers, and clinicians. This tool was designed to assess the occurrence, nature, seriousness, avoidability, and impact of safety concerns. It was incorporated into the UK COVID-19 Public Experiences (COPE) study [9], a longitudinal mixed-methods study conducted in a UK community setting. The COPE study aimed to identify determinants of health behaviour over the course of the pandemic, guided by the Capability, Opportunity, Motivation—Behaviour model. A total of 13 604 participants completed the COPE survey across three time points, with 1363 reporting patient safety concerns [10]. This demonstrates the tool’s potential to capture real-time safety issues during a rapidly evolving public health crisis.
The tool was subsequently used in Evaluation of the Shielding Initiative in Wales (EVITE Immunity) study [11], which examined the impact of a UK government policy introduced in 2020 advising clinically extremely vulnerable individuals to self-isolate for 12–16 weeks. As the Shielding Initiative was implemented without prior evidence of its effects on health or behaviour, the tool was employed to capture patient-reported safety concerns arising from this intervention—such as unintended consequences related to disrupted access to healthcare and essential services, as well as emotional and psychological distress. While these applications demonstrated the tool’s utility, its expedited development process necessitated further testing and refinement to enhance its comprehensiveness, clarity, and feasibility. These insights informed the current study’s iterative development process.
Patient-reported outcome and experience measures (PROMs and PREMs) offer valuable frameworks for capturing safety concerns, but their integration into learning healthcare systems remains limited [12]. Despite promising feasibility results for the validated primary care patient measure of safety (PC PMOS) [13], we were not aware at the time of any validated multisetting patient-reported safety measures, and none that captured free-text qualitative data. Addressing this gap, the current study aims to undertake preliminary development and validation of the Patient Reported Safety Concern Tool (version 1.0), ensuring its content and face validity and its potential for embedding within organizational safety learning systems. This tool (version 1.0) was rapidly adapted from the Patient Incident Reporting Tool, originally developed for research into patient involvement in safety within the hospital setting [7, 14]. Following initial scoping review of literature of existing patient-reported safety tools, our objectives were to (i) describe what ‘patient safety’ means to the public, (ii) explore content validity of the preliminary tool, (iii) determine face validity with the target population, and (iv) produce a revised version of the Patient Safety Concern Tool (version 3.0) for further testing, development, and validation in diverse patient populations and healthcare settings, and consultation with key stakeholders in learning organizations to explore how it can be embedded into the learning systems.
Methods
The methods were informed by COSMIN guidelines for developing patient-reported measures [15] and conducted in two phases. This qualitative study took place virtually in Wales between January and September 2023, using online platforms (Zoom or MS Teams). Phase 1 focused on content validation and refinement to create version 2.0, while Phase 2 involved cognitive testing to assess face validity and refine the tool further, resulting in version 3.0.
Participants
Eligible participants were adults (18+) residing in Wales who had accessed healthcare services within 6 months. Exclusion criteria included professional expertise in patient safety or communication impairments precluding participation. Recruitment targeted diverse demographics through social media, charities (e.g. The Patients Association), and a study webpage, which provided detailed information and a screening questionnaire. In Phase 1, 47 individuals responded to the screening questionnaire, and 26 were purposively selected to ensure diversity in age, gender, ethnicity, and educational attainment. In Phase 2, 28 responded, and 10 were purposively sampled using the same criteria. The screening questionnaire confirmed eligibility, captured demographic information, and supported purposive sampling. Participants in Phase 1 (focus groups) and Phase 2 (cognitive interviews) were from separate cohorts. Focus groups lasted ∼90 min, while cognitive interviews ranged from 45 to 60 min. Eligible participants were contacted for consent and scheduling of focus groups or cognitive interviews.
Data collection
Phase 1: Focus groups—item refinement and development
Conducted online via Microsoft Teams between January and March 2023, focus groups were video recorded and facilitated by researchers (A.S./N.J.W.) using a semi-structured topic guide. The topic guide was co-developed with a public partner, informed by prior literature on co-production [16], patient engagement and safety reporting [17, 18], and pilot tested with two public contributors unaffiliated with the study to refine question clarity, structure, and flow. Discussions explored participants’ understanding of patient safety and their feedback on version 1.0 of the tool, previously used in the COPE [9] and EVITE Immunity [11] studies for comprehension, completeness, usefulness, and feasibility. Participants also assessed alternative items from other published tools [7, 19–21] to identify gaps and evaluate relevance. Researchers practiced reflexivity by maintaining reflective notes and meeting regularly to consider how their perspectives might shape data collection and interpretation. Feedback informed revisions, resulting in version 2.0 of the tool (Table 3).
Table 3.
Changes to the patient safety concern tool following Phase 1 focus groups and Phase 2 cognitive interviews
| Item No. | Version 1.0 Item and response options | Phase 1 focus group suggested changes | Version 2.0 Item and response options | Phase 2 cognitive interview suggested changes | Version 3.0 Item and response options |
|---|---|---|---|---|---|
| 1 |
|
|
|
|
|
| 2 |
|
The original response options were selected as the survey was specific to the pandemic period. Discussions focused on preferred maximum timeframe for recalling incidents. Participants recommended 6 months as the maximum reference period, due to recall bias. |
|
In line with changes to Item 1, recall period changed to 12 months. |
|
| 3 |
|
No changes required. |
|
No changes required. |
|
| 4 |
|
|
|
|
|
| 5 |
|
Choices are updated to match the postpandemic period (response option ‘concerns specific to the coronavirus outbreak…’ was removed). |
|
No changes required. |
|
| 6 |
|
|
|
|
|
| 7 | Question: Do you think it would have been possible to have stopped this safety concern from happening?
|
No changes required. | Question: Do you think it would have been possible to have stopped this safety concern from happening?
|
|
Do you think anything could have been done to stop it from happening?
|
| 8 | N/A | This item on patients’ experience of various forms of harm was adapted from a study by Ricci-Cabello, Avery [25], as acknowledged by certain participants who believed it would effectively encompass their first-hand experiences related to safety concerns. |
|
In line with changes to Item 1, recall period changed to 12 months. |
|
Private healthcare services were included in version 1.0 as this was within remit for the COPE study. Subsequent versions were amended to only include NHS settings; the target setting for the tool.
Phase 2: Cognitive interviews
From July to September 2023, individual cognitive interviews using a ‘think aloud’ method were conducted online using Microsoft Teams or Zoom. Researchers (A.B./A.T.B.), both trained in qualitative methods, employed a concurrent ‘think aloud’ approach [22], in which participants verbalized their thought processes as they read and responded to each item in version 2.0. A semi-structured interview script (co-developed [16] with a public partner, informed by best practices in cognitive interviewing [23]) was used to guide the interviews and pilot tested with two public contributors not involved in the main study. The script ensured consistency across interviews while allowing flexibility to probe emerging insights. Interviews focused on assessing face validity, including item clarity, comprehensibility, and overall tool format. Researchers maintained detailed field notes and practiced reflexivity throughout, discussing their own interpretations and assumptions during analysis. Participant feedback guided refinements incorporated into version 3.0 of the tool.
Data analysis
Focus groups
Focus group transcripts were transcribed verbatim and analysed using qualitative content analysis [24] in NVivo 12. An inductive approach was taken, with initial open coding applied to capture participants’ own language and interpretations. The coding framework was developed by A.S. through repeated readings of one transcript and expanded to include emergent codes based on recurring patterns across groups. Codes were then grouped into categories related to patient safety definitions, item comprehension, and feasibility. This framework was then reviewed and refined through discussion with N.J.W. to ensure alignment with the study’s aims. Transcripts were reviewed iteratively, with the refined coding framework applied across all transcripts to explore how themes developed and diverged between focus groups. Regular team meetings supported ongoing interpretation, reflexivity, and rigour in analysis.
Cognitive interviews
Interview data underwent similar content analysis in NVivo 12 to assess the face validity of version 2.0. Researchers (A.S./N.J.W.) identified themes related to item clarity and usability. Emerging themes were validated through discussion and incorporated into version 3.0.
Results
A summary of participants, tool refinements, and versions are available in Fig. 1.
Figure 1.
Summary of content validity (Phase 1: focus groups) and face validity (Phase 2: cognitive interviews) stages of the Patient Reported Safety Concern Tool
Participants
Demographic characteristics of focus group and cognitive interview participants are presented in Table 1.
Table 1.
Demographic characteristics of focus group (n = 26) and cognitive testing (n = 10) participants
| Focus groups | Cognitive testing | |
|---|---|---|
| Number (%) | Number (%) | |
| Ethnicity | ||
| Welsh, English, Scottish, Northern Irish, or British | 7 (26.9) | 4 (40) |
| Black, African, Caribbean, or Black British | 6 (23.1) | 1 (10) |
| White and Black African/Caribbean | 8 (30.8) | 1 (10) |
| African/Caribbean | 4 (15.4) | 1 (10) |
| Roma | 1 (3.8) | |
| Arab | 1 (10) | |
| Indian | 1 (10) | |
| Bangladeshi | 1 (10) | |
| Age (years) | ||
| 18–24 | 4 (15.4) | |
| 25–34 | 16 (61.5) | 2 (20) |
| 35–44 | 5 (19.2) | 5 (50) |
| 45–54 | 1 (3.8) | |
| 55–64 | 1 (10) | |
| 65–74 | 1 (10) | |
| 75–84 | 1 (10) | |
| Gender | ||
| Male | 17 (65.4) | 2 (20) |
| Female | 6 (23.1) | 7 (70) |
| Prefer not to say | 3 (11.5) | 1 (10) |
| Highest educational level | ||
| University level qualification | 16 (61.5) | 7 (70) |
| School or college leaver qualification | 9 (34.6) | 3 (30) |
| No formal qualifications | 1 (3.8) |
Phase 1: focus groups—item refinement and development
There were 26 participants across three focus groups (n = 9, n = 9, n = 8). Ages ranged from 25–54 years (mean = 32) and 23.1% were female; three participants chose not to disclose their gender. Focus groups were 60–90 min in duration.
Phase 2: Cognitive interviews
There were 10 cognitive interview participants. Ages ranged from 25–84 years (mean = 46.5) and 70% were female; one person preferred not to disclose their gender. Interview length ranged between 30 and 60 min.
Key findings
Phase 1: Focus groups
Participants shared their views on what ‘safety concerns’ meant to them (Table 2) across the three focus groups (‘F’ indicates focus group number and ‘P’ indicates participant identifier). There was a range of responses from ‘how well patients are taken care of’ [F1P3, male] to ‘avoidance of unintended or unexpected harm to people’ [F1P8, male]. Key themes emerging across the responses included being ‘cared’ for, ‘protecting’ patients, patient ‘wellbeing’, preventing ‘harm’, ‘fairness’, and ‘equity’. All participants felt it was an essential part of healthcare, e.g. ‘patient safety is kind of very important and shouldn’t be neglected in any way, because it ensures the general care and wellbeing of a patient’ [F1P2, male].
Table 2.
What ‘patient safety’ means to the focus group participants
| What ‘patient safety’ means (direct quotes from participants) | Focus group/participant/gender |
|---|---|
| We’re looking at the general overview of what it means to care for someone who is sick | F1P2, male |
| Patient safety just means a way of preventing harm to the patient | F1P4, male |
| Avoidance of unintended or unexpected harm to people during the provision of their healthcare | F1P8, male |
| Operation safety is an aspect where [the] operation experiences a very good outcome, there is equity in our system, and it's being treated fairly. The patient has self-confidence on every strategy he has confided in the facility in which he found himself…this makes him or her feel safe… | F2P19, male |
| Patient safety is the prevention of healthcare errors and the elimination of the admission and mitigation of patients’ injury caused by health error…so when a patient is given proper treatment, and the treatment also goes along with the patient’s illnesses | F2P10, male |
| Patient safety means the system used to help the patients…keeping them safe from harm, other possible complexities that come up within the healthcare | F2P11, male |
| It can be emotional safety and also physical safety | F2P13, female |
| Keeping patients safe from health problems | F2P14, female |
| Patient safety means like the general wellbeing of patients and the communication between patient and healthcare workers | F2P16, male |
| How we can make sure our patients have a good living and are able to really communicate very well with our healthcare workers to be able to get efficient and sustainable healthcare services | F2P17, male |
| For me, as a person, and as for my family, I feel safe in any medical facility, when I realize that I’ve been treated fairly, and everybody there makes me feel okay, and everything is accurate on time, there is real reliability | F2P19, male |
| I think safety concern means the care and the well-being of patients | F3P23, male |
| To me safety concern means how the patient has been cared for, their well-being… it really has to do with your safety and them being in a safe space in a healthy, safe, safe space | F3P20, male |
In version 1.0, the term ‘safety concern’ was chosen over the term ‘incident’, commonly used in clinical safety processes, to describe patient safety events. Participants reviewed alternatives (e.g. incident, event, harm, episode) and agreed that ‘safety concern’ was the most suitable. They found it understandable, easy to relate to, and all-encompassing. Some participants noted that they liked the term ‘concern’ because it had a bidimensional meaning, indicating that the healthcare system cared and was also concerned about patients and their safety.
Patient safety concern means a lot to me, because it shows that there are some concerns that the NHS is having for the patient [F3P20, male]
Other participants shared COVID-19 pandemic-specific unsafe experiences, including staff shortages, poor hygiene, inadequate infection control, bias, and racism, noting that the term ‘concern’ accurately captured such experiences. Further discussions emphasized the clarity and comprehensiveness of tool items. Participants felt that the items used lay terminology that were easy to understand and complete, with refinements detailed in Table 3.
For Item 2 (when the safety concerned happened), Phase 1 participants highlighted that an ideal recall interval depends on individual circumstances, concern severity, and personal significance. A 6-month timeframe was deemed optimal for most, minimizing recall bias. Item 3 (severity) was universally regarded as clear and effective, with participants noting its utility in helping organizations prioritize actionable concerns.
Response options in Item 4 (setting) and Item 5 (what the concern related to) were seen as comprehensive, encompassing most healthcare settings and issues. However, participants recommended removing pandemic-specific options to ensure long-term relevance. The multiresponse format of Item 4 raised concerns about linking issues to specific settings. To streamline analysis and enhance organizational utility, participants suggested revising this to a single-response format, focusing on the most relevant setting.
All participants valued the additional information that would be captured in free text in Item 6 (asking them to describe the concern), e.g.:
This question will help expressing more information in the way it’s supposed to, it’s more like a space for vital information [F2P13, female]
However, the phrasing ‘in a few sentences’ might indicate that a thorough and detailed description of the concern was unnecessary or not desired. As a result, patients might only provide a superficial overview, which could lack the depth needed to adequately understand the concern and be useful for organizational learning. Therefore, Item 6 was rephrased to ‘please tell us a bit more about what happened’.
During the focus groups, we presented alternative versions of items that were included in version 1.0 and additional items that were not included (drawn from the literature review). Participants noted that the current survey does not make it clear that patient safety concerns could be physical and emotional in nature:
But it didn't really specify the kind of harm because, when you talk about harm, it could be physical harm. It could be emotional harm. So, I believe on something should be added to it [F1P1, male]
To address this, an additional item identified in our literature review [25] was included in version 2.0 (Item 8, Table 3).
Overall, participants reported that they would be comfortable completing this survey routinely, and the majority preference for distribution was an online survey sent via email; although some participants recommended raising awareness of the survey in local community settings (e.g. supermarkets, barber shops, religious venues).
Phase 2: Interviews—cognitive testing
All participants found the majority of version 2.0 items easy to understand with only minimal changes suggested. Results from the ‘Think Aloud’ cognitive interviews are presented in Supplementary Appendix S1, which shows the number of participants who understood the main item statements and response options. The suggested changes were minor and included sentence shortening, enhancing clarity, and extending the recall period from 6 to 12 months to align with routine National Health Service (NHS) or General Practice (GP) follow-up intervals (Table 3). Amendments were made to Items 1, 2, and 8 to reflect the extended 12-month recall period. ‘Ambulance services’ was added as an additional response option for Item 4. The wording of Items 6 and 7 and one response option from Item 8 were identified as lengthy and overly complex; these were changed to enhance clarity. Items 3 and 5 remained unaltered. The refined tool (version 3.0) integrating these recommendations is presented in Table 3.
Discussion
Statement of principal findings
This study presents the refinement and preliminary testing of a Patient Reported Safety Concern Tool (version 3.0), with a focus on its content and face validity. Through insights gained from focus groups and cognitive interviews, we enhanced the tool’s clarity, relevance, and comprehensibility. Participants affirmed the tool’s usability, demonstrating a willingness to complete it to contribute to organizational learning. They viewed patient safety as an integral part of healthcare delivery rather than a distinct activity, emphasizing the interplay between safety processes and care provision. Key findings from Phase 1 (focus groups) included broad consensus on the relevance and need for the tool, but there were disagreements regarding the clarity and wording of specific items. Some participants emphasized the need for more emotional or relational aspects of safety to be captured, while others focused on procedural and clinical safety concerns. Differences sometimes emerged within focus groups, reflecting variation in individual healthcare experiences, literacy levels, and expectations [26]. These inconsistencies were addressed by refining item wording for clarity and adding an item for generalizability. In Phase 2 (cognitive interviews), participants generally found version 2.0 clear and usable but flagged areas where phrasing could still be improved.
Participants expressed motivation to use the tool beyond merely reporting concerns, recognizing its potential to inform safety improvements. However, this motivation was contextualized by the study’s research setting, necessitating further investigation into real-world implementation across diverse populations.
Our findings also highlighted the concept of ‘voiceable concerns’, with participants expressing uncertainty about when and how to report certain experiences. This subjective process (determining whether an event was significant enough to warrant reporting) suggests that supportive resources and clear definitions will be critical for successful implementation.
Interpretation within the context of the wider literature
This study also offers important insights into how the public conceptualizes ‘patient safety’. Participants in our study described safety as a multidimensional construct that encompassed physical, emotional, interpersonal, and systemic elements. Safety was not seen merely as the absence of harm, but as the presence of care, respect, clear communication, fairness, and confidence in health systems. This broad conceptualization aligns with earlier findings that patients frequently embed safety within their wider experience of care [17, 27–29]. They perceive safety as relational and embedded in trust, timeliness, and reliability, rather than only defined by technical outcomes.
This contrasts with traditional clinical definitions of patient safety [30], which tend to focus on preventable adverse events and system failures. Our findings add nuance to the literature by reaffirming that public understandings are shaped by lived experience and a holistic view of care, including emotional safety and equity. Some participants also showed awareness of systemic aspects of safety, suggesting that the public can engage with both micro- (individual care) and macrolevel (organizational and structural) dimensions of safety when given the appropriate tools and language. These perspectives support the design of safety concern tools that reflect the full breadth of what patients experience and value.
This study supports existing evidence that patient-reported data can provide unique insights into safety concerns not captured by organizational systems [6, 8]. The findings align with research highlighting patient-reported barriers to engagement, such as fear of repercussions and power imbalances, which can discourage reporting even in urgent safety scenarios [31–33]. In this context, the anonymous nature of the Patient-Reported Safety Concern Tool may help to mitigate such barriers, supporting existing evidence that anonymity can increase willingness to report safety concerns [34, 35].
Addressing these barriers requires an understanding of the broader factors influencing patient engagement, including task complexity, healthcare professional attitudes, and contextual factors such as busy work environments or perceived importance of the concern [32]. In addition to enhancing the tool itself, leveraging organizational processes known to support engagement with patient feedback—such as accreditation requirements, quality improvement cycles, and existing safety monitoring systems—could maximize the tool’s impact. Prior research has shown that integrating patient feedback into structured safety systems, like the PRASE intervention, can improve staff responsiveness and organizational learning [36–38]. Embedding the Patient-Reported Safety Concern Tool within these established processes may therefore offer a practical and sustainable route for implementation.
The challenge of distinguishing between ‘negative experiences’ and ‘voiceable concerns’ is also well-documented [39]. Healthcare workers themselves often struggle with subjective interpretations of safety concerns, suggesting the need for clear guidance and support for patients navigating similar decisions [40]. Integrating frameworks such as Systems Engineering Initiative for Patient Safety extended model (SEIPS 2.0) and the Healthcare Complaints Analysis Tool can help systematize the analysis of patient-reported data, providing actionable insights for organizational learning.
Strengths and limitations
A key strength of this study is the iterative development process, which ensured that the tool aligns with patient perspectives and enhances its relevance and accessibility. The inclusion of focus groups and cognitive interviews with a diverse sample further strengthened the tool’s applicability across different healthcare settings. Additionally, this work contributes to the growing body of research [41–43] that underscores the importance of patient-reported data in identifying safety concerns that may otherwise go unnoticed by organizational systems. The anonymous nature of the Patient-Reported Safety Concern Tool may also help mitigate commonly reported barriers to patient engagement in safety reporting, such as fear of blame, repercussions, or power imbalances [31–33]. This aligns with previous findings suggesting that anonymity increases patient or healthcare professionals’ willingness to raise safety concerns [34, 35].
The exploration of ‘voiceable concerns’ during tool refinement is an additional strength, as it provides insight into how subjective decision-making processes can influence patient reporting behaviour and may inform targeted support strategies to facilitate reporting. However, some limitations must be acknowledged. The study was conducted with highly motivated participants in the recovery phase of the disruptive pandemic, who were engaged in the testing and refining the tool. This could have inflated willingness and perceived ease of tool completion. Further work is required to explore the reality of patients completing the tool outside of the research context. Furthermore, broader testing with ‘less heard’ patients, such as individuals with low literacy or limited digital access, is necessary to ensure that the tool is inclusive and equitable. Finally, the long-term utility and integration of the tool into routine safety learning systems remain to be demonstrated.
Implications for policy, practice, and research
Embedding patient-reported tools into routine care aligns with ‘policy’ trends towards person-centred healthcare systems [44] and quality frameworks emphasizing timely, safe, and effective care [2]. Policymakers should prioritize the integration of such tools into existing safety strategies to complement organizational reporting systems and ensure alignment with overarching healthcare goals. Embedding patient-reported tools into routine care aligns with current policy trends towards person-centred healthcare systems and quality frameworks that emphasize timely, safe, and effective care.
From a ‘practice perspective’, successful implementation of the tool requires careful consideration of several factors. First, patient burden must be minimized by addressing common barriers, such as time constraints, complexity, and potentially unclear utility of the data collected. It is essential to identify preferred modes of delivery, timing, and the level of support required for effective tool use. Second, inclusive design must accommodate diverse populations by ensuring the tool is accessible to individuals with varying literacy levels, digital access, and cultural perspectives. Tailored approaches for underrepresented groups will be critical in achieving equitable data collection. Third, the tool’s data must integrate seamlessly into existing safety reporting mechanisms, aligning with established terminology and workflows [45]. Collaboration with learning organizations and safety teams will be essential to achieve this goal.
‘Future research’ should focus on further validation, including construct validity, relationship to incident reporting (distinguished from ‘concerns’), reliability, and validation with underrepresented groups to ensure it is generalizable and inclusive. Additional studies are needed to investigate how patients distinguish between negative experiences and safety concerns, which will inform guidance and support for their reporting decisions. Drawing on findings from studies of PROMs and PREMs [46], future development should also prioritize reducing response burden while ensuring meaningful data collection. Leveraging artificial intelligence and deep learning methods could further enhance the scalability and efficiency of analysing patient-reported data, enabling the identification of actionable insights on a larger scale.
Implications for routine safety learning systems
Developing a valid and feasible tool is only the first step; its successful implementation requires that the data collected move beyond basic metrics, such as incident frequency and severity, to provide actionable insights into the causes and prevention of safety concerns. Tools like the PMOS [47] and PC PMOS [7] have demonstrated how patient-reported data can help identify latent conditions and error-producing factors that contribute to safety incidents in both secondary and primary care settings. These approaches may offer valuable lessons for enhancing the depth and utility of the Patient-Reported Safety Concern Tool. Collaborations with learning teams in NHS Wales and other organizations across the UK will be instrumental in exploring how the tool could be used to sit alongside and complement current organizational patient safety learning and integrate into existing strategies and processes [(e.g. routine patient experience PROMs/PREMS, ongoing work with Patient Advice and Liaison Service, Safety and Learning Network, digital systems (e.g. Civica Experience Platform), and patient portals]. Understanding the needs of organizational stakeholders will guide refinements to data collection strategies, ensuring that the tool aligns with safety goals and is both usable and informative.
Consistency in terminology, such as the use of terms like ‘concerns’, ‘incidents’, and ‘complaints’, is also crucial to enhance data interpretability and application. Ongoing work analysing free-text data from earlier tool iterations in the COPE [9] and EVITE studies [11] has demonstrated the richness of patient-reported insights [10]. Applying frameworks like SEIPS 2.0 [48] and patient safety incidents (PISA) [45] has shown potential for identifying system-level improvements. Future analyses with larger samples and advanced methodologies, including automated coding, will further refine these approaches and potentially enhance the tool’s utility.
Conclusions
We have refined the Patient Reported Safety Concern Tool (version 3.0), demonstrating content and face validity and identifying opportunities for further development and validation. Participants expressed motivation to contribute to safety learning, emphasizing the tool’s potential to provide actionable insights for healthcare organizations. Future work will address barriers to completion, explore patient decision-making processes around ‘voiceable’ safety concerns, and optimize the tool’s integration into routine systems. These efforts will advance co-produced, data-informed safety systems that prioritize patient voices, ultimately enhancing healthcare quality and safety.
Supplementary Material
Acknowledgements
We are grateful for the input of Nigel Pearson (Patient Partner) during the initial development of the Patient Reported Safety Concerns Tool (version 1.0).
Contributor Information
Abubakar Sha’aban, Health and Care Research Wales Evidence Centre, Division of Population Medicine, Cardiff University, Heath Park, Cardiff CF14 4YS, United Kingdom.
Anna Torrens-Burton, PRIME Centre Wales, Division of Population Medicine, Cardiff University, Heath Park, Cardiff CF14 4YS, United Kingdom.
Denitza Williams, Health and Care Research Wales Evidence Centre, Division of Population Medicine, Cardiff University, Heath Park, Cardiff CF14 4YS, United Kingdom.
Andrew Carson-Stevens, PRIME Centre Wales, Division of Population Medicine, Cardiff University, Heath Park, Cardiff CF14 4YS, United Kingdom.
Adrian Edwards, Health and Care Research Wales Evidence Centre, Division of Population Medicine, Cardiff University, Heath Park, Cardiff CF14 4YS, United Kingdom; PRIME Centre Wales, Division of Population Medicine, Cardiff University, Heath Park, Cardiff CF14 4YS, United Kingdom.
Lee Joseph, NHS Wales Performance & Improvement, Oldfield Road, Bocam Park, Pencoed, Bridgend, CF35 5LJ, United Kingdom.
Natalie Joseph-Williams, Health and Care Research Wales Evidence Centre, Division of Population Medicine, Cardiff University, Heath Park, Cardiff CF14 4YS, United Kingdom.
Author contributions
Abubakar Sha’aban (Conceptualization, Design of the focus groups and cognitive testing, Data analysis and interpretation, Drafting first version of article, Critical revision of article, Final approval), Anna Torrens-Burton (Design of the focus groups and cognitive testing, Critical revision of article, Final approval), Denitza Williams (Conceptualization, Design of the focus groups and cognitive testing, Data analysis and interpretation, Critical revision of article, Final approval), Andrew Carson-Stevens (Conceptualization, Design of the focus groups and cognitive testing, Critical revision of article, Final approval), Adrian Edwards (Conceptualization, Design of the focus groups and cognitive testing, Critical revision of article, Final approval), Lee Joseph (Critical revision of article, Final approval), and Natalie Joseph-Williams (Conceptualization, Design of the focus groups and cognitive testing, Data analysis and interpretation, Drafting first version of article, Critical revision of article, Final approval)
Supplementary data
Supplementary data is available at IJQHC online.
Conflict of interest
No known conflict of interests.
Funding
This article presents research conducted by the Wales COVID-19 Evidence Centre, which was funded 2021-2023 by Health and Care Research Wales on behalf of the Welsh Government.
Data availability
The datasets generated and analysed during the current study are not publicly available due to the sensitive nature of the data and the confidentiality agreements with participants. However, de-identified data may be made available upon reasonable request to the corresponding author, subject to approval by the relevant ethics committee and in compliance with institutional and data protection regulations.
Ethics and other permissions
Ethical approval was obtained from the Cardiff University School of Medicine’s Research Ethics Committee (SMREC 22/90). Informed consent was sought from all participants before the focus groups and cognitive testing interviews.
References
- 1. World Health Organization. Implementation of the Global Patient Safety Acton Plan 2021-2030—Interim Report: Based on the first survey of patient safety in WHO Member States (10 May 2023). 2023. https://www.who.int/publications/m/item/interim-report---based-on-the-first-survey-of-patient-safety-in-who-member-states (10 June 2025, date last accessed).
- 2. Welsh Government. Quality and Safety Framework: Learning and Improving. 2021. https://www.gov.wales/sites/default/files/publications/2021-09/quality-and-safety-framework-learning-and-improving_0.pdf (10 June 2025, date last accessed)
- 3. Donaldson LJ. The wisdom of patients and families: ignore it at our peril. BMJ Qual Saf 2015;24:603–4. [DOI] [PubMed] [Google Scholar]
- 4. World Health Organization. Patients for Patient Safety. 2024. https://www.who.int/initiatives/patients-for-patient-safety (01 May 2024, date last accessed).
- 5. Lachman P, Linkson L, Evans T et al. Developing person-centred analysis of harm in a paediatric hospital: a quality improvement report. BMJ Qual Saf 2015;24:337–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Weingart SN, Pagovich O, Sands DZ et al. What can hospitalized patients tell us about adverse events? Learning from patient-reported incidents. J Gen Internal Med 2005;20:830–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Hernan AL, Giles SJ, Carson-Stevens A et al. Nature and type of patient-reported safety incidents in primary care: cross-sectional survey of patients from Australia and England. BMJ Open 2021;11:e042551. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Weissman JS, Schneider EC, Weingart SN et al. Comparing patient-reported hospital adverse events with medical record review: do patients know something that hospitals do not? Ann Intern Med 2008;149:100–8. [DOI] [PubMed] [Google Scholar]
- 9. Phillips R, Taiyari K, Torrens-Burton A et al. Cohort profile: the UK COVID-19 public experiences (COPE) prospective longitudinal mixed-methods study of health and well-being during the SARSCoV2 coronavirus pandemic. PLoS One 2021;16:e0258484. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. McFadzean IJ, Bilal M, Davies K et al. The evolution of patient-reported safety concerns during the COVID-19 pandemic within a series of study questionnaires: a multi-method analysis. Int J Qual Health Care 2025;37:1–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Snooks H, Watkins A, Lyons J et al. Did the UK's public health shielding policy protect the clinically extremely vulnerable during the COVID-19 pandemic in Wales? Results of EVITE immunity, a linked data retrospective study. Public Health 2023;218:12–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Kingsley C, Patel S. Patient-reported outcome measures and patient-reported experience measures. BJA Educ 2017;17:137–44. [Google Scholar]
- 13. Hernan AL, Giles SJ, O'Hara JK et al. Developing a primary care patient measure of safety (PC PMOS): a modified Delphi process and face validity testing. BMJ Qual Saf 2016;25:273–80. [DOI] [PubMed] [Google Scholar]
- 14. O’Hara JK, Lawton RJ, Armitage G et al. The patient reporting and action for a safe environment (PRASE) intervention: a feasibility study. BMC Health Serv Res 2016;16:676. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Prinsen CAC, Mokkink LB, Bouter LM et al. COSMIN guideline for systematic reviews of patient-reported outcome measures. Qual Life Res 2018;27:1147–57. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Gremyr A, Andersson Gäre B, Thor J et al. The role of co-production in learning health systems. Int J Qual Health Care 2021;33:ii26–32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Rhodes P, McDonald R, Campbell S et al. Sensemaking and the co‐production of safety: a qualitative study of primary medical care patients. Sociol Health Illn 2016;38:270–85. [DOI] [PubMed] [Google Scholar]
- 18. Ocloo J, Garfield S, Franklin BD et al. Exploring the theory, barriers and enablers for patient and public involvement across health, social care and patient safety: a systematic review of reviews. Health Res Policy Syst 2021;19:8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Stahl K, Reisinger A, Groene O. Assessing patient experience with patient safety in primary care: development and validation of the ASK-ME-questionnaire. BMJ Open 2022;12:e049237. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Schnock KO, Snyder JE, Gershanik E et al. Unique patient-reported hospital safety concerns with online tool: MySafeCare. J Patient Saf 2022;18:E33–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Mounce LTA, Salema N-E, Gangannagaripalli J et al. Development of 2 short patient-report questionnaires of patient safety in primary care. J Patient Saf 2022;18:161–70. [DOI] [PubMed] [Google Scholar]
- 22. Willis GB. Cognitive interviewing revisited: a useful technique, in theory? In: Groves RM, Kalton G, Rao JNK et al. Methods for Testing and Evaluating Survey Questionnaires. New York: John Wiley & Sons, Inc., 2004, 23–43.
- 23. Willis G. Cognitive Interviewing.Thousand Oaks: SAGE Publications, Inc., 2011.
- 24. Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005;15:1277–88. [DOI] [PubMed] [Google Scholar]
- 25. Ricci-Cabello I, Avery AJ, Reeves D et al. Measuring patient safety in primary care: the development and validation of the “patient reported experiences and outcomes of safety in primary care” (PREOS-PC). Ann Fam Med 2016;14:253–61. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Barbour R. Doing Focus Groups. London: SAGE Publications Ltd, 2018. [Google Scholar]
- 27. Rhodes P, Campbell S, Sanders C. Trust, temporality and systems: how do patients understand patient safety in primary care? A qualitative study. Health Expect 2016;19:253–63. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Ricci-Cabello I, Pons-Vigués M, Berenguera A et al. Patients’ perceptions and experiences of patient safety in primary care in England. Fam Pract 2016;33:535–42. [DOI] [PubMed] [Google Scholar]
- 29. Lyndon A, Davis D-A, Sharma AE et al. Emotional safety is patient safety. BMJ Qual Saf 2023;32:369–72. [DOI] [PubMed] [Google Scholar]
- 30. World Health Organization. Conceptual Framework for the International Classification for Patient Safety. Geneva: World Health Organization, 2009. [Google Scholar]
- 31. Francis RQ. Report of the Mid Staffordshire NHS Foundation Trust Public Inquiry. London: Her Majesty’s Stationery Office, 2013. [Google Scholar]
- 32. Davis RE, Jacklin R, Sevdalis N et al. Patient involvement in patient safety: what factors influence patient participation and engagement? Health Expect 2007;10:259–67. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Entwistle VA, McCaughan D, Watt IS et al. ; Patient Involvement in Patient Safety Group. Speaking up about safety concerns: multi-setting qualitative study of patients’ views and experiences. Qual Saf Health Care 2010;19:e33. [DOI] [PubMed] [Google Scholar]
- 34. Runciman B, Merry A, Smith AM. Improving patients’ safety by gathering information. BMJ 2001;323:298. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Johnson S. A quality improvement project to tackle under-reporting of hazards by doctors by using an anonymous telephone hotline. BMJ Qual Improv Rep 2016;5:u208718.w3660. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Hernan AL, Giles SJ, Beks H et al. Patient feedback for safety improvement in primary care: results from a feasibility study. BMJ Open 2020;10:e037887. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Lawton R, O'Hara JK, Sheard L et al. Can patient involvement improve patient safety? A cluster randomised control trial of the patient reporting and action for a safe environment (PRASE) intervention. BMJ Qual Saf 2017;26:622–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38. Beks H, Hernan AL, Giles S et al. Theorizing factors mediating with the implementation of a patient feedback on safety intervention implemented in the primary care setting. Qual Health Res 2021;31:2260–73. [DOI] [PubMed] [Google Scholar]
- 39. Dixon-Woods M, Aveling EL, Campbell A et al. What counts as a voiceable concern in decisions about speaking out in hospitals: a qualitative study. J Health Serv Res Policy 2022;27:88–95. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Williamson R. Forgive and remember: managing medical failure, 2nd edition. J R Soc Med 2004;97:147. [Google Scholar]
- 41. O'Hara JK, Armitage G, Reynolds C et al. How might health services capture patient-reported safety concerns in a hospital setting? An exploratory pilot study of three mechanisms. BMJ Qual Saf 2017;26:42–53. [DOI] [PubMed] [Google Scholar]
- 42. Gillespie A, Reader TW. Online patient feedback as a safety valve: an automated language analysis of unnoticed and unresolved safety incidents. Risk Anal 2023;43:1463–77. [DOI] [PubMed] [Google Scholar]
- 43. Armitage G, Moore S, Reynolds C et al. Patient-reported safety incidents as a new source of patient safety data: an exploratory comparative study in an acute hospital in England. J Health Serv Res Policy 2018;23:36–43. [DOI] [PubMed] [Google Scholar]
- 44. Care Quality Commission. Better Care in My Hands: A Review of How People Are Involved in Their Care. 2016. https://www.cqc.org.uk/sites/default/files/20160519_Better_care_in_my_hands_FINAL.pdf (10 June 2025, date last accessed).
- 45. Carson-Stevens A, Hibbert P, Williams H et al. Characterising the nature of primary care patient safety incident reports in the England and Wales national reporting and learning system: a mixed-methods agenda-setting study for general practice. Health Serv Deliv Res 2016;4:1–76. [PubMed] [Google Scholar]
- 46. Aiyegbusi OL, Roydhouse J, Rivera SC et al. Key considerations to reduce or address respondent burden in patient-reported outcome (PRO) data collection. Nat Commun 2022;13:6026. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47. McEachan RRC, Lawton RJ, O'Hara JK et al. ; Yorkshire Quality and Safety Research Group. Developing a reliable and valid patient measure of safety in hospitals (PMOS): a validation study. BMJ Qual Saf 2014;23:565–73. [DOI] [PubMed] [Google Scholar]
- 48. Holden RJ, Carayon P, Gurses AP et al. SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics 2013;56:1669–86. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The datasets generated and analysed during the current study are not publicly available due to the sensitive nature of the data and the confidentiality agreements with participants. However, de-identified data may be made available upon reasonable request to the corresponding author, subject to approval by the relevant ethics committee and in compliance with institutional and data protection regulations.