Abstract
Abstract:
Objectives
Patient-reported experience measures (PREMs) offer unique insights into patient perceptions of care. However, their implementation at the micro level with regards to patient safety remains limited. This study aimed to repurpose PREMs to co-produce validated, context-specific PREMs tools aimed at improving patient safety.
Methods
A prospective mixed method approach was used to design PREMs-based tools. This multiphase study was conducted from September 2023 to April 2024. In phase 1, a core group identified 17 key processes to devise the PREMs tools. In phase 2, focus group discussions were conducted by the multidisciplinary principal teams to finalise the standard operating procedures for each key process and prepare the respective PREMs questionnaires. Patient safety was the primary construct. Relevant stakeholders, including patients, participated in the face and content validation of each PREMs tool during phase 3. Content Validity Index (CVI) was calculated using three indices, namely, item level CVI (I-CVI), scale-level CVI (S-CVI) and content validity ratio (CVR). The reliability of the questionnaires was checked using Kuder-Richardson Formula 20 (KR-20) values with pilot tests on patients.
Results
17 checklist format PREMs tools were developed. The I-CVI values for all the items in the questionnaires ranged from 0.73 to 1.00 and the CVR values ranged from 0.46 to 1.00. The S-CVI/AVG ranged from 0.89 to 1.00. These indicated strong content validity for all items and questionnaires. The reliability analysis for the 17 studies, based on the KR-20 values, ranged from 0.4324 to 0.9455. Except for ‘fall prevention’, all tools showed good internal consistency.
Conclusions
We offer a battery of patient safety oriented PREMs tools. Co-production of PREMs tools across an extensive range of patient care processes offers significant potential in patient safety implementation in addition to patient engagement.
Keywords: Patient safety, Patient-centred care, Surveys, Healthcare quality improvement, Checklists
WHAT IS ALREADY KNOWN ON THIS TOPIC.
WHAT THIS STUDY ADDS
Here, we repurposed PREMs to co-produce a set of validated, context-specific PREMs tools with patients, aimed at assessing key processes in the hospital in relation to safety.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
Utilisation of our suite of validated PREMs tools would possibly help improve patient safety implementations and further encourage patient engagement in the safety journey. This project opens up new avenues for research in the form of compliance studies, validation of the tools in other parts of the world and creation of more tools for additional patient care processes.
Introduction
Despite significant advances made in medical science and continuous efforts aimed at improving care quality over the past few decades, patient harm in healthcare settings remains alarmingly high. The Office of Inspector General of the US Department of Health and Human Services reported that in October 2018, 25% of Medicare patients experienced harm, with 43% of these incidents deemed avoidable.1 Similarly, the National Health Service (NHS) in England recorded 384 serious and avoidable ‘never events’ between 1 April 2022 and 31 March 2023.2 These figures underscore the long-standing issue of the healthcare system’s failure to consistently implement safety protocols. Several studies and reports have documented inadequate application of safety measures in healthcare settings,3 4 highlighting the urgent need for robust mechanisms that validate adherence to safety protocols. To address this critical issue, we propose to use patient-reported experience measures (PREMs) as an additional tool to assess the implementation of safety protocols at the individual provider level.
Patient experience measures aid us in evaluating the quality and effectiveness of healthcare services as they offer unique insights into patient perceptions of care. However, their implementation into broader context remains limited, more so in developing countries, due to challenges such as system complexity, resistance to change and concerns around data confidentiality.5 Application of PREMs has traditionally been limited to macro-level (eg, system performance) and meso-level (eg, network-level performance) evaluations. Gilmore et al,6 in their systematic review, observed that the use of PREMs at the micro level—specifically to enhance patient safety and individual care quality—remains sparse.
Many countries have adopted the PREMs tool as part of national health initiatives. For example, NHS England launched its first national patient survey in 2001, and the USA introduced the Hospital Consumer Assessment of Healthcare Providers and Systems survey soon after. Countries including Australia, New Zealand, Germany, Norway and Sweden have also been collecting PREMs data for years.6 However, so far, these data have primarily been used for making policy-level decisions such as pay-for-performance schemes or publicly reporting patient satisfaction scores to guide consumer choices. While such initiatives are valuable, there is limited evidence on their impact on tangible improvements at the patient level.6 7 We hope to offer targeted and impactful use of PREMs tools at the patient level through this study.
Patient perspectives remain central to the design and validation process of PREMs tools. Co-production of these tools with patients and their families ensures their active engagement, and this was a core component of our study. The Indian healthcare landscape is vast and complex, comprising a mix of public and private providers delivering care, spanning from primary to quaternary levels. The current use of PREMs in India has been largely limited to evaluating service excellence metrics, such as waiting times, facility cleanliness or the quality of non-clinical services like food. Patient safety, however, has not been the focus of PREMs in India so far. To the best of our knowledge, this is the first study in India to collaboratively produce a collection of PREMs, with input from a broad range of stakeholders including healthcare providers and patients, to assess the implementation of safety protocols by individual healthcare providers.
Methods
In this multiphase study, we applied a prospective mixed method approach to design PREMs-based tools (figure 1). The study was conducted from September 2023 to April 2024.
Figure 1. Flowchart depicting the methodology used for PREMs tool validation. CVI, Content Validity Index; CVR, content validity ratio; PI, principal investigator; PREMs, patient-reported experience measures.
Phase 1: identification of the key processes by a PREMs core group
A core group (including a physician, a surgeon, a healthcare administrator, a quality professional, an infection control professional and a statistician) deliberated and identified the 17 key processes for PREMs tools development. This was done after multiple discussions on patient experience, patient engagement, patient safety issues and common incidents at hospitals. For example, reported incidents of retained instruments, wrong site surgeries, poor compliance to surgical site markings, inadequately filled informed consents, among others, in various hospitals of India led us to choose ‘surgical safety’ as one of the key processes. The finalised list included ‘surgical safety’, ‘preoperative care by anaesthesiologists for elective surgeries’, ‘antenatal care’, ‘cardiac catheterisation’, ‘cataract surgery’, ‘discharge process’, ‘blood donor safety’, ‘emergency care’, ‘endoscopy’, ‘fall prevention’, ‘geriatric care’, ‘haemodialysis’, ‘peripheral line infection prevention’, ‘medication administration’, ‘MRI’, ‘orthopaedic implant surgeries’ and ‘wound care management’.
Phase 2: PREMs tools development by multidisciplinary principal teams
Two subject experts were identified and designated as principal investigator (PI) and co-PI, respectively, for each key process. Next, the PIs of each key process built their respective ‘principal teams’ which included the relevant stakeholders. For example, the ‘surgical safety’ principal team included surgeons, anaesthetists, nurses and quality professionals, along with the PIs. Each principal team had a minimum of eight members with representation from across India. Ethics committee approvals for the individual PREMs tools creation were obtained from the institutes of the PIs.
The principal teams reviewed the literature for any existing PREMs tools related to their specific key process. The team members participated in focus group discussions to create a consensus Standard operating procedure (SOP) for the respective key process, which incorporated all the important safety-related steps. Each principal team then prepared questionnaires (PREMs tools) specific to their respective key process based on the finalised SOPs, while capturing diverse viewpoints. The key focus areas in each tool included the core domains of patient safety, along with clinical care, patient communication, patient identification, patient privacy, infection prevention and control and financial awareness.
Phase 3: psychometric testing of the PREMs tools
Face and content validation of the individual PREMs tools was done by experts (relevant stakeholders) from across India. For example, the experts for face and content validation of the ‘surgical safety’ PREMs tool included five representatives from each of the following categories: surgeons, anaesthetists, operating room nurses, operating room technicians, quality managers/ administrators and patients/ patient attendants. These experts opined on the relevance, understandability, language use and technical jargon for each question in the tool.
Face validity was assessed using expert opinion as described above. Content Validity Index (CVI) was calculated using three indices, namely, item level CVI (I-CVI), scale-level CVI (S-CVI) and content validity ratio (CVR) for each item’s relevance.8 The I-CVI represented the proportion of experts stating the relevance. The S-CVI was calculated using two methods,9 namely S-CVI/universal agreement (UA), which presented the proportion of the items stated as relevant by all the experts (UA by experts) divided by the total number of items and S-CVI/AVG, which presented the sum of the I-CVI scores of all items divided by the total number of items (average of the I-CVI scores for all items across all experts). The CVR was calculated by dividing the subtraction product of the number of experts indicating an item as relevant and half the number of experts, divided by half the number of experts, represented by the formula (CVR=(ne−N/2)/(N/2)).10 11 The CVI, S-CVI and CVR values were discussed with the core group. The questions that had no UA were modified, irrelevant questions were deleted, and some new questions were added based on the inputs of the PREMs core group.
Once the questionnaires were finalised, the PREMs tools were pilot-tested12 at 15 hospitals comprising a mix of small, medium and large hospitals from north, south, east, west and central zones of the country with patients relevant to the specific tool. Administrative approvals were obtained from each participating hospital for pilot testing. The pilot study used purposive sampling technique with a minimum sample size of 150 patients to assess the internal consistency of the questionnaires based on Kuder-Richardson Formula 20 (KR-20), an indicator of internal consistency for dichotomous items (eg, questions with true/false or yes/no answers). All statistical analyses were conducted using Stata V.16 (StataCorp, 2019).
Patient and public involvement
Patients were involved during content validation of the PREMs questionnaires as experts. They were also involved during the pilot study for reliability testing.
Results
A total of 17 questionnaires were developed (online supplemental file 1). 268 questions were validated across the 17 key processes. The number of questions ranged from 11 (‘emergency care’) to 21 (‘MRI’). The number of experts who participated in the face and content validation ranged from 25 (‘wound care’) to 45 (‘haemodialysis’).
The I-CVI and S-CVR for each tool are listed in tables1 2, respectively. The I-CVI values for all the items in the 17 questionnaires ranged from 0.73 to 1.00 and CVR values ranged from 0.46 to 1.00.
Table 1. I-CVI and CVR values of individual questions of all processes.
| Sr. no. | Key process/PREMs tool | I-CVI and CVR values of individual questions of all processes | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 | Q11 | |||||||||||||
| I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | ||
| 1 | Anaesthesia safety | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.97 | 0.93 | 0.97 | 0.93 | 0.97 | 0.93 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 2 | Antenatal care | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.97 | 0.94 | 1 | 1 | 1 | 1 |
| 3 | Cardiac catheterisation | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9 | 0.8 | 0.9 | 0.8 | 0.9 | 0.8 |
| 4 | Cataract surgery | 1 | 1 | 0.97 | 0.94 | 0.87 | 0.74 | 0.97 | 0.94 | 1 | 1 | 0.97 | 0.94 | 1 | 1 | 1 | 1 | 1 | 1 | 0.97 | 0.94 | 1 | 1 |
| 5 | Discharge Process | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.97 | 0.95 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 6 | Blood donor safety | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 7 | Emergency care | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 8 | Endoscopy | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.96 | 0.92 | 1 | 1 | 1 | 1 | 1 | 1 |
| 9 | Fall prevention | 1 | 1 | 1 | 1 | 1 | 1 | 0.97 | 0.93 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | x | x |
| 10 | Geriatric care | 0.9 | 0.8 | 0.83 | 0.67 | 0.83 | 0.67 | 0.87 | 0.73 | 0.9 | 0.8 | 0.9 | 0.8 | 0.9 | 0.8 | 0.9 | 0.8 | 0.9 | 0.8 | 0.9 | 0.8 | 0.9 | 0.8 |
| 11 | Haemodialysis | 1 | 1 | 1 | 1 | 0.93 | 0.87 | 1 | 1 | 0.98 | 0.96 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 12 | Peripheral line infection prevention | 0.83 | 0.67 | 0.97 | 0.93 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 13 | Medication safety | 0.93 | 0.86 | 1 | 1 | 1 | 1 | 0.91 | 0.82 | 0.95 | 0.91 | 1 | 1 | 0.98 | 0.95 | 1 | 1 | 0.95 | 0.91 | 0.93 | 0.86 | 0.91 | 0.82 |
| 14 | MRI safety | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0.97 | 0.94 | 1 | 1 | 1 | 1 | 1 | 1 |
| 15 | Orthopaedic implant surgery | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 16 | Surgical safety | 0.95 | 0.89 | 0.97 | 0.95 | 0.97 | 0.95 | 0.97 | 0.95 | 0.97 | 0.95 | 0.97 | 0.95 | 0.89 | 0.79 | 0.92 | 0.84 | 0.95 | 0.89 | 0.97 | 0.95 | 0.97 | 0.95 |
| 17 | Wound care | 0.96 | 0.92 | 0.92 | 0.84 | 0.96 | 0.92 | 0.92 | 0.84 | 1 | 1 | 1 | 1 | 0.92 | 0.84 | 0.92 | 0.84 | 0.92 | 0.84 | 0.88 | 0.76 | 0.92 | 0.84 |
| Sr. No. | Key process/PREMs tool | Q12 | Q13 | Q14 | Q15 | Q16 | Q17 | Q18 | Q19 | Q20 | Q21 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | I-CVI | CVR | ||
| 1 | Anaesthesia safety | 1 | 1 | 1 | 1 | 1 | 1 | x | x | x | x | x | x | x | x | x | x | x | x | x | x |
| 2 | Antenatal care | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | x | x |
| 3 | Cardiac Catheterisation | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | x | x | x | x | x | x | x | x | x | x |
| 4 | Cataract surgery | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | x | x | x | x | x | x | x | x | x | x | x | x |
| 5 | Discharge process | 1 | 1 | 0.73 | 0.46 | 1 | 1 | 0.97 | 0.95 | 0.92 | 0.84 | 1 | 1 | x | x | x | x | x | x | x | x |
| 6 | Blood donor safety | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | x | x |
| 7 | Emergency care | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x |
| 8 | Endoscopy | 1 | 1 | 1 | 1 | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x |
| 9 | Fall prevention | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x |
| 10 | Geriatric care | 0.9 | 0.8 | 0.9 | 0.8 | 0.9 | 0.8 | 0.9 | 0.8 | 0.9 | 0.8 | 0.9 | 0.8 | x | x | x | x | x | x | x | x |
| 11 | Haemodialysis | 1 | 1 | 1 | 1 | 1 | 1 | 0.98 | 0.96 | 1 | 1 | x | x | x | x | x | x | x | x | x | x |
| 12 | Peripheral line infection prevention | 0.97 | 0.93 | 1 | 1 | 0.93 | 0.87 | 1 | 1 | 0.97 | 0.93 | 1 | 1 | 1 | 1 | 0.93 | 0.87 | 1 | 1 | x | x |
| 13 | Medication safety | 0.93 | 0.86 | 0.95 | 0.91 | 0.91 | 0.82 | 1 | 1 | 1 | 1 | x | x | x | x | x | x | x | x | x | x |
| 14 | MRI safety | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 15 | Orthopaedic implant surgery | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | x | x | x | x | x | x | x | x |
| 16 | Surgical safety | 0.97 | 0.95 | 0.95 | 0.89 | 0.95 | 0.89 | x | x | x | x | x | x | x | x | x | x | x | x | x | x |
| 17 | Wound care | 0.92 | 0.84 | 0.92 | 0.84 | 0.88 | 0.76 | 1 | 1 | 1 | 1 | 1 | 1 | 0.96 | 0.92 | x | x | x | x | x | x |
CVR, content validity ratio; I-CVI, Item Level Content Validity Index; PREMs, patient-reported experience measures; Q, questionnaire.
Table 2. S-CVI values, no. of questions and experts in each process.
| Sr. no. | Key process/PREMs tool | S-CVI/ Avg | S-CVI/UA values | No. of questions | No. of experts |
|---|---|---|---|---|---|
| 1 | Anaesthesia safety | 0.992 | 0.785 | 14 | 30 |
| 2 | Antenatal care | 0.998 | 0.95 | 20 | 31 |
| 3 | Cardiac catheterisation | 0.981 | 0.813 | 16 | 40 |
| 4 | Cataract surgery | 0.982 | 0.667 | 15 | 31 |
| 5 | Discharge process | 0.976 | 0.765 | 17 | 37 |
| 6 | Blood donor safety | 1 | 1 | 20 | 27 |
| 7 | Emergency care | 1 | 1 | 11 | 30 |
| 8 | Endoscopy | 0.997 | 0.923 | 13 | 24 |
| 9 | Fall prevention | 0.997 | 0.9 | 10 | 30 |
| 10 | Geriatric care | 0.89 | 0 | 17 | 30 |
| 11 | Haemodialysis | 0.993 | 0.813 | 16 | 45 |
| 12 | Peripheral line infection prevention | 0.98 | 0.7 | 20 | 30 |
| 13 | Medication safety | 0.96 | 0.864 | 16 | 44 |
| 14 | MRI | 0.998 | 0.952 | 21 | 32 |
| 15 | Orthopaedic implant surgery | 1 | 1 | 17 | 32 |
| 16 | Surgical safety | 0.957 | 0 | 14 | 38 |
| 17 | Wound care | 0.944 | 0.278 | 18 | 25 |
Avg, average; PREMs, patient-reported experience measures; S-CVI, Scale Level Content Validity Index; UA, universal agreement.
The PREMs tools on ‘orthopaedic implant’, ‘blood donor safety’ and ‘emergency care’ key processes demonstrated a perfect S-CVI/AVG (mean I-CVI value) of 1, indicating a UA among the experts on the relevance of the questions/items. Also, the ‘antenatal’ and ‘MRI’ tools showed a very high S-CVI/AVG of 0.998 supporting the strength of the item scale. The S-CVI/AVG of other PREMs tools items ranged between 0.89 and 0.997, showing a strong overall agreement on the relevance of questions/items.
The S-CVI/UA values of the key processes ‘blood donor safety’, ‘emergency care’ and ‘ortho implants’ were all 1, which again demonstrated perfect agreement among experts, indicating that the PREMs for these areas were well-defined and contextually relevant.
The key processes like ‘antenatal care’ (0.95), ‘medication safety’ (0.864) and ‘MRI’ (0.952) also showed S-CVI/UA values close to 1, indicating near-perfect validity. The high overall S-CVI/UA values suggest that these tools are robust and reliable for assessing patient-reported experiences. However, the S-CVI/UA for ‘surgical safety’ and ‘geriatrics’ items received a value of 0, indicating that some specific items might require further attention to achieve universal consensus. These items were modified after discussion with the respective principal teams and the core group.
The reliability analysis for the 17 studies, based on the KR-20 reliability value, ranged from 0.4324 to 0.9455 (table 3). Four PREMs tools, ‘fall prevention’(0.43), ‘geriatrics’(0.65), ‘haemodialysis’(0.62) and ‘medication safety’(0.66), had a KR-20 value of <0.7 (figure 2).
Table 3. KR-20 reliability values of dichotomous item questionnaires across multiple pilot studies.
| No. | PREMs tool | Sample size (n) | Number of items | KR-20 reliability value | Interpretation |
|---|---|---|---|---|---|
| 1 | Anaesthesia safety | 210 | 20 | 0.8029 | Acceptable |
| 2 | Antenatal care | 220 | 22 | 0.7205 | Acceptable |
| 3 | Cardiac catheterisation | 285 | 16 | 0.8506 | Good |
| 4 | Cataract surgery | 240 | 14 | 0.8011 | Good |
| 5 | Discharge process | 225 | 18 | 0.9455 | Excellent |
| 6 | Blood donor safety | 180 | 21 | 0.8094 | Good |
| 7 | Emergency care | 290 | 11 | 0.7896 | Acceptable |
| 8 | Endoscopy | 210 | 14 | 0.7775 | Acceptable |
| 9 | Fall prevention | 225 | 10 | 0.4324 | Poor |
| 10 | Geriatric care | 165 | 17 | 0.6493 | Questionable |
| 11 | Haemodialysis | 222 | 16 | 0.6188 | Questionable |
| 12 | Peripheral line infection prevention | 225 | 20 | 0.8073 | Acceptable |
| 13 | Medication safety | 240 | 15 | 0.6603 | Questionable |
| 14 | MRI | 213 | 21 | 0.9421 | Excellent |
| 15 | Orthopaedic implant surgery | 230 | 15 | 0.7091 | Acceptable |
| 16 | Surgical safety | 225 | 16 | 0.8312 | Good |
| 17 | Wound care | 254 | 19 | 0.7480 | Acceptable |
KR-20, Kuder-Richardson Formula 20; PREMs, patient-reported experience measures.
Figure 2. KR-20 reliability values of the PREMs tools. KR-20, Kuder-Richardson Formula 20; PREMs, patient-reported experience measures.
Discussion
‘Quality 3.0’ is focused on the co-production of healthcare services.13 The patient has a vital role to play in this endeavour. One of the components of co-production involves understanding the ‘lived reality of the person’, who is the patient. Another component is the navigation of the current healthcare system, which provides valuable perception of the ‘safety, reliability and coordination’. Listening to the patients and entrusting them with an active role is the way forward to healthcare quality improvement. This study actively involved patients in the development and validation of all the PREMs tools, thus being a prime example of co-production.
A review of the 100 most cited articles on PREMs showed that while Asia, Africa and South America collectively accounted for just six publications as of January 2024,14 North America and Europe had contributed 88 articles. This demonstrates the limited in-roads PREMs have made in the Global South. Furthermore, there is sparse literature on PREMs from India. Symon et al published a new tool called ‘Participant-Generated Experience and Satisfaction Index’ for obstetric patients, based on a pilot study conducted in India.15 Bawankar et al16 used PREMs as one of the tools to evaluate patient satisfaction while comparing two treatments for deficient alveolar ridges. Notably, our multicentric pan-India PREMs project with 17 different tools is the first step at bridging this gap.
Vincent et al17 highlighted the lack of studies on patient contribution to safety. They further added that patient involvement would improve diagnostic accuracy, ensure appropriate treatment and administration of the same, help choose a safer provider and lead to quicker identification of adverse events or side effects. Rathert et al,18 in a qualitative study, reported the views of acute care patients on their role in safety. The study showed that while patients expected competent care (8%), the right to ask questions (7%), be well-informed of their clinical condition (6%) and be an active participant in their care (4%), they also acknowledged their duty to follow the instructions of the healthcare providers (23%). Although PREMs can be used to monitor patient safety,6 19 there is little evidence in the form of implementation. Doyle et al20 summarised 55 studies and provided indirect evidence that patient experience is positively associated with clinical effectiveness and patient safety. Weingart et al21 interviewed 228 medical patients in a prospective cohort study. 20 adverse events and thirteen near misses were reported by the patients. However, only 55% of the adverse events and 31% of the near misses had been documented in the medical record and none in the hospital incident reporting system. In a similar study, Weissman et al22 interviewed 998 patients and found 21 serious preventable adverse events which had not been documented in the medical record. These studies demonstrate the patients’ capacity to recognise medical errors, thus providing evidence that patient involvement is pivotal in ensuring patient safety. The tools developed and validated in our study focus on patient safety and cover many important patient care processes in hospital settings. To the best of our knowledge, this is the first study to report use of PREMs tools for patient safety and paves the way to unlock the full potential of PREMs.
We chose 17 patient care processes based on their significant impact on patient safety. Development of the process-specific SOPs with the involvement of domain experts was integral to our methodology, which ensured that the designed questionnaires were comprehensive. Patients were involved in the content validation phase and pilot phase of the questionnaire development. This co-production with patients was another significant characteristic of the project. The patient sample for the pilot studies was gathered equally from small, medium and large hospitals, irrespective of accreditation status. The hospitals were chosen randomly from the list of volunteers from all five zones of the country. The above measures ensured equal representation from across the country.
Our study followed the standard methodology for questionnaire development including item generation based on SOPs, creation of an initial questionnaire, revision of the questionnaire based on focused group discussions, testing for validity, revision of the questionnaire if needed, pilot testing for reliability and final revision, if required.12 The pilot questionnaires in our study underwent suitable revisions, which included deletion if a question was redundant or not relevant, addition of a relevant question that had been missed and modification, if a question was not comprehensible or incomplete. An important differentiator of our tools from other PREMs questionnaires is that this is a checklist-based questionnaire with dichotomous items with yes/no answers, highlighting its ease of use and thus potential for wider applicability. PREMs items are usually frequency-based scales. Furthermore, unlike other PREMs, which may have multiple domains, in our case the predominant domain was patient safety with a focus on the functional aspects of the patient care process.
The I-CVI values for all items across all tools were more than 0.8, which is a cut-off for retaining an item in the questionnaire.8 23 The CVR values for all items too were above the critical value for acceptance based on Lawshe’s method.10 24 The minimum S-CVI/AVG of the tools was 0.89. Our overall findings suggest that the items within the scales demonstrated strong content validity based on the high I-CVI and S-CVI values. These findings support the relevance and representativeness of the items selected in the respective PREM tool. A minimum sample size of 150 was used for pilot testing the questionnaires, which is beyond the upper limit of the sample size advised for pilot studies.25 We chose KR-20 instead of Cronbach’s alpha for internal consistency reliability testing since the items were binary/dichotomous.26 A cut-off of 0.7 suggests good reliability,26 and this was achieved in 13 of the 17 tools. Of the four PREMs tools with a KR-20 value of less than 0.7, three questionnaires with marginal KR-20 values (0.6–0.7) were re-evaluated by the individual principal teams as well as the core group. The teams reached a consensus that all the questions in these tools were relevant, important and comprehensible. Hence, no changes were made to these questionnaires. Thus, except for ‘fall prevention’ (0.43), all the other 16 tools were deemed valid and reliable. The low KR-20 value could have been due to inconsistent item difficulty and homogeneity in responses. A mismatch between the questionnaire’s complexity and the respondent group, along with variability in how the survey was administered, could further contribute to low reliability. Since this was a checklist with binary responses, the psychometric evaluation was restricted to face validity, content validity and internal consistency.27 This psychometric testing methodology was also used by Harris et al. for validation of ‘surgical patient safety checklist’.28 The appropriateness of a PREMs tool for quality assessment is determined by its clinical relevance, the domains covered and its validity and reliability.29 Our findings showed that our tools are clinically relevant, valid and reliable in the patient safety domain.
Strengths
This is the first detailed description of PREMs being used as a patient safety tool in India. These tools were co-produced with patients, allowing their active patient participation to widen the application of PREMs, something that has remained underexplored globally. Further, these questionnaires in English were prepared based on internationally accepted standards of care and were finalised by the subject experts after multiple rounds of discussions. Hence, these questionnaires are likely to be relevant across the world, in similar settings. The inclusion of accredited as well as non-accredited hospitals of all sizes across the country in the validation process ensured better representation. A robust methodology was used to validate these checklist-based questionnaires, thus improving their reproducibility. Finally, these PREMs safety tools cover an extensive range of patient care processes and may be used together as a battery at any hospital.
Limitations
Although our study presents a set of validated PREMs tools towards patient safety, they do not comprehensively cover all patient care processes. India is a multilingual country, and the content validity/pilot testing process may have had instances of patients being helped with translation of the English questionnaire. This may have affected the quality of our tools. The application of this set of PREMs tools is limited to the English-speaking populations globally. SOPs for the various care processes could vary in different geographical regions with their respective resource constraints, and this may impact the applicability of the questionnaire. Some of the longer questionnaires in this set may be burdensome for patients. The demographic factors, data collection methods and the knowledge level of healthcare professionals in interpretation of PREMs could affect the results of these tools. PREMs tools may also be generally limited by recall bias and subjectivity. There are implementation challenges with PREMs at system, service and individual level.29 The barriers and the mitigation strategies at various levels29 are summarised in table 4.
Table 4. Implementation barriers and mitigation strategies in PREMs.
| Barriers | Mitigation strategies |
|---|---|
| System level | |
| Resistance to pay for performance model Dearth of appropriate PREMs tools Shortage of value based benchmarking data |
Mandatory requirement of pay for performance model Mandatory implementation of tailor-made PREMs tools Development and implementation of value based schemes |
| Service level | |
| Inadequate prioritisation of PREMs implementation Poor information technology infrastructure support for PREMs implementation Costs of PREMs implementation |
Incorporation of PREMs implementation as a strategic priority Inclusion of PREMs implementation in information technology infrastructure Adequate funding |
| Individual level | |
| Healthcare workers lack the required skills for PREMs implementation, interpretation and its use in quality improvement. Healthcare workers’ distrust about PREMs |
Tailored training programmes Transparency and easy accessibility of PREMs data for healthcare workers. PREMs data is presented along with other relevant service metrics and adjusted for confounding patient variables. |
PREMs, patient-reported experience measures.
Conclusions
This set of PREMs tools has been created with a combination of patient-centric approach and rigorous statistical validation. Our suite of validated PREMs tools is an initial step at using PREMs in patient safety implementation, and its utilisation may not only help enhance patient safety but possibly also help improve the quality of patient care across India. The co-production of these tools with patients is likely to improve patient engagement in patient safety. Furthermore, this project opens up new avenues for further research, including creation of tools for additional care processes, validation of the tools in other countries, evaluation of domains, validation studies of the tool translations into various languages and field studies using the tools to ascertain compliance with patient safety standards.
Supplementary material
Acknowledgements
We would like to acknowledge the 34 PIs and co-PIs, the 17 hospitals where the ethics approval was obtained and the hospitals (250+) which participated in the study. We would also like to acknowledge the members of the principal teams (85+), the experts who contributed to validation (250+) and patients (2500+) who participated in the validation and pilot studies. We thank the leadership of the Consortium of Accredited Healthcare Organizations (CAHO) for their support and the Research Committee of CAHO for the guidance.
Footnotes
Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Provenance and peer review: Not commissioned; externally peer reviewed.
Patient consent for publication: Not applicable.
Ethics approval: This study involves human participants and the Ethics Committee approvals of each tool are mentioned hereby as "Sr. No.", "Name of the study", "Ethics committee institution", "Reference number". 1. Anaesthesia Safety: Institutional Research and Ethics committee of Christian Medical College and Hospital (IRB Min. No.16129 dated 28.02.2024). 2. Antenatal: Ethics committee of Pramukhswami Medical College, Bhaikaka University, Karamsad, Gujarat. (Reference number: BU/2024/Ex.11/33/2024 dated 29-01-2024). 3. Cardiac catheterisation: Father Muller Institutional Ethics Committee (Reference number: FMIEC/CCM/186/2024 dated 20/02/2024). 4. Cataract surgery: the Institutional Review Board (IRB) at Sankara Eye Hospital, Bangalore (Reference number: SEH/BLR/EC/2024/122). 5. Discharge process: Institutional Ethics Committee of Apollo Hospitals, Chennai (Registration No. EC/NEW/INST/2020/527). 6. Blood donor: Institutional Ethics Committee at Bhaikka University in Karmsad, Anand, Gujarat (Approval No. IEC/BU/2024/Ex.28/144/2024). 7. Emergency care: Ethics Committee of Manipal Hospitals, Bangalore, approved the study protocol (dated 31/05//2024). 8. Endoscopy: Ethics Committee of Sundaram Medical Foundation dated 03/04/20249. Fall prevention: Institutional Ethics committee of Fortis Hospital Mohali (Reference number: IEC/2024/OAS/01). 10. Geriatric care: Ethics Committee of JSS Medical College, Mysuru (Reference number: JSSMC/IEC/18122023/39NCT/2023-24). 11. Hemodialysis: Ethics Committee of Kalyani Kidney Care Centre dated 21 Feb 2024 12. Infection Prevention: Ethics committee of JSS Medical College and Research Institute (Reference number: JSSMC/IEC/070324/01 NCT 24-25). 13. Medication Safety: Ethics Committee of Peerless Hospital & Research Center Limited (PHH&RCL-CREC/S02/2023)14. MRI: Institution Ethics Committee (IEC), AIIMS Bhubaneswar (T/EMF/Hosadm/2023-24/209). 15. Ortho implants: Ethics committee of Ganga Hospital, Coimbatore (Reference number: 2024/IN/01). 16. Surgical safety - Ethics committee of Mahatma Gandhi Medical College and Research Institute (Reference number: MGMCRI/2024/04/IHEC/12). 17. Wound care: Ethics committee of Gleneagles Health City, Chennai (Reference number: BMHR/2024/0072). Participants gave informed consent to participate in the study before taking part.
Patient and public involvement: Patients and/or the public were involved in the design, conduct, reporting or dissemination plans of this research. Refer to the Methods section for further details.
Data availability statement
Data are available upon reasonable request.
References
- 1.Grimm CA. Adverse events in hospitals: a quarter of medicare patients experienced harm in october 2018. Office of Inspector General. 2022
- 2.England NH, Improvement NH. Provisional publication of Never Events reported as occurring between. 2018;2:2020. [Google Scholar]
- 3.Joseph L, Gulati B, Raju U, et al. A multicentric study of practice of surgical site marking. QAI J Healthc Qual Patient Saf . 2020;2:9. doi: 10.4103/QAIJ.QAIJ_6_20. [DOI] [Google Scholar]
- 4.| WHO | Regional Office for Africa. WHO | Regional Office for Africa. 2024. https://www.afro.who.int/node Available.
- 5.Moore L, Britten N, Lydahl D, et al. Barriers and facilitators to the implementation of person-centred care in different healthcare contexts. Scand J Caring Sci. 2017;31:662–73. doi: 10.1111/scs.12376. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Jamieson Gilmore K, Corazza I, Coletta L, et al. The uses of Patient Reported Experience Measures in health systems: A systematic narrative review. Health Policy. 2023;128:1–10. doi: 10.1016/j.healthpol.2022.07.008. [DOI] [PubMed] [Google Scholar]
- 7.DeCourcy A, West E, Barron D. The National Adult Inpatient Survey conducted in the English National Health Service from 2002 to 2009: how have the data been used and what do we know as a result? BMC Health Serv Res. 2012;12:71. doi: 10.1186/1472-6963-12-71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Lynn MR. Determination and quantification of content validity. Nurs Res. 1986;35:382–5. doi: 10.1097/00006199-198611000-00017. [DOI] [PubMed] [Google Scholar]
- 9.Polit DF, Beck CT. The content validity index: Are you sure you know what’s being reported? critique and recommendations. Res Nurs Health. 2006;29:489–97. doi: 10.1002/nur.20147. [DOI] [PubMed] [Google Scholar]
- 10.Lawshe CH. A Quantitative Approach to Content Validity. Personnel psychology/Berrett-Koehler Publishers; 1975. 10.1111/j.1744-6570.1975.tb01393.x. Available. [DOI] [Google Scholar]
- 11.Frey BB. The Sage Encyclopedia of Educational Research, Measurement, and Evaluation. 2455 Teller Road, Thousand Oaks, California 91320: Sage Publications; Jan 29, 2018. https://methods.sagepub.com/reference/the-sage-encyclopedia-of-educational-research-measurement-and-evaluation Available. [Google Scholar]
- 12.Kishore K, Jaswal V, Kulkarni V, et al. Practical Guidelines to Develop and Evaluate a Questionnaire. Indian Dermatol Online J. 2021;12:266–75. doi: 10.4103/idoj.IDOJ_674_20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Batalden P, Lachman P, von Plessen C, et al. Coproduction of healthcare services-from concept to implementation. Int J Qual Health Care. 2023;35:mzad083. doi: 10.1093/intqhc/mzad083. [DOI] [PubMed] [Google Scholar]
- 14.Attar A, Shukla K, Mulay P. Top 100 most cited articles on Patient Reported Experience Measures (PREM): insights and perspectives. J Patient Rep Outcomes. 2024;8:114. doi: 10.1186/s41687-024-00791-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Symon A, Lightly K, Howard R, et al. Introducing the participant-generated experience and satisfaction (PaGES) index: a novel, longitudinal mixed-methods evaluation tool. BMC Med Res Methodol. 2023;23:214. doi: 10.1186/s12874-023-02016-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Bawankar PV, Kolte AP, Kolte RA. Patient-centered outcome measures comparing the autogenous and allogenic bone blocks in the augmentation of deficient alveolar ridges: A pilot study. J Indian Soc Periodontol. 2023;27:87–94. doi: 10.4103/jisp.jisp_733_21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Vincent CA, Coulter A. Patient safety: what about the patient? Quality and Safety in Health Care. 2002;11:76–80. doi: 10.1136/qhc.11.1.76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Rathert C, Huddleston N, Pak Y. Acute care patients discuss the patient role in patient safety. Health Care Manage Rev. 2011;36:134–44. doi: 10.1097/HMR.0b013e318208cd31. [DOI] [PubMed] [Google Scholar]
- 19.Ahmed F, Burt J, Roland M. Measuring patient experience: concepts and methods. Patient . 2014;7:235–41. doi: 10.1007/s40271-014-0060-5. [DOI] [PubMed] [Google Scholar]
- 20.Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open. 2013;3:e001570. doi: 10.1136/bmjopen-2012-001570. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Weingart SN, Pagovich O, Sands DZ, et al. What can hospitalized patients tell us about adverse events? Learning from patient-reported incidents. J Gen Intern Med. 2005;20:830–6. doi: 10.1111/j.1525-1497.2005.0180.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Weissman JS, Schneider EC, Weingart SN, et al. Comparing patient-reported hospital adverse events with medical record review: do patients know something that hospitals do not? Ann Intern Med. 2008;149:100–8. doi: 10.7326/0003-4819-149-2-200807150-00006. [DOI] [PubMed] [Google Scholar]
- 23.Yusoff MSB, Department of Medical Education, School of Medical Sciences, Universiti Sains Malaysia, MALAYSIA ABC of Content Validation and Content Validity Index Calculation. EIMJ. 2019;11:49–54. doi: 10.21315/eimj2019.11.2.6. [DOI] [Google Scholar]
- 24.Ayre C, Scally AJ. Critical values for Lawshe’s content validity ratio: revisiting the original methods of calculation. Meas Eval Couns Dev. 2014;47:79–86. doi: 10.1177/0748175613513808. [DOI] [Google Scholar]
- 25.Rubio DM, Berg-Weger M, Tebb SS, et al. Objectifying content validity: Conducting a content validity study in social work research. Soc Work Res. 2003;27:94–104. doi: 10.1093/swr/27.2.94. [DOI] [Google Scholar]
- 26.Ntumi S, Agbenyo S, Bulala T. Estimating the Psychometric Properties (Item Difficulty, Discrimination and Reliability Indices) of Test Items using Kuder-Richardson Approach (KR-20) education . 2023;11:18–28. doi: 10.34293/education.v11i3.6081. [DOI] [Google Scholar]
- 27.Philip M. Questionnaire, rating scale and checklist- How do they differ? Methods in Psychology. 2024;1:100145. doi: 10.1016/j.metip.2024.100145. [DOI] [Google Scholar]
- 28.Harris K, Søfteland E, Moi AL, et al. Development and validation of patients’ surgical safety checklist. BMC Health Serv Res. 2022;22:259. doi: 10.1186/s12913-022-07470-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Bull C, Teede H, Watson D, et al. Selecting and Implementing Patient-Reported Outcome and Experience Measures to Assess Health System Performance. JAMA Health Forum. 2022;3:e220326. doi: 10.1001/jamahealthforum.2022.0326. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Data are available upon reasonable request.