Abstract
Objectives
The UK-developed patient measure of safety (PMOS) is a validated tool which captures patient perceptions of safety in hospitals. We aimed (1) to investigate the extent to which the PMOS is appropriate for use with stroke, acute myocardial infarction (AMI) and hip fracture patients in Australian hospitals and (2) to pilot the PMOS for use in a large-scale, national study ‘Deepening our Understanding of Quality in Australia’ (DUQuA).
Participants
Stroke, AMI and hip fracture patients (n=34) receiving care in 3 wards in 1 large hospital.
Methods
2 phases were conducted. First, a ‘think aloud’ study was used to determine the validity of PMOS with this population in an international setting, and to make amendments based on patient feedback. The second phase tested the revised measure to establish the internal consistency reliability of the revised subscales, and piloted the recruitment and administration processes to ensure feasibility of the PMOS for use in DUQuA.
Results
Of the 43 questions in the PMOS, 13 (30%) were amended based on issues patients highlighted for improvement in phase 1. In phase 2, a total of 34 patients were approached and 29 included, with a mean age of 71.3 years (SD=16.39). Internal consistency reliability was established using interitem correlation and Cronbach's α for all but 1 subscale. The most and least favourably rated aspects of safety differed between the 3 wards. A study log was categorised into 10 key feasibility factors, including liaising with wards to understand operational procedures and identify patterns of patient discharge.
Conclusions
Capturing patient perceptions of care is crucial in improving patient safety. The revised PMOS is appropriate for use with vulnerable older adult groups. The findings from this study have informed key decisions made for the deployment of this measure as part of the DUQuA study.
Keywords: Patient perceptions, patient safety, vulnerable older adults, aged care, mixed methods
Strengths and limitations of this study.
This was the first time the patient measure of safety (PMOS) has been used in Australia.
The sample size was targeted and modest, and limited to one site. Nonetheless, the participants were representative of the population that will be surveyed in the forthcoming national Deepening our Understanding of Quality in Australia (DUQuA) study.
Researchers partnered with hospital leaders and healthcare professionals to implement participant recruitment and data collection processes that can be replicated.
This study investigated feasibility of data collection with vulnerable older adult patients, which is of importance given the expected growth in older persons globally.
Background
Over the past two decades, the patient safety movement has urged hospitals to improve the safety of their practices.1–4 Although patients are frequently surveyed on the satisfaction of their experiences in hospital, these surveys do not capture information that can be used to improve safety of care.5 Patients’ own perceptions of the care provided to them is an overlooked yet well-placed resource for understanding the factors responsible for safety of care.6 Patients have demonstrated a willingness and capability to provide valuable feedback on both the safety of care delivered to them, and of the care environment (eg, delays to treatment, faulty equipment, miscommunication)—factors that busy healthcare professionals may not observe or report.5 7 With the right tools, patients can help hospitals to proactively identify safety issues, in order to intervene and prevent incidents from occurring.6
In recognition of this, a group of healthcare quality and safety experts in the UK developed the patient measure of safety (PMOS).5 6 The PMOS is the first valid and reliable tool that asks patients to comment on the factors that influence patient safety, according to a validated contributory factors framework of patient safety.8 It has been designed such that hospitals and wards can use it as a diagnostic guide for the design of patient safety improvement interventions by front-line staff.
While a measure to gain information about patient perceptions of safety can be useful, identifying whether patient perceptions of safety are associated with (1) how hospitals set themselves up for quality and safety, (2) the way care is delivered and (3) health outcomes for patients, is yet to be established. Understanding the extent of these relationships can assist policymakers and hospital leaders in making decisions about the target areas to invest resources to improve the quality and safety of healthcare.9 A large national study, ‘Deepening our Understanding of Quality in Australia’ (DUQuA;9 a modified replication of a European study of 188 hospitals (DUQuE)10), aims to explore the relationships between organisation and department-level quality management systems and patient factors in 60 large public hospitals across Australia.9 One of the factors that will be measured in DUQuA is patient-reported perceptions of safety, and for this the PMOS will be used.i
The patient conditions selected for the DUQuA study are stroke, acute myocardial infarction (AMI) and hip fracture. Thirty patients within each of these condition groups will be surveyed in each hospital (estimated total of 5400 patient participants) using the PMOS. However, to date, the PMOS has neither been tested in the Australian setting, nor with a principally vulnerable older adult population (the mean (SD) age of the original validation sample was 54 (18.13) years). Since stroke, AMI and hip fracture predominantly affect older adults in Australia,12–14 they are a particularly vulnerable sample due to the prevalence of condition-related complications or age-related comorbidities.15–19 As such, it is highly probable that these patients may experience pain or fatigue, and may suffer cognitive, hearing or visual impairments; factors that must be considered when planning recruitment, consent and data collection, and which can severely impact the quality and reliability of data.20–22 Therefore, it is important to identify and improve the suitability of the PMOS for use with this population prior to the full deployment of the measure. Finding strategies to overcome the challenges of undertaking research with hospitalised older adults23 is of significance due to an ageing population, and expected global growth of older persons—a factor associated with an increase in health expenditure.24
It is also crucial to ensure that the processes for conducting research with in-hospital AMI, stroke or hip fracture patients are appropriate, risk-free, consistent and effective. The large-scale nature of the national DUQuA study necessitates piloting the PMOS to explore ways to establish efficient, rigorous, pragmatic and ethical processes that can be replicated by the data collectors who will administer the measure in 60 Australian hospitals.9
Objectives
The objective of this study was to investigate whether the PMOS is appropriate to be used for stroke, AMI and hip fracture patients in Australian hospitals in order to ensure the feasibility of using this measure in the DUQuA study. The study had two phases. Phase 1, ‘think aloud’, aimed to test the face validity of the PMOS with the intended population for DUQuA (ie, Australian hospitalised older and vulnerable adults), and identify if amending the PMOS would improve understanding by, and relevance for, this population. Phase 2 aimed to test the revised measure to establish the internal consistency reliability of the revised subscales, and to pilot the PMOS to identify key areas within the recruitment and administration processes that could inform and enhance the feasibility of the DUQuA study.
Methods
Hospital engagement
The research team contacted the quality manager (QM) at a large metropolitan public hospital that has also enrolled in the DUQuA study. The research team advised the QM of two benefits for participating as the PMOS pilot site: provision of a report on patients’ perceptions of safety across the three wards, and that the experience would familiarise the hospital with data collection processes prior to the DUQuA study. The QM agreed that the hospital would participate as the pilot site for this study.
Participants
Sample type and size
Given that (1) the first five participants should usually reveal around 85% of the problems available for discovery in that iteration of an instrument25 26 and (2) an extensive amount of ‘think aloud’ research was undertaken on the originally developed PMOS, five patients with stroke were approached to participate in the ‘think aloud’ (phase 1). Stroke was selected as the most vulnerable cohort of the three conditions, due to the sensorimotor, communication and cognitive complications (eg, speech or hand impairments) faced by hospitalised patients with stroke.27 Recruitment continued until there was evidence of data saturation, consistent with other ‘think aloud’ studies.28 29
For phase 2, guidance was taken from the literature on appropriate sample sizes for pilot studies testing a new scale (n=30).30 As this was a pilot study, it was deemed appropriate to collect 10 PMOS questionnaires per ward, giving a total of 30 participants.
Eligibility
Detailed criteria were required to ensure the exclusion of patients who lacked the cognitive ability to give informed consent or who would be overly burdened by participation. Inclusion and exclusion criteria for both study phases are presented in table 1.
Table 1.
Participant inclusion and exclusion criteria
| Inclusion criteria | Exclusion criteria |
|---|---|
|
|
*Refer to the full DUQuA study protocol9 for primary diagnosis admission codes for each condition.
AMI, acute myocardial infarction; DUQuA, Deepening our Understanding of Quality in Australia.
Measure
The PMOS items were developed based on 33 interviews with inpatients from six units in one UK teaching hospital about their perceptions of patient safety.5 The face validity of the measure was established via 12 patients and 12 healthcare professionals. Following this, 297 patients from 11 hospital wards completed the PMOS questionnaire. Factor analysis revealed nine key domains of safety6 indicated in table 2, which align with a validated framework of contributory factors to patient safety.8 Items on the PMOS are phrased both positively and negatively, and scored using a five-point Likert scale (strongly disagree to strongly agree). Following reverse scoring for negative items, higher total scores represent more favourable responses.
Table 2.
Patient measure of safety subscales and example items
| Subscale (n items) | Example item |
|---|---|
| Communication and team work (10) | I always felt staff listened to me about my concerns |
| Organisation and care planning (5) | Staff gave me different information about my care |
| Access to resources (4) | Staff seemed to struggle to get help when they needed it |
| Ward type and layout (11) | Staff were prompt in answering my buzzer |
| Information flow (3) | After a shift change staff appeared to know important information about my care |
| Roles and responsibilities (4) | It wasn't clear to me who was in charge of my care |
| Staff training (2) | On at least one occasion a member of staff was not able to use the necessary equipment |
| Equipment design and functioning (2) | Equipment needed for my care was always working properly |
| Delays (2) | My treatment/procedure/operation always happened on time |
Data collection
Identifying and approaching patients
The research lead (NT) was introduced to the nurse unit managers (NUMs) for each ward by the QM. The research lead liaised with each NUM to determine a communication process to identify patients efficiently, causing as little disruption to normal working practice as possible. Over a 3-month period, a member of the research team called the wards on 1–2 mornings per week to determine if any eligible patients were due to be discharged from hospital. In the orthogeriatric ward, the research lead was provided with a list of highlighted patient names and bed numbers. In the stroke and cardiac wards, the NUM informed the research lead of the patient name and bed number.
Once identified, the research lead approached the patient with a standard introduction (box 1), which was adjusted (eg, through language simplification, emphasising word pronunciation, reducing the speed of word flow or repetition) based on individual patient responses or needs.
Box 1. Standard introduction for approaching patients.
▸ Hi, are you Mrs X? My name is Natalie. I have been told by the nurse in charge that you are due to be discharged soon—is that correct? (Patient response…)
▸ I am doing some research with Macquarie University to help improve this hospital and wondered if you would be willing to take part in a short patient questionnaire? (Patient response…)
▸ It is just to try and understand a bit more about your experience of care and safety since you have been on this [AMI/stroke/hip fracture] ward. It takes about 15 minutes and is confidential. (Patient response…)
▸ I have an information sheet for you to read and if you agree to take part there is just a consent form to sign. It is up to you whether you would like to complete the questionnaire yourself, have me read the questions out to you and you point to the answers, or a member of your family can help you to complete it too. (Patient response…)
Consent and screening
From the initial approach to finalising questionnaire administration, the research lead screened the consenting patients using a list of cues developed by Harris and Dyson22 (box 2). If the patient at any time demonstrated a lack of understanding of consent as per these cues, the patient was excluded from the study, regardless of the provision of consent.
Box 2. Harris and Dyson's22 screening cues to assess current cognitive understanding.
Was the patient able to concentrate on the brief introduction?
Did they appear to listen and engage in what the researcher was saying?
Did they respond appropriately? Did they ask appropriate questions?
Were there any signs of inconsistency in what they said?
Was there a lot of repetition in what the patient was asking?
Was the conversation appropriate or did the patient talk at length, constantly going off at tangents?
Were there other agendas and preoccupations that were far more important to the patient?
If the patient did not appear to understand, how did they deal with this? Did they deny not understanding? Or make appropriate suggestions, for example, using their next of kin to help them?
Did they appreciate the consequences of consenting or not consenting?
Did they appear to understand that there was a need for a decision?
Considerations for using the PMOS with vulnerable older adult patients
The PMOS could be completed either by the patient, or with assistance from a researcher.6 The research team anticipated that in implementing this measure with this group of participants, it would be likely that they would request assistance.31 The research lead used evidence-based techniques for interacting with older adult participants (eg, sit facing the patient, ensure lips are visible to the patient, check patients have their hearing aids or glasses if necessary, speak slowly and clearly, use visuals aids and large fonts).32 To assist patients with visual, speech or physical impairments,31 and to protect privacy of participants who may be concerned about others overhearing their answers on the ward,33 an enlarged Likert scale was designed for patients to point to as a way of indicating answers. The research lead also offered to read questions aloud, and at a slower pace, for patients with visual or hearing impairments, or both. ‘Thank you’ cards were provided for all patients who participated in the study.34
Phase 1—‘think aloud’
The ‘think aloud’ method has previously been used to assess validity and feasibility of health and social care measures for use with older adults,28 and was also used to assess face validity during the development of the original PMOS.5 During the ‘think aloud’ phase, the research lead sat with patients as they worked through the PMOS and prompted them to identify and elaborate on aspects of the questionnaire that were difficult to understand or presented cultural differences in terminology. Patients were asked to consider and discuss the following features with the research lead, who made notes during the conversation: presentation, terminology, content, phrasing and the number of questions.
Phase 2—testing the revised PMOS
Following amendments to the PMOS based on the ‘think aloud’, the revised measure was tested with patients recruited across the stroke, cardiac and orthogeriatric wards. The majority of participants requested assistance from the research lead in the form of reading out the questions and writing down answers. The average length of time to complete was ∼20 min per patient, ranging from 10 to 30 min.
Pilot test for the DUQuA study
From the outset of the study, the research team kept a project log of potentially relevant and important occurrences—any facet that may affect the feasibility of using the questionnaire. This included records of communication with the hospital's research office, conversations with ward staff, the research lead's observations of accessing wards, interacting with patients and family members, and administering the questionnaire. These occurrences were discussed by the research team during regular meetings. This was undertaken to inform and justify key decisions about data collection materials and processes for future use of the PMOS, in particular for the DUQuA study.
Analysis
Phase 1—‘think aloud’
The notes made by the research lead were collated and categorised, generating a list of recommended changes to the PMOS. Given the extensive development process in the UK, the research team consulted with the UK PMOS lead (RL) to discuss the suggested amendments. A consensus was reached based on findings from additional studies that had been undertaken or were underway in the UK, and revisions were made to the Australian version of the PMOS.
Phase 2—revised PMOS reliability test
The reliability of the PMOS subscales were assessed using interitem correlation for subscales with two items (r>0.2 acceptable),35–37 and Cronbach's α for subscales with more than two items (α>0.7 acceptable).38 For the total sample and the individual wards, mean (M) and standard deviation (SD) scores were calculated for each subscale, as well as a ‘negative index’, representing the mean number of items that each patient scored negatively (ie, ≤2).6
Pilot test for the DUQuA study
The project log was categorised into a set of key feasibility factors, with an explanation of how each had affected the decisions made for the DUQuA study. These key feasibility factors informed the development of training materials and resources for DUQuA data collectors.
Results
Phase 1—‘think aloud’
Five patients agreed to participate in the ‘think aloud’ study but on continuous screening during the third consultation, it was clear that this patient was not engaging with the researcher to answer questions appropriately. Therefore, the researcher thanked the patient and informed the NUM that, for this reason, the patient would be excluded. Consequently, four patient consultations were included in this phase. All four patients commented on three main areas: item phrasing, terminology and wording. As no new comments or suggestions were made, no additional patients were approached for further consultation.
Of the 43 questions in the PMOS, 13 (30%) were amended based on areas these patients highlighted for improvement (table 3). Recommended changes included amending healthcare terms that were less common in the Australian context, and rephrasing negatively phrased questions to positively framed questions. Following consultation with the UK team, the decision to reword all negatively phrased questions was not taken, as the research team believed in some instances the intended meaning of the questions would be altered as a result. The proportion of negatively phrased questions was therefore reduced from 56% to 28%. The new structure and reliability levels of the adapted PMOS are reported following phase 2.
Table 3.
Modifications to the patient measure of safety following phase 1
| Issues and areas for improvement | PMOS amendments made |
|---|---|
Item phrasing
|
Phrasing reworded
|
Terminology
|
Terminology changed
|
Wording
|
Wording modified
|
Phase 2—revised PMOS reliability test
Descriptive statistics and revised PMOS reliability
A total of 34 patients were approached following identification by the NUMs. One patient declined and two agreed to participate and signed a consent form, but later withdrew, citing reasons including feelings of tiredness, or the arrival of a physiotherapist for final assessment before discharge. Patients agreeing to participate (n=31) were from the cardiology (n=8), stroke (n=13) and orthogeriatic wards (n=10). One stroke and one hip fracture patient agreed to participate but on continuous screening by the research lead, it appeared they were confused and consequently these responses were excluded. Therefore, the number of included patients was 29 (n=8 cardiology, 12 stroke and 9 hip fracture).
The mean age of participants was 71.3 (SD=16.39) across all wards (stroke M=69.83, SD=18.89; cardiology M=62.25, SD=12.81; hip fracture M=82.10, SD=9.56). Two items with response rates lower than 25% were removed from analysis: ‘a doctor changed my plan of care and other staff know about it’, and ‘staff were kept waiting for my test results’. The indications were that patients did not know the answers to these questions. All PMOS subscales demonstrated acceptable reliability, based on either Cronbach's α (α>0.7) or interitem correlation (r>0.2), except for the organisation and care planning subscale, which had a Cronbach's α of 0.52. Additional file 1 compares the Cronbach's α, interitem correlations and mean subscale interitem correlations for the originally tested PMOS and our revised version.
Revised PMOS scores
As illustrated in table 4, the mean PMOS negative index for the entire sample was 2.59 (SD=2.72), suggesting that on average people answered 2–3 out of 43 questions negatively (once all items were reverse scored, a low score, ie, ≤2/5 indicates a negative answer). Individual item analysis revealed scores between M (SD)=3.26 (0.99; ‘staff were kept waiting for my test results’) and M (SD)=4.69 (0.47; ‘the drugs I have been prescribed were always available’)—this item also had the lowest SD, indicating limited use of the rating scale. The extent to which items were answered with ‘not applicable’ (NA) ranged from 0 (19 items) to 10 (1 item), with ‘a doctor changed my plan of care and other staff knew about it’, and ‘the following aspects of the ward made it difficult for staff to do their jobs: position of nurses’ station receiving n=8 NAs and ‘staff were kept waiting for my test results’ receiving 10.
Table 4.
PMOS scores
| Subscales (5-point Likert scale) | Stroke M (SD) | AMI M (SD) | Hip M (SD) | Total |
| Communication and team work | 4.53 (0.61)* | 4.50 (0.71) | 4.13 (0.90) | 4.40 (0.76) |
| Organisation and care planning | 4.45 (0.65) | 4.50 (0.94) | 4.17 (0.86) | 4.38 (0.81) |
| Access to resources | 4.38 (0.80) | 4.34 (0.87) | 3.74 (1.23) | 4.21 (0.97) |
| Ward type and layout | 4.40 (0.83) | 4.25 (0.88) | 4.20 (1.12) | 4.31 (0.93) |
| Information flow | 4.50 (0.59) | 4.38 (0.81) | 4.29 (0.61)* | 4.41 (0.66) |
| Roles and responsibilities | 3.96 (1.13)† | 4.13 (1.18)† | 3.68 (1.36) | 3.92 (1.22) |
| Staff training | 4.46 (0.66) | 4.69 (0.60)* | 4.18 (0.53) | 4.44 (0.63)* |
| Equipment (design and function) | 4.43 (0.88) | 4.47 (0.83) | 4.00 (0.94) | 4.31 (0.84) |
| Delays | 4.30 (0.88) | 4.20 (0.94) | 3.11 (1.13)† | 3.89 (1.11)† |
| PMOS negative index‡ | 1.67 (1.56) | 2.13 (2.23) | 4.22 (3.70)† | 2.59 (2.72) |
*Ward-based subscale most favourable scores from patients.
†Ward-based subscale with least favourable scores from patients.
‡Represents the mean number of items that each patient scored negatively (ie, ≤2).6
AMI, acute myocardial infarction; M, median; PMOS, patient measure of safety.
Among the nine key domains measured in the PMOS, patients rated staff training most favourably (M=4.44, SD=0.63) and the delays subscale least favourably (M=3.89, SD=1.11). It was noted that the delays subscale only contained two questions, and thus had a larger SD.
Ward-level results
Patients with stroke had the most favourable answers to the communication and team work subscale (eg, patients felt staff listened to their concerns). The roles and responsibilities subscale received the least favourable scores (eg, patients were uncertain about who was in charge of their care, or of the different roles of those caring for them).
Patients with AMI rated staff training most favourably (eg, patients felt staff could use equipment properly and carry out tasks they should be able to do). The roles and responsibilities subscale received the least favourable scores (eg, patients were uncertain about who was in charge of their care, or of the different roles of those caring for them).
Hip fracture patients answered more negatively on average than patients with stroke and AMI. The information flow subscale (eg, staff handed information over between shifts effectively) was the most favourable to hip fracture patients, and delays (eg, to treatment, or having too few staff to get things done on time) received the lowest mean score.
Pilot test for the DUQuA study
A list of 10 key feasibility factors is presented in table 5. For each factor, the details of findings from the pilot study are provided, alongside an explanation of how each factor has influenced the decisions made for the DUQuA study. In addition, the research lead provided hospital leaders with a report of the results and subsequently met to discuss the results in order to gain feedback about the report (ie, if the tone was appropriate and the information valuable), and to discuss how best to provide the hospital with recommendations for improvement.
Table 5.
Key feasibility factors and implications for the DUQuA study
| Key factors | Findings from the pilot | Implications for the DUQuA study |
|---|---|---|
| Ethics | Identifying possible ethical issues for working with vulnerable populations | Provide training in ethical research and clear recruitment instructions to data collectors coordinating PMOS |
| Liaising with wards | Timing—understanding likely times for discharge and working around this Appreciating the nature of the individual working wards and the best way to contact/ engage with the NUM |
Design instructions for data collectors to ensure they understand the need to find information about discharge timing to make the process efficient. Provide options in the instructions for data collectors to present to wards to arrange data collection |
| Questionnaire timing | Identifying how long it takes to undertake the questionnaire | Provide an experience-based estimate of the time required in instructions for data collectors |
| Identifying patients | Logistics for obtaining eligible patient names | Emphasise importance of awareness of patient's name before approaching them, that is, train data collectors to liaise with the NUMs to create a system suited for the individual wards |
| Approaching patients | Learning how to approach and talk to these cohorts of patients | Provide scripts for data collectors detailing how to approach the patients, and the order in which information should be provided and requests for participation made |
| Providing participant with assistance | Assistance from family members or carers, or the data collector | Provide scripts for data collectors to determine whether a family member or carer may be willing to assist the patient to complete the PMOS if the patient is physically unable (eg, reading out the questions, and/or circling the scales), interpret the questionnaire (for non-English-speaking patients), or if the patient would like assistance from the data collector |
| Presentation of information | Presentation of information sheets and consent forms in an easy to use format | Provide data collectors with colour-coded paper to simplify which documents belong to the patient or data collector |
| Additional data | Decisions about additional data that will be important to factor into the analysis | Include item on questionnaire form to indicate whether the patient completed the PMOS independently, with assistance from the data collector, or from a family member or carer |
| Specialised tools | Tested the use of specialised tools (eg, paper-based enlarged Likert scales) | Provide data collectors with laminated and enlarged versions of the questionnaire and Likert scale |
| Providing hospitals with recommendations | Identifying an appropriate level of practical recommendations for the hospital and the wards based on the findings | Consult with hospital leaders to ensure (1) the hospital is provided with appropriate and feasible recommendations and (2) the format of hospital reports is pitched at an appropriate and valuable level |
DUQuA, Deepening our Understanding of Quality in Australia; NUM, nurse unit manager; PMOS, patient measure of safety.
Discussion
Patients have the capacity to signal potential safety incidents early,39 40 and the PMOS has the capacity to reliably and efficiently tap into this resource.6 41 Testing the extent to which the PMOS can capture this information from vulnerable older adult patients in a reliable and risk-free way was a key aim of this study. Following a ‘think aloud’, the revised PMOS was tested with patients with stroke, AMI and hip fracture in one Australian hospital to establish internal consistency reliability of the revised subscales. While the ‘organisation and care planning’ produced a Cronbach's α of 0.52, the comparisons for the Cronbach's α, interitem correlations and mean subscale interitem correlations for the originally tested PMOS and our revised version demonstrate similar findings (additional file 1), despite our smaller sample size in our pilot study. Throughout the study, the research team identified key factors relating to hospital engagement, participant recruitment, questionnaire administration and providing feedback to hospitals that will inform and enhance the feasibility of using this measure for the national DUQuA study.
One key amendment to the PMOS, based on the results of the ‘think aloud’, was the reduction of negatively worded items from 56% to 28% (24–12 items) to reduce the possibility of confusing, distressing or eliciting unreliable answers from participants. The decision to make these changes is supported by previous research with acutely ill hospitalised older adult patients, which found that simpler questions are more effective.31 On testing the revised version in phase 2, internal consistency reliability was established for all but one subscale, and differences were observed for the most and least favourably rated areas of safety on the three wards. However, the modest sample size within each ward did not warrant statistical analysis of these differences. Item analysis revealed three questions had been completed with between 8 and 10 NA answers. It is understandable that patients may not know the answers to these questions (eg, they may have had no way of knowing if their test results had been delayed, or where the nurse’s station is positioned to enable an opinion to be formed about how this might have affected the difficulty of a nurse's job). In the deployment of the PMOS, participants should be encouraged to comment on why a particular question is not applicable to them in order to support the explanation of results.
By piloting the PMOS, the research team have experienced the process of administering the questionnaire and have developed ideas for improving the feasibility of using this measure with an older, vulnerable cohort of patients (the mean age of patients in this study was 71.3 years, 17.3 years older than the sample used for the original PMOS validation) for the DUQuA study. Examples of some of the key feasibility findings include liaising with wards to understand operational procedures and identify patterns of patient discharge. Making initial contact with each of the NUMs to make a collaborative decision about the days and times of day to call to learn of new patient discharges and arrange subsequent visits to the ward was crucial for the smooth recruitment of patients. This was particularly important due to the unpredictable nature of the discharge process, which has been previously highlighted as a challenging point at which to attempt to collect data from patients.32
While specific resources were developed for data collection based on recommendations from previous research with acutely ill hospitalised older adult patients, this work highlighted areas for improvement to the type and format of resources that need to be provided. For example, in this study, patients typically needed the research lead to read aloud and fill out the questionnaire based on their answers, and were also provided with an enlarged (font size 28) Likert scale to point to the rating of their choice. However, findings indicated patients also preferred to see the questionnaire while it was read aloud and often leaned over from their bed towards the research lead's version of the PMOS, causing patients to move into a position that was not ideal for their condition. As the research lead read the questions aloud, it would have been useful to have enlarged versions of the entire questionnaire and scale for patients to view (eg, a minimum font size of 15 to ensure letter height based on visual arc is sufficient for viewing capability)42 to ensure they were comfortable throughout participation, a factor previously identified as important.34 Laminating these resources would enable the research lead to easily identify them among the different sets of documentation (eg, consent forms, information sheets, revocation sheets, the questionnaire itself), and to reuse them with all patients. Colour coding the documentation would also have been beneficial, to enable easy identification of the different paperwork.
Implications for health services research
The experiences in this PMOS pilot and feasibility study have influenced key decisions for the use of this measure in the national DUQuA study (table 5), and will inform the design of future health services research studies measuring patient perceptions. For example, detailed instructions and webinars have been developed for the DUQuA data collectors to ensure they understand the need to develop approaches for (1) engaging with the wards; (2) finding information about timing of discharge; and (3) determining when it is appropriate to visit the ward, to minimise disruption for staff and to make the process efficient. These instructions include strategies that data collectors can present to wards to arrange data collection. Furthermore, the data collection resources (ie, laminated versions of enlarged scales and complete questionnaires) have been modified to maximise the ease for patients to visualise and use the materials while maintaining a comfortable position, and to improve the logistics of the process for the data collector.
This preliminary work also enabled the research team to consider the format of feedback to provide to the participating hospital and respective wards, an important factor for bolstering senior healthcare management decision-making for improvement based on research evidence.43 Throughout the production and dissemination of the hospital report based on this work, the research team liaised and met with senior hospital leaders to discuss the possible approaches in presenting feedback to ensure that (1) the hospital and participating wards were provided with appropriate and realistic recommendations and (2) the reports for the cohort of DUQuA hospitals will be pitched at an appropriate and valuable level.
Strengths and limitations
Factors including selection and investigator bias, randomisation and the use of a control group could have been improved within the study. For practical reasons, we had to rely on the ward managers to identify the patients based on their discharge lists for the day. While patients were not randomly selected, the full list of patients due to be discharged was used, and all patients on this list were approached, removing an element of selection bias. The investigator was chosen as an impartial, unbiased person to administer the questionnaire, rather than members of staff working on the ward, in order to reduce the risk of patients providing responses that were more favourable than their genuine perceptions (eg, for fear of offending staff who may have directly cared for them). Limited timeframes and the difficulty recruiting these types of patients meant the decision to use a control group or comparison questionnaire was not taken.
While only one hospital was used, it is typical of the DUQuA hospital sample (ie, a tertiary metropolitan public hospital that treats all three conditions, with ∼300 beds). Furthermore, although this study used modest samples of patients in each phase, the sample was representative of the forthcoming DUQuA study population (ie, in terms of the inclusion criteria, including the primary diagnosis for admission—fully specified in the DUQuA study protocol9), which bolsters the ecological validity of this feasibility test. The sample size across individual wards did not allow for statistical analysis of ward differences that were observed. Nonetheless, internal consistency reliability was established for all but one subscale, and the DUQuA study aims to test the PMOS with 1800 patients from each condition (5400 total) across 60 hospitals. Therefore, sample sizes to detect differences between wards for patient ratings of these areas of safety will be adequately powered. Despite the small sample size, feedback to the research team from the hospital has indicated value in the results for generating ideas for targeted improvement.
Conclusion
For the first time internationally, a cohort of vulnerable patients provided their perceptions of the care they received on the respective wards using the PMOS, a tool that highlights evidence-based contributory factors to the delivery of safe patient care, and signals areas for hospitals to target for improvement. Ensuring that the questionnaire is appropriate for use with these patient groups is highly significant for the DUQuA study, and will inform the deployment of this questionnaire nationally, and of future health services research projects. With a larger, more representative sample, DUQuA will further strengthen the validity and reliability of the PMOS for the benefit of Australian and international hospitals and institutions. The DUQuA study will also assess the strength of relationships between the PMOS and the way hospitals are set up for quality and safety, the way care is delivered, and patient outcomes.
Acknowledgments
The authors would like to acknowledge the Quality Manager and Director of Medical Services of the participating hospital for their assistance in facilitating the pilot study, and the NUMs of the three wards involved for their help in recruiting participants. They would also like to thank Dr Annette Pantle for her insight into engaging and providing appropriate feedback to hospitals, Gemma Louch for her time taken to consult on the revised PMOS items, and Sally Moore for providing advice about procedures for identifying and recruiting patients.
Footnotes
Twitter: Follow Natalie Taylor at @njt14
Contributors: NT conceived the study design, and led the data collection, analysis and development of the manuscript. EH contributed to the design of participant recruitment and data collection strategies, the ethics application, and the development of the manuscript. RC-W contributed to design of data collection strategies and revision of the manuscript. ZL contributed to the design of data collection strategies and the revision of the manuscript. RL contributed to amendments of PMOS and to the development of the manuscript. JB is the chief investigator (CI) for the NHMRC grant (APP1054146) which funded this research and contributed to the development of the manuscript. All authors read, commented on and agree with the final version of the manuscript.
Funding: This work was supported by National Health and Medical Research Council (NHMRC) Program Grant APP1054146 (CI: JB).
Competing interests: None declared.
Ethics approval: Ethical approval was granted for this study by a National Health and Medical Research Council (NHMRC) certified Human Research Ethics Committee (HREC), May 2014. An approval letter containing HREC name and reference number has been submitted to the editors. The DUQuA study has also received ethical approval from multiple NHMRC certified HRECs.9
Provenance and peer review: Not commissioned; externally peer reviewed.
Data sharing statement: Data will be made publicly available to the extent that individual participants or participating hospitals cannot be identified, in accordance with requirements of the approving Human Research Ethics Committees.
The original DUQuE study found that patient experience showed no relationship with hospital quality management strategies.11 However, in the DUQuE study, the Nordic Patient Experiences Questionnaire (NORPEQ) was used, for which we identified some limitations that may have influenced the strength of the relationship. More specifically, in the NORPEQ development and validation study, the questionnaire was completed by patients independently at home following in-patient care, and the measure does not have any scales to measure experiences of safety; whereas in the DUQuE study used in-hospital patient completion, and the measures of quality management systems that were compared with the NORPEQ results focused primarily on ‘patient safety’ (as opposed to general experience).
The original PMOS was developed and validated with an in-hospital patient sample, and measures patient experiences of safety on the ward which provided the majority of their care. The PMOS is also designed based on contributory factors to patient safety framework. For these reasons, and the issues outlined with the measures used in the original DUQuE study, the PMOS was identified as an appropriate tool for our national DUQuA study.
References
- 1.Wilson AR, Fabri PJ, Wolfson J. Human error and patient safety: interdisciplinary course. Teach Learn Med 2012;24:18–25. 10.1080/10401334.2012.641482 [DOI] [PubMed] [Google Scholar]
- 2.Kohn K, Corrigan JM, Donaldson MS, eds. To err is human: building a safer health system. Washington DC: National Academies Press, 2000. [PubMed] [Google Scholar]
- 3.Leape LL. Errors in medicine. Clin Chim Acta 2009;404:2–5. 10.1016/j.cca.2009.03.020 [DOI] [PubMed] [Google Scholar]
- 4.Reason J. Human error: models and management. BMJ 2000;320:768–70. 10.1136/bmj.320.7237.768 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Giles SJ, Lawton RJ, Din I et al. Developing a patient measure of safety (PMOS). BMJ Qual Saf 2013;22:554–62. 10.1136/bmjqs-2012-000843 [DOI] [PubMed] [Google Scholar]
- 6.McEachan RR, Lawton RJ, O'Hara JK et al. , Yorkshire Quality and Safety Research Group. Developing a reliable and valid patient measure of safety in hospitals (PMOS): a validation study. BMJ Qual Saf 2014;23:565–73. 10.1136/bmjqs-2013-002312 [DOI] [PubMed] [Google Scholar]
- 7.Weissman JS, Schneider EC, Weingart SN et al. Comparing patient-reported hospital adverse events with medical record review: do patients know something that hospitals do not? Ann Intern Med 2008;149:100–8. 10.7326/0003-4819-149-2-200807150-00006 [DOI] [PubMed] [Google Scholar]
- 8.Lawton R, McEachan RRC, Giles SJ et al. Development of an evidence-based framework of factors contributing to patient safety incidents in hospital settings: a systematic review. BMJ Qual Saf 2012;21:369–80. 10.1136/bmjqs-2011-000443 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Taylor N, Clay-Williams R, Hogden E et al. Deepening our Understanding of Quality in Australia (DUQuA): a study protocol for a nationwide, multilevel analysis of relationships between hospital quality management systems and patient factors. BMJ Open 2015;5:e010349 10.1136/bmjopen-2015-010349 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Secanell M, Groene O, Arah OA et al. DUQuE Project Consortium. Deepening our understanding of quality improvement in Europe (DUQuE): overview of a study of hospital quality management in seven countries. Int J Qual Health Care 2014;26(Suppl 1):5–15. 10.1093/intqhc/mzu025 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Groene O, Arah OA, Klazinga NS et al. Patient experience shows little relationship with hospital quality management strategies. PLoS ONE 2015;10:e0131805 10.1371/journal.pone.0131805 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.AIHW. Stroke and its management in Australia: an update. Cardiovascular disease series 37. Cat. no. CVD 61 Canberra: Australian Institute of Health and Welfare, 2013. [Google Scholar]
- 13.Nichols M, Peterson K, Alston L et al. Australian heart disease statistics 2014. Melbourne: National Heart Foundation of Australia, 2014. [Google Scholar]
- 14.AIHW. The problem of osteoporotic hip fracture in Australia. Bulletin no. 76. Cat. no. AUS 121 Canberra: Australian Institute of Health and Welfare, 2010. [Google Scholar]
- 15.Langhorne P, Stott DJ, Robertson L et al. Medical complications after stroke: a multicenter study. Stroke 2000;31:1223–9. 10.1161/01.STR.31.6.1223 [DOI] [PubMed] [Google Scholar]
- 16.Taylor A, Price K, Gill TK et al. Multimorbidity—not just an older person's issue. Results from an Australian biomedical study. BMC Public Health 2010;10:718 10.1186/1471-2458-10-718 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Carpintero P, Caeiro JR, Carpintero R et al. Complications of hip fractures: a review. World J Orthop 2014;5:402–11. 10.5312/wjo.v5.i4.402 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Ostwald SK, Wasserman J, Davis S. Medications, comorbidities, and medical complications in stroke survivors: the CAReS Study. Rehabil Nurs 2006;31:10–14. 10.1002/j.2048-7940.2006.tb00004.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.McManus DD, Nguyen HL, Saczynski JS et al. Multiple cardiovascular comorbidities and acute myocardial infarction: temporal trends (1990–2007) and impact on death rates at 30 days and 1 year. Clin Epidemiol 2012;4:115–23. 10.2147/CLEP.S30883 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Hall S, Longhurst S, Higginson IJ. Challenges to conducting research with older people living in nursing homes. BMC Geriatr 2009;9:38 10.1186/1471-2318-9-38 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Uman GC, Urman HN. The challenges of conducting clinical nursing research with elderly populations. AORN J 1990;52:400–6. 10.1016/S0001-2092(07)68173-3 [DOI] [PubMed] [Google Scholar]
- 22.Harris R, Dyson E. Recruitment of frail older people to research: lessons learnt through experience. J Adv Nurs 2001;36:643–51. 10.1046/j.1365-2648.2001.02029.x [DOI] [PubMed] [Google Scholar]
- 23.Chouliara Z, Kearney N, Worth A et al. Challenges in conducting research with hospitalized older people with cancer: drawing from the experience of an ongoing interview-based project. Eur J Cancer Care (Engl) 2004;13:409–15. 10.1111/j.1365-2354.2004.00543.x [DOI] [PubMed] [Google Scholar]
- 24.United Nations. World population ageing 2013. ST/ESA/SER.A/348 New York: United Nations, Department of Economic and Social Affairs, Population Division, 2013. [Google Scholar]
- 25.Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems; ACM, 1993:206–13. [Google Scholar]
- 26.Borsci S, Federici S. The partial concurrent thinking aloud: a new usability evaluation technique for blind users. Assistive technology from adapted equipment to inclusive environments. Assistive Technology Research Series 2009;25:421–5. [Google Scholar]
- 27.National Stroke Foundation. Clinical guidelines for stroke management 2010. Melbourne, Australia: National Stroke Foundation, 2010. [Google Scholar]
- 28.van Leeuwen KM, Jansen AP, Muntinga ME et al. Exploration of the content validity and feasibility of the EQ-5D-3L, ICECAP-O and ASCOT in older adults. BMC Health Serv Res 2015;15:201 10.1186/s12913-015-0862-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Whitty JA, Walker R, Golenko X et al. A think aloud study comparing the validity and acceptability of discrete choice and best worst scaling methods. PLoS ONE 2014;9:e90635 10.1371/journal.pone.0090635 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Johanson GA, Brooks GP. Initial scale development: sample size for pilot studies. Educ Psychol Meas 2010;70:394–400. 10.1177/0013164409355692 [DOI] [Google Scholar]
- 31.Hancock K, Chenoweth L, Chang E. Challenges in conducting research with acutely ill hospitalized older patients. Nurs Health Sci 2003;5:253–9. 10.1046/j.1442-2018.2003.00159.x [DOI] [PubMed] [Google Scholar]
- 32.Berkman CS, Leipzig RM, Greenberg SA et al. Methodologic issues in conducting research on hospitalized older people. J Am Geriatr Soc 2001;49:172–8. 10.1046/j.1532-5415.2001.49039.x [DOI] [PubMed] [Google Scholar]
- 33.Samelson EJ, Kelsey JL, Kiel DP et al. Issues in conducting epidemiologic research among elders: lessons from the MOBILIZE Boston study. Am J Epidemiol 2008;168:1444–51. 10.1093/aje/kwn277 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Peel NM, Wilson C. Frail older people as participants in research. Educ Gerontol 2008;34:407–17. 10.1080/03601270701835973 [DOI] [Google Scholar]
- 35.Hooper D, Coughlan J, Mullen M. Structural equation modelling: guidelines for determining model fit. Electron J Bus Res Methods 2008;6:53–60. [Google Scholar]
- 36.Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Modeling 1999;6:1–55. 10.1080/10705519909540118 [DOI] [Google Scholar]
- 37.Motl RW, Conroy DE. Confirmatory factor analysis of the physical self-efficacy scale with a college-aged sample of men and women. Meas Phys Educ Exerc Sci 2000;4:13–27. 10.1207/S15327841Mpee0401_3 [DOI] [Google Scholar]
- 38.George D, Mallery P. SPSS for windows step by step: a simple guide and reference, 11.0 update. 4th edn Boston: Allyn & Bacon, 2003. [Google Scholar]
- 39.Bacon N. A smoke-alarm for patient safety and healthcare quality. Neil Bacon Blog, 2010. [Google Scholar]
- 40.Ward JK, Armitage G. Can patients report patient safety incidents in a hospital setting? A systematic review. BMJ Qual Saf 2012;21:685–99. 10.1136/bmjqs-2011-000213 [DOI] [PubMed] [Google Scholar]
- 41.Lawton R, O'Hara JK, Sheard L et al. Can staff and patient perspectives on hospital safety predict harm-free care? An analysis of staff and patient survey data and routinely collected outcomes. BMJ Qual Saf 2015;24:369–76. 10.1136/bmjqs-2014-003691 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Weinger M, Wiklund M, Gardner-Bonneau D, eds. Handbook of human factors in medical device design. Boca Raton, FL: CRC Press, 2011. [Google Scholar]
- 43.Oxman AD, Lavis JN, Fretheim A et al. SUPPORT Tools for evidence-informed health Policymaking (STP) 16: using research evidence in balancing the pros and cons of policies. Health Res Policy Syst 2009;7(Suppl 1):S16 10.1186/1478-4505-7-S1-S16 [DOI] [PMC free article] [PubMed] [Google Scholar]