A systematic review of the validity and reliability of patient‐reported experience measures

Abstract

Objectives

To identify patient‐reported experience measures (PREMs), assess their validity and reliability, and assess any bias in the study design of PREM validity and reliability testing.

Data Sources/Study Setting

Articles reporting on PREM development and testing sourced from MEDLINE, CINAHL and Scopus databases up to March 13, 2018.

Study Design

Systematic review.

Data Collection/Extraction Methods

Critical appraisal of PREM study design was undertaken using the Appraisal tool for Cross‐Sectional Studies (AXIS). Critical appraisal of PREM validity and reliability was undertaken using a revised version of the COSMIN checklist.

Principal Findings

Eighty‐eight PREMs were identified, spanning across four main health care contexts. PREM validity and reliability was supported by appropriate study designs. Internal consistency (n = 58, 65.2 percent), structural validity (n = 49, 55.1 percent), and content validity (n = 34, 38.2 percent) were the most frequently reported validity and reliability tests.

Conclusions

Careful consideration should be given when selecting PREMs, particularly as seven of the 10 validity and reliability criteria were not undertaken in ≥50 percent of the PREMs. Testing PREM responsiveness should be prioritized for the application of PREMs where the end user is measuring change over time. Assessing measurement error/agreement of PREMs is important to understand the clinical relevancy of PREM scores used in a health care evaluation capacity.

1. INTRODUCTION

Patient‐reported experience measures (PREMs) are tools that capture “what” happened during an episode of care, and “how” it happened from the perspective of the patient.1, 2, 3 PREMs differ from patient‐reported outcome measures (PROMs), which aim to measure patients’ health status,4 and the more subjective patient satisfaction measures, which are an indication of how well a patient's expectations were met,5 a benchmark which is criticized for being too heavily influenced by past health care encounters.6

Patient‐reported experience measures are gaining attention as an indicator of health care quality and can provide information regarding the patient‐centeredness of existing services as well as areas for potential improvement regarding health care delivery.7 The purpose of employing PREMs is consistent with the Institute of Medicine's (IOM) definition of health care quality, defined as care that is patient‐centered, effective, efficient, timely, and equitable.8 In recent years, PREMs have been used to inform pay‐for‐performance (P4P) and benchmarking schemes, adjunct with other health care quality domains, including clinical quality/effectiveness, health information technology, and resource use.9, 10 Such schemes see health care services financially rewarded for their performance across these domains of health care quality, as opposed to the traditional fee‐for‐service payment system, which may inadvertently promote low‐value care.10, 11

While there is evident merit behind utilizing PREMs in health care quality evaluations, there remains some conjecture regarding their use. Manary and colleagues12 identify three main limitations expressed by critics of PREMs. Firstly, patient‐reported experience is largely seen as congruent with terms such as “patient satisfaction” and “patient expectation,” both of which are subjective terms that can be reflective of judgments on the adequacy of health care and not the quality.12, 13, 14 Secondly, PREMs may be confounded by factors not directly related to the quality of health care experienced by the patient, such as health outcomes.12 And finally, PREMs can be a reflection of patients’ preconceived health care “ideals” or expectations and not their actual care experience.12 All three limitations are indicative of a blurring of concept boundaries and inappropriate interchanging of concepts. While this is not unique to PREMs, it does suggest a low level of concept maturity regarding patient‐reported experiences15 and, consequently, is an area of research that warrants greater attention.

Despite these limitations, PREMs have gained international recognition as an indicator of health care quality. This is largely because: (a) they enable patients to comprehensively reflect on the interpersonal aspects of their care experience16; (b) they can be utilized as a common measure for public reporting, benchmarking of institutions/centers and health care plans10; and (c) they can provide patient‐level information that is useful in driving service quality improvement strategies.17, 18

Understanding the validity and reliability of PREMs is integral to the appropriate selection of instruments for quality assessment of health care services, in conjunction with other aspects, such as the clinical relevance of an instrument and the domains of patient‐reported experience that the PREM covers. Validity refers to the ability of an instrument to measure what it intends to measure, and reliability refers to the ability of an instrument to produce consistent results under similar circumstances, as well as to discriminate between the performance of different providers.19, 20 It is important to assess these properties in order to understand the risk of bias that may arise in employing certain instruments21 and whether instruments are suitable for capturing patient‐reported experience data.

While two previously published systematic reviews have examined the psychometric testing of PREMs, one related to PREMs for inpatients,16 and the other for emergency care service provision,22 there has been no comprehensive examination of the tools available across a range of health care contexts. The aim of this systematic review was to identify PREMs, assess their validity and reliability, and assess any bias in the study design of PREM validity and reliability testing, irrespective of the health care context the PREMs are designed to be used in.

1.1. Objectives

1. To identify existing tools for measuring patient‐reported experiences in health care, irrespective of the context

2. To critically appraise bias in the study design employed in PREM validity and reliability testing, and

3. To critically appraise the results of validity and reliability testing undertaken for these PREMs.

2. METHODS

In conducting this systematic review, the authors conformed to the Preferred Reporting Items for Systematic Reviews and Meta‐Analysis (PRISMA) statement.23 This review was registered with PROSPERO (registration number: CRD42018089935).

2.1. Search strategy and eligibility criteria

The databases searched were MEDLINE (Ovid), CINAHL Plus with full text (EBSCOhost), and Scopus (Elsevier). No date restriction was applied to the search strategy; records were searched up to March 13, 2018. Patient “satisfaction” was included in search terms in order to not to limit our results as there is a blurring of these terms and some PREMs may be labeled as satisfaction measures.

Articles were included in the systematic review if they met all the following criteria:

• Described the development and evaluation of PREMs

• Published in English

• Full‐text published in peer‐reviewed journals

• Labeled as a satisfaction questionnaire, but framed around measuring patients’ experiences (eg, the Surgical In‐Patient Satisfaction (SIPs) instrument24)

Articles were excluded if they met any of the following criteria:

• Instruments labeled as a satisfaction questionnaire which were: (a) framed around measuring patient levels of satisfaction; (b) inclusive of a global satisfaction question or visual analogue scale; and (c) developed based on satisfaction frameworks or content analyses

• Patient expectation questionnaires

• Quality of care questionnaires

• Patient participation questionnaires

• Related to patient experiences of a specific treatment or intervention (eg, insulin devices, hearing aids, food services, anesthesia, and medication/pharmaceutical dispensary) or specific program (eg, education programs)

• Measuring emotional care received by patients (eg, empathy)

• Studies where PREMs were completed entirely by proxy (completed by populations not receiving the care); however, if proxy‐reported data comprised only a small proportion of data collected (patient‐reported data also reported), then the study was still included

• Quality improvement initiatives

• Patient attitude scales

• Checklists

• Patient experience questionnaires comprised of a single domain, or

• PREMs superseded by a more up‐to‐date version of the same PREM with corresponding updated validity and reliability testing

The full search strategy for each database is provided in Appendix S1. All references were imported into EndNote (Version 8, Clarivate Analytics), and duplicates were removed. Two reviewers independently screened paper titles and abstracts for inclusion. Where the title and abstract were not informative enough to make a decision, the full‐text article was reviewed. Figure 1 depicts the PRISMA flow diagram of this process. The two reviewers handled disagreements regarding article inclusion or exclusion. Where a decision could not be made, a third reviewer adjudicated the final decision. Reference list handsearching was also employed for the identification of PREMs not identified through database searching, and updates for PREMs originally identified through the database searching.

Figure 1.

PRISMA diagram of patient‐reported experience measure search [Color figure can be viewed at wileyonlinelibrary.com]

2.2. Data extraction

Descriptive data were independently extracted from the included articles by two reviewers into a standardized excel extraction form (refer to Appendix S2). Discrepancies in the extracted data were discussed between the two reviewers, or adjudicated by a third if necessary. If there was insufficient information in the full‐text article regarding the validity and reliability testing undertaken, the article was excluded.

2.3. Critical appraisal

To critically appraise bias in the study design employed in PREM validity and reliability testing, the Appraisal tool for Cross‐Sectional Studies (AXIS)25 was used. This is a 20‐item appraisal tool developed in response to the increase in cross‐sectional studies informing evidence‐based medicine and the consequent importance of ensuring that these studies are of high quality and low bias.25 The purpose of employing the AXIS tool in the present systematic review was to ensure that the results of PREM validity and reliability testing were supported by appropriate study designs and thus able to be interpreted as a robust representation of how valid and/or reliable a PREM is. The AXIS assesses the quality of cross‐sectional studies based on the following criteria: clarity of aims/objectives and target population; appropriate study design and sampling framework; justification for the sample size; measures taken to address nonresponders and the potential for response bias; risk factors/outcome variables measured in the study; clarity of methods and statistical approach; appropriate result presentation, including internal consistency; justified discussion points and conclusion; discussion of limitations; and identification of ethical approval and any conflicts of interest.25

The scoring system conforms to a “yes,” “no,” or “do not know/comment” design. PREMs were categorized into quartiles: >15 AXIS criteria met, 10‐15 AXIS criteria met, 5‐9 AXIS criteria met, and ≤4 AXIS criteria met. The AXIS tool was used to appraise the most recent publication for each PREM as this was also reflective of the most recent version of validity and reliability testing that the PREM that had undergone.

To assess the validity and reliability testing undertaken for PREMs included in this review, we employed a revised version of the COSMIN checklist (COnsensus‐based Standards for the selection of health status Measurement INstruments) published in a recent systematic review of quality in shared decision making (SDM) tools.19 These criteria is comprised of 10 psychometric measurement properties and subproperties, including internal consistency; reliability; measurement error/agreement; validity (inclusive of content validity, construct validity [structural validity, hypothesis testing, and cross‐cultural validity]; and criterion validity); responsiveness; and item response theory (IRT). Appendix S3 provides definitions for each of these measurement properties and identifies the appraisal parameters used to assess them.26

Reporting of these measurement properties conforms to the following: “+” (meets criteria), “−” (does not meet criteria), or  "?” (unclear or missing information). These scores were numerically coded, and PREMs were ranked within their corresponding context(s) (refer to Appendix S4). Where more than one article was identified for the validity and reliability testing of a PREM, all articles were used to critically appraise the PREM. If the same criteria were assessed in separate studies for a given PREM and provided conflicting results (eg, a “+” and a “−” score), then the more favorable result was recorded.

Appraisals with both tools were undertaken by one author. A sample of the revised COSMIN checklist appraisal data was cross‐checked with a second reviewer. A Kappa measure was used to assess the level of inter‐rater agreement. A Kappa value of 0.5 depicted moderate agreement, >0.7 good agreement, and >0.8 very good agreement.27

3. RESULTS

A total of 88 PREMs were identified through the systematic literature search. Greater than one‐third of these instruments were contextually designed for inpatient care services (36.4 percent), 23.9 percent for primary care services and 12.5 percent for outpatient care services. Table 1 depicts the other contexts and conditions covered by the PREMs. Roughly 20 percent of instruments were developed in the UK, while other countries included the United States (19.3 percent), Norway (14.8 percent), and the Netherlands (13.6 percent). The most common mode of PREM administration was postal (45.7 percent), followed by face to face (33.1 percent), telephone (13.6 percent), and electronic (7.6 percent). The earliest PREMs detected through the systematic search were developed in 1993.28, 29 The median number of items per PREM was 27 (IQR: 21‐35; range: 4‐82), and the median number of domains was 5 (IQR: 4‐7; distribution: 2‐13). Extracted data can be identified in Appendix S2.

Table 1.

PREMs identified by individual, condition, setting, and country‐specific context

Contexta	Number of PREMs (%)
Individual‐specific
Child/adolescent care57, 58	2 (2.3)
Low‐income59	2 (2.3)
Homeless60	1 (1.1)
Not individual‐specific	83 (94.3)
Condition‐specific
Mental health61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73	10 (11.4)
Palliative care/cancer36, 74, 75, 76	5 (5.7)
Renal (including dialysis)77, 78, 79	3 (3.4
Rheumatoid arthritis80, 81, 82	3 (3.4)
Substance dependence64, 65, 83	2 (2.3)
Trauma84, 85, 86, 87	2 (2.3)
Chronic disease88	1 (1.1)
Cystic fibrosis89	1 (1.1)
Maternity90, 91	1 (1.1)
Parkinson's disease92	1 (1.1)
Not condition‐specific	59 (67.1)
Setting
Inpatient services28, 30, 31, 59, 61, 63, 66, 72, 73, 76, 81, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108	32 (36.4)
Day surgery28	1 (1.1)
Rehabilitation81, 99	2 (2.3)
Preoperative109	1 (1.1)
Postoperative47, 110	3 (3.4)
Primary care services34, 46, 60, 69, 88, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132	21 (23.9)
Medical home133	1 (1.1)
Out‐of‐hours care113, 118	2 (2.3)
Home care29	1 (1.1)
Outpatient services59, 67, 68, 71, 75, 76, 109, 134, 135, 136, 137, 138	11 (12.5)
Accident and emergency department services139, 140, 141, 142	3 (3.4)
Dental services143, 144	2 (2.3)
Integrated care services145, 146	2 (2.3)
Not specified35, 37, 147, 148, 149, 150, 151, 152, 153, 154, 155	5 (5.7)
Not setting‐specific	1 (1.1)
Country
UK28, 58, 66, 69, 75, 80, 87, 98, 99, 104, 112, 113, 114, 119, 122, 124, 126, 129, 139, 142, 147, 148	18 (20.5)
USA29, 34, 35, 57, 60, 64, 65, 74, 78, 79, 89, 116, 117, 123, 127, 131, 133, 143, 146, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165	17 (19.4)
Norway31, 37, 61, 67, 71, 76, 81, 83, 102, 103, 118, 121, 130, 135	13 (14.8)
Netherlands36, 62, 77, 82, 90, 91, 92, 93, 105, 140, 141, 166, 167	13 (14.8)
Australia70, 115, 120, 144	4 (4.5)
Spain88, 94, 96, 109	4 (4.5)
Canada47, 84, 85, 86, 137, 138	3 (3.4)
Hong Kong24, 95, 106, 107, 108	3 (3.4)
Sweden68, 72, 73, 122	3 (3.4)
China30, 136	2 (2.3)
Ethiopia59	2 (2.3)
Germany63, 145	2 (2.3)
Europeb , 46, 111, 125, 132	1 (1.1)
France100	1 (1.1)
Saudi Arabia/UAE149	1 (1.1)
Taiwan134	1 (1.1)
Total	88 (100)

Abbreviations: PREM, patient‐reported experience measure; UAE, United Arab Emirates; UK, United Kingdom; USA, United States of America.

^aSome tools were embedded across contexts.

^bOne study was conducted across 14 countries in Europe.

A proxy, not the recipient of care, completed PREMs on the behalf of patients in 11.4 percent of the PREMs. This was typically only for a small portion (10‐12 percent) of any given study's population. Over 40 percent of the PREMs were developed and tested in languages other than English. Few papers discuss formal translation processes being undertaken for PREMs.

3.1. AXIS critical appraisal

Table 2 identifies that 63 (70.5 percent) of the papers reporting on PREMs met >15 AXIS criteria. Over a quarter of studies met 10‐15 criteria (28.4 percent), and 1.1 percent (n = 1) met 5‐9 criteria. No PREM met ≤4 AXIS criteria. The median number of “yes” scores was 16 (IQR: 15‐17). The lowest scoring of all PREMs answered “yes” to only five of the 20 AXIS questions.30 The highest scoring PREM answered “yes” to all questions.31 Appendix S5 presents the AXIS results for all PREMs from highest to lowest number of AXIS criteria met.

Table 2.

PREMs categorized according to proportion of AXIS criteria met

Total AXIS score	PREM	n (%)
>15 AXIS criteria met	NORPEQ31; AEDQ139; CACHE94; CAHPS HIT123; CQI‐CSD105; CQI‐Hip Knee110; MPOC‐A47; OPEQ135; OPEQ‐China136; PEPAP‐Q109; POPEQ71; QTAC‐PREM86; SF‐HKIEQ108; AIPS104; CAHPS Dental plan143; CQI A&E141; CSS29; EUROPEP125; González, Quintana, Bilbao, Escobar, Aizpuru, Thompson, Esteban, San Sebastián, De la Sierra96; OPQ113; PCQ‐PD92; PEACS 1.0145; PEQ‐GP121; PEQ‐ITSD83; PEQ‐OHC118; PFQ134; PIPEQ‐OS61; QPC73; Re‐PEQ81; VOICE66; ADAPT57; Bruyneel, Van Houdt, Coeckelberghs, Sermeus, Tambuyzer, Cosemans, Peeters, Van den Broeck, Weeghmans, Vanhaecht62; CABHS65; CAHPS CC155; CAHPS PCMH133; CPEQa,76; CPEQb,76; CQI‐Cataract167; CQI‐RA82; GPAQ124; GPPS112; I‐PAHCc,59;LifeCourse74; MCQ75; O‐PAHCd,59; PEPAC166; PEQ MH69; ReproQ91; Walker, Stewart, Grumbach146; Bruyneel, Tambuyzer, Coeckelberghs, De Wachter, Sermeus, De Ridder, Ramaekers, Weeghmans, Vanhaecht93; CAHPS Health Plan151; CEO‐MHS70; ChASE58; Homa, Sabadosa, Nelson, Rogers, Marshall89; ICEQ87; IDES63; IEXPAC88; Labarère, Fourny, Jean‐Phillippe, Marin‐Pache, Patrice100; NREQ99; PSQ MD138; Steine, Finset, Laerum130; UCSQ142	62 (70.5)
10‐15 criteria met	ACES127; Black, Sanderson28; CAHPS C&G131; CQI‐CHD77; CQI‐PHHD77; GS‐PEQ37; HKIEQ107; HQ68; I‐PEQ CHD98; IPQ119; PAIS120; PCQ‐H60; PEQ103; PESS115; SIPS24; CAHPS ICH78; DPQ144; Drain116; HCAHPS157; howRwe148; Malott, Fulton, Rigamonti, Myers149; PREM RA and Other80; Picker MSD122; PPE‐1597; CQI‐PC36	25 (28.4)
5‐9 criteria met	PEES‐5030	1 (1.1)
≤4 criteria met	Nil	0 (0)

Abbreviations: AXIS, Appraisal tool for Cross‐Sectional Studies; PREM, patient‐reported experience measure.

^aOutpatient version of CPEQ.

^bInpatient version of CPEQ.

^cReports I‐PAHC.

^dReports O‐PAHC.

Appendix S6 identifies that all studies we assessed as presenting clear study aims and utilizing appropriate study designs to answer their research questions (Q1 and Q2). Greater than 95 percent of PREMs appropriately sampled participants to be representative of the target population under investigation (Q5). Over 95 percent of the studies reported that there was no conflict of interest related to a funding source and the interpretation of results (Q19). Questions 13 (potential for response rate bias), 14 (description of nonresponders), and 20 (attainment of ethical approval or participant consent) were the criteria least frequently met by PREM papers.

3.2. Revised COSMIN checklist validity and reliability appraisal

Appendix S4 details the validity and reliability testing undertaken for the PREMs according to the revised COMSIN checklist. PREMs are ranked within their specified contexts according to the number of positive results obtained for the validity and reliability tests. Inter‐rater reliability between two assessors for a portion of the COSMIN checklist appraisals was κ = 0.761, indicative of good agreement.

Some validity and reliability tests were undertaken more often than others (Table 3). The three psychometric tests most commonly meeting “+” criteria were internal consistency (n = 58, 65.9 percent), structural validity (n = 49, 55.7 percent), and content validity (n = 33, 37.5 percent). Seven of the 10 revised COSMIN checklist criteria were not undertaken in ≥50 percent of the PREMs: (a) reliability (n = 44, 50.0 percent); (b) hypotheses testing (n = 53, 60.2 percent); (c) cross‐cultural validity (n = 65, 73.9 percent); (d) criterion validity (n = 79, 89.8 percent); (e) responsiveness (82, 93.2 percent); (f) item response theory (n = 84, 95.5 percent); and (g) measurement error/agreement (n = 86, 97.8 percent). None of the studies undertook testing for all 10 validity and reliability criteria.

Table 3.

Psychometric quality of PREMs according to the Revised COSMIN checklist

Psychometric quality criteria	Criteria met, n (%)	Criteria not met, n (%)	Unknown or unclear information, n (%)
Internal consistency	58 (65.9)	18 (20.5)	12 (13.6)
Reliability	18 (20.5)	26 (29.5)	44 (50.0)
Measurement error/agreement	1 (1.1)	1 (1.1)	86 (97.8)
Content validity	33 (37.5)	0 (0)	55 (62.5)
Construct validity
Structural validity	49 (55.7)	6 (6.8)	33 (37.5)
Hypotheses testing	21 (23.9)	14 (15.9)	53 (60.2)
Cross‐cultural validity	13 (14.8)	10 (11.4)	65 (73.8)
Criterion validity	3 (3.4)	6 (6.8)	79 (89.8)
Responsiveness	4 (4.5)	2 (2.3)	82 (93.2)
IRT	3 (3.4)	1 (1.1)	84 (95.5)

Abbreviations: COSMIN, COnsensus Standards for the selection of health Measurement INstruments; IRT, item response theory; PREM, patient‐reported experience measure.

4. DISCUSSION

The purpose of this systematic review was threefold: to identify and describe peer‐reviewed PREMs, irrespective of their contextual basis; to critically appraise PREM validity and reliability; and to critically appraise any bias in the study design of PREM validity and reliability testing. It is integral to understand whether PREMs have been subject to rigorous validity and reliability testing as this reflects whether an instrument is able to appropriately capture patient‐reported experiences of health care. In turn, it is also important to ensure that the results of PREM validity and reliability testing are underpinned by a rigorous study design so that readers can be assured that the validity and reliability results are a robust representation of how valid and/or reliable that PREM is. To our knowledge, this is the first systematic review to examine PREMs across a range of health care contexts and settings.

This systematic review identified a total of 88 PREMs. Interestingly, roughly 20 percent of the identified PREMs were developed from 2015 onwards, and a quarter of all PREMs received some form of additional validity and reliability testing in this time frame as well. Given that 1993 was the earliest PREM development year identified through the search strategy, this indicates a significant increase in the desire for instruments that measure patient experiences.

Generally, the PREMs identified in this systematic review reflect a heavy emphasis on measuring singular events of health care. The Institute of Medicine's (IOM) 2001 report on crossing the quality chasm identified that despite a significant increase in chronic and complex conditions, health care systems are still devoted to acute episodes of care.32 Overwhelmingly, this sentiment still reigns true today, despite efforts to promote greater integration and coordination of care across and within health care services, as well as patient‐centric, high‐quality health care.8, 33 For example, this systematic review identified only one peer‐reviewed PREM targeting chronic disease holistically (as opposed to a singular disease focus) and two PREMs focusing on the integration of care. Most PREMs related to short‐term care episodes, largely in the hospital setting, though there are PREMs (eg, the CG‐CAHPS34 and health plan CAHPS35) that examine patient experiences of care delivered over 6‐ to 12‐month periods. By developing and utilizing PREMs that maintain a single  event, unidimensional focus of health care, we are inhibiting our ability to strive for international health care goals related to reducing health care fragmentation, and optimizing continuity, coordination, and quality within and between services. Consequently, future PREM development should aim to capture patient experiences of the continuity and coordination within and between their health care services and providers in order to mirror international shifts toward greater health care integration.

Encouragingly, nearly all PREM evaluation papers met ≥10 AXIS criteria (98.9 percent). Furthermore, all papers possessed appropriate study designs for their stated aims, and >95 percent of papers demonstrated appropriate sampling of participants to be representative of the target population under investigation. One PREM,30 however, met only five out of 20 AXIS criteria, implying that this PREM should undergo further evaluative testing prior to use in patient experience evaluations. Generally, the results of the AXIS critical appraisal indicate that the study designs underpinning PREM validity and reliability testing were sound.

Unlike the recent systematic review of hospital‐related PREMs16 where all instruments presented some form of validity and reliability testing, we identified two PREMs (CQI‐PC and GS‐PEQ) that did not present any testing in accordance with the revised COSMIN checklist.36, 37 This was either a consequence of not having done the testing, not presenting clear enough information to be scored “+” or “−,” or not having published this information in the peer‐reviewed literature. Evidently, both the CQI‐PC and GS‐PEQ instruments require further validity and reliability testing before being used in patient experience evaluations.

The most frequently undertaken reliability and validity criteria that also received positive results included internal consistency (a measure of reliability), structural validity, and content validity. This ultimately indicates that most PREMs measure the concept which they set out to measure, and do so consistently. Responsiveness—an instrument's ability to detect changes overtime19—was not evident for >90 percent of PREMs. While some of the identified PREMs appear to have been developed for a once‐off purpose, and thus exhibiting the ability to detect changes in patient experiences overtime is not a property of significant importance, it was surprising to identify that responsiveness was not evident in most of the CAHPS suite of surveys. Most CAHPS surveys are employed annually on a nationwide scale, such as the HCAHPS, which has been used in this capacity since 2002 in US hospital reimbursement and benchmarking schemes.38, 39 However, only the CAHPS PCMH scored positively for responsiveness. The GPPS PREM also scored positively. This is the UK National General Practitioner Patient Survey which has been undertaken annually since 2007.40 It is important to note though that this information may be presented outside of the peer‐reviewed literature, and consequently, what was captured in this systematic review may be an underrepresentation of all testing undertaken for these measures. The lack of testing for instrument responsiveness is consistent with previous systematic reviews16, 22, both of which identified that responsiveness was not undertaken by any of the PREMs that they assessed. Evidently, testing responsiveness should be prioritized for instruments that are to be utilized on an annual or repeated basis.

The least prevalent property assessed using the COSMIN checklist was measurement error/agreement. Measurement error, in accordance with the revised COSMIN checklist, assesses whether the minimally important change (MIC) (the smallest measured change in participant experience scores that implies practicable importance41) is greater than or equal to the smallest detectable change (SDC) in participants scores, or outside of the limits of agreement (LOA) (a technique used when comparing a new measuring technique to what is already practiced42). Thus, in the clinical context, the MIC enables researchers to define a threshold of clinical relevancy. That is to say, a score above that threshold (as defined by the MIC) demonstrates that the intervention/program/service was clinically relevant and responsive to improving the patient experience. Given that the patient experience of health care is internationally recognized as a key determinant of health care quality,32, 43 and there is evidence to support the relationship between patient experience data and health care quality,44, 45 the clinical relevancy of improving patient experiences is likely to have implications for resource allocation and decision making in optimizing the quality of health care provided to patients. As such, assessing PREM measurement error/agreement should be undertaken, particularly in instances where PREM scores are being used to inform decision making and funding.

None of the PREMs were tested for all of the revised COSMIN checklist criteria. There are several reasons that this may be the case. For example, criterion validity was only undertaken in roughly 10 percent of the PREMs as some authors recognized that there simply is no gold standard PREM available as a comparator in their given context.46, 47 Another reason could be inconsistencies in psychometric reporting guidelines and journal guidance regarding what constitutes adequate validity and reliability testing. A previous systematic review48 examined the quality of survey reporting guidelines. The authors identified that there is generally a lack of validated reporting guidelines for survey instruments. Furthermore, the review highlighted that only a small portion of medical journals, where papers such as those included in this review may be published, provide guidance for the reporting of survey quality.48 This indicates an area of research generally that warrants greater attention as this is not just a limitation that impacts upon the quality of PREMs, but a wide range of instruments.

4.1. Limitations

One major limitation of the current study was that grey sources of literature were not considered in the identification of PREMs. Consequently, we may have missed PREMs that otherwise would have fit the inclusion criteria. Furthermore, there were PREMs that we excluded because they had not yet published their supporting validity and reliability results. This was the case for the UK Renal Registry Patient‐Reported Experience Measure (UKRR‐PREM) who had published their instrument,49 but were still in the process of developing psychometric evaluation publications at the time that this review was undertaken. However, the purposeful selection of PREMs that were published in peer‐reviewed journals was to maximize the quality of the instruments evaluated.

A limitation of the AXIS appraisal tool is that a summative score cannot be derived to interpret the overall quality of the study being assessed25 (ie, whether a study is deemed poor, moderate, or high quality). However, assessment of risk of bias imposed by a study design is standard practice in the appraisal of studies for systematic reviews.50 For this study, PREMs were categorized into quartiles according to the proportion of AXIS criteria met, with full details of each PREM assessment provided in Appendix S5 to enable readers to make an informed decision about PREMs that they may use in their own patient experience evaluations and research.

The revised COSMIN checklist also possessed some important limitations. Firstly, the revised version of the COSMIN checklist was used instead of the original checklist51 as it was more user‐friendly to use given the large proportion of PREMs included in this systematic review. Secondly, the parameters of measure for the validity and reliability testing comprising the checklist are very prescriptive. For example, the “structural validity” criteria stated that factors identified through exploratory factor analysis (EFA) had to explain at least 50 percent of the variance.19 Yet other parameters such as a significant Bartlett's test of sphericity (P < 0.05), the Kaiser‐Meyer‐Olkin (KMO) measure of sampling adequacy (acceptability typically regarded as >0.6), or factor loading >0.4 (acceptable strength of loading on a factor)52, 53 can also be used to assess the quality of EFA. As such, this limited the authors’ ability to assess the reliability and validity of the instruments where tests other than those prescribed in the checklist were undertaken. Thirdly, the checklist fails to attribute rigor to the multidomain design of the included PREMs in measuring the same construct, which may positively impact upon how well the PREM captures a broad array of the attributes of a patient‐reported experience.54 Fourthly, the COSMIN fails to capture the importance of floor and ceiling effects, as well as the percentage of missing data. These were commonly reported statistics among the included PREMs and demonstrate: (a) the ability of the instrument to discern meaningful differences between patients reporting extremes of low and high experience scores; and (b) the burden and feasibility of completing the instrument.55 Fifthly, the revised COSMIN checklist fails to provide a summative score indicative of whether, overall, a PREM is or is not valid and reliable. Moreover, whether some tests of validity and reliability are more relevant or suitable than other tests to the overall validity and reliability of a PREM remains unknown. Further, it is unclear whether all tests ultimately need to be undertaken in order for a PREM to be labeled as a valid and reliable measure. Thus, in order to assist the reader to make an informed choice in their PREM selection, Appendix S4 ranks the PREMs within their specified contexts, according to the number of “+” scores obtained. Despite these limitations, the COSMIN checklist is currently the most comprehensive psychometric quality criteria for developing outcome measurement instruments and evaluating the method of development for these instruments.56 Furthermore, the checklist has been applied to other similar systematic reviews16, 22 and was the most appropriate means of systematically measuring the psychometric rigor of the included PREMs.

5. CONCLUSION

Patient‐reported experience measures are internationally recognized instruments for measuring the quality of health care services from the patients perspective. The construct of patient‐reported experience appears to still be evolving, and though this systematic review identified PREMs across a range of contexts, PREMs remain largely designed to assess singular events of health care. The key messages of this systematic review are that while the testing of PREM validity and reliability has generally been undertaken in the context of appropriate study designs, there is large variability in both the number and type of validity and reliability testing undertaken for the PREMs identified. As such, it is important that PREM users are aware of the validity and reliability already undertaken for the PREM they have selected, and whether they themselves should undertake more robust testing. Further, the selection of PREMs for research and evaluation purposes should also be considerate of other important selection criteria such as whether a disease/condition or setting‐specific measure is more appropriate than a generic measure, and whether a PREM designed in the researcher's country is more appropriate than one designed in a different country, potentially with a different health care system in mind.

Supporting information

 

 

 

 

 

 

 

Bull C, Byrnes J, Hettiarachchi R, Downes M. A systematic review of the validity and reliability of patient‐reported experience measures. Health Serv Res. 2019;54:1023‐1035. 10.1111/1475-6773.13187