Involving patients in quality indicator development – a systematic review

Abstract

Background

Quality indicators (QI) are used in many health care areas to measure, compare, and improve the quality of care. Ideas of quality differ between health care providers and patients, yet patients are not regularly involved in QI development nor does a methodological standard for patient involvement in QI development exist. In this study we systematically reviewed the medical journal articles and gray literature for published approaches for involving patients in QI development.

Methods

We searched medical literature databases (Medline, Excerpta Medica database, and Cumulative Index to Nursing and Allied Health Literature), screened websites, and contacted experts in the field of QI development for publications on approaches to patient involvement in QI development.

Results

Eleven relevant journal articles and four web-published documents were included. Four major approaches to patient involvement were extracted from the literature: (1) focus group interviews, (2) self-administered questionnaires, (3) individual interviews, and (4) participation in panels during systematic consensus processes. Patients’ views were collected by involving patients, patient representatives, or family members.

Conclusion

Although there is a large body of literature on QI, publications that describe approaches to patient involvement in QI development are scarce. In principle, indirect and direct methods of patient involvement can be distinguished, and it seems most promising to combine different approaches. However, the limited number of publications identified clearly shows that further research in this field is overdue and that the quality of reporting found in studies within this field needs to be improved.

According to the Institute of Medicine’s definition, quality in health care is the “degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge.”^1,2 However, different stakeholders in health care may have diverging conceptions of quality and may desire different health outcomes.³ For instance, patients may wish for other health outcomes than doctors, health care managers, or politicians.^4,5 This has important implications for the measurement of quality.

Quality indicators (QI) are widely used in many health care areas to measure and improve the quality of care.^6,7 QI are measurable items that refer to health care outcomes and to the structures and processes of health care services.⁸ Several international working groups have described methods for the development of QI.⁹ Most of the methods have certain methodological elements in common, such as the systematic review of available evidence and the use of consensus methods to augment evidence with expert opinion.¹⁰ There is a general consensus that QI should also reflect patient views on quality in health care.^11,12 To achieve this, patients or patient representatives should either be directly involved in the development process or they should be given the opportunity to contribute their views on desired outcomes, quality targets, and health care priorities.^13,14 Despite this and despite the impact patient involvement has on the acceptance of QI in clinical application, patients are not systematically involved in the development of QI. There are numerous barriers to patient involvement, including practical and financial structures, differing knowledge bases, cultural barriers, variable patient interest, and the attitudes of health professionals.¹⁵ As yet, there is no international consensus on how patients should be involved in QI development.^13,16 A methodological “gold standard” for patient involvement in QI development might help to overcome these barriers and might contribute to making patient involvement a routine procedure in QI development as is the case for patient involvement in medical decision making and patient safety.^17,18

In this study we systematically reviewed the existing medical journal articles and gray literature to identify the current state of methods used to involve patients or patient representatives in the QI development process of expert teams. The results are aimed at forming a basis for the development of a methodological standard.

Methods

Literature search

We conducted a systematic literature search in three electronic databases (Query date August 15, 2012): Medline and the Excerpta Medica database (EMBASE), both via OvidSP^® to cover medical journals that are not included in Medline; and the Cumulative Index to Nursing and Allied Health Literature (CINAHL) via EBSCOhost^® to include articles published in the field of nursing and the allied health professions. We used search terms related to quality indicators, quality of care, and patient involvement. In addition, we searched the reference lists of all publications included in the final review to obtain additional articles not identified through the database searches. We used the Guidelines International Network (G-I-N) database (www.g-i-n.net) and the identified journal articles to acquire contact information for institutions and working groups in the field of QI development. G-I-N is an international, not-for-profit association of more than 200 organizations and individuals from 48 countries involved in the development and use of clinical practice guidelines in quality improvement and quality assessment (http://www.g-i-n.net). We extracted all those institutions from the G-I-N database that were marked as being involved in QI development (15 institutions) and scanned their websites for web-published methods papers. Finally, we consulted colleagues with a research interest in QI development to identify articles not detected by our search.¹⁹

Study selection criteria

Two reviewers independently screened all obtained references for eligibility in a three-stage screening process. Discrepancies were resolved by consensus. Articles were included if they provided any information on how patients or patient representatives were involved in QI development. We set no restrictions for publication type, study design, clinical setting, or medical condition. All articles published in English, French, or German were considered (Figure 1).

Figure 1.

Flowchart summarizing the screening process.

Data extraction and quality assessment

Two researchers independently extracted data from the relevant literature to a predesigned and piloted data extraction form. Discrepancies were resolved by consensus. For journal articles we extracted the clinical topic and setting for which the QI were developed, the number and characteristics of the involved patients or patient representatives, criteria for their selection, time point or period of involvement within the QI development process, methods of involvement (ie, focus groups, individual interviews), outcome of the QI development process (ie, number and type of indicators), and performance of a practice test of the selected QI. For quality assessment of the original research articles included, we extracted information regarding source of funding, study and publication type, duration and time frame of the study, potential sources of bias, explicit criteria for the selection of recommendations from pre-existing clinical practice guidelines (if applicable), and the (predefined) target setting for the resulting QI. Data extraction from web-published documents included general methods of QI development described in the methods paper, participating patients or patient representatives, methods used for patient involvement, and performance of a practice test of the selected QI. Given the fact that the web-published documents did not follow the classical structure of an original research paper, a quality assessment was not possible for these papers. Data extraction was performed using Excel 2007 (Microsoft Corporation, Redmond, WA).

Data synthesis and analysis

Data is summarized in narrative form separately for journal articles and web-published documents. We used a framework for the methodological steps in QI development that we developed as part of a prior systematic review of QI development methods to illustrate the stage of the QI development process at which patients were involved (Figure 2).²⁰

Figure 2.

Patient involvement in different steps of quality indicator (QI) development.

Results

Search findings and literature selection

We screened 156 potentially relevant full texts for eligibility (Figure 1). No publications were identified through expert consultation or the search of trial registries. In total, 15 publications (eleven journal articles^23,33 and four web-published documents)^12–15 met our eligibility criteria and were included in this review.

Journal articles

All of the eleven journal articles included were original articles reporting on the development of QI for specific clinical contexts. Three articles were published by the IQ health care research group^25,30,31 (University of Radbound, Nijmegen, The Netherlands) and two journal articles were authored by the RAND Corporation^27,33 (Santa Monica, CA). Notably, the approaches to patient involvement in QI development varied even among publications of the same institution. Eight of the eleven journal articles focused on cancer care (Table 1).

Table 1.

Characteristics of included journal articles

Paper	Whom to involve	Topic	Purpose	Setting	Selection criteria for participants	When to involve	How to involve	Participation of experts	Results	Practice test
Earle et al23	2 panels including 12 patients and 1 panel including 4 members	Cancer care	To improve palliative services	Not specified	Patients with advanced incurable cancer	Assembly of the QI set	Focus group interviews	Yes	7 process indicators, 1 outcome indicator	No
Gagliardi et al24	15 patients	Cancer care	To compare views of different stakeholders on types of QI	Hospital care	English-speaking, ≥18 years, confirmed colorectal cancer	Assembly of the QI set	Individual interviews	Yes	6 process indicators, 4 outcome indicators	No
Hermens et al25	7 patient representatives	Lung cancer	To foster the implementation of a clinical practice guideline	Hospital care	Members of the national board of patient representatives for lung cancer	Selection of sources for QI development	Self-administered questionnaire	Yes	15 process indicators	– Assessment of measurability, improvement potential, discriminating capacity and feasibility by extracting data from medical records of 276 patients – Questionnaire survey in 132 of 276 patients
Kesmodel and Jolving26	1 patient representative	Obstetrics	To measure and improve the quality of deliveries in Denmark	Not specified	Member of an association of women giving birth	Expert involvement	Member of the indicator group	Yes	3 process indicators, 5 outcome indicators	– Assessment of clinical meaningfulness and interpretability by testing in a clinical setting in 6 delivery units – Hearing of stakeholders
Malin et al27	Patient representatives (number unclear)	Breast and colorectal cancer	To measure the quality of breast and colorectal cancer care across the US	Hospital care	Unclear	QI selection	Unclear	Unclear	61 process indicators (36 for breast cancer)	– Measurement of adherence to QI by extracting data from medical records of 2002 patients – Questionnaire survey of 2002 patients
Martin et al28	10 patients	Ophthalmology	To determine key areas for the development of QI in eye outpatient services	Ambulatory care	Outpatients of the RVEEH	Selection of topics for QI development	Individual interviews	Yes	4 topics for QI development	No
Miyashitaet al29	183 bereaved family members	Cancer care	To rate QI on end-of-life cancer care and to explore factors related to the evaluation of QI by bereaved family members	Palliative care	Patients who died in PCU because of cancer, ≥20 years, in hospital for ≥3 days	QI selection	Self-administered questionnaire	No	27 process indicators, 6 outcome indicators	No
Ouwens et al30	30 patients	Head and neck cancer	To measure the quality of integrated care for patients with head and neck cancer	Cross-sectoral care	Patients diagnosed with head and neck cancer between May and December 2003	Selection of sources for QI development	Individual interviews	Yes	12 structure indicators, 19 process indicators	– Assessment of current practice: Questionnaire completed by 189 patients with head and neck cancer and 15 experts – Analysis of 189 medical records – Assessment of feasibility, opportunity for quality improvement and reliability by research team
Ouwens et al31	30 patients and 7 patient representatives	Head and neck cancer	To measure and improve the patient-centerdness of cancer care	Not specified	Patients with head and neck cancer or members of the national board of patient representatives for lung cancer	Selection of sources for QI development	Individual interviews	No	26 process indicators	– Analysis of medical records of 276 patients – Questionnaire survey in 132 of 276 patients – Assessment of room for quality improvement, applicability, discriminating capacity and reliability by research team
Shieldet al32	Focus group interview: unclear Delphi survey: 1 panel including 9–12 patients and patient representatives	Mental health	To develop a set of QI for primary mental health care that reflects stakeholder perspectives	Primary care	Participants reflecting geographical spread across Great Britain	QI selection, expert involvement	Focus group interviews, self-administered questionnaire (Delphi survey)	Yes	QI categorized into 21 aspects of care	No
Spencer et al33	3 panels including 10 patients and spouses	Prostate cancer	To develop an infrastructure to evaluate variations in quality of care for prostate cancer	Not specified	English-speaking patients, 6–12 months after surgical or radiation treatment for localized prostate cancer	Selection of topics for QI development	Focus group interviews	Yes	5 structure indicators, 23 process indicators, 16 outcome indicators	No

Abbreviations: RVEEH, royal Victorian Eye and Ear Hospital; PCU, Palliative Care Unit.

Quality assessment of journal articles

The funding source was disclosed and public funding was reported in only four articles.^25,27,30,32 All articles reported on the criteria for the selection of specific recommendations, discussed strengths and limitations of their study, and mentioned the target setting of the QI. Five articles reported potential sources of bias, which included the recruitment of patients from teaching hospitals only (selection bias),^23,24 limiting participation to patients in certain stages of particular diseases (selection bias),²⁷ the exclusion of data of deceased or non-English speaking participants (selection bias),²⁷ and the time elapsed (1 or 2 years) between interviewing participants and the death of a relative (recall bias).²⁹

Web-published documents

Two of the four web-published documents included were methods papers for QI development in general.^12,13 Another web-published document discussed methods for QI development for human immunodeficiency virus infection/acquired immunodeficiency syndrome care.¹⁴ One document published online by the National Health Service, Scotland, described a project for the development of QI for nursing (Table 2).¹⁵

Table 2.

Characteristics of included web-published documents

References	Institution	General methods	Participants	Recruitment of participants	Specific methods	Practice test
AQUA12	AQUA-Institute (Göttingen, DE)	– Literature review and generating a list of potential Ql – Selection of experts and panel composition – Assessment by the panel – Development of instruments and documentation – Adjustment with G-BA and participants – Publication	– 2 patient representatives in each panel – Panel size: approximately 12 members	From a particular federal state in Germany	– Opening workshop about topics, questions and planned processes – 3 rounds of the RAND/UCLA method	Not mentioned
FACCT14	FACCT (Portland, US)	– Selection of Ql – Field test – Focus group interviews	8 patients diagnosed with HIV/AIDS	Unclear	Focus group interviews: sorting potential Ql into groups (more important versus less important) and creating a ranking	Feasibility assessment
NHS15	NHS (Edinburgh/Glasgow, UK)	– Defining, developing and testing agreed Ql in a year-long pilot project – Literature review – Patient focus group interviews – Project steering group – Pilot testing	– Focus group interviews: 4 male and 4 female patients – Project steering group: 1 male and 1 female patient or patient representative per 16 members	3 focus group interviews for northern, central and southern Scotland, project steering group: unclear	– Focus group interviews: semi-structured discussion about specific aspects of nursing care – Project steering group: 5 meetings to define the project aim, objectives, project deliverables, involvement of stakeholders and interface with NHS Scotland – Combination of Ql arising from the literature review with results of the focus group interviews and project steering groups – Creation of a matrix with 19 factors and questions against which all suggested Ql can be judged
NICE13	NICE (London, UK)	– Collation of existing and potential new Ql – Committee meetings: review and prioritization of Ql – Development of Ql by NPCRDC/YHEC: pilot process and consultation – Approval and publication by the committee	Patient representatives (number unclear)	Committee members appointed for 3 years by a panel consisting of an executive director, a nonexecutive director and the Chair of the Committee	– Membership of the committee will be published on the NICE-website – Registration of individuals and stakeholders on a mailing list – Annual notification of the opportunity to comment on suggested areas for new Ql – Review of recommendations and comments on the proposed new Ql by the committee	Exploration of feasibility, acceptability, implementation, reliability, validity and unintended consequences

Abbreviations: RAND/UCLA, Research and Development /University of California, Los Angeles; FACCT, Foundation for Accountability; NHS, National Health Service; NICE, National Institute for Clinical Excellence; NPCRDC/YHEC, National Primary Care Research and Development Centre/York Health Economics Consortium.

Whom to involve

In seven journal articles and two web-published documents, patients were involved in the QI development process.^{14,15,23,24,28,30–33} In five publications patient representatives participated in the development process.^{25–27,31,32} Three studies involved family members, for instance because the patients themselves had already passed away by the time of the study^23,29,33

Criteria for patient selection varied. Miyashita et al provided a full description of eligibility criteria for patient representatives.²⁹ They included bereaved family members of cancer patients who died in a palliative care unit during a certain time period. Potential representatives had to be above the age of 20 years, and the deceased patients must have spent at least 3 days in the hospital. Exclusion criteria for patient representatives (ie, the bereaved relatives) were participation in other questionnaire surveys, serious psychological distress, inability to reply to self-administered questionnaires, and the unawareness of their relative’s cancer diagnosis. In contrast, Ouwens et al quasi-randomly selected eligible participants.³⁰ They included every third consecutive patient who visited a specialized follow-up clinic. The remaining studies reported the following selection criteria, among others:

• a certain stage of the concerned disease^23,27,33

• ability to communicate in the national language^24,27,33

• age (eg, minimum age of 18 or 21 years)^24,27

• membership in a patient organization.^25,26,31,32

In its methods paper, the German AQUA Institute stipulates that, if available, appropriate patient organizations should nominate “qualified” patient representatives.¹² The number of patients, patient representatives, or family members involved in QI development ranged from 1 to 183; the majority of studies included between 9 and 15 participants.

When to involve

Figure 2 shows the methodological steps of QI development in chronological order.²² In about half of the journal articles, patients were involved early in the development process. In two studies participants were asked to assess the utility of existing quality indicators prior to their adaptation to another setting,^28,33 and in three studies participants rated guideline recommendations prior to their possible transformation into QI (as opposed to pre-existing QI).^25,30,31

Patients views related to QI selection were included in one web-published document and three journal articles.^15,27,29,32 In three of the four web-published documents and in two journal articles, patients were involved in the selection of “final” QI from a set of candidate QI.^{12–14,23,24}

How to involve

Four different approaches to patient involvement were described in the included publications: (1) focus group interviews,^{14,15,23,32,33} (2) self-administered questionnaires,^25,29,32 (3) individual interviews of patients, patient representatives, or family members,^{24,28,30,31,33} and (4) participation of patients in expert panels during a systematic consensus process.^12,13,26,32 Only one study used more than one of these approaches.³²

Focus group interviews. In one study that used focus group interviews, patients and relatives were asked to contribute their opinions about potential QI and were given the opportunity to propose potential QI during meetings.²³ One of the web-published documents stated that focus group interviews were conducted “in order to gain a sample of patient opinion about important issues in hospital care.”¹⁵ In another web-published document, focus group interviews were used to gain insight into patients’ perceptions of quality of care.¹⁴ Further, patients were asked to rate the importance of candidate QI (more versus less important). The remaining two studies did not describe the focus group interviews in detail.^32,33

Self-administered questionnaires. Among the three journal articles that used self-administered questionnaires, two different strategies were followed. In two studies participants were sent a list of candidate QI and were asked to rank these by importance.^29,32 In the third study, participants were asked to judge recommendations from clinical practice guidelines.²⁵ In all three studies patients were requested to rate QI/guideline recommendations based on predefined criteria, such as the relevance of health benefits and treatment outcomes to the patients, the relevance of continuity and efficiency of care, and the relevance of patient orientation.

Individual interviews. When individual interviews were used to involve patients, participants were asked to review either candidate QI²⁴ or recommendations from clinical practice guidelines^30,31 or to suggest QI for a specific clinical context.²⁸ One article described a combined approach as the authors conducted focus group interviews as well as sending out self-administered questionnaires.³² It was not reported whether the participants of the focus group interviews were the same as the participants of the self-administered questionnaires.

Participation of patients in expert panels. Two of the web-published documents and one journal article described that patients or representatives were members of the expert panels.^12,13,26 In contrast to the approaches described above, patients were able to directly discuss their views with expert panel members during the panel meetings.

No information was retrieved on the input that patients or patient representatives contributed (apart from the methods used to collect their views) nor was it mentioned how their input was used in concrete terms or whether their contributions led to a change in the resulting QI. Additionally, we did not retrieve information on whether cultural factors influenced patient involvement.

Discussion

Despite a growing body of literature on QI in general, only few publications describe methodological aspects of patient involvement. We were not able to identify any publications that compare different approaches to patient involvement. Even the article that used a combined approach (focus group interviews and self-administered questionnaires) did not compare strengths and limitations of the two methods. The reasons for the low number of publications might be that patient involvement in QI development has not yet been a focus of research or that the particular methodological aspects of patient involvement do not have high priority in the QI development community.

Clearly, involved patients or patient representatives are not meant to be experts in methods of evidence summary and guideline or QI development. In contrast, the process of QI development in expert teams needs to be supplemented with patients’ views on quality of care. For this reason, our review identified four different strategies for collecting patients’ views on QI development. Generally, these strategies include indirect methods of patient involvement whereby patients’ views are collected using focus groups, self-administered questionnaires, or individual interviews. Alternatively, patients or patient representatives may be directly involved when they are members of the expert panel in charge of the QI development.

Indirect methods are time consuming and costly. Interviews using an open format have the advantage that patients’ views can be collected independently of the interviewer’s agenda. Dominant opinion leaders may bias the discussions in a group, whereas participants of individual interviews can answer the questions in an undisturbed and uninterrupted manner. Using a more formal and rigid schedule and closed questions for an interview is less time consuming but carries the risk of not allowing the participants to contribute their own ideas. Self-administered questionnaires avoid the problem of opinion leaders influencing participants’ contributions and are less costly. They do, however, bear the risk of misunderstandings, and there is usually no possibility for participants to clarify questions immediately.

Some authors combined different approaches to balance out the shortcomings of direct and indirect methods. Shield and colleagues combined two indirect methods (focus group interviews after the analysis of self-administered questionnaires).³² Hermens et al used a modified version of the RAND/University of California, Los Angeles appropriateness method.^25,34 They conducted one round of a self-administered questionnaire survey followed by a panel meeting including all participants. In a third round a group of experts reviewed the results from the questionnaires and the panel meeting. Patients were not involved in this expert panel meeting but they were asked to rate candidate QI in the self-administered questionnaire.

One of the main challenges is the recruitment of patients or patient representatives who are able to contribute significantly to the discussion among experts. At the same time, they should be “legitimized” to represent the broad spectrum of patients with different socioeconomic and health characteristics and attitudes toward health care. Hermens et al considered participants’ lack of understanding of the medical and technical issues to be a major limitation to patient involvement in QI development.²⁵ In contrast, Gagliardi et al argued that patients understand medical and technical issues quite well and that their involvement is both feasible and fruitful.²⁴ Based on the complexity of the clinical topics and the resulting QI as reported in the included journal articles, we believe that involved patients or patient representatives should be required to have a relatively high level of knowledge of their disease, exceeding that of the average patient. It would therefore appear crucial to sufficiently inform patients of important medical facts that are relevant to the development, rating, or selection of specific QI. Additionally, patients or patient representatives should have a basic understanding of the concept of quality measurement using QI.

Patient organizations may help recruit patient representatives who have an appropriate level of knowledge about the disease, the health care system, and quality measurement in general. However, for many diseases the number of patient organizations available, such as self-help groups, is very low. Clearly, given the small number of patients or representatives usually involved in QI development (9 to 15), the challenge is to gather a well-balanced group that represents different age, socioeconomic, and ethnic groups and which consists of patients in different stages of the disease.³³ Certain patient groups, such as comparably young patients or persons with a low level of impairment, may be not interested in a time-consuming and usually voluntary commitment. For elderly and severely ill patients, in contrast, the burden of participation may be too large. Family members may be included if patients are not able to participate themselves, for example, because they are too sick, they are not able to communicate because of their disease (eg, dementia, speech disorders), or because they are deceased (eg, palliative care).

Our review is based on a broad literature search, and it seems unlikely that we missed relevant studies. Trial selection and data extraction, including quality assessment, were performed independently by two of the authors to minimize bias and transcription errors. However, when interpreting the results of the present review, several limitations need to be considered. As with any systematic review, our study is limited by the quantity and quality of the included publications. Several aspects relevant to the assessment of the different methodologies used to involve patients in QI development were consistently not reported in the retrieved publications, such as the specific input patients or patient representatives contributed (apart from the methods used to collect their views), how their input was used in concrete terms, and whether their contributions led to a change in the resulting QI. Future studies are needed to evaluate the effects that patient involvement has on the validity and acceptance of QI, and the different methods used for patient involvement should be evaluated in comparative studies. Another limitation is that the concept of quality in health care may have different meanings in different cultural contexts.³⁵ Due to a lack of reporting on this topic, the present paper did not take account of cultural competency.

Patient involvement is already standard for clinical practice guideline development where patient involvement even serves as an indicator for the quality of the guidelines.^36,37 Légaré et al recently published a systematic review on patient involvement in clinical practice guideline development and reported methodological approaches similar to those this review identified for patient involvement in QI development: indirect participation through focus group interviews, individual interviews, self-administered questionnaires, or direct participation in expert panels.³⁸ The German Agency for Quality in Medicine, for example, provides detailed information about its method of patient involvement in clinical guideline development; a panel of patient representatives selects a spokesperson on the basis of specific criteria, such as clinical and methodological expertise and no conflicts of interest. As a full member of the guideline development panel, the spokesperson attends all panel meetings and participates in the consensus process for the selection of graded recommendations.³⁹ Given the fact that the processes of guideline and QI development have much in common, the standards used for guideline development might serve as a model for patient involvement in QI development.

There is consensus that patients’ views can and should be involved in many areas of health care.¹⁷ This review is limited by the low level of reporting on details of patient involvement in the publications included, and based on this review we cannot clearly recommend one method of patient involvement over another. However, these two facts have several implications:

• – Patients or patient representatives should systematically be involved in QI development.

• – Patient involvement should be reported in more detail (ie, who was involved when involved how involved).

• – International standards for patient involvement in QI development would be helpful. Organizations connected to the subject, such as G-I-N, might take the lead in suggesting, consenting, and evaluating such standards.

• – Future qualitative and quantitative research projects should evaluate the most efficient ways of collecting patients’ views for quality measurement purposes.