A multimethod research investigation of consumer involvement in Australian health service accreditation programmes: the ACCREDIT-SCI study protocol

Abstract

Introduction

Health service accreditation programmes are a regulatory mechanism adopted to drive improvements inpatient safety and quality. Research investigating the benefits or limitations, of consumer involvement in accreditation programmes is negligible. To develop our knowledge in this area the ACCREDIT collaboration (Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork) has developed a research plan, known as the ACCREDIT-SCI (Standards of Consumer Involvement) study protocol. Two complementary studies have been designed: one, to examine the effectiveness of a standard for consumer participation and two, to explore how patient experiences vary across a range of settings with differing accreditation results.

Methods and design

The research setting is the Australian healthcare system, and the two studies focus on three accreditation programmes in the primary, acute and aged care domains. The studies will use multimethods: document analysis; interviews and surveys. Participants will be stakeholders across the three domains including: policy officers; frontline healthcare professionals; accreditation agency personnel, including surveyors and healthcare consumers. Drawing on previous experience, the research team has developed purpose-designed tools. Data will be analysed using thematic, narrative and statistical (descriptive and inferential) procedures.

Ethics and dissemination

The University of New South Wales Human Research Ethics Committee has approved the two studies (HREC 10274). Findings will be disseminated through seminars, conference presentations, academic publications and research partner websites. The findings will be formulated to facilitate uptake by policy and accreditation agency professionals, researchers and academics, and consumers, nationally and internationally.

Article summary.

Article focus

• To provide a research protocol that aims to investigate consumer involvement in Australian health service accreditation programmes. Two studies with multiple components have been conceptualised. The first examines the effectiveness of a standard for consumer participation, and the second explores how patient experiences vary across a range of settings with differing accreditation results.

Key messages

• Governments and healthcare organisations around the world have adopted accreditation programmes as a strategy to regulate the quality and safety of clinical care and organisational performance.

• Empirical research into the value of consumer involvement in accreditation programmes is limited and where there are studies, the findings are ambiguous.

• The two studies presented in the Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork-Standards of Consumer Involvement study protocol will use multimethods to examine consumer involvement in accreditation programmes, and triangulate findings.

Strengths and limitations of this study

• The investigation of health service accreditation programmes across the healthcare continuum of primary, acute and aged care is a strength of this study. Additionally, the collaborative research partnership between researchers and accreditation stakeholders, presents the opportunity for translational research to drive improvements in the health system.

• The focus on accreditation programmes within a single country is a limitation of the study.

Introduction

Health service accreditation

Health service accreditation programmes are a regulatory mechanism adopted globally to drive improvements in patient safety and quality.^1–3 They are complex organisational interventions that seek to reassure external stakeholders, including consumers, that healthcare industry safety and quality standards are being achieved and improved. Accreditation programmes are used across the healthcare continuum of primary, acute and aged care services.⁴ While there are variations between countries, and the health systems within them, in the regulatory frameworks that contextualise and operationalise accreditation programmes, a common model has evolved.⁵ Healthcare organisations (HCOs) enrol with an accrediting agency and self-assess against their standards. Accreditation standards comprise a set of performance statements and associated criteria and outcomes or actions. HCOs produce reports that document their quality maintenance and improvement activities against the accreditation standards. Reports are reviewed by the accrediting agency which sends teams of peer-reviewers, usually known as surveyors, to conduct onsite visits to validate the claims. When visiting HCOs, surveyors conduct observations of facilities, interviews with staff and document analysis of organisational reports, meeting minutes, quality improvement projects and policies and procedures.⁶ Surveyors provide verbal feedback to the HCO on their assessment of their achievement against the accreditation standards, including, where appropriate, remedial actions recommended. Written survey reports are provided to the accrediting agency that decides on conferring accreditation status, or not. Accreditation status is the declaration by the accreditation agency, an external authority that HCOs have demonstrated competency against recognised industry-based safety and quality standards.⁵ Accreditation status is typically for a period of 3–5 years.

Consumer participation in health service accreditation programmes

Consumer participation in healthcare decision-making and health services governance has become an internationally accepted target.⁷ Studies have shown benefits from consumer participation in both the clinical^{8 9} and organisational¹⁰ domains. However, research investigating the benefits, or limitations, of consumer participation in accreditation programmes is negligible;^{1 4} only two studies have been identified.^{11 12} In one study, an evaluation of consumers as surveyors in an accreditation programme highlighted stakeholder perceptions of an increase in survey teams’ objectivity and credibility. The role of the consumer surveyor was identified as needing to be clarified and more consistently applied. In addition, their capacity to contribute to rating of criteria and writing of the survey report was noted as problematical.¹¹ A second study examined accreditation performance against the consumer involvement in health services, reporting no relationship between the two.¹²

Beyond this research, there are a limited number of studies that relate consumer views about care received or patient satisfaction with services, with accreditation outcomes. Where they have been done, studies show no clear or consistent findings.⁴ No relationship between hospital accreditation outcomes and patient satisfaction have been identified.^13–15 Accredited health services have not been rated higher than non-accredited organisations against patient-reported quality measures.¹⁶ Conversely, another study demonstrated that accredited health units performed better than non-accredited units on patient satisfaction measures.¹⁷ Similarities and differences in patient and health professionals’ assessment of accreditation standards of care have been identified.¹⁸ In the related area of accreditation of medical practices in primary care, patient experience data have been shown to assist in improving medical practice quality.¹⁹ Nevertheless, the study concluded that knowing how and to what extent patient experience data are to be included in accreditation decisions has yet to be defined.¹⁹ In short, the existing empirical evidence is both limited and ambiguous as to the value of consumer involvement in health accreditation programmes.^{5 6 15–17} The motivation for the present study is to address this knowledge deficit.

The research context: the Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork collaboration

In Australia, to investigate healthcare accreditation a collaborative research partnership has been formed involving academic healthcare researchers, accreditation agencies from across the care continuum of primary, acute and aged care and quality improvement bodies at national and state levels: the Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork (ACCREDIT).¹ The organisational partners are: the Centre for Clinical Governance Research and Centre for Health Systems and Safety Research, in the Australian Institute of Health Innovation at the University of New South Wales (UNSW); Australian General Practice Accreditation Limited (AGPAL); the Australian Council on Healthcare Standards (ACHS); Aged Care and Standards Accreditation Agency (ACSAA); the Australian Commission on Safety and Quality in Health Care (ACSQHC); and the Clinical Excellence Commission (CEC) from the state of New South Wales. The ACCREDIT collaboration also has an international advisory group (IAG) providing expert advice and critique to their work. The IAG members are prominent healthcare quality and safety and health services researchers based in Europe.

The ACCREDIT collaboration have scoped a research project with four aims: evaluate current accreditation processes; analyse the costs and benefits of accreditation; improve future accreditation via evidence and develop and apply new standards of consumer involvement in accreditation. These aims emerged from the melding of ideas from: previous research into healthcare accreditation;^{2 4 6 12 20–26} literature reviews;^{4 27 28} a workshop consultation with the partners and other accreditation stakeholders;²⁹ and extended negotiations to clarify and focus the research agenda. In 2010, the ACCREDIT collaboration became the recipient of an Australian Research Council linkage grant (LP100200586). These grants are awarded to research projects that have a significant academic merit and the ability to generate findings to benefit Australian industry and society,³⁰ and international parties with interest in the area.

Methods and analysis

This research protocol details the investigation of consumer involvement in Australian health service accreditation programmes within the ACCREDIT project; known as the ACCREDIT-SCI (Standards of Consumer Involvement) study protocol. To achieve this aim, two studies have been conceptualised with multiple components. The studies correspond to studies 4 and 5 outlined in the overarching ACCREDIT design.¹ The first examines the effectiveness of a standard for consumer participation, and the second explores how patient experiences vary across a range of settings with differing accreditation results. To counteract potential limitations of any single research method, a multi-method approach will be used. This strategy enables triangulation of data sources and promotes credibility of findings.³¹

Study 1: examining a standard for consumer participation

In the intervening period since the key questions and details were agreed to by the partners and encoded into a funded proposal,¹ the ACSQHC developed a new Australian national standard for consumer participation. The standard Partnering with Consumers is one of the new 10 Australian National Safety and Quality Health Service Standards (NSQHSS).³² The work of the ACSQHC has superseded the first three aspects of the consumer involvement study as proposed in the overarching ACCREDIT project design.¹ Consequently, the study has been revised to accommodate the changed contextual circumstances and is now comprised of three parts: conduct a retrospective analysis of the development process for the NSQHSS consumer standard; apply the standard in the field (n=30) and evaluate its use and efficacy with survey and qualitative methods.

Retrospective analysis of consumer standard development processes

The research team will retrospectively analyse the activities and processes used by the ACSQHC to develop the NSQHSS Partnering with Consumers standard. Two evaluation methods will be used to triangulate findings, including documentary analysis³³ of ACSQHC Standards Development Committee deliberations and workshop reports. Additionally, semistructured interviews with members of the ACSQHC Standards Development Committee will be conducted. Recruitment and access to key participants and relevant reports will be facilitated by ACSQHC, who will email potential participants inviting them to take part in the study. Interviews will be recorded, transcribed and thematically analysed. Drawing on the research team's knowledge from prior work,^23–25 the issues focusing the evaluation, either by document analysis or interview method, will include: what evidence was drawn upon for the standard; how was the evidence assessed; what was the decision making process to include or exclude evidence; to what extent were the stakeholders engaged and how does the standard integrate with the other standards of the NSQHCS?

Study information and consent forms, approved by the UNSW Human Research Ethics Committee (HREC),³⁴ will be provided to potential participants electronically via email. The potential informants will be followed up, via telephone calls, after 2 weeks if there has not been a response to the invitation. Interviews will be conducted face-to-face where possible, or alternatively via telephone, in locations suitable for those involved. To promote participant responses, face-to-face interviews will be the first option. To enhance reliability and internal comparison of data³⁵ the researchers involved in interviews will complete a training session together and use a purpose designed, standardised semi-interview schedule. Furthermore, the schedule will provide structure for uniformity while allowing scope for respondents to expand on issues important to them. Interviews will be digitally recorded and professionally transcribed.

A two-step process will be used to analyse the data. First, the issues employed to direct the evaluation will be used to collate and thematically group the data from the documentation and interview transcriptions. Second, a joint approach of a narrative strategy³² and temporal bracketing³² will be used to interpret the data. This strategy involves constructing an account of the standard development process with time used as the framework to structure the narrative. This approach is suitable for ordering³² and examining change events³⁶ and capturing the complexity of proceedings.³⁷

Apply and evaluate the NSQHCS partnering with consumers standard

Thirty accreditation surveys, comprising 10 from each healthcare domain, will be chosen using stratified randomised sampling³⁸ to examine the implementation and assessment of the NSQHSS Partnering with Consumers standard. Based on prior research experience, including accreditation research studies,^{6 12 22 24} this cohort is expected to be appropriate to provide both depth and breadth of data to assess the standard. A multimethod approach will be used, involving document analysis, interviews and a survey questionnaire.

The research team, in collaboration with the study partners, will map the NSQHSS Partnering with Consumers standard to the accreditation standards used in each domain. That is, the NSQHSS to the ACSAA Accreditation Standards, The Royal College of General Practitioners Standards for General Practice (4th edition), and the ACHS accreditation programme, Evaluation and Quality Improvement Program (EQuIP) V.5, respectively, to identify corresponding standards and criteria related to consumer participation. These details will be used to focus the document analysis of the accreditation outcomes and reports, and provide topics and suitable phrasing of language for the interviews and survey questionnaire.

The HCOs accreditation outcomes and survey reports will be collected. Using purposeful sampling³⁸ accreditation agencies’ representatives and surveyors, and key informants from the HCOs involved with each survey, will be invited to participate in the study. They will be offered an individual or group interview and asked to complete a questionnaire. Informed by their previous work^{6 21 22} in this field, the research team will examine the following issues with participants: what do you understand the standard is aiming to achieve; is your understanding similar or different to your colleagues and other stakeholders; is the standard easy to survey against or provide evidence for; what criteria or actions do HCOs implement well and which others are difficult to implement; what criteria or actions stimulated most discussion between survey participants; and what, if any, resources are required to implement the Partnering with Consumers standard? The same processes for ethics approval, study information and data collection outlined above will be applied to the interview data. Analysis will follow accepted norms for systematic classification of interview data.³⁹ It will be directed by the principles of addressing the significant points in the data while incorporating key interpretations, with analytical decisions documented by an audit trail and shaped by previous research experience and findings.^{6 24 25 40–42} An inductive process will guide the analysis and the textual grouping software, NVivo V.9⁴³ will be employed.

A web-based survey, using KeySurvey⁴⁴ software, which will take approximately 15 min to complete, will complement the interviews. This technology enables complex question routing and ease of data collection at a reasonable cost.⁴⁵ Focusing on similar topics as the interviews, the survey will employ a five-point Likert scale⁴⁶ with ratings from ‘strongly agree’ to ‘strongly disagree’. Demographic details on participants will be also collected including: occupational position; professional background; highest qualification and level of experience with accreditation surveys. Piloting of the survey will be undertaken. Feedback regarding comprehensiveness of instructions and phrasing of questions and time for completion will be reviewed and the survey amended as necessary. Survey data will be analysed using descriptive and inferential statistics, including generalised linear modelling applied to Likert-type outcomes,⁴⁷ assisted by other categorical data analysis techniques.⁴⁸ To identify participant variations regarding views of key themes, responses will be compared both within and between stakeholder groups and accreditation domains.

Study 2: the patient experience across a range of settings

Study two aims to examine how the patient experience varies across a range of settings with different accreditation results. The study will be undertaken using the approach detailed below.

The research team will review and update or adapt the partners’ existing patient journey tools²² for application in the acute, aged care and general practice settings. Three new purpose designed research tools, detailed below, will be produced: a ‘patient experience questionnaire’; an ‘HCO patient trajectory tool’ and a ‘health professional patient experience questionnaire’. The content and structure of the tools will also be shaped by experience from the research team's previous accreditation research work.^{3 5 12} A three-stage recruitment and data collection process, indicating participants and research tools, is outlined below and represented in figure 1.

Figure 1.

Patient experience study recruitment and data collection process.

Stage one: identify HCOs to participate

Stage one aims to identify HCOs to participate in the study. To recruit subjects the research team will send an electronic request, that is, an invitation to participate in the study, to the accreditation partners to forward on to the HCOs they accredit. The request will contain the UNSW HREC approved study information and consent forms.³⁴

Stage two: identify potential patients journey survey subjects

Identifying potential subjects within the nominated HCOs is the goal of stage two. The HCOs nominating to participate will be approached by the research team to identify potential subjects to track at the time of, or immediately following, the accreditation survey. Potential patients will be those with healthcare journeys characterised as a complex case involving multiple organisations, services, departments and health professionals. The research team and an HCO representative using purposeful sampling³⁸ will together review potential subjects for the study. Once a potential subject is identified, an HCO representative will forward an electronic or written request from the research team to the potential subject to participate in the study. The request will contain the UNSW HREC approved study information and consent forms.³⁴ Using this process, the research team will seek to identify 20 patient journeys to investigate in each domain. Based on prior research experience, including accreditation research studies,^22–25 this cohort is expected to be appropriate to provide both depth and breadth of data to assess the patient experience.

Stage three: construction of individual case studies

Stage three involves the collection of data and construction of individual case studies. After confirmation of suitable subjects, the research team will initiate four actions. First, to capture the patients’ experience of the care and services they received, a ‘patient experience questionnaire’ will be administered. The research team will speak with each patient to give them the choice to complete the questionnaire on paper or electronically. The patients, in addition to providing basic demographic data and reasons for attending the HCO, will be asked to report upon their experience of: making appointments; arrival; waiting for appointments; assessments; interactions with health professionals, including communication and understanding of issues; referral information; discharge; timeliness and accessibility and negotiation of the physical environment. The questionnaire will use a five-point Likert scale and is expected to take 20 min to complete.

Second, an ‘HCO patient trajectory audit tool’ will be used to map the interorganisational and intraorganisational trajectory of individual patients. This tool will enable documentation of the HCOs, and the various services or departments within them, and health professionals or teams who provided significant assessment, intervention or advice to the patient or the primary health professionals or team caring for them. A researcher and HCO representative will examine the medical record to document these issues; it is anticipated that this task will take between 1 and 4 h, depending on the domain.

Third, the health professionals or teams identified will be approached, in person or via telephone, by the research team or HCO representative and asked to participate. They will be surveyed using a ‘health professional patient experience questionnaire’. The health professionals or team will be asked to comment upon: services and care provided, including assessments, interventions and referrals undertaken; information provided; length and quality of interactions with the patient and timeliness in provision of care. The questionnaire will use a five-point Likert scale and is expected to take 20 min to complete. Fourth, the research team will collect, from the accrediting bodies, each participating HCO's accreditation surveyor report and outcomes.

A case study of each patient experience will be constructed using the data collected from the three tools and documentary analysis³³ of the associated accreditation report and outcomes. Case study methodology, framed by complexity theory, will be used to seek to understand the system as an integrated whole.⁴⁹ Within the case study framework, the analysis activity will be the same as that defined for the retrospective analysis of the development of the consumer standard.

Ethics and dissemination

The ACCREDIT-SCI study has been approved by the UNSW HREC (HREC 10274). Study information sheets and consent forms explaining activities, processes and participant roles have been developed. A research team contact document has also been finalised for distribution to enable participants to raise questions or concerns. In accordance with UNSWHREC guidelines: complaints will be systematically recorded and actioned; prior to publication or presentation participant information will be de-identified; findings will be made available to participants and research data will be stored in a secure location, accessible only to the research team, and deleted after a minimum of 7 years.

Dissemination of study findings by the research team will occur through a variety of forms. Seminars will be conducted, with targeted invitations made to partners and stakeholders. Presentations will be made at national and international conferences, and journal articles developed for academic and industry publications. Additionally, information, updates and outcomes will be made via UNSW and partner websites.

Conclusion

The investigation of consumer involvement in health service accreditation programmes is an important task that addresses a significant gap in the knowledge base. The ACCREDIT-SCI study protocol details two studies—one examining the effectiveness of a standard for consumer participation and the other exploring how patient experiences vary across a range of settings with differing accreditation results—that are designed to complement each other and meet this need. The ACCREDIT research collaboration engages and uses the skills and experience of a diverse range of academics, accrediting agency personnel and policy makers. In this way, the collaboration provides capacity to implement the ACCREDIT-SCI study protocol and distribute findings across the continuum of the healthcare industry.