Development of a Best Practice Guide to Optimise the Reporting of Patient Reported Measures by Clinical Quality Registries for Quality Improvement Purposes

Abstract

Background:

Clinical quality registries (CQRs) systematically monitor the quality of healthcare by routinely collecting and reporting health-related information. The collection of patient reported measures (PRMs) by CQRs provides a personal perspective on the expectations and impacts of treatment. Reporting of CQR-collected PRMs for quality improvement (QI) is highly variable.

Objectives:

To develop a best practice guide (BPG) for CQRs, clinicians and health services to support high-quality and transparent reporting of PRM data for QI purposes.

Methods:

The project comprised four stages. The first sought to describe how PRMs were reported for QI purposes in Australia and internationally. The second stage included seven focus groups with 20 Australian CQRs to identify existing practices, issues and impacts regarding PRMs reporting. During stage 3, findings from the literature and focus groups were used to draft a preliminary BPG. Finally, expert workshops involving PRMs experts, consumers, clinicians and representatives from CQRs were convened to refine a preliminary BPG.

Results:

We identified 61 international and 45 Australian CQRs that reported PRMs for QI purposes. PRMs were used for shared decision-making in clinical encounters, for developing clinical decision aids, to revise treatment guidelines and to monitor complications after hospital discharge. Several themes emerged from the focus groups. These included: purpose and context, funding and resource requirements, consumer involvement, clinician training, instrument selection and administration, outlier identification, visualisation and interpretation of the data. A preliminary BPG was refined during the workshop discussions.

Conclusion:

An increasing number of CQRs use PRMs to enhance QI reporting, however there are no published guidelines currently to support this. Through identifying existing practices, methods and techniques that CQRs use to report PRMs, we developed a practical guideline to support CQRs and standardise their PRMs reporting for QI purposes, with the overarching goal of optimising the value of PRM data within CQRs.

Introduction

Patient reported measures (PRMs) are increasingly being introduced into clinical quality registries (CQRs), where they assist in monitoring and evaluating outcomes from care for both acute and chronic conditions.^1,2 Clinical quality registries are common internationally; in Australia, more than 100 have been registered by the Australian Commission on Safety and Quality in Health Care. ¹ Including PRMs in CQRs offers many advantages. Incorporating patient reported outcomes ensures that measurement of healthcare outcomes is patient-centred. Symptom burden, health-related quality of life (HRQoL) and satisfaction with care are essentially lost if not captured in “real time,” which may not coincide with clinical care episodes. Capturing of comprehensive PRMs data can inform health service planning, research and evaluation, and facilitate benchmarking of participating health services.^3,4

Traditionally, registries have reported clinician-collected data back to health services to inform quality improvement (QI). ⁵ Yet, existing published evidence indicates that reporting of CQR-collected PRMs data to health services for QI is highly variable in its extent and nature. Patient reported outcome (PROMs) and experience measures (PREMs) are key components of patient-centred care that can underpin CQR monitoring and QI efforts.^6-8 Together, these data provide important information about how healthcare systems are performing from the perspective of the people accessing care.

The National Clinical Quality Registry Program (the Program) aims to improve the quality of health care and ensure better health outcomes for Australian patients. As part of this Program, the Australian Government Department of Health and Aged Care is leading a range of activities under the National Clinical Quality Registry and Virtual Strategy 2020 to 2030 (the Strategy). ⁹ Increasing the use of PROMs and PREMs in national CQRs is a key priority of the Strategy.

The aim of this project was to develop a best practice guide (BPG) and recommendations for CQRs, clinicians and health service managers for the planning, collection, analysis, reporting and interpretation of PRM data, including hospital or clinician-level PRMs from CQRs for QI purposes. These recommendations aim to improve the quality and transparency of PRM reporting, and to facilitate standardised approaches to PRM data reporting across clinical registries. This paper is intended to provide a comprehensive overview of the process used to develop the BPG, which was informed by the extant literature and significant multisectoral and multidisciplinary stakeholder input.

Methods

Study Design

The BPG development process occurred over July 2023 to June 2024 and included the following stages: (1) A scoping review of the academic and grey literature to identify how PRMs were reported by CQRs for QI purposes in Australia and internationally; (2) Focus groups with representatives of Australian registries collecting PRMs to identify existing practices, issues and impact regarding PRMs reporting for QI purposes; (3) Drafting of a preliminary BPG based on the findings from the literature and focus groups and (4) Expert workshops with CQRs, consumers, clinicians and health service representatives to discuss the preliminary guidelines (Figure 1). The detailed methods used for each stage are described here.

Figure 1.

Diagrammatic representation of stages underpinning BPG development.

Literature Search

A literature review was performed by two members of the group (CM and RR). We searched MEDLINE and EMBASE databases for peer-reviewed journal articles, annual reports and websites in English that described PRM reporting in CQRs, and their impact for QI. Search terms included the words “PROMs,” “PREMs,” “clinical registries,” “CQRs,” “reporting” and “QI.” The terms were combined by means of Boolean operators and are listed in Additional File 1 (Supplemental Material). Grey literature was also searched using these terms to identify registry websites and annual reports containing potentially relevant information on PRMs data collection. In addition, a list of Australian registries collecting PRMs was compiled through a grey literature search performed in Google Scholar (www.googlescholar.com.au) and via the website of the Australian Register of Clinical Registries (https://www.safetyandquality.gov.au/publications-and-resources/australian-register-clinical-registries). Articles meeting inclusion criteria proceeded to data extraction.

The following information was extracted from the identified publications, websites and reports: country, registry name, condition, PRMs reporting process, types of the reports, reporting frequency, to whom the reports were provided, impact made from reporting and any other relevant details.

Focus Groups

Focus groups were conducted to identify existing practices and considerations for PRMs reporting for QI with registry managers, clinical and academic leads, and other registry staff in Australia. A list of 47 potential participants across diverse clinical areas and stakeholder groups was compiled through a Google search, snowballing and project team networks. Potential participants were sent an invitation email and explanatory statement. The discussion schedule comprised 19 semi-structured questions regarding PRM data collection and reporting practices that were based on the findings from the literature review (Additional File 2, Supplemental Material).

The first focus group was conducted on the 23rd January 2024. Informed verbal consent was obtained and recorded from each participant prior to commencing the session. The lead researcher (RR) led the discussions with notes taken by a research assistant (CM). All sessions were recorded on the videoconferencing software, Zoom. ¹⁰ Zoom recordings were transcribed by a paid transcription service. To ensure data quality, the research team checked all transcriptions against the recorded audio files. Data analysis involved coding and categorising the data from the transcripts using NVivo software (version 12, QSR, Australia). The researchers immersed themselves in the transcripts and thematically analysed the data by identifying quotes and words and grouping them according to question areas. The themes were discussed internally among the research team and combined with the findings from the literature review to inform development of the BPG.

Workshops

Academic and clinical leads, registry staff, consumers, clinicians and researchers were invited to review the preliminary BPG and participate in an online workshop. Forty-eight external stakeholders were invited including CQRs, consumer representatives, PRMs experts, clinicians and health service representatives. Seven days before the meeting, the attendees were emailed meeting details and agenda, and a draft of the BPG. Attendees were asked to review the document prior to the meeting and provide their feedback.

A facilitator (RR) presented a summary of the BPG recommendations to the online workshop participants. The workshop discussions were guided by the same moderator using open-ended questions to achieve a final consensus on each recommendation (eg, “Could this recommendation be modified?,” “Do you agree this is acceptable for PROM reporting, why/why not?”). At each workshop, the facilitator (RR) summarised the recommendations and feedback from the workshops and subsequent work required to develop or amend the draft before outlining the proposed content and layout for the document.

Four workshops were held on Zoom in May 2024. All workshops were recorded, and a research assistant (CM) documented notes. The notes were discussed internally, resulting in an updated list of recommendations in the next iteration of the BPG.

Final Recommendations

Feedback from the Australian Government Department of Health and Aged Care was sought and incorporated into the final version of the BPG. Refinements to the document were made and then circulated to the funder.

Results

Literature Review

International Registries

The titles and abstracts of 1473 journal articles, annual reports and internet sources were screened according to the inclusion criteria. Of those, 306 full text articles were assessed for eligibility. The screening of full texts resulted in 117 journal articles eligible for inclusion. Combined with the 20 reports and 4 websites, a total of 141 information sources were included in the review. Data were extracted from 100 international registries capturing PRMs, with the majority based in Europe and the majority capturing data related to arthroplasty procedures related data. Sixty-one (61%) of these registries reported PROMs and only three reported PREMs (Additional File 3, Supplemental Table 1). Forty-nine registries (80%) reported PRMs in their annual reports, with 14 (23%) reporting PRMs information quarterly or biannually in local or regional reports, surgeon reports, webpages, publications, press releases or presentations at meetings. Thirteen (20%) registries reported PRMs through newsletters, publications or institutional outcome reports.

Forty-eight registries (79%) sent their reports to hospitals, orthopaedic units, clinics or clinicians. Clinician-specific reports were distributed to surgeons, and in some cases, midwives. Twenty-three registries (37.7%) completed site specific reports, where PROMs reporting was completed for hospitals, clinics or other sites that were providing treatment. In this manner, sites had access to their own data. Other specialised reports with PRM data were distributed to steering committees, product manufacturers, policymakers, health authorities, patients, educators and family members. For most of the registries, PRMs data were reported in the form of text, tables, figures or graphs.

Forty-four registries (72%) demonstrated some form of QI impact through reporting PRMs. Examples of impact included a reduced number of postoperative complications, often resulting in improved quality of treatment, shorter length of hospital stay, decreased utilisation of health services and decreased postoperative in-hospital mortality.

Australian Registries

One hundred and fourteen Australian registries were included in this review. Sixty-eight registries (60%) were identified as already capturing or planning to collect PRMs. Of these, 45 (66%) reported PRMs (Additional File 3, Supplemental Table 2). Half (50%) reported PRMs through annual reports in the form of text, figures, tables or graphs. Some of the registries (eg, TrueNTH Global Registry-Prostate Cancer Outcomes, Myeloma and Related Diseases Registry and Australian and New Zealand Extracorporeal Membrane Oxygenation Registry) also produced 6-month benchmark site reports that incorporated PRMs data.

The Australasian Myositis Registry, the Prostate Cancer Outcomes Registry-Australia and New Zealand (PCOR-ANZ) and the Coronary Angiogram Database of South Australia provided reports with PRMs to clinicians. The Australian and New Zealand Hip Fracture Registry’s data are available through real-time data dashboard and audit reports, annual reports to participating sites and state reports. Of the 45 registries that were reporting PRMs, 10 (22.2%) produced site reports. For most of the other registries, PRMs data were reported primarily in scientific publications.

In comparison to international registries, the use of PRMs data for QI purposes was infrequent in Australian registries. The PRM data was predominantly used for comparative research rather than site-level reporting. Thirteen registries (29%) utilised PROMs data for research and publications. Eight (18%) registries utilised PRMs data to evaluate treatments and procedures. Only the Victorian Orthopaedic Trauma Outcomes Registry and Victorian State Trauma Registry appeared to be utilising PRMs to support policy changes.

Focus Groups

Twenty (11 female and 9 male) registry managers, clinical and academic leads, coordinators and other registry staff from Australia participated in seven focus groups. The participants were from registries capturing PRM data on asthma, trauma, cancers, chronic pain, diabetes, stroke, dementia, burns, retinal, cardiac, anaesthesia, pelvic floor and breast device procedures, life-support, angiograms and orthopaedic related procedures. Ten (50%) were from Victoria, six (30%) from New South Wales, two (10%) from Queensland, one (5%) from South Australia and one (5%) from Western Australia.

The main themes discussed at the focus groups included: (1) aims for PRMs data collection for QI reporting, (2) outcomes collected and measures administered, (3) who PRMs are reported to for QI purposes, (4) how PRMs for QI purposes are presented, (5) management/reporting of PRMs outliers, (6) clinician perspectives of PRMs reporting for QI purposes, (7) data quality issues and reporting for QI purposes and (8) PRMs impact on local clinical care.

Aims for PRMs Data Collection for QI Reporting

Most of the registry representatives reported that PRMs were collected for the registries since their inception. Examples of specified aims included to “predict device performance, revision rates or complication rates [or] to compare outcomes and do interventions” (Reg003) and to “streamline data collection” (Reg001).

Outcomes Collected and Measures Administered

Registries mostly captured generic quality of life measures such as the EQ-5D or SF-36. Nearly half of the participants indicated their registry had a >50% response rate for PRMs. In general, response rates ranged from 30% to 85%. Although a majority of registries were unable to capture PRMs at baseline or 6 months (Reg001, Reg00, Reg012, Reg017, Reg019), most captured PRMs at 12 month or 24 months (Reg002, Reg019).

Who PRMs for QI Purposes Are Reported To

Most registries provided annual reports which were available to the public via a public-facing website. Some registries reported PRMs data to individual clinicians (Reg005, Reg006, Reg008, Reg016, Reg017, Reg022). Reg004 indicated reporting PRM data to special interest groups, and Reg008 mentioned that “clinicians can download their reports themselves.”

How PRMs for QI Purposes Are Presented

Most of the registries presented their PRMs data in text, tables, graphs and/or figures (Reg008, Reg009, Reg011). One registry implemented a traffic light system: “[their] reporting is really at the national level and they put a lot of effort into forming working groups and they have comms, they make lots of really beautiful charts and infographics available online” (Reg008). Many participants communicated that there is “[to do] a lot of work with participating sites on reporting for quality improvement purposes. So [they] now do a mixture of tables and figures.”

Management/Reporting of PRMs Outliers

Few registries reported implementing processes to manage sub-optimal scores by alerting sites or clinicians where these were identified (Reg004, Reg001). Reg001 indicated that “data collection is occurring across all of the sites via staff at those sites. [. . .] So obviously any flags that come up to them, they’re going to deal with through their routine management plans.” In other cases, outliers were reported in their benchmarking reports: “[they] report that in the benchmarking reports, [they] identify the patients who have reported that big bother and this is with consent from the participant when they participate in the registry so that they can follow up care for that patient.” (Reg006).

Clinician Perspectives Around PRMs Reporting for QI Purposes

The majority of registries indicated that clinicians were interested in utilising PRM data for QI purposes. Five registries discussed the importance of PRMs education to clinicians since “clinicians don’t know what PRMs can and can’t do” (Reg007, Reg008, Reg013, Reg021, Reg022). These registries highlighted that PRMs were “viewed as a research or academic endeavour” and that they “[were] trying to bridge that gap” (Reg007, Reg008, Reg013, Reg021, Reg022). Some registries thought that “the mindset of a cultural shift” and “education on what [PRMs] can and can’t do” would improve QI reporting and consequently impact patient care and health outcomes (Reg001, Reg002, Reg005, Reg021, Reg022).

Data Quality and Reporting

Several registries highlighted varying issues with data quality and reporting of PRMs for QI purposes. The most prominent issue identified by most participants was inadequate funding, followed by data completeness and follow up issues (Reg006, Reg007, Reg015, Reg016) and interoperability of IT systems (Reg009, Reg008). Additionally, registries highlighted single data entry (Reg008), administrative (Reg015, Reg010) and governance barriers (Reg013) and language barriers (Reg014), which also affected data quality and reporting in registries.

PRMs Impact on Local Clinical Care

Some registries indicated they had achieved impact from reporting PRMs, including perceived large-scale impact. For example, one registry said that capturing PRMs data has “helped [them] to fund a new trauma ward,” and “to set up models of care and funding based on reporting” (Reg002). Another highlighted that “PRM tracking has led to better access to medication with low costs for patients” (Reg021) and a further two registries indicated “improved quality at both a procedural, service delivery and patient care levels” (Reg009, Reg020) through PRMs reporting. At a system level Reg019 stated that PRMs reporting resulted in “algorithms [being] developed to predict patient recovery, which has subsequently improved workload management and resource allocation for the registry” and Reg016 “developed clinical quality indicators from PRM data.”

Drafting Recommendations for a BPG

Draft recommendations for a BPG were developed from the project findings with domains aligned with the CQR PROMs Framework, developed by Ruseckaite et al,^3,4 together with vignettes and examples.

The draft BPG containing ten domains was designed to provide a set of guiding principles and recommendations for CQR staff, clinicians and health service managers (Figure 2).

Figure 2.

The best practice guide domains.

The first domain “Planning PRM activities” described the initial steps required for setting up PRMs program and data collection. Key recommendations included careful planning, having a clear rationale and establishing a governing group with patients, consumers and representatives from culturally and linguistically diverse and First Nations communities.

The second domain provided guidance regarding instrument selection for CQRs wanting to collect PRMs. QI opportunities differ depending on the condition and the information collected in the CQR (eg, measuring access to care vs monitoring adherence to clinical care guidelines). The choice of PRM tool(s) should be driven by outcomes capable of contributing to improved patient care for the CQR condition/disease, that can be feasibly monitored.

“PRMs administration” domain listed recommendations regarding PRMs administration in CQRs, timing and frequency for data collection, ethical requirements and response rates. “Stakeholder engagement section and education” addressed stakeholder engagement, data presentation and education of clinicians and health services. PRM use is relatively new in CQRs, so there is a variable level of knowledge regarding PRMs among clinicians and health service managers. This should be considered when reporting PRMs, that is, results should be presented clearly and informatively.

“PRM data preparation and analysis” domain was intended to provide high-level guidance on the appropriate statistical techniques to consider. The recommendations addressed data missingness, analysis of baseline and follow-up data, risk adjustment to control for the role of confounding and case mix in PRMs data analysis and adjusting confounding and case-mix factors.

Examples of QI use cases using PROMs and PREMs were included under the “Use cases for quality improvement” domain. These included measuring and comparing mean/median PRM scores between groups or over time, reviewing variation in PRM scores via comparison between multiple individuals/groups using visual mechanisms or visualising the distribution of the change in PRMs to evaluate the effectiveness of the intervention.

The “Variation and outlier identification” domain provided recommendations and examples for PRMs outlier identification and visualisation, including funnel plot curves, histograms and control charts.

“PRM data visualisation” presented a brief overview and examples (eg, tables, graphs and figures) for PRM data visualisation to aid the interpretation of PRM results.

The success of a PRM program lies not only in the successful collection and analysis of data but also in how this information is reported and translated to advance patient care and health system goals. This includes determining the target audience of the information gathered from the PRM program. The reporting and use of PRM data require that decisions relating to how the data will be used be made early on in the planning phase. A brief guidance regarding timing, frequency and ethical consideration for PRM data reporting has been described under the “PRM data reporting and dissemination” domain.

Systematic evaluation of PRMs usage within a registry should be performed regularly and should consider facilitators/ barriers that were identified during preparation and implementation phase. The “PRM program evaluation” domain provided guidance regarding PRM program evaluation and components.

Appendices to the BPG were developed to summarise evidence for developing the recommendations, to provide an overview of psychometric terms used in PRMs research, to highlight statistical considerations for PRM data analysis, and provide further information regarding best practices for data visualisation.

Workshops

Of 48 invited registry academic leads, clinicians and PRMs experts, 29 (20 female and 9 male) accepted the invitation and participated in four workshops that were conducted on different days, each containing 7 to 8 participants. Eleven (38%) were from Victoria, seven (24%) from New South Wales, three (10%) from Queensland, three (10%) from South Australia, three (10%) from the Australian Capital Territory and two (7%) from Western Australia. Two (7%) participants were consumers, 11 (38%) participants were clinicians, and the remaining participants were academics (including registry managers and PRM experts).

Several themes emerged from the workshop discussions. These included: purpose and context, funding and resource requirements, consumer involvement, clinician training, instrument selection and administration, data preparation, analysis and outlier management, visualisation and use of cases and examples. Participants made several amendments and recommendations to BPG, as outlined below.

Purpose and Context

Participants recommended the need for a clear purpose of the BPG, clarification on target audience, explanation on guideline application, especially in hospitals for evaluating clinical services, as well as the need for a clear distinction between using the terms PREMs and PROMs throughout the BPG. Participants further indicated that it was important to clarify to CQRs that although PRMs were collected, they might not always be reported back to health services.

Funding and Resource Requirements

Participants requested specific amendments to the BPG such as amending the “Planning PRM Activities” section by adding the following statement “A sufficient budget should be allocated to support PRM data collection, analysis and reporting activities in the CQRs.” Additionally, funding details were recommended to be clarified.

Consumer Involvement

To clarify the role of consumers, it was recommended that the BPG sections relevant to consumer involvement be amended to ensure cultural appropriateness, response burden, literacy levels (eg, languages other than English, cultural relevance), and the real-world context in which people with lived experience and their families live, work and play. Participants also recommended confirmation that selected instruments address health outcomes or experiences that are relevant to patients and capture outcomes and experiences in a comprehensive and understandable manner.

Clinician Training

It was suggested that clinicians and health service managers are not as familiar in reviewing PRMs info as they are with clinical data. Hence it was recommended that the “Stakeholder engagement” section of the BPG be amended by adding “PRM use is relatively new in CQRs, so there is a variable level of knowledge regarding PRMs among clinicians and health service managers. This should be considered when reporting PRMs i.e. results should be presented clearly and informatively.”

Instrument Selection and Administration

From the discussions, it was evident that PRM administration, time points and frequency vary for different patient groups. For example, for cancer patients PRMs may be collected from the start of cancer journey and can include different aspects of care. Contrastingly, for patients with chronic disease, PRMs may be collected after acute hospital admissions. Therefore, statements regarding patient burden, length of the questionnaires and modes of administration were recommended to be added to the BPG.

Data Preparation, Analysis and Outlier Identification

Several recommendations were made by workshop participants regarding data collected by CQRs. These include providing examples of missing data, clarifying the difference between ceiling and floor effects and providing more examples of control charts that highlight outliers. In a technical sense, participants recommended to separate the statistical section of the BPG from the data preparation section, and to move the statistical considerations highlighted in the BPG to the appendices.

Visualisation and Use of Cases and Examples

To assist BPG users with preparing visualisations, it was recommended to include additional details regarding existing sources of information and to provide more figures and explanatory statements regarding the reporting and presentation of the data. Participants also emphasised that PRMs need to be reported by the registry if this can be achieved in a timely manner so that clinicians can access the information in real time. Therefore, how CQR PRMs provide a longitudinal picture of patient outcomes after discharge from hospital can be complementary to hospital PRMs data, although there is the risk of duplication if there is overlap in data collection time points.

The suggestions and recommendations made during workshops were reviewed and discussed internally, and subsequently implemented in the BPG.

Discussion

Patient reported measures are key components of patient-centred care that can underpin CQR monitoring and QI efforts. These data provide important information about how healthcare systems are performing from the perspective of the people accessing care. In order for this to occur, it is essential that PRMs are reported in a manner that is both useful and understandable. ¹¹

Developed in collaboration with partners across CQRs, this BPG presents a framework and recommendations to guide the collection, analysis, reporting, and use of PROMs and PREMs. This is the first study in the field, outside of clinical practice and trials,^12-14 that has developed recommendations for clinicians and health services when reporting PRM data for QI purposes.

The process of guideline development process often includes a review of the available literature and engages technical expertise to inform development of a position statement, advice and/or recommendations about a topic of interest.^15,16 In this paper, we have described the process undertaken to develop a BPG targeted to a specific audience and which encompassed a technical topic that required input and opinions from multiple stakeholders. We engaged an interdisciplinary sample of clinical, academic and government representatives who joined focus groups discussions, provided detailed feedback on preliminary concepts and attended workshop meetings to discuss the draft BPG. It is the first step in a coordinated approach, where PRM data are collected from patients and made readily available to guide continuous QI, support clinician-patient decision making, and inform health system policy decisions as part of a learning health system. The limitations of our study include that our BPG was based on experience of the CQRs already collecting PRMs. Some subjective interpretation of how to analyse and present PRMs data was unavoidable. We acknowledge that the ability to detect references among different professional audience groups (eg, medical staff vs researchers and government representatives) was limited by the composition of mostly female participants who self-selected to participate in our focus groups and workshops.

We next intend to pilot test and validate these newly developed recommendations. This will be conducted in three phases: (1) Piloting and validating the initial guidelines with clinicians, sites, jurisdictions and registries; (2) Adapting the guidelines to a consumer audience; and (3) Developing accompanying practical resources for consumers and clinicians.

The final set of clinician-focused and consumer-focused guidelines and the accompanying material will be made available on the Australian Government Department of Health’s National Clinical Quality Registry Program website and broadly disseminated via the Australian Clinical Trials Alliance Registry Special Interest Group and presentations at relevant CQR forums and conferences. It is expected that this initial set of guidelines will evolve over time and will incorporate practical feedback from real-world implementation, as well as further research to evaluate their uptake and usefulness.