Using feedback on patient health outcomes to improve orthopaedic physical therapist practice: a quality improvement study

Abstract

Objective

Measuring health outcomes plays an important role in patient-centred healthcare. When aggregated across patients, outcomes can provide data for quality improvement (QI). However, most physical therapists are not familiar with QI methods based on patient outcomes. This mixed-methods study aimed to develop and evaluate a QI programme in outpatient physical therapy care based on routinely collected health outcomes of patients with low-back pain and neck pain.

Methods

The QI programme was conducted by three teams of 5–6 physical therapists from outpatient settings. Plan-do-study-act cycles were used based on team-selected goals. Monthly feedback reports of process and outcomes of care, including pre–post treatment changes in Oswestry Disability Index (ODI) and Neck Disability Index (NDI), guided the QI efforts. Primary outcomes were pre-QI and post-QI changes in knowledge and attitudes towards outcome measures through a survey, and administered and self-reported compliance with using the ODI and NDI. Semistructured interviews and a focus group were conducted to evaluate the perceived value of the programme.

Results

Post-QI, the survey showed improvements in two items related to the role of patients and implementation of outcome measures. Registered pre-QI and post-QI completion rates were high at intake (ODI:91% pre, 88% post; NDI:75% pre, 84% post), while completion rates at discharge improved post-QI (ODI:14% pre, 66% post; NDI: 32% pre, 50% post). Perceived benefits of the QI programme included clinician and institutional accountability to processes and strategies aimed at continuous improvement in patient care. An important facilitator for programme participation was autonomy in project selection and development, while a main barrier was the time required to set up the QI project.

Conclusion

A QI programme based on the feedback of routinely collected health outcomes of patients with low back pain and neck pain was feasible and well accepted by three pilot teams of physical therapists.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Measuring health outcomes plays an important role in patient-centred healthcare. When aggregated across patients, outcomes can provide data for quality improvement.

WHAT THIS STUDY ADDS

• A quality improvement programme in outpatient physical therapy care using plan-do-study-act cycles is feasible and well accepted by the pilot teams of physical therapists in the USA.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• The quality improvement programme improved the use of outcome measures, allowing for comparison of treatment outcomes over time within and between physical therapy practices.

Background

Measuring health outcomes can play an important role in patient-centred healthcare.^1–3 Discussing outcomes in patient–clinician interactions could lead to patients becoming more involved in goal setting and may increase patients’ self-efficacy.^{4 5} When outcomes are aggregated across patients in clinical registries, they provide data for managing clinical quality, benchmarking and public reporting across organisations, and in clinical research.^{6 7} However, current evidence shows a clear lack of understanding when and how feedback of aggregated health outcomes may improve quality of care.^{6 8}

Outcome measures include patient-reported outcomes (PROs) or performance-based outcomes. PROs assess aspects of a patient’s health status coming directly from the patient. PROs are assessed with PRO measures (PROMs), usually through a questionnaire. Performance-based outcomes include measures of physical function and mobility that are performed by a patient and quantified in terms of level of assistance needed to complete the task, time, distance or quality of movement. To use PROs or performance-based health outcomes for quality improvement (QI), plan-do-study-act (PDSA) cycles may be initiated through feedback reports of routinely collected outcomes.^{9 10} Such feedback reports can be generated by local or national databases, including professional registries.^{11 12}

In physical therapist (PT) practice, the use of outcome measures in patient care is common practice, although their aggregated use for QI is far from routine practice. Most PTs are not familiar with QI methods, including PDSA cycles, based on patient outcomes. Improving the use of outcome measures requires a systematic approach, using targeted strategies in which barriers for implementation are addressed.¹³ For developing strategies for implementing QI in PT practice, it is important to identify mechanisms that facilitate the use of aggregated health outcomes to improve quality of care.

The purpose of this QI study was to develop and evaluate a QI programme in outpatient PT care based on routinely collected health outcomes of patients with two common health conditions: low back pain and neck pain. Specifically, we aimed to identify the facilitators and barriers to implement QI in PT practice to promote continuous QI based on outcome data.

Methods

Design and setting

This pilot QI study was conducted from June 2019 to March 2022 within the University of Pittsburgh Medical Center (UPMC) Centers for Rehab Services (CRS) network. UPMC CRS is a large rehabilitation network in western Pennsylvania in which health outcomes are collected through the electronic medical record (EMR). QI projects of three groups of PT offices within the network aimed to improve care for patients with low back pain and neck pain. A QI programme to support the offices in their projects was developed by the research team (PJvdW, EJB, PLM and KJH) in a design-based approach with knowledge brokers from UMPC (HZ and JJI). A mixed-methods design, theoretically informed by a systematic implementation approach,¹³ was used for evaluating the QI programme. We used the Standards for Quality Improvement Reporting Excellence guidelines for reporting this study.

Participants

Three PT offices were selected through purposive sampling in the CRS network of UPMC. Inclusion criteria for offices were as follows: (1) included at least two PTs; (2) collected data electronically through the EMR and (3) committed to implementing a QI initiative using outcomes. Each office formed a QI team with an assigned quality manager who acted as liaison with the researchers and UPMC knowledge brokers. Participating PTs provided informed consent for participation in this QI project.

Patient and public involvement

Patients or the public were not involved in the design, conduct or reporting of our research. The project was conducted as an internal QI study of healthcare professionals without a direct relationship with patients.

Programme for QI

The QI programme was developed as an iterative process through strategic sessions between the researchers and UPMC knowledge brokers, with the aim of integrating the projects of the QI teams in the overall UPMC quality programme. The QI programme consisted of the following components:

Online training module for participants

The 2-hour online training module included an introduction to the project and study logistics, the foundations of the Institute for Healthcare Improvement (IHI),¹⁴ PDSA cycles and PT-specific case examples of outcomes-based QI. The asynchronous module was accessed through a learning management system using Qualtrics software, V.2019. Participants completed the online module individually at a convenient time prior to starting the QI project.

QI project

Each QI team was guided in developing a QI project using PDSA cycles aimed at improving outcomes for patients with low back or neck pain. For guiding the PDSA cycles, participants used the IHI resources including the PDSA-worksheet, a fishbone diagram, mind-map and scatter plot.¹⁵ All tools were made available to the participants through a project website linked to the IHI resource website.¹⁵ The specific steps in the QI project development are listed in online supplemental appendix 1.

Timeline

Training of participants in the three QI teams was completed between November 2019 and January 2020. Initial feedback reports (step 1) and meetings with the three groups for goal setting (steps 2–4) were finalised in February 2020, just prior to the outbreak of the COVID-19 pandemic. The study was put on hold due to the effect of COVID-19 on healthcare until October 2020. The study was reinitiated with a virtual meeting and confirmation of the goals of each team. The projects of the three QI teams with accompanying feedback reports and evaluation (steps 5–7) were conducted between October 2020 and June 2021 (groups 2 and 3 from October 2020 to March 2021, and group 1 from January to June 2021).

Quantitative data collection

The following patient demographics were collected at baseline: age, gender, primary diagnosis (neck pain or low back pain), body mass index and comorbidity (Charlson Comorbidity Index (CCI)). Quantitative outcomes of the QI projects were evaluated with pre-QI and post-QI measurements through surveys and data from the UPMC EMR. Data from the EMR are pulled into a clinical dashboard of which reports are able to be customised and downloaded. Survey data were collected using Qualtrics software, V.2021.

Primary outcomes

Self-assessment survey

A self-assessment survey was used to evaluate knowledge, attitudes and barriers for the use of outcome measures in PT practice (table 1). The 27-item survey was adapted from a survey by Meerhoff et al. ¹⁶

Table 1.

The pre–post QI programme difference in mean change of self-assessment of knowledge and attitudes in the use of outcome measures*

#		Pre-QI Mean (SD)	Post-QI Mean (SD)	Change Mean (SD)	t†	P value
1	I know where to find outcome measures.	3.75 (0.20)	3.94 (0.06)	0.19	1.38	0.19
2	I am able to implement outcome measures together with my patients	3.56 (0.26)	3.88 (0.12)	0.31	1.78	0.10
3	I am able to interpret outcome measure results	3.69 (0.23)	3.38 (0.92)	−0.19	−1.16	0.26
4	Using outcome measures does not affect my professional authority to make my own decisions	3.19 (1.23)	3.38 (0.92)	0.19	0.56	0.58
5	Using outcome measures helps me to formulate a PT diagnosis	2.56 (0.93)	2.88 (1.18)	0.31	1.78	0.10
6	Outcome measures are useful in the evaluation of a treatment	3.31 (0.50)	3.38 (0.38)	0.06	0.44	0.67
7	I am able to use outcome measures within my clinical reasoning process	3.44 (0.26)	3.56 (0.26)	0.13	0.70	0.50
8	Outcome measures have a positive influence on the quality of PT practice	3.31 (0.50)	3.25 (0.47)	−0.06	−0.32	0.75
9	It is important to document the patient’ perceived health objectively with outcome measures	3.38 (0.52)	3.75 (0.20)	0.38	2.09	0.05
10	Using outcome measures in clinical practice takes too much time	1.19 (0.70)	1.56 (1.60)	0.38	1.25	0.23
11	I think my patients do think that using outcome measures takes too much time	2.88 (0.52)	2.63 (1.05)	−0.25	−0.77	0.45
12	When using PROMs I am able to include the wishes of my patients	2.44 (0.53)	2.5 (1.20)	0.06	0.21	0.84
13	I would like to use outcome measures more often	1.85 (0.52)	2.13 (1.58)	0.25	0.72	0.48
14	I use outcome measures in daily practice	3.44 (0.66)	3.25 (0.73)	−0.19	−0.90	0.38
15	I have experienced the added value of outcome measures	3.13 (0.52)	3.19 (0.70)	0.06	0.32	0.75
16	Patients are cooperative in using outcome measures	2.0 (0.80)	2.69 (0.50)	0.69	3.15	0.007‡
17	Using outcome measures is a standard part of my clinical reasoning	2.94 (0.60)	3.13 (0.52)	0.19	0.90	0.38
18	Outcome measures are available in our practice	3.88 (0.12)	3.94 (0.06)	0.06	−0.56	0.58
19	Within our practice we have made arrangements on how to use outcome measures	3.56 (0.40)	3.75 (0.33)	0.19	−1.38	0.19
20	My supervisor(s) support the use of outcome measures	3.88 (0.12)	3.94 (0.06)	0.06	−1.0)	0.33
21	My supervisor(s) use outcome measures in clinical practice themselves	3.69 (0.36)	3.94 (0.06)	0.25	−1.73	0.10
22	My supervisor(s) require that outcome measures data are reported in our electronic health records	3.88 (0.12)	3.81 (0.16)	−0.06	1.0	0.33
23	My colleagues also use outcome measures in clinical practice	3.69 (0.36)	3.69 (0.23)	0.00	0	1.00
24	In our practice we discuss how to implement outcome measures	3.13 (0.65)	3.63 (0.38)	0.50	−2.74)	0.02‡
25	The use of outcome measures fits our work routine	3.25 (0.47)	3.31 (0.50)	0.06	−0.25	0.81
26	When possible we send patient reported outcome measures to our patients electronically.	2.13 (1.45)	1.69 (1.83)	−0.44	0.91)	0.38
27	Patient-reported outcome measures are administered by the patients themselves.	2.06 (1.53)	1.62 (2.12)	−0.44	0.92	0.37

*Using 5-point Likert scale ranging from 0: completely disagree to 4: completely agree.

†Paired t-test with df=15.

‡∗p<0.05.

PROMs, patient-reported outcome measures; PT, physical therapist; QI, quality improvement.

Administered compliance with the use of outcome measures

The administered compliance with the use of the Oswestry Disability Index (ODI) and Neck Disability Index (NDI) in patients with neck pain and low back pain, respectively, was evaluated from data in the UPMC EMR dashboard reports.

Self-reported compliance with the use of outcome measures

The use of the ODI and NDI for patients with low back pain and neck pain, respectively, was measured through self-report of the participating PTs (online supplemental appendix 2).

Secondary outcomes

Health outcomes before and after the implementation of the QI programme

Aggregated pre–post treatment change scores of the ODI and NDI were determined for patients with low back pain and neck pain, respectively. Pre-QI measurement comprised a 6-month time frame before implementation of the QI programme, and post-QI measurement comprised a 6-month time frame after completion of the QI project for each team.

Commitment to change

The perceived impact of the QI programme was measured through Commitment to Change statements,¹⁷ representing the goals of each QI team, collected at the start of the project. At project end, participants were requested to evaluate to what extent the goals for their team were met on a 10-point Numeric Rating Scale.

Qualitative data collection

After quantitative data collection, semistructured virtual interviews were conducted with two representatives from each QI team (six interviews) and one focus group with the Quality Managers from the QI teams (three participants) to evaluate the perceived value of the QI programme. Interview and focus group guides were developed by the research team.

Data analysis

Quantitative analysis

Pre–post QI scores of the survey for the use of outcome measures, administered compliance with the use of outcome measures (ODI, NDI), self-reported compliance with the use of outcome measures (ODI, NDI) and health outcomes (ODI, NDI, Pain) were calculated. Improvements in pre-QI and post-QI processes and outcomes were expressed as percent change. Non-parametric outcomes were evaluated using McNemar’s test statistics. Parametric outcomes were evaluated using Student’s paired t-test statistics. Outcomes were reported as mean pre–post treatment change scores, aggregated at office level. Submitted health outcomes data were presented by primary diagnosis (neck pain vs low back pain).

Qualitative analysis

Recordings of the interviews and focus groups (n=7) were transcribed verbatim. All members of the research team (PJvdW, EJB, PLM and KJH) open-coded the first three transcripts by analysing the content to identify potential codes related to the perceived value of participating in the programme, challenges to participation, the perceived value of specific programme components and recommendations for future iterations. A consensus process was employed including all researchers to establish the code-book (transcripts 1–3), then transcripts 4–7 were coded by two members of the research team (EJB and PLM) using the established codebook. Resulting codes were grouped through research team consensus into the following categories: barriers, facilitators, programme components, benefit/value, recommendations for future iterations of the programme and recommendations for future QI projects.

Results

Three PT offices with 17 participating PTs (6 in groups 1 and 2; 5 in group 3) comprised the QI teams. Twelve PTs identified as female; age distribution was: 20–29 (n=4), 30–39 (n=10), 40–49 (n=2) and 50–59 (n=1).

Online supplemental appendix 3 presents characteristics of the included patients in the QI projects of the three teams. Teams 1 and 3 collected data from patients with neck pain, while team 2 collected data from patients with low back pain and neck pain.

Self-assessment survey

All participants completed the Self-Assessment Questionnaire on The Use of Outcome Measures in Physical Therapist Practice before and after conducting the QI project. Table 1 presents the results of the survey. The paired t-test demonstrated a statically significant difference on two items: ‘Patients are cooperative in using outcome measures’ (item 16) and ‘In our practice, we discuss how to implement outcome measures’ (item 24); results indicated greater agreement with the statement following the QI project.

To further evaluate the pre–post measurements of the survey items, we conducted a sensitivity analysis by comparing the results of the original 5-point rating scale with a 2-point scale (agree/disagree). Survey data were transformed by collapsing 0–2 into 0 (disagree) and 3–4 as 1 (agree), replicating the process used by Meerhoff et al.¹⁶ Transformed data were analysed using McNemar’s test and resulted in question 16 ‘patients are cooperative in using outcomes measures’ maintaining significance (p=0.016). Data transformation resulted in a majority of the questions showing no change between pre-QI and post-QI programme.

Use of outcome measures

For the ODI, pre-QI and post-QI compliance with the administration of the ODI at intake remained similar (pre-QI:91%, post-QI:88%), and there was an increase of the administration of the ODI at discharge (pre-QI:14%, post-QI:66%). For the NDI, compliance showed an increase at intake (pre-QI:75%, post-QI:84%) and at discharge (pre-QI:32%, post-QI:50%) (table 2). Findings between administered and self-reported compliance with use of outcome measures were generally consistent (online supplemental appendix 2).

Table 2.

Administered compliance with the use of outcome measures

	Pre-QI			Post-QI
Low back pain (ODI)*	Episodes	Intake	Discharge	Episodes	Intake	Discharge
Team 2	146	91.1%	14.2%	196	88.3%	66.1%
Neck pain (NDI)	Episodes	Intake	Discharge	Episodes	Intake	Discharge
Overall	256	75.4%	32.4%	236	83.9%	49.6%
Team 1	104	63.5%	16.7%	43	79.1%	32.5%
Team 2	51	88.5%	66.7%	79	86.3%	71.4%
Team 3	101	80.2%	42.0%	114	83.3%	56.8%

*Only team two focused on patients with low back pain.

ODI, Oswestry Disability Index; QI, quality improvement.

Health outcomes

Overall, health outcomes measured with the ODI and NDI were improved at discharge compared with intake, both pre-QI and post-QI (online supplemental appendix 4). No consistent patterns were seen in comparing pre-QI and post-QI outcomes in terms of reaching the minimal clinical important difference or mild disability threshold for the ODI or NDI at discharge.

Achievement of QI project objectives

On determining whether the groups achieved their desired outcomes, participants rated their success on achieving these goals. The average scores of the participating PTs on the 10-point rating scale for objectives achieved in the QI project were 6.2, 6.5 and 7.0 for group 1, 2 and 3, respectively.

Interviews and focus group

Analysis of the qualitative interviews yielded insight into the facilitators and barriers to participation in the QI programme, the perceived value/benefit of programme participation, recommendations for future iterations of the programme and recommendations for future QI projects as detailed below. Referenced quotes of participants are listed in table 3.

Table 3.

Quotes of participants in the interviews and focus group

#	Quote
1	‘You guys left it very open to us to decide what we were going to do… which was cool because… we could kind of individualize what we thought would be implemented best into our practice… So I think that’s what was helpful for us, was keeping it kind of broad so that we could choose what we were doing.’ (GR1-PT1)
2	‘So I think we kind of would always start out in a group setting, so whether we did the PowerPoint presentation and went over some didactic stuff, then we would go into our hands on portion. And again, be kind of like in a group setting. Then [team leads] would show us different things. We try it on our partners and on different conditions and stuff. And usually it was through that hands on practice that we were getting into discussions about which ones were working, which ones weren't…. So I think we would go group--didactic--group, like hands on and then kind of break down and once we were in that breakdown, that’s where we get into those more individual conversations.’ (GR1-PT1)
3	‘It was kind of a positive feedback loop there. Definitely seeing our numbers go up was reassuring and encouraging for that. This is actually helping us.’ (GR3-PT1)
4	‘Sometimes we have a busy clinic and you're practicing and you have working with multiple conditions and different schedules and offices. Sometimes, it’s hard to get that meeting time and that time to get together and talk about it and discuss it.’ (GR2-PT1)
5	‘The big thing that would be improvement is actually giving us feedback on that. We all put the information in the computer, but then never hear … are our patients getting better? Again, whether that be as a clinic or as an individual clinician. It’s great that they have all the data, but what does that mean to me, I guess?’ (GR1-PT2)
6	‘I don't know if we had them (feedback reports) as timely as we probably could have…. I'd like to see once a month and then if we're going to do it for extended period of time, compare quarters like a rolling quarter 3 months and just see how we're doing with that.’ (GR2-PT2)
7	‘I think in terms of just number of patients. We just didn't have the volume that we had under normal circumstances…. Then also too, just us developing and using these Zoom platforms wasn't as common either. So, I think that changed over time too. Like we got all used to it and better at it. You know what I mean? It wasn't like we were going down to (another clinic) and having a meeting in a conference room. We were actually having a meeting over a virtual platform. That took some getting used to it was too.’ (GR2-PT2)
8	‘I think we always kind of did that, but never really wrote it down. So again, that whole circling back to the accountability thing where it made us say yes, we did it and we wrote it down in the treatment diary and we checked it off. So that was good. So, I think we always kind of subconsciously did it, but just never formally and consistently did it across the board.’ (GR2-PT2)
9	‘Being able to bounce those ideas off of someone right then and there, or saying, ‘Hey, I tried this. This didn't really work. Do you have another idea of what I can do to maybe get the segment to loosen up or get that muscle to relax,’ has been really beneficial as well, especially since they've all went through the training too.’ (GR1-PT2)
10	‘There’s definitely a usefulness to this idea of constantly questioning what you're… keeping a consistent questioning of what you're doing. How can you do it better and then following up on if it’s actually better.’ (GR3-PT2)
11	‘I think it was really helpful in that having something there that’s evidence-based in front of us to use. So yeah. Hence or thus forth, [I would have] gone onto the APTA website and looked up many more practice guidelines to keep fresh in my mind.’ (GR3-PT1)
12	‘But I kind of see how I can implement that almost on an individual basis. Like, I don't have to do this as part of a research project with a university, I can just kind of do this within my own practice and figure out for other areas where else I can improve.’ (GR1-PT1)
13	‘You know, better and further educate our patients, if neck and back patients to see if we can use a little more patient education, more consistent patient education, talk about their impairment or functional limitations.’ (GR2-PT1)
14	‘And I liked that in the beginning it was a lot more of you guys leading us, and then after that period it transitioned into more of us dictating the conversation.’ (GR3-PT2)

Facilitators to programme participation

The category ‘facilitators’ represents perceptions of enablers to program participation. Facilitators of note related to the process by which projects were designed and conducted and to the data and feedback guiding project completion. Participants commented on how autonomy in project selection and development, accompanied by feedback and guidance from the research team, enabled selection of projects aligned to a clinics’ unique needs, eventuating successful project completion [quote 1].

Periodic communication and feedback from the research team throughout project completion was also noted as an essential facilitator. Additionally, collaboration within the QI teams facilitated learning and knowledge sharing essential to the QI process. Teams noted that a positive approach to learning and engaging in the program supported collaboration and communication among clinicians in the same office and in other offices. A collaborative learning approach invited knowledge sharing across members of a QI team related to new techniques, leveraging expertise of more knowledgeable peers and creating communities of practice in which less experienced clinicians could learn from those more experienced. Collaborative learning was also supported by one QI team’s combination of didactic and hand-on approaches, in which more knowledgeable peers guided skill development. Team leads and their expertise proved essential to guiding the learning process and collaboration within the QI teams [quote 2].

Feedback on performance was also essential to the process of learning and improvement. Feedback reports related to individual and team performance on specific metrics served several purposes. They created awareness of missing data, inviting reflection upon process to ensure accurate data capture. Also, seeing numbers and scores and reviewing reports within the QI team, particularly when scores improved over time, provided motivation toward change and validation of change efforts [quote 3].

Barriers to programme participation

The category ‘barriers’ indicates perceived impediments to participating in the QI Program. Participants noted barriers related to time, data and feedback, and the external environment as barriers. Time was noted by all participants as an impediment to participation. Time and scheduling conflicts served as individual and team level barriers. Competing demands on individual time and team scheduling conflicts posed challenges to project implementation and collaborative learning [quote 4].

In addition, time required to set up the QI project served as an initial barrier; however, the requisite time for set up reduced the required time to implement projects. While noted as facilitators, outcome data and feedback on performance were also emphasized as potential barriers to participation. A lack of outcome data or feedback on an individual’s or groups’ could inhibit determination of progress and influence motivation [quote 5]. Correspondingly, participants emphasized the need for more regular and consistent feedback [quote 6].

Finally, participants noted external constraints beyond control of participating clinicians or the clinics as significant barriers. The pilot occurred during the COVID-19 pandemic, and participants noted how it eventuated reduced patient volume and reduced in person team time and also necessitated an adjustment to virtual meetings [quote 7].

Benefits/value to participating in the programme

The category ‘benefit/value’ relates to the perceived positive outcomes of participating in the QI programme. Noted benefits related to the clinic and clinic practices, the individual clinician and the patient.

Benefits to the clinic/clinic practices

Participation in the QI program provided accountability to explicit processes or educational strategies aimed at continuous improvement in patient care. Specifically, it provided accountability to a streamlined process toward improved patient care [quote 8]. Participants also noted that the QI program invited collaboration and knowledge sharing among clinicians that supported learning and practice change [quote 9].

Additionally, participants emphasised how the programme ensured application of best practice/best evidence within a specific local context aligned to a specific clinic-related problem in providing patient care. From their perspectives, participation in a programme such as this enables improved implementation of clinical practice guidelines. Correspondingly, an often-noted result was improved clinical care (ie, improved patient outcomes and change scores).

Benefits to individual clinicians

Specific benefits for individual clinicians included increased learning and awareness related to the quality improvement process, specific clinical practice guidelines, and new techniques for patient care. Participation in the program also fostered an increase in reflective practice by inviting reflection upon performance and identification of areas of improvement [quote 10]. One participant commented that participation in this pilot inspired them to look at other clinical practice guidelines for future improvement in their own practice [quote 11].

Participants also noted improved confidence in participating in QI initiatives and in applying specific clinical practice guidelines and new techniques for patient care as an important benefit. One even noted feeling empowered to engage independently in future QI projects [quote 12].

Benefits to the patient

In addition to noting improved patient outcomes as results of program participation, the participants also noted how improved processes yielded better patient education [quote 13].

Recommendations for future iterations of this programme

Participants noted several recommendations for programme improvement. First, they recommended that clinics keep the scope of their projects to something quick to implement and that would readily yield improvement/change. They also recommended that feedback reports be available on a monthly basis with a quarterly comparison of progress and that the reports include both clinician and clinic data. As one participant noted, if feedback reports were ‘broken down by therapist’ then they could more readily discern their improvement/progress relative to their peers.

Participants also noted specific components of the QI programme that were essential to future iterations such as meetings between the Research Team and the QI project teams, particularly when the teams were deciding on projects and reviewing data (quote 14). They also indicated meetings with other QI project teams involved in the pilot as helpful in sharing process tips and lessons learnt during implementation.

Recommendations for future QI projects in the clinic network

Qualitative results also revealed recommendations for future QI projects within the UPMC network. To facilitate continuous QI and integration of most recent evidence-based practices, participants recommended that UPMC regularly send clinical practice guideline updates to clinicians, perhaps on a monthly basis. They also recommended mandatory QI participation for clinicians with less than 5 years’ practice experience. However, they cautioned that future projects must carefully consider the time requirement involved for clinicians.

Integration of results

Figure 1 represents the integration of qualitative and quantitative findings for this study, showing the barriers, facilitators, benefits and outcomes for participating in the QI programme. Bidirectional arrows reflect the influence of internal and external environments and the application of knowledge across boundaries.

Figure 1.

Integration of quantitative and qualitative results.

Discussion

This pilot study shows that a QI programme using PDSA cycles based on the feedback of routinely collected health outcomes improved the use of outcome measures at discharge, allowing for comparison of treatment outcomes over time within and between practices. Overall, questionnaire results indicated that participants reported being knowledgeable with positive attitudes towards using outcome measures. No consistent patterns were seen in comparing pre-QI and post-QI treatment outcomes. Perceived benefits of the QI programme related to the team, the individual clinician and the patient. Facilitators related to the process by which projects were designed and conducted and to the data and feedback guiding project completion. Noted barriers included time, data and feedback and the external environment.

Pre-QI, all three teams showed high completion rates of outcome measures for low back pain and neck pain at intake, although the completion rates dropped significantly at discharge. During their QI projects, participants became aware of the limited use of outcomes at discharge and increased their efforts of completing outcome measures, which resulted in increased completion rates at discharge. For QI team 2, outcome measure compliance dropped slightly following the QI project for both the NDI and ODI, however, the differences are low (NDI: 91% vs 88%; ODI 89% vs 86%). These differences did not appear significant to us as we also considered the effects COVID-19 had on patient-related processes in the post-QI period.

The UPMC CRS practice was concurrently focusing on continued development of their clinical dashboard and efforts were underway at an organisational level to increase PROM collection through methods of engaging leadership and key clinicians in addressing barriers to collection, audit and feedback to leadership teams and continued efforts to improve the data presented in the dashboard. Data from a Dutch study also showed improvements in the use of outcomes based on a systematic process, supporting the importance of a robust implementation process.¹⁶

Participants perceived several benefits of the QI programme. Autonomy in tailoring the project to the needs and objectives of the participants was perceived to be an important facilitator. We also found that collaboration, communication and trust were essential to all phases of the project for establishing a continuous learning cycle. While outcome data and feedback on performance were noted as facilitators, these can become barriers when feedback is not provided on a regular basis or data are lacking robustness. Such mechanisms have also been identified in a realist review that examined the circumstances and processes through which PROMs feedback improves patient care.⁸ Implementation strategies are, therefore, important to address barriers and facilitators, and QI efforts should be tailored to the local context.¹⁸ Our iterative approach in designing the QI strategy with the UPMC knowledge brokers and collaborating with the Quality Managers from the QI teams throughout the project was important for tailoring the QI programme and created consistency across the teams in providing feedback on performance.

Our study fits well in an era of using outcomes for QI and improving value. Routine outcome measurement is deemed important for learning health systems that integrate knowledge and practice through cycles of continuous QI and learning.¹⁹ However, the research base for using aggregated PROMs data for QI is limited.¹⁸ Several studies have described the use of outcome measures in PT practice,^{16 20 21} but evaluative studies of using outcomes in continuous QI are needed. Our study adds to the current knowledge base that the systematic use of PDSA cycles supports PTs in their QI activities.

An important prerequisite for producing and using feedback reports of treatment outcomes is the availability of a database in which data are collected and the robustness of the data for reliable and valid reporting.⁹ Furthermore, outcome data should be collected on a routine basis, and it is recommended that healthcare organisations implement processes and policies to sustain compliance with data collection. This is especially important for collecting outcomes at discharge. UPMC is a large system that has implemented processes for routine data collection and continuous QI.²² Other healthcare organisations have also published such initiatives using embedded learning health system researchers.²³ At the national level, professional bodies of health professionals have established national registries, allowing for benchmarking at national level.¹⁶ Such registries can be used to encourage projects with specific QI training directed at smaller institutions or practices that do not have their own data infrastructure.

This study is subject to several limitations. First, the small sample size of three QI teams with 5–6 PTs prohibited advanced statistical analyses of the study. Second, patients or the public were not involved in the study. Although the study was aimed at internal QI of health professionals, inclusion of the patient’s perspective in the design, conduct or reporting would have strengthened our research. Third, inclusion of a voluntary sample with motivated PTs from a large networked system may limit the generalisability of the results of this study. Encouragement of similar projects should also be directed to smaller institutions or practices. Fourth, the QI programme was interrupted due to the pandemic. The specific QI projects of the three teams were put on hold between March and October 2020. Fifth, selection bias could also have played a role in patients being more or less cooperative in completing questionnaires at discharge, although we have no indications that this happened in a specific direction. In addition, the low scores on the CCI suggest a healthy population, and the selective sample of patients may have influenced the findings.

Conclusion

Our study shows that the QI programme is feasible and well accepted by the pilot teams of PTs, with improved processes for measuring outcomes. The researchers suggest active clinician participation in developing an outcomes-based QI programme supported the use, feasibility and value of outcome measurement for clinical practice. The results are promising for further evaluation and implementation of using treatment outcomes for QI efforts in PT practice.

Data availability statement

Data are available on reasonable request.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

This study involves human participants and was approved by the George Washington University Committee on Human Research, Institutional Review Board (IRB), GWU IRB# 180659; and through the UPMC Quality Improvement Review Committee (Project ID #1982).