Management of musculoskeletal conditions with remotely delivered physiotherapy versus face-to-face physiotherapy: process evaluation of the REFORM trial using the Realist Evaluation and the RE-AIM Framework

Abstract

Objectives

The objectives of this process evaluation were to explain the results of the REFORM (REhabilitation FOR Musculoskeletal conditions) trial and identify potential facilitators and barriers to the future rollout of a remotely delivered physiotherapy model of care.

Setting

Outpatient physiotherapy units in five government-funded public hospitals in Sydney, Australia.

Design

This process evaluation was run alongside the REFORM trial. The REFORM trial (n=210) set out to determine whether remotely delivered physiotherapy (with one initial face-to-face session with a physiotherapist) was as good or better than a course of face-to-face physiotherapy. The process evaluation was informed by The UK Medical Research Council Process Evaluation Guidance. It was also based on the Realist Evaluation and the dimensions of the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) Framework. Qualitative and quantitative data were collected. The qualitative data included semistructured interviews. The quantitative data included audits of screening logs and participants’ self-reported satisfaction with service delivery. A thematic analysis using both inductive and deductive approaches was used for qualitative analysis. Multiple data were used to triangulate the process evaluation findings.

Participants

Quantitative data were collected on 210 participants. Qualitative interviews were conducted with 20 participants, 15 physiotherapists and 5 stakeholders.

Results

Some participants valued the convenience and accessibility of remotely delivered physiotherapy. It was also deemed to have the potential of improving the efficiency in the way physiotherapy is delivered. The findings from the RE-AIM framework were mixed. For example, the Reach was limited, and the Adoption and Maintenance were inconsistent across sites.

Conclusion

Remotely delivered physiotherapy has the potential to be rolled out across Australian clinical settings to participants similar to the REFORM trial. However, further research involving participants with different types and severity of musculoskeletal conditions is needed to improve the generalisability of our findings. Adoption might be improved with a hybrid model of care where physiotherapists’ and patients’ preferences are met.

Trial registration number

ACTRN12619000065190.

Strength and limitations of this study.

• This process evaluation was run alongside a randomised controlled trial and used triangulation of qualitative and quantitative data from many different sources.

• This process evaluation followed a prespecified protocol which outlined the underlying causal assumptions of the remotely delivered physiotherapy model of care.

• This process evaluation was conducted in a small number of public hospitals in Sydney, Australia, with a restricted range of patients, thereby limiting the generalisability of the results.

• The COVID pandemic may have influenced participants’, physiotherapists’ and stakeholders’ acceptability of this model of care.

Introduction

Musculoskeletal conditions can prevent individuals from performing daily activities and restrict participation in employment and social activities.¹ Current evidence shows that most musculoskeletal conditions are best managed by primary healthcare professionals such as physiotherapists using exercises, support and advice.¹,³ In Sydney, Australia, physiotherapy is provided free of charge in government-funded public hospitals. However, there are often long waiting times and some patients find it difficult and costly to regularly travel to hospital for physiotherapy.⁴,⁷ To address these problems, our research team set up a randomised controlled trial to compare a remotely delivered physiotherapy model of care with usual face-to-face physiotherapy. We called this the REFORM (REhabilitation FOR Musculoskeletal conditions) trial.⁸

The results of the REFORM trial indicated that our model of care was as good as regular face-to-face physiotherapy.⁹ While these results tell us about the effectiveness of the intervention, we needed to do parallel work to better understand why the intervention proved to be as good as face-to-face physiotherapy. We had assumed that our model of care would be accessible and convenient for patients, and may prove to be less costly for healthcare providers.¹⁰ We also wanted to explore barriers and facilitators so we could improve the implementation of our model of care in real world clinical settings.¹¹,¹⁴ For these reasons, we conducted a process evaluation alongside the REFORM trial (see Withers et al¹⁰ for the protocol of the process evaluation). This paper reports the findings from the REFORM process evaluation. Our aims were to (1) explain the trial results and the generalisability of the REFORM model of care, and (2) to identify barriers and facilitators to the future rollout of the remotely delivered physiotherapy intervention.

Methods

Summary of REFORM trial

The REFORM trial was a multicentre pragmatic non-inferiority randomised controlled trial (see Withers et al⁸ for the protocol of the trial). The trial ran from March 2019 to May 2023 with some interruptions due to COVID-19. The aim was to determine whether remotely delivered physiotherapy with ongoing support and advice was as good or better than usual face-to-face physiotherapy for patients with musculoskeletal conditions. 210 participants were recruited from five public hospitals in Sydney, Australia. Participants were randomised to either the remotely delivered physiotherapy group or the face-to-face physiotherapy group for 6 weeks. Participants allocated to the remotely delivered physiotherapy group received one face-to-face physiotherapy session in which they were assessed and provided with an individualised exercise programme designed by the trial physiotherapist. The trial physiotherapist then sent the participants’ exercise programmes to their mobile devices using a freely available app developed by some of the authors (www.physiotherapyexercises.com). The app was downloaded to participants’ devices during the initial face-to-face physiotherapy sessions. The trial physiotherapist educated the participants on how to access the app and use it to record their exercise adherence. Recorded exercise adherence was monitored remotely by the trial physiotherapist. Participants also received weekly text messages and fortnightly telephone calls for ongoing support and advice from the trial physiotherapist. Outcomes were collected at 6 and 26 weeks. The results indicated that outcomes at 6 weeks on the Patient Specific Functional Scale (the primary outcome) were as good with remotely delivered physiotherapy as usual face-to-face physiotherapy.⁹

Theoretical frameworks for the process evaluation of the REFORM trial

This process evaluation is guided by the recommendations of the UK Medical Research Council’s guidelines for process evaluations of complex interventions.¹¹ They recommend first articulating the causal assumptions underpinning the trial. For this trial, these were why we believed that remotely delivered physiotherapy may be as good or better than face-to-face physiotherapy. These causal assumptions have been outlined in our protocol paper¹⁰ and are summarised in table 1. The guidelines then recommend explaining the trial results and identifying barriers and facilitators by examining each causal assumption and exploring how each is influenced by three factors, namely context, implementation and mechanism of impacts.¹¹ We followed this process using two frameworks, namely, the Realist Evaluation and the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) Framework. Reporting of this process evaluation is also guided by the Standards of Reporting Qualitative Research.¹⁵ The details are provided below and in the supplementary file (see online supplemental tables 1 and 2).

Table 1. Realist Evaluation programme theories and configuration of Context, Mechanism, Outcome (CMO) for the REFORM trial along with the data sources.

Programme theories	Context (For whom it worked?)	Mechanism (How did it work?)	Outcomes	Data sources
Increased convenience for participants.	Participants who are time poor and have other commitments, and hence prefer the convenience of a supported home exercise programme.	Participants can exercise at times that suit them. No travel and hence time not spent travelling can be used to meet other commitments.	Supported home exercise programmes are more convenient and less time-consuming than face-to-face physiotherapy for some people.	Interviews with trial participants, physiotherapists and other stakeholders. Travel time to and from physiotherapy appointments provided as part of the trial.
Increased accessibility to physiotherapy for participants.	Participants who have mobility issues and difficulty regularly getting to physiotherapy appointments at the hospitals.	Negates the need for sufficient mobility to get to regular physiotherapy appointments at the hospitals.	People with significant mobility or difficulty travelling were able to access the physiotherapy more easily.	Interviews with trial participants, physiotherapists and other stakeholders.
Increased efficiency in the delivery of physiotherapy services for the funders and healthcare system.	Physiotherapy departments in public hospitals (where physiotherapy services are government funded) with limited budgets and long waiting lists.	Physiotherapists require less overall time to provide home-supported exercise programmes than face-to-face physiotherapy.	It is a more efficient method of service delivery because it requires less overall physiotherapy time.	Economic data collected alongside the REFORM trial. Interviews with trial participants, physiotherapists and other stakeholders.
Participant empowerment.	Participants who are motivated to improve their own health conditions and take control of their own health.	Provides a sense of responsibility for own outcomes. Enables participants to take control. Provides a sense of self-empowerment.	Supported home exercise programmes provide motivation and accountability.	Interviews with trial participants, physiotherapists and other stakeholders.
Participants can regularly perform their own exercises at home without the need for ongoing physiotherapy supervision, provided they also receive an initial face-to-face session with a physiotherapist and support and advice via telephone and text messages.	Participants who feel confident using apps and technology.	Requires an easy-to-use app.	People can be shown exercises once during an initial face-to-face physiotherapy session and continue exercising at home with the app and appropriate support and advice via text messages and telephone calls.	Number of phone calls and text messages received/sent and the duration of the telephone calls provided to participants in the remotely delivered physiotherapy group. Content of phone calls to participants in the remotely delivered physiotherapy group at weeks 2 and 4. Participants’ self-reported satisfaction with service delivery. Interviews with trial participants, physiotherapists and other stakeholders.
	Participants who are content for a physiotherapist to monitor their exercises by the telephone and an app.	Requires an app that physiotherapists can monitor adherence. Participants need to know that a physiotherapist is remotely monitoring their exercise adherence.
	Participants who believe in the importance of exercise and do not have strongly held beliefs about the importance of hands-on treatment and/or ongoing face-to-face physiotherapy.	Participants’ beliefs about the type of intervention they require influence outcomes and satisfaction.
	Participants who are content to receive support and advice by texts and telephone calls.	Support and advice by text and telephone can be as good or better than the support and advice provided during face-to-face physiotherapy. An initial face-to-face physiotherapy session is sufficient to establish a relationship that is meaningful so as to provide motivation.
	Participants who can maintain motivation to regularly exercise at home.	Text messages and telephone calls provide motivation. Participants’ knowledge that their physiotherapist is remotely monitoring exercise adherence provides motivation.

The Realist Evaluation was used to explain the trial results and trial generalisability (see online supplemental file for a summary of the key findings).

REFORM, REhabilitation FOR Musculoskeletal conditions.

The Realist Evaluation

The Realist Evaluation¹⁶,²⁰ was primarily relied on to explain the trial results and trial generalisability. Five programme theories were produced based on the evidence from the current literature and the clinical experience of the research team. These are presented in the context–mechanism–outcome configuration that encompasses the many different factors that could trigger the mechanisms linking the REFORM intervention to the outcome.¹⁶,²¹ In turn, the context and mechanisms through which the intervention may prove to be as effective as face-to-face physiotherapy were defined. Our data sources were then mapped to each of the three components of our context–mechanism–outcome configuration. The Realist And Meta-narrative Evidence Syntheses: Evolving Standards (RAMESES II) reporting standards for Realist Evaluation²⁰ were used to guide our reporting.

The reach effectiveness adoption implementation maintenance (RE-AIM) framework

The RE-AIM dimensions²²,²⁶ were incorporated in the process evaluation framework to help identify barriers and facilitators to the future rollout of remotely delivered physiotherapy in the real-world clinical setting. This was considered from the perspectives of the participants, physiotherapists, stakeholders and health systems. The five dimensions of the RE-AIM were then used to explore barriers and facilitators from each of these perspectives. That is: Reach, this was used to reflect how representative the participants were of the target population. This was determined by both the sites’ abilities to recruit participants as well as the demographics of the patients that presented to the included hospitals; Effectiveness, this was used to reflect the impact of the REFORM trial intervention on outcomes including both positive and negative consequences, quality of life, participant satisfaction and economic outcomes; Adoption, this was used to reflect how well the sites and participants embraced the intervention as reflected through the interviews, sites’ willingness to recruit participants and participants’ exercise adherence data; Implementation, this was used to determine how well the intervention was delivered, namely, fidelity; Maintenance, this was used to gauge how well remotely delivered physiotherapy could be maintained or implemented into usual clinical practice. The maintenance dimension included consideration of the time, cost and organisational support required for the future rollout of the intervention.

Questions within each of the five dimensions of the RE-AIM Framework were articulated and data sources were mapped to each question (see online supplemental table 2).

Data sources and analyses

A range of qualitative and quantitative data were collected and analysed. These are detailed in the published protocol paper¹⁰ and online supplemental file. The online supplemental file contains details about how each data source was collected. In brief, the data sources were:

Interviews with trial participants, physiotherapists and other stakeholders

15 physiotherapists, 5 stakeholders and 20 trial participants (9 from the face-to-face physiotherapy group and 11 from the remotely delivered physiotherapy group) were interviewed (see online supplemental file: interview guide). Thematic analysis using a hybrid inductive and deductive approach was used for qualitative analysis.

Audit of the screening logs

The screening logs collected as part of the trial were audited to gauge the types of people who were recruited to the trial, and the reasons why people were ineligible.

Demographic characteristics of the included participants

The demographic characteristics of the included participants were examined to gauge the types of people who were recruited to the trial.

Number of phone calls and text messages received and sent, and the duration of the telephone calls provided to participants in the remotely delivered physiotherapy group

The number of phone calls and text messages that were provided as part of the intervention to the participants in the remotely delivered physiotherapy group, as well as the duration of the telephone calls, was analysed.

Content of phone calls to participants in the remotely delivered physiotherapy group at weeks 2 and 4

The content of the phone calls made to participants in the remotely delivered physiotherapy group at weeks 2 and 4 was analysed.

Exercise adherence of participants in the remotely delivered physiotherapy group as recorded through the app and telephone calls at 2, 4 and 6 weeks

Participants in the remotely delivered physiotherapy group were asked to record on the app each time they exercised. In addition, participants were asked during the phone calls at 2, 4, and 6 weeks whether they had practised their exercises. These data were analysed.

Participants’ self-reported satisfaction with service delivery

All participants were asked at 6 weeks about their satisfaction with the physiotherapy they had received over the course of the trial using a scale from 0 (not at all satisfied) to 10 (extremely satisfied).

Adverse and serious adverse events at weeks 6 and 26

All participants were asked about any serious adverse events or adverse events they had experienced at their 6-week and 26-week assessments.

Economic data collected alongside the REFORM trial

Economic data were collected from health funder and patients’ perspectives. Health funder costs included physiotherapists’ time to provide services in both the face-to-face physiotherapy group and the remotely delivered physiotherapy group. Patients’ costs included their time devoted to other healthcare use and associated costs (eg, distance travelled to attend appointments and purchase of equipment if needed). The results of the economic evaluation collected for the REFORM trial are published elsewhere.²⁷

Travel time to and from physiotherapy appointments provided as part of the trial

All participants were asked how long they spent travelling to and from their appointments with healthcare professionals as part of economic data collected.

All data were analysed separately and iteratively as per UK Medical Research Council Guidance for complex interventions.¹¹ Triangulation of the data was conducted to increase the validity of findings between different data sources.^{28 29} The results of all quantitative data are presented descriptively using means (SD), medians (IQRs), percentages and counts as appropriate. All statistical analyses were performed using Stata Statistical Software.³⁰

Patient and public involvement

Experiences, opinions and perspectives of participants, physiotherapists and stakeholders were captured through the interviews. This provided qualitative data for the interpretation of the trial results and future rollout of the remotely delivered physiotherapy model of care.

Results

The findings from all our data sources are summarised in tables1 2 and presented in the online supplemental file. A summary of results is presented in the following section.

Table 2. Key findings from the five dimensions of the RE-AIM Framework.

Dimensions	Main findings
Reach	Audit of the screening logs: An estimated 1535 people presented to the five hospitals. The participation rate was low, with only 14% of those screened for the trial being included. The participants represented those who typically present to the included hospitals. However, it did not represent those unable to speak English. In addition, the catchment of the included hospitals was not reflective of all hospitals in Sydney or Australia. Demographic characteristics of the included participants: Participants had a median (IQR) age of 53 (41–66). Most participants in the trial had chronic conditions (84%) with back (30%) and knee (23%) being the most common location of injury. This may not reflect the range of participants that we intended to target.
Effectiveness	Trial results: The Patient Specific Functional Outcome (primary outcome of the REFORM trial) showed that remotely delivered physiotherapy was as good as face-to-face physiotherapy. Adverse events at weeks 6 and 26: There were no serious adverse events. There was no notable difference in adverse events reported between the two groups.
Adoption	Recruitment rates at each site: There was a large range in the number of participants recruited at sites (ranging from 17 to 71). This may reflect how well the staff at the hospital embraced the REFORM intervention and the trial. Interviews with trial participants, physiotherapists and other stakeholders: ‘It was nice to see some of the chronic patients being treated in a way….that was appropriate for them … through evidence based and minimal hands on treatment and all that.’ – physiotherapist 5 ‘I think we’re very evidence-based and [we] make sure [our treatments are] best and patient centred. I think if we know that it works, with the evidence, …and know who we can deliver it to effectively and things like that. I think if we overcome the barriers like we have appropriate technology based on the patient and therapist, and good systems and programs and so on. I think most of these things are still kind of in their infancy so once we get a more mature system in a few years’ time, like a lot of barriers won’t be issues.’ – physiotherapist 1 ‘… some of our clients… particularly some of the older ones who are not as confident in their abilities to use technology, just [like] that sort of good old-fashioned…they like to come in and actually have a face-to-face conversation. That’s the therapeutic relationship kind of thing’ – physiotherapist 10 ‘I [would] recommend my friends see her [if they have] the same issue …. I am really thankful for [the trial physiotherapist). She … gave me step by step exercise, without cortisone injection, without any surgery. Even the specialist, … was surprised as well [by] the improvement which I had….’ – participant 5 ‘(With face-to-face physiotherapy) that rapport builds up, plus the motivation to keep you going. You know, you can hide at home and not do the exercises correctly and not really see any gains. But I think going to the face-to-face… sort of give(s) yourself that little bit of added pressure to do well….’ – participant 7 Exercise adherence of participants in the remotely delivered physiotherapy group as recorded through the app and telephone calls at 2, 4 and 6 weeks: Exercise adherence recorded through the app showed 50% of participants practised two or more exercises on 3 days in week 1. This figure dropped to 1 day by week 2–5 and 0 days in week 6. Exercise adherence reported during phone calls at 2, 4 and 6 weeks showed 86% of participants stated at week 2 that they practised exercises each day, spending a median (IQR) of 20 (15–30) min per day. This figure remained largely unchanged over the 6 weeks.
Implementation	Number of phone calls and text messages received and sent, and the duration of the telephone calls provided to participants in the remotely delivered physiotherapy group: 94% and 90% of participants spoke to the trial physiotherapist at weeks 2 and 4, respectively. 100% of participants received texts during the 6 weeks. Content of phone calls to participants in the remotely delivered physiotherapy group at weeks 2 and 4: During the 2-week phone calls, 64% of participants received advice, 83% of participants were assessed and 23% and 24% of participants received encouragement and support. This was similar during the 4-week phone calls.
Maintenance	Economic data collected alongside the REFORM trial: The health system plus patient costs were AUD $742 per participant for remotely delivered physiotherapy group and AUD $910 per participant for face-to-face physiotherapy group. Participants’ self-reported satisfaction with service delivery: The median (IQR) score was 8 (5–10). This was not notably different to participants in the face-to-face physiotherapy group.
	Interviews with trial participants, physiotherapists and other stakeholders: ‘Quite honestly, I’m a bit old fashioned. I prefer face-to-face. Yeah. Like, I don’t like doing things online and on the phone.’ – participant 8 ‘…making sure that the clinician is comfortable using it and understands…what wording to use to generate an exercise etc. Because that could be so much more time consuming.’ – physiotherapist 12 ‘We found that new staff [and]… junior staff struggled with the concept….more so being able to clinically reason and I think the confidence of actually having patients in front of them gives them a bit of reassurance…if you have that clinical knowledge and expertise behind you then you have more of the confidence.’ – physiotherapist 11 ‘I would be very worried if, for example, it was mandated that we only ever treated everyone forever remotely. I’d be very worried about lots of our patients slipping through the cracks in that context. I just think that our patient groups, they always come with their own individual sort of preferences and complications.’ – physiotherapist 10 ‘Moving forward we’re hoping to offer a flexible service delivery model, whereby we have all of these means and we can, treat people by any of these means and as long as it’s not contraindicated, or we feel like there’s risk, we will go ahead with that. And rather than sort of saying that everyone needs a face-to-face appointment, it will be based on the patient and the presentation and their preferences.’ – physiotherapist 12 ‘Certainly, for people in rural or more remote areas, obviously there is a lot of benefit there in terms of people not having to travel long distances. Deal with parking also is what people find challenging. Those would be the instances where I think the benefits lie.’ – physiotherapist 10 ‘I feel like people can adapt to it definitely, but it depends on their resources.’ – participant 4 ‘…it certainly relies on both the therapist and patients having the resources, so the technology and stable internet connection and all of those things…and the confidence in being able to operate it and use it….’ – physiotherapist 10 ‘…we need to ensure for those … that have been given an app and an exercise program that there’s support for that.’—physiotherapist 12 ‘I think …. provided we were all given enough information and given enough training and educated on the best way to get our patients involved in this. I think if we have to get onboard, we will get onboard.’ – physiotherapist 5

REFORM, REhabilitation FOR Musculoskeletal conditions.

The Realist Evaluation

Our findings largely supported the following proposed programme theories as to why remotely delivered physiotherapy may be appealing to patients, physiotherapists, funders and healthcare systems:¹⁶,²⁰

Increased convenience for participants

The findings from the interviews of participants and physiotherapists supported our first programme theory that some participants would welcome the convenience of remotely delivered physiotherapy. For example, one participant stated:

The fact that you can…do it anytime at home. You’re not locked into a time like an appointment at one o’clock… You can do it whenever you want. – participant 1

The convenience was also reflected in the travel times. For example, the median total travel time for the face-to-face physiotherapy group was 2.3 hours as compared with only 45 min for the remotely delivered physiotherapy group (this was time to attend the first appointment).

Increased accessibility to physiotherapy for participants

Participants and physiotherapists interviewed supported our second programme theory that the REFORM intervention increased accessibility to physiotherapy for some participants. One participant stated:

… I don’t have to organise a time to go to the hospital. And I have a baby and nowhere to put him. A lot of things. And also the parking. So yep – it is good when you can do it at home. And also, you don’t have to maintain any time. Any time you are free you can do it. – participant 3

Other illustrative quotes from participants, physiotherapists and stakeholders are included in the online supplemental table 1.

Increased efficiency in the delivery of physiotherapy services for the funders and healthcare system

The economic analysis found that the REFORM intervention was more cost efficient. The costs to the health system and patients were AUD $742 per participant in the remotely delivered physiotherapy group and AUD $910 per participant in the face-to-face physiotherapy group.²⁷ This cost difference was primarily due to the reduced time required by the physiotherapists to manage their participants in the remotely delivered physiotherapy group compared to the face-to-face physiotherapy group. However, the time (and hence cost) difference was not as great as one might anticipate because participants in the face-to-face physiotherapy group only received a median of three face-to-face physiotherapy sessions (see online supplemental table 3).

Quotes from interviews showed that physiotherapists and stakeholders indicated that remotely delivered physiotherapy had the potential to increase efficiencies within the healthcare system. This belief was reflected by the following quote (see online supplemental table 1 for more quotes):

I think personally to me, equity in healthcare is very important, and so having this program, which reduces the waiting list and people are seen in a timelier manner, that definitely appeals to me. For this program to come in and reduce our waiting times, that’s a definite huge bonus. – physiotherapist 2

Participant empowerment

There was some evidence from data sources that the REFORM intervention might be particularly beneficial for participants who are motivated to improve their own health conditions and take control of their therapy. This was evident in some participants who commented on feeling motivated to practise their exercises. For example, two participants stated:

It motivated me, because I sort of felt like, ‘oh, I’d better do them’, because otherwise she’s gonna text me. – participant 1

So my knee feels……90%. I can do what I have to. I can run around. I can exercise again. So, I don’t need that operation. So, in my case the app was really good. I did the exercises every day. I could see that it was getting better. – participant 2

Physiotherapists believed that it was important to empower patients in their care in general and often talked about a model of care that involved coaching rather than regular face-to-face treatments. This was reflected in comments like this (see online supplemental table 1 for more quotes):

People need to be empowered. They need to be educated appropriately and then they need [to] be empowered to have the knowledge and skills to be able to do it themselves……While physios have a massive role in showing people what to do and changing their mindset, it’s kind of the person that’s the one that’s got to action that change. – physiotherapist 1

The initial face-to-face physiotherapy session along with telephone calls and text messages supported ongoing exercise at home

All participants had an initial face-to-face session with the trial physiotherapist, and most received their subsequent phone calls and text messages which provided advice and support to encourage ongoing exercises at home. For example, 94% and 90% of participants received a phone call at 2 and 4 weeks, respectively, 56% and 64% were provided with advice at each phone call and approximately 40% were provided with encouragement and support (see online supplemental table 4). Most also received some form of assessment during each phone call. While we cannot verify the participants’ perceptions about the quality of advice and support they received, we do know that they expressed high levels of satisfaction with remotely delivered physiotherapy at 6 weeks when asked (see online supplemental table 5). We also know from the interviews that most participants felt confident using the exercise app. This is reflected in this quote:

It was really easy, when you open the app, all your stuff was on it…so if you are ever unsure of how to do it, it had a little description of how to do it. – participant 4

Although this view was not shared by all, with one participant saying:

I wouldn’t recommend it to anyone who doesn’t have at least basic knowledge of the mobile phone. – participant 1

Another participant implied that they were encouraged to exercise at home because the participant knew that the physiotherapist could see when the participant recorded practising their exercises on the app. The participant stated:

Pokes me a little bit, and maybe because I was slacking off a little. In a nicest way of course, but that’s important too … You know, just a little poke here and there, like she’s saying, ‘I know what you’re doing, I can see it’; yeah, okay, alright. – participant 1

Participants also reported feeling supported by texts and telephone calls from the trial physiotherapist (see online supplemental table 1 for more quotes).

… she [the trial physiotherapist] prompted me a lot via text message, which [was] very helpful, for me to do the reminding… and like providing motivation to me. – participant 5

The RE-AIM framework

The results specific to each of the five dimensions of the RE-AIM Framework are summarised below.

Reach

The findings from our data sources indicated that a large number of potentially eligible participants were excluded. For example, our recruitment logs indicated that only 14% of those screened (∼1535 people) were ultimately randomised to the trial. 176 eligible patients declined to participate despite knowing that they would be moved off the waiting list. A large number of potential participants were excluded due to the need for an interpreter, and some were excluded because they did not have access to smart devices (see online supplemental table 6). Some of the reasons for exclusion were due to the characteristics of the five sites. They were large public hospitals in Western and Northern areas of Sydney. They provide services to a particular subgroup of the general population. For example, four of the sites serve culturally diverse populations with an over-representation of people from low socioeconomic backgrounds. And many potential participants were not familiar with apps or technology. One physiotherapist stated:

…you do get a lot of people… who aren’t going to have phones or tablets … – physiotherapist 1

In addition, the median (IQR) age of participants was 53 (41–66) and most had chronic conditions (see online supplemental table 7). In all, those recruited to the trial may not be typical of those who seek physiotherapy for musculoskeletal conditions in the wider community.

Effectiveness

The primary results of the REFORM trial indicated that remotely delivered physiotherapy was as good as face-to-face physiotherapy.⁹ Importantly, there were no serious adverse events reported, and the number of adverse events reported was similar between the two groups at weeks 6 and 26 (see online supplemental table 8).

Adoption

The adoption of the REFORM intervention by the sites was somewhat reflected by the number of participants recruited at each site. For example, the number of participants recruited at each site was highly variable, with one site recruiting 17 participants while another site recruited 71 participants. This most likely reflects the types of patients that these hospitals managed. However, it may also reflect how well the staff of the hospital that recruited 71 participants embraced both the REFORM intervention and the trial. Differences in staff attitudes to the REFORM intervention are reflected in the below three quotes from physiotherapists at recruiting sites. The first quote suggests strong support, the second indicates possible future support if systems are in place and the third expresses reluctance to embrace the intervention.

It was nice to see some of the chronic patients being treated in a way…that was appropriate for them … through evidence-based and minimal hands-on treatment and all that. – physiotherapist 5

I think we’re very evidence-based and [we] make sure [our treatments are] best and patient centred. I think if we know that it works, with the evidence, …and know who we can deliver it to effectively and things like that. I think if we overcome the barriers like we have appropriate technology based on the patient and therapist, and good systems and programs and so on. I think most of these things are still kind of in their infancy so once we get a more mature system in a few years’ time, like a lot of barriers won’t be issues. – physiotherapist 1

… some of our clients…particularly some of the older ones who are not as confident in their abilities to use technology, just [like] that sort of good old-fashioned…they like to come in and actually have a face-to-face conversation. That’s the therapeutic relationship kind of thing. – physiotherapist 10

Participants likewise had varying opinions about the intervention which would influence the success of any future rollout. For example, one participant clearly valued the intervention, stating:

I [would] recommend my friends see her [if they have] the same issue … I am really thankful for [the trial physiotherapist]. She … gave me step by step exercise, without cortisone injection, without any surgery. Even the specialist, … was surprised as well [by] the improvement which I had… – participant 5

However, some clearly described their need for regular face-to-face contact with a physiotherapist to maintain motivation to practise their exercises. This quote reflects this belief:

[With face-to-face physiotherapy] that rapport builds up, plus the motivation to keep you going. You know, you can hide at home and not do the exercises correctly and not really see any gains. But I think going to the face-to-face … sort of give[s] yourself that little bit of added pressure to do well… – participant 7

The adoption of the REFORM intervention by the participants is also reflected by the exercise adherence data. We used two data sources to gauge whether patients in the remotely delivered physiotherapy group adopted and adhered to their home exercise programmes. The data from these two sources gave contradictory information. For example, the data that participants recorded on the app indicated that they only practised two or more exercises on a median (IQR) of 3 (0–5) days per week during the first week. This dropped to 1 (0–4) day per week by the second week and to 0 (0–2) days per week by the 6th week (see online supplemental table 9). Yet the data attained during each phone call at week 2, 4 and 6 indicated a much higher rate of adherence. Participants were asked if they had practised their exercises as instructed and to estimate how long they had spent doing this (they were instructed to practise their exercises each day). 86% of participants stated at week 2 that they had practised their exercises each day, spending a median (IQR) of 20 (15–30) min per day. This remained largely unchanged over the 6 weeks (see online supplemental table 10). Given the limitations of self-report and the contradictory information, it is not clear which source of data is more accurate and whether participants did regularly exercise at home.

Implementation

Information about how well the intervention was applied (ie, fidelity) in the context of the trial provides insights into how feasible it will be to roll out the intervention in the future. The fidelity of the REFORM intervention was assessed by analysing several data sources (see table 2). Our data sources showed that the trial intervention was delivered as intended. For example, the participants largely received phone calls at week 2 and 4, and text messages each week (see above and online supplemental table 11), in which they received advice, encouragement and support. In addition, they all received a home exercise programme via the app. The home exercise programme consisted of a median (IQR) of 12 (9–14) exercises with 55% of the exercises aimed at increasing strength and 17% directed at improving function (see online supplemental table 12). It was relatively easy to ensure that the intervention was administered as intended because only one physiotherapist (ie, the trial physiotherapist) was responsible for delivering the intervention to participants in the remotely delivered physiotherapy group. This physiotherapist had 15 years of clinical experience in musculoskeletal physiotherapy and was familiar with the website and the app. This helped ensure that the intervention was delivered as intended.

Maintenance

Findings from three data sources (interviews, satisfaction with service delivery and results from the economic analysis) suggest that the REFORM trial intervention could be rolled out and institutionalised into routine practice. Most importantly, the economic analysis indicated that the intervention was cost effective. This is likely to be a major driver for health funders (see table 2). Despite this, the successful roll out will also depend on the support and buy-in of patients, physiotherapists and stakeholders. The findings from our interviews indicate mixed support, with the main concerns coming from patients and physiotherapists. For example, some patients expressed a preference for the social contact associated with face-to-face physiotherapy. A comment was

Quite honestly, I’m a bit old fashioned. I prefer face-to-face. Yeah. Like, I don’t like doing things online and on the phone. – participant 8

The concerns expressed by the physiotherapists were varied. For example, one physiotherapist stated:

…making sure that the clinician is comfortable using it and understands…what wording to use to generate an exercise etc. Because that could be so much more time consuming. – physiotherapist 12

Others were concerned about the abilities of junior physiotherapists to largely manage patients over the telephone. For example, one physiotherapist stated:

We found that new staff [and]… junior staff struggled with the concept…more so being able to clinically reason and I think the confidence of actually having patients in front of them gives them a bit of reassurance…if you have that clinical knowledge and expertise behind you then you have more of the confidence. – physiotherapist 11

A number of the physiotherapists voiced concerns about this model of care being imposed on them or being the only option. For instance:

I would be very worried if, for example, it was mandated that we only ever treated everyone forever remotely. I’d be very worried about lots of our patients slipping through the cracks in that context. I just think that our patient groups, they always come with their own individual sort of preferences and complications. – physiotherapist 10

Similarly, others articulated the desire to maintain flexibility in the way they delivered physiotherapy. For example:

Moving forward we’re hoping to offer a flexible service delivery model, whereby we have all of these means and we can treat people by any of these means and as long as it’s not contraindicated, or we feel like there’s risk, we will go ahead with that. And rather than sort of saying that everyone needs a face-to-face appointment, it will be based on the patient and the presentation and their preferences. – physiotherapist 12

However, many patients and physiotherapists expressed support for remotely delivered physiotherapy primarily on the basis of its convenience and ability to empower patients (as captured in the previous sections), as well as its potential to reduce waiting times and increase access for all to physiotherapy. This was also reflected in participants’ rating of satisfaction with remotely delivered physiotherapy at 6 weeks. The median (IQR) score was 8/10 (5–10) points (see online supplemental table 7). It was perceived to be particularly beneficial for those from rural and isolated parts of the country. For example:

Certainly, for people in rural or more remote areas, obviously there is a lot of benefit there in terms of people not having to travel long distances. Deal[ing] with parking also is what people find challenging. Those would be the instances where I think the benefits lie. – physiotherapist 10

An important additional consideration to the future rollout is whether public hospitals have the resources to provide remotely delivered physiotherapy such as staffs’ skills to provide this model of care, and whether the patients have access to mobile devices and the internet. This was expressed by a number of physiotherapists and participants (see table 2).

The findings from this process evaluation using the Realist Evaluation and the RE-AIM Framework identified a number of facilitators and barriers to rolling out this model of care in the real-world clinical setting. These are presented in table 3.

Table 3. Key barriers and facilitators to the future rollout of the remotely delivered physiotherapy model of care identified from REFORM process evaluation.

Roles	Barriers	Facilitators
Patients	Do not have smart devices or access to the internet. Unable to use smart devices. Patients’ preferences for regular face-to-face contact with physiotherapists (eg, enjoy the social aspect or find it more motivating). Patients’ beliefs about the need for hands-on physiotherapy. Patients’ beliefs about the need for physiotherapists to regularly check how they perform their exercises.	Convenience of remotely delivered physiotherapy (eg, can exercise in your own time; less time and cost associated with travel). Access to the easy-to-use app. Patients’ beliefs that regular phone calls are a way of maintaining contact with their physiotherapists. Patients’ beliefs that this model of care increased their sense of empowerment.
Clinicians	Physiotherapists’ beliefs that patients prefer hands-on therapy. Physiotherapists’ beliefs that hands-on physiotherapy is important for some patients. Concerns from physiotherapists that red flags might be missed with only one face-to-face physiotherapy session and they may miss serious underlying pathology. Physiotherapists’ beliefs that they need to regularly check that patients are performing their exercises correctly. Senior physiotherapists’ beliefs that junior physiotherapists are not sufficiently experienced to be able to assess and then provide a home exercise programme in one initial face-to-face physiotherapy session. Concerns that this model of care is not suitable for all.	Having the option to provide a hybrid model of care and/or reinstating face-to-face physiotherapy sessions if necessary. Evidence that some patients attain similar outcomes with this model of care as face-to-face physiotherapy. Having free and ready access to the exercise app. Physiotherapists’ beliefs that this model of care empowers some patients. Physiotherapists’ beliefs that this model of care will increase access to physiotherapy for all. Access to training and support to provide remotely delivered physiotherapy. Physiotherapists’ understanding of the types of patients suitable for this model of care.
Healthcare systems	Need for flexibility around deciding who is appropriate for this model of care (eg, results of the REFORM trial cannot be used to assume this model of care is appropriate for those with acute conditions or those presenting to public hospitals that service a different demographic than those used in the REFORM trial). Resources and structures in place to move patients from face-to-face physiotherapy to this model of care (eg, education of administrative staff to enable them to book telephone calls; appropriate areas and times for physiotherapists to make telephone calls; systems to support the sending of automated text messages).	Evidence that remotely delivered physiotherapy is as effective and more cost efficient than face-to-face physiotherapy. Evidence that remotely delivered physiotherapy can reduce waiting times for physiotherapy. Evidence that physiotherapists are supportive (or at least not opposed) to this model of care.

REFORM, REhabilitation FOR Musculoskeletal conditions.

Discussions and conclusions

This process evaluation set out to explain the trial results and identify the barriers and facilitators to the future rollout of the REFORM intervention.¹¹ It was found that some patients valued the convenience and accessibility of the remotely delivered physiotherapy. For funders and healthcare systems, this model of care has the potential to improve efficiency of delivering physiotherapy.²⁷ Some participants reported feeling more motivated and empowered with this model of care. They felt one initial face-to-face physiotherapy session with ongoing support through text messages and telephone calls was adequate. However, this was not supported by all participants and physiotherapists (see online supplemental table 1).

Findings from the RE-AIM Framework showed that Reach was limited, as evident from the screening log audit and demographic characteristics of included participants (see table 2). Only 14% of those screened were included in the trial, and 84% of those included had chronic conditions (see table 2). This is not representative of typical patients seeking physiotherapy. Effectiveness and Implementation of the trial supported the roll out of this model of care. Effectiveness was addressed with the primary and secondary outcomes of the REFORM trial. It showed that remotely delivered physiotherapy was as good as face-to-face physiotherapy.⁹ The intervention was delivered as intended and supported by several data sources (see table 2). Adoption and Maintenance showed inconsistent findings with the results from the quantitative and qualitative data contradicting each other (see table 2).

We identified a number of barriers and facilitators to the future rollout of the REFORM intervention (see table 3). The main barriers included patients’ preferences for regular face-to-face contact with physiotherapists, physiotherapists’ beliefs that patients prefer hands-on therapy, and inadequate resources and structures in the healthcare systems to move patients onto this model of care. And the main facilitators will be access to the easy-to-use app, having the option to provide a hybrid model of care and evidence that this model of care is as effective and more cost-efficient than face-to-face physiotherapy. An important next step will be co-designing implementation strategies to either address the barriers or to take advantage of the facilitators. The strategies will need to be adapted to the context and ultimately tested within a large implementation trial.

To facilitate the future rollout of remotely delivered physiotherapy, it will be important for the treating physio-therapists to know whether patients are adhering to their home exercise programmes. Exercise adherence in the REFORM trial was collected in two ways with contradictory results. One was through the app and the other was by self-report at the time of the phone calls from the trial physiotherapist. Adherence reported through the app was lower than what the participants reported when asked over the telephone. It is hard to know which source of data is more accurate, although probably true adherence lies somewhere in between. The discrepancy may be because participants did not accurately recall how often they exercised when asked during the phone calls and overestimated adherence in an effort to please the trial physiotherapist. Alternatively, participants may have underestimated how often they exercised on the app because they may not have always recorded what they did (as suggested during the interviews).¹⁰ Importantly, we do not know how often those in the face-to-face physiotherapy group exercised. However, we do know that regardless of any differences, those in the remotely delivered physiotherapy group did as well as those in the face-to-face physiotherapy group. These results suggest that either there was little difference in exercise adherence between the two groups, or if there was a difference in exercise adherence between the two groups, this did not affect outcomes.

There are two strengths of this process evaluation. First, all interviews were conducted by external researchers with experience in qualitative research and who were not involved in the REFORM trial. This reduced bias and increased credibility of the qualitative data. Second, we used many different data sources to triangulate our findings. This increased the validity of the main findings.

The main limitation of the REFORM trial as reflected in the results of this process evaluation is the possibility of poor generalisability. All participants recruited were screened from five large public hospitals in Sydney, Australia, with four of the sites providing physiotherapy to a culturally diverse population. Reasons for exclusions included language barriers and lack of access to smart devices. In addition, most participants had chronic musculoskeletal conditions (see online supplemental table 7). These factors together suggest that the participants enrolled in the REFORM trial did not represent typical patients seeking physiotherapy for musculoskeletal conditions. It cannot be assumed that a more diverse group of people would respond in the same way. The effect of this model of care is unknown for people with more acute musculoskeletal conditions, people with non-English speaking backgrounds or people attending private physiotherapy clinics. The generalisability of the results is also limited because the face-to-face physiotherapy may not reflect usual care across all hospitals and services. For example, participants in the face-to-face physiotherapy group only received a median of three sessions over 6 weeks. This clearly reflects usual care in the five included hospitals, but this may not be usual care elsewhere. It is not clear whether the physiotherapists thought that three sessions was adequate for their patients or whether the physiotherapists were merely responding to the high demands and long waiting times for physiotherapy services across the five hospitals.⁹ Regardless, it is not known whether the results of the REFORM trial would be similar if usual care consisted of more than three face-to-face sessions of physiotherapy.

Another limitation of this process evaluation is that our method of sampling for the interview may have led to bias. That is, we may have only interviewed those with positive experiences and attitudes to the intervention. We tried to guard against this possibility by ensuring we randomly selected an equal balance of those who had indicated good and low satisfaction with the intervention (see online supplemental file). However, despite our best efforts, only one participant with a low satisfaction score agreed to be interviewed. Those that did not agree to be interviewed may have provided a different perspective.

This trial was conducted during the COVID-19 pandemic. During the pandemic, public hospitals’ outpatient departments provided limited face-to-face physiotherapy services. Instead, outpatient services were delivered mostly through telehealth. Therefore, it is possible that participants’ and healthcare professionals’ attitudes and acceptance of the REFORM intervention changed with increasing exposure to telehealth services.

The findings of this process evaluation help explain the REFORM trial results and identify facilitators and barriers for the future rollout of this model of care. Care will need to be taken before generalising the results of this trial to all patients with musculoskeletal conditions. However, even if the roll out is restricted to patients similar to those studied in the REFORM trial, work will need to be done to address some of the concerns about this model of care expressed by some physiotherapists. Patients themselves are unlikely to pose a major barrier to the future rollout, and healthcare providers are likely to welcome the potential cost saving and improved accessibility to physiotherapy for all that comes with moving the appropriate patients onto remotely delivered physiotherapy.

Reflexivity statement

Our research team consists of mainly academics and clinical physiotherapists; some with specific skills in musculoskeletal conditions and others with experience in qualitative and/or quantitative research. Most members work in public hospitals or public educational institutions. Some members of the team had developed the free exercise app which may have influenced their interpretation of data relating to this. The REFORM trial was motivated by a desire to increase access to physiotherapy for all in a sustainable and cost-effective way. Over the course of the trial, we reevaluated our initial beliefs and refined our hypothesis about the range of patients that could potentially benefit from our model of care. We set out to conduct the trial and process evaluation in a way that minimised bias and we focused on trying to interpret the findings in an impartial way. In addition, an independent consultancy company with expertise in qualitative research and programme evaluation conducted and coded all interviews. We had no direct or indirect conflicts of interest that may have inadvertently encouraged us to promote either face-to-face physiotherapy or our model of remotely delivered physiotherapy, although some clinical physiotherapists in our team may have felt that their and/or their colleagues’ employment could be threatened if this model of care replaced face-to-face physiotherapy. They may have also perceived that their assessment and treatment skills were being undervalued by investigating whether remotely delivered physiotherapy could be as good as face-to-face physiotherapy. Our team reflected on our backgrounds, beliefs and hopes for this body of research in an attempt to ensure our findings were as credible and trustworthy as possible.

Data availability statement

Data are available upon reasonable request.