Why do common sense trials fail in the UK? Lessons learned from a trial which tested the effectiveness and cost-effectiveness of a community falls prevention programme (the Firefli study)

Abstract

Background

The Firefli study was funded from a commissioned call to conduct a individually randomised controlled trial to assess the effectiveness and cost-effectiveness of Home Fire Safety Visits (also known as Safe and Well Visits) in their ability to reduce falls and improve quality of life in older adults living in the community. These visits are routinely carried out by fire and rescue services in England and aim to reduce risk of fire, support independent living and improve quality of life. In this paper, we reflect on our experience of attempting to deliver a definitive trial within the fire service, with the aim of informing future commissioning and methodological practice for non-National Health Service hosted trials in the UK.

Lesson learned

It proved impossible to conduct the trial as planned in the current research landscape, randomising only 63 participants from a target of 1156. Whilst there were challenges associated with the COVID-19 pandemic, it was key issues pertaining to current regulatory requirements, the acquisition of data and lack of research culture and infrastructure with the fire service which were fundamental barriers to successful research delivery. Specifically, these barriers meant it was not feasible to implement the trial as designed to reflect actual service delivery. The adapted trial design had very low recruitment and resulted in differences between the target population and the trial population.

Conclusions

Conducting trials outside of health is extremely challenging in the UK. We recommend an urgent review of research governance processes which are primarily designed for health-related research in the National Health Service and are not fit for purpose when conducting research within other sectors. Many of the challenges identified are not exclusive to delivering trials in the fire service and have wider implications as the scope for evidence-based practice expands outside of health.

Trial registration

ClinicalTrials.gov: NCT 04717258.

Background

In 2018 the National Institute of Health and Care Research (NIHR) Public Health Research Programme [1] launched a commissioned call (reference 18/70)—Fire and Rescue Visits to Improve Health Outcomes. The call specified ‘research on the effectiveness and cost-effectiveness of interventions that have been, or have the potential to be delivered in the UK by Fire and Rescue Services to improve health outcomes’. Included in the list of examples of interventions were ‘Safe and Well visits within the homes of the most vulnerable people in a community’ and ‘Falls prevention’. An overarching theme of the call was reducing health inequalities.

Fire and rescue services (FRS) in England routinely carry out Home Fire Safety Visits (HFSV) to people’s homes to reduce the risk of fire and advise on actions to be taken in the event of a fire. Over recent years, these visits have expanded to include health-related topics, such as: falls prevention; smoking cessation; social isolation; and winter warmth, and have been called Safe and Well Visits. The aim is to reduce risk of fire, support independent living and help prevent avoidable hospital admissions and excess winter deaths. The FRS carry out approximately 670,000 home safety visits in England each year [2]. There is some evidence to indicate that HFSV are effective: the Winter Pressures Pilot service evaluation commissioned by Public Health England and the Chief Fire Officers’ Association, which was observational, found that visits were effective in addressing falls, cold homes and social isolation, but less effective at influencing the uptake of flu vaccinations [3]. However, given that the evidence for HFSV was limited to observational studies and yet were rolled out almost universally across FRS in England, we felt effectiveness and cost-effectiveness evidence on this intervention was an unanswered, important and practise relevant question.

The Firefli study was designed as an individually randomised, pragmatic, multicentre, two arm, open randomised controlled trial (RCT) with embedded qualitative and economic evaluations. Full details of the planned study were published as a protocol [4], and key information is summarised in Table 1. The trial was designed to mirror the existing delivery of the HFSV service as closely as possible. The FRS routinely approaches members of the public to offer a HFSV by mail—targeting specific households at greatest risk based on General Practice (GP) registration data. Therefore, the service is intended for largely underserved populations. We designed the study to closely align with this process, whereby as members of the community were contacted by the FRS, we would attempt to recruit into the trial and households would be randomised to obtain a HFSV as soon as possible (the intervention arm) or 12 months later (the control arm). The NIHR brief specified that the primary outcome must be health related and falls prevention was specifically earmarked, therefore, we selected a measure of falls as our primary outcome alongside a generic quality of life measure as a co-primary. As the outcome of most interest to FRS relates to fire prevention, we included this as a secondary outcome. Consent included trial participation and the collection of baselines and follow-up data, with primary outcome measure taken over 12 months from-randomisation.

Table 1.

Summary of original Firefli trial protocol

Methods	Details
Objectives	To evaluate the effectiveness and cost-effectiveness of the HFSV to reduce falls and improve quality of life in older adults living in the community
Design	A multicentre, randomised controlled trial with economic and qualitative evaluations, involving two fire and rescue services in England, to recruit 1156 community dwelling adults aged 65 years and over, randomised 1:1 to intervention versus usual care
Intervention	The intervention group were offered a HFSV usually within 3 weeks of randomisation, the HFSV were delivered by either a firefighter, day duty safety advocate or Safe and Well Officer and took around 45–60 min
Outcome measures	We had co-primary outcomes of falls (the number of self-reported falls per participant over the 12 months from randomisation) and health-related quality of life measured by the EQ-5D-5L over the 12 months from randomisation
FRS collaboration	A collaborative approach was adopted with members of the FRS as co-applicants on the study. This included a data analyst, a Customer Experience Strategic Lead/Customer Engagement and Safety Officer, and a Prevention Inclusion Manager
Nested qualitative study	Experience of managing/delivery of the HFSV was explored through interviews with 17 service providers (6 firefighters, 11 advocates) and 11 service leaders 15 trial participants were interviewed to discuss acceptability and experience of the intervention
Additional qualitative interviews	At the end of the trial period we conducted an additional 7 interviews with key members of the trial team to gather information and reflections on the trial delivery process

It was impossible to conduct the trial as planned in the current research landscape, randomising only 63 participants from a target of 1156 across the period of the study. To our knowledge this was the first large NIHR-funded evaluation of a fire service intervention; therefore, it was a new context for research. Whilst trials and evidence-based practice are well established and have long-standing infrastructure in health, this is not replicated in other sectors. Successful completion of trials in contexts including the police service and social care have been noted to have additional and different threats [5, 6] to those encountered in health. The aim of this commentary is to outline the challenges encountered in attempting to deliver a RCT of an existing service, which had already been implemented widely across England that was designed to provide key information to FRS on the value of the HSWV. It is important to alert the research community to the difficulties that can be encountered in the delivery of such ‘common sense’ studies through this example, but which have broader generalisability across other contexts as evidence-based practice expands into more sectors outside of health. Here we draw on our trial process documentation, data from our trial process evaluation and additional interviews we conducted with key members from the trial delivery team.

Challenges in delivering the research

Delivery of the Firefli trial took place against the backdrop of the COVID-19 pandemic, which severely impacted the study. The grant commenced just one month prior to the first UK COVID-19 lockdown (March 2020). Our study set up coincided with a period of unprecedented demand on research sponsors and approvals committees, whose remit was amended to prioritise studies relating to COVID-19. This caused delays (approximately five months) in obtaining the initial approvals necessary for our study and significantly reduced the time we had available to deliver the trial. However, we do not believe this to be the key contributing factor for the failure of the trial. The challenges we set out below are independent of those brought about by COVID-19 and need to be understood by researchers planning similar research.

Regulatory approval

We originally planned to identify potential participants from GP registration data, as this was the dataset routinely used by the FRS to offer households HFSV. Firefli was designed as a pragmatic trial, to reflect normal practice as much as possible. The FRS obtain GP registration data on address, sex and year of birth of people living within the areas they cover from NHS England and we planned to recruit using this data. However, the data sharing agreement between the FRS and NHS England did not allow these data to be used for research purposes and NHS England were unwilling to change this, as this is a national level agreement with all FRSs in England. This resulted in an unforeseen request for GP registration data having to be completed by the FRS via a Primary Care Registration Management Data Extract Authority request. This is a lengthy and complex process which requires ethics, Confidentiality Advisory Group [7] and Health Research Authority [8] approvals to be in place. Neither the research team, nor the fire service collaborators were aware in advance that the data sharing for research purposes would be such a major issue. As future researchers are likely to encounter this issue we believe a change in the required bureaucracy in circumstances such as this, whereby the research is attempting to embed into routine service provision practices is required. Using the same data and processes in the research as is utilised on a daily basis by the FRS would result in the most scientifically rigorous evaluation and would be a timely and cost-efficient way to provide this important evidence to the service provider.

Whilst the Confidentiality Advisory Group application is generally complex and time consuming this process was further exacerbated within an organisation not aware or accustomed to these processes. In this instance this application was particularly burdensome, with regard to the submission of the Data Security Protection Toolkit (DSPT) application. Any organisation processing patient information under a Confidentiality Advisory Group application needs to have their DSPT self-assessment submission reviewed by NHS England, to provide assurances that the organisation had achieved the appropriate standards. Remembering the FRS use this data routinely for HFSV provision, however for such purposes they are not required to submit a DSPT application. For one of the FRS participating in the study, service demands and staffing issues meant this aspect was difficult to deliver. For example, to meet DSPT requirements, a high proportion of all staff in the service are required to have research governance training every year. This frequency of training is not a normal requirement of FRS staff and therefore accreditation had to wait until training had been refreshed for the entire organisation, even though only a small number of staff were involved directly in the trial or would have access to the NHS data. This unanticipated requirement was burdensome and caused delays, which manifested in disillusionment from the research collaborators in the FRS. Unless participation in research is attractive to potential partners and does not become an additional drain on their resources, future collaborations are in jeopardy. We call for a more flexible approach to the governance requirements, which should be determined by the circumstances of the particular study. This would prevent research teams jumping through nonsensical hoops to use data that is already routinely in use by the FRS. In this case, the Confidentiality Advisory Group requested a change to the proposed data flow, which required an ethical amendment to the approved protocol, causing further delays.

Whilst NIHR are committed to increasing research capacity outside of the NHS, this study within the FRS made it clear that collaborators who lack a research culture and the formal infrastructure associated with that, alongside complex regulatory requirements, imposed severe limits on the ability to deliver research in this space. This circular, fragmented series of regulatory requirements seriously inhibited the progress and ultimately made, what could have been, an efficient, pragmatic ‘common-sense’ trial impossible to deliver. There is a clear irony—that unfortunately those who regularly carry out health research have become accustomed to—pertaining to the inability to easily provide robust evaluative information on non-evidenced based services that have unquestioned widespread implementation. The way bureaucratic blockers do not permit the same data to be used in the same way for service delivery and research purposes is unfathomable to new research partners outside of the NHS and patient and public contributors alike.

Capacity in the system outside of the control of those funding and delivering research

The ability to deliver NIHR funded studies relies on several other external organisations, who are not answerable to the NIHR. On this occasion, despite all of the regulatory approvals being put in place (Health Research Authority and Care Research Wales ethical approval (Health Research Authority Research Ethics Committee, Confidentiality Advisory Group approval and Data Release Authorisation Board approval), due to capacity issues within NHS England the FRS still did not receive the requested data before the end of the trial. Capacity issues within NHS England are unpredictable and outside of the control of the research team and NIHR, however, this should be considered a risk to research delivery by NIHR.

As the GP registration data was never obtained, an alternative method of contacting potential trial participants had to be found, outside of the system used for routine service delivery of HFSV. As an adjunct to their usual practice of contacting households via GP registration data, FRS use Consumer Classification Platform (CCP) data as part of this risk profiling. It was decided to use this data as the best alternative to GP registration data to mailshot potential trial participants. However, further permissions and adjustments to the regulatory approvals had to be made, including agreement from the data supplier for use for research purposes, agreement from the Sponsor, the University of York Data Protection Officer, the Trial Steering Committee and NIHR as funder. Following agreement, a protocol amendment to use CCP data was submitted and approved by Health Research Authority research ethics.

As already alluded to, the additional burden on research collaborators in the FRS meant senior staff in these organisations became increasingly frustrated by the seemingly ridiculous nature of the demands of the trial research process. Even the provision of additional resources to the FRS would not have meant these demands were any more easily met as any funds available would not have easily resolved a general staffing issue. Such organisations are often stretched due to staffing and/or demand that can inhibit the research process, especially where an evidence-based practice and research culture is lacking. There was an increasing sense that the findings from the research would not be worth the additional demands, especially for a service that was already successfully operationalised.

Equality, diversity and inclusion

The HFSV are intended to target those households at greatest risk of fire, because of which deprived communities are over-represented. Therefore, the trial was attempting to recruit from underserved populations. Due to issues outlined above which resulted in GP registration data not being available, participants were recruited by the FRS mailing out postal recruitment packs that could only be addressed to ‘The Occupier’ rather than personally addressed to potential recruits from the FRS. Recruitment packs contained an invitation letter, study information, consent form, screening questionnaire and pre-paid return envelopes. Those interested in taking part were asked to return study documentation to the University of York, and consenting participants were assessed for eligibility. Eligible participants were then sent a baseline questionnaire and pack of falls calendars. Participants had to return the baseline questionnaire and at least one falls calendar before they were randomised to demonstrate engagement with this method of data collection, as this was used to collect co-primary outcome data of falls post-randomisation.

The overall randomisation rate, from the total number of full recruitment packs sent out (n = 5118), was 1.2%. This figure may be a useful guide for recruitment of underserved populations using non-personalised ‘cold calling’ by mail and should be factored into study planning. Steps taken within the study team, that were feasible within a short time-frame and limited budget (see Table 2), to attempt to mitigate this low randomisation rate were not effective. Therefore, this is likely to be an accurate predicted response to approaches of this kind. This compares to an approximate 8% response rate the fire service would normally expect to receive to an individually addressed mail shot to offer HFSV. Feedback from the FRSs suggests some potential participants found the recruitment information too lengthy and complex. However, trial governance and ethical approval are contingent on the provision of certain information, which makes it impossible to heed to PPI requests for shorter and less complex provision of information as part of the study recruitment process. This may be particularly important when attempting to recruit in areas of deprivation.

Table 2.

Factors put in place to mitigate low recruitment rate

Issue	Mitigation
Mailshot addressed to ‘The Occupier’ mistaken for junk mail or scam	Legitimising the correspondence with branding from FRS, University of York and NIHR. Envelope used to send out the invitation, was franked with details of the FRS sending out the invitation Community engagement via the FRS’ websites, Twitter, neighbourhood social media (Nextdoor for Public Services) and the University of the Third Age
Resentful demoralisation that may be randomised to usual care (wait 12 months for HFSV)	The invitation letter contained information about how to access a HFSV for those not wishing to be in the study. However, neither FRS reported a significant increase in the number of requests for HFSV outside of the trial following the mail-out of recruitment packs
Large information-heavy recruitment packs	Simplification of the study documentation and reduce the cost of mailing out recruitment packs, a shorter two-page Expression of Interest (EoI) letter

In addition to the particularly low randomisation rate, whilst all of those participating in the trial would have been eligible for a HFSV visit, it may be that those who agreed to participate in the trial were from a ‘healthier’ demographic than the average of would ordinarily be in receipt of the service. Participants from the qualitative research often reported possessing characteristics that made the visit less appropriate to them, such as being independent, mobile, and well supported in terms of home safety. When asked about the utility of the visit, some participants discussed hoping that their participation in the trial is helpful, rather than any potential utility from the visit. This was exacerbated in this study as the fire service insisted that those who were categorised as at greatest risk of fire were excluded from the mailshot as it was deemed unethical to potentially withhold a HFSV visit—from the FRS perspective, the primary aim of HFSV were to prevent fires with health outcomes secondary to that.

Discussion

HSFV are an integral part of the fire service provision across the country, with an associated cost. However, there is no certainty as to whether this intervention is positively impacting on health outcomes for individuals in receipt of these visits. The Firefli study was funded in response to a commissioned call to evaluate the effectiveness and cost-effectiveness of the HFSV to reduce falls and improve quality of life in older adults living in the community. It was, therefore, deemed an important question from both FRS and public health perspectives. We designed a randomised evaluative study to slipstream the existing service delivery as closely as possible, to generate the gold-standard in effectiveness evidence. Whilst the trial on paper should have been deliverable from a common-sense perspective, it was not feasible.

We found huge challenges associated with the interface between complex regulatory processes and lack of research culture and infrastructure in FRS. This is not the first study to highlight the impact of regulation and bureaucracy on trial delivery. This issue has been discussed extensively over the previous 20 years in the clinical research space, and remains a deterrent to the delivery of clinically relevant research [9], in particular data governance—which is perceived as complex, daunting and time-consuming [10]. Whilst these issues remain a barrier in health research, where evidence-based practice is the norm, with strong infrastructure in place to support the research process, this was amplified when working with FRS. Lack of familiarity with research processes and being situated in an emerging research culture has been noted to introduce methodological and practical challenges in research across policing and social care—being described as ‘navigating uncharted territory’ [5]. It is important that these issues are subsequently addressed, as research activity, particularly trials, across a range of non-medical sectors is set to rise exponentially [11]. In October 2022, a new Academic Collaboration, Evaluation and Research Group (ACER) [12] was convened by the National Fire Chief Council to work alongside the Digital, Data and Technology (NFCC DDaT) function that aims to create better links between the FRS and academia. Whilst this signals a commitment to a shift in research culture within the organisation, the success of future trials in this area are in jeopardy as long as the bureaucratic challenges described remain static.

FIREFLI was seeking to determine the effectiveness of an intervention aimed at high-risk groups; the FRS use profiling data to identify those most at risk of fire in the community, and therefore, those eligible for receipt of HFSV have higher levels of deprivation. Reducing health inequalities and EDI were central to the commissioned call and NIHR more generally [13]. NIHR stresses that as a research community, ‘we need to learn from our successes and failures, learn from others both within and outside our sector, and learn from the wider community’. Our experience from the Firefli trial has shown us that non-individualised postal recruitment into randomised trials is unlikely to result in sufficient uptake for a study to be feasible and will likely result in very low rates of randomisation. In addition, those recruited into the trial may not reflect the population in receipt of the service i.e. those with better health and less deprived, which has been observed in other trials. Not only is there scepticism about the legitimacy of this type of contact amongst potential participants, the amount of information required in order to consent individuals into a trial may well have been off putting. It has long since been acknowledged that printed participant information tends to be long, technical and difficult to navigate [14]. Non-participation in the Firefli trial is likely to reflect a combination of known factors that act as barriers to the inclusion of underserved populations in research. From Autumn 2024, inclusive research design will become a condition of NIHR funding [15] supported by guidance including the NIHR Learn Research Inclusion Hub and the NIHR INCLUDE roadmap [16]. In addition, outputs from the increasing research interest in this topic will also support improvements, for example, the recently published STEP-UP guidance to help researchers design inclusive trials [17]. Whilst some methods to improve inclusion have been utilised, the effectiveness of these strategies has rarely been rigorously evaluated [18]. Designing research to enhance the inclusion of deprived populations will require innovative approaches over and above tokenistic gestures, but will not be possible without the availability of appropriate funding and the ability to convince communities improvement will result to stand any chance of sustained community engagement.

Conclusion

We have set out the challenges we encountered in attempting to deliver the Firefli trial—a study to evaluate whether a widely implemented programme, delivered by FRS, is achieving improvements in health-related outcomes. We had significant buy-in from colleagues with the FRS and substantial funding from NIHR. Despite this, the trial was not feasible. Having to adhere to regulatory processes designed primarily for the NHS was a major inhibiting factor and is likely to be encountered by other teams attempting to run trials in sectors outside of healthcare. This could cost the NIHR millions in undelivered research and will mean that local authorities and their service users will not benefit from high-quality evidence. We would call for essential reforms to the bureaucratic processes required in order for innovative and efficient research to be feasibly implemented in a timely manner to inform practice. There is a danger that under the current regulatory framework, a ‘legacy of failure’ may develop, the unintended consequences of which could be a decline in interest from local authority organisations to participate in research. This is especially where the perceived value and kudos associated with evidence-based practice, in particular RCTs, is already less than it is within the NHS.

Abbreviations

CCP

Consumer Classification Platform

DSPT

Data Security Protection Toolkit

FRS

Fire and rescue services

GP

General Practice

HFSV

Home Fire Safety Visits

NHS

National Health Service

NIHR

National Institute of Health and Care Research

RCT

Randomised controlled trial

Authors’ contributions

JA, AS and AD drafted the manuscript and were involved in the acquisition of funding, design and delivery of the project. SC and CF were involved in the acquisition of funding and made a substantial contribution to the design, delivery, analysis and interpretation of data in the study.

All authors read and approved the final manuscript.

Funding

The study is funded by the National Institute for Health and Care Research (NIHR) Public Health Research (PHR; project reference 128341) and Humberside and Kent Fire and Rescue Services. The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

Health Research Authority and Care Research Wales ethical approval (Health Research Authority West Midlands—Coventry and Warwickshire Research Ethics Committee (Ref 21/WM/0050), Confidentiality Advisory Group approval (Ref 21/CAG/049).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.