Outcome measurement in speech and language therapy: a digital journey

Abstract

Background

Evidencing the impact of speech and language therapy interventions is challenging. The UK’s professional body for speech and language therapists (SLTs) is supporting a consistent approach to outcome measurement and analysis using Therapy Outcome Measures (TOMs).

Objective

To develop a digital solution for collecting TOMs data, evaluate the impact of therapeutic interventions and explore contributing factors to outcome variation across clinical areas.

Method

Agile methodology was applied to software development. Organisations were recruited to provide data. Criteria were identified to exemplify outcome variability.

Results

A digital tool was developed. 21 organisations provided data on 16 356 individuals. Improvement in at least one domain of TOMs occurred in 77.1% of instances. Data for two clinical areas exemplify the tool’s effectiveness in highlighting the impact of speech and language therapy.

Conclusion

This established outcomes data set can be used to evaluate the impact of speech and language therapy, and explore variation in outcomes.

Introduction

Speech and language therapists (SLTs), who treat individuals of all ages with a broad range of communication and swallowing needs, face the challenge of providing evidence-based practice and continually improving service delivery. Clinicians recognise the importance of quality improvement, but in practice, achieving this can be challenging due to gaps in research.¹ Furthermore, the existence of different service delivery models, including variation in the level, type and frequency of provision,² may impact on patient outcomes.^{3 4} Comparing the outcomes of individuals accessing different services and reviewing the factors which affect progress could complement the existing evidence base. Furthermore, collecting data on all individuals would place the information from randomised controlled trials (RCTs) into the broader clinical context. RCTs often exclude individuals with certain personal, demographic or multimorbid clinical attributes in order to maximise internal validity⁵ yet such attributes describe the majority of patients.

Increasingly, the value of applying ‘real-world’ data to support quality services, is recognised, with the role of information and digital technology frequently highlighted in national policies.^6–10 Nevertheless, there are numerous barriers for SLTs in the UK, including the absence of tools to support data collection and analytics. In recognition of this, the Royal College of Speech and Language Therapists (RCSLT) is undertaking a programme of work on outcome measurement, data collection and analysis with the aim of supporting SLTs with delivering quality services.

To promote a more consistent approach to outcome measurement across the profession the RCSLT undertook a review of 63 candidate measures.¹¹ Following a synthesis and Delphi consensus approach,¹² Therapy Outcome Measures for Rehabilitation Professionals (TOMs)^13–16 was identified as an appropriate existing measure that satisfied key criteria and was selected for the project.¹⁷

Aims

1. To develop a digital solution to support collection and analysis of outcomes data to evaluate the overall impact of speech and language therapy interventions.

2. To evaluate the impact of speech and language therapy for individuals presenting with the same speech, language, communication or swallowing diagnosis, but with different underlying medical conditions.

3. To evaluate the impact of speech and language therapy for individuals presenting with the same speech, language, communication or swallowing diagnosis receiving care from different services.

Methods

A basic prototype tool for data collection was developed by Different Class Solutions Ltd, who took an Agile approach¹⁸ to develop the RCSLT Online Outcome Tool (ROOT).¹⁹ Agile methodology prioritises the production and delivery of working software in consultation with end users. End users (‘pilot sites’) were recruited through the RCSLT membership. To ensure the pilot group represented the profession, applicants who were experienced in using TOMs were profiled against a range of criteria, including organisational funding, clinical speciality and geographical region. Sites were consulted and suggested changes to the initial ROOT prototype. Since the initial pilot period, the number of sites involved has expanded.

To develop, test and use the ROOT, sites provided de-personalised data on all individuals referred for speech and language therapy intervention. There were no recruitment/inclusion criteria. A privacy-friendly approach was adopted to minimise the amount of personal data collected by the ROOT, including age, gender, diagnoses and TOMs ratings at the beginning and end of an episode of care. Advice regarding ethics permission for use of this anonymised audit data for this project was sought and deemed not to be necessary. Sites were required to seek approval from their employing authorities to be involved and followed local processes about informing individuals about how information was being used.

The General Data Protection Regulation²⁰ came into force during the course of this project. The RCSLT sought support from national bodies, including the Information Commissioner’s Office,²¹ to ensure that the guidance provided for the ROOT was in line with best practice.

To demonstrate the ROOT’s utility, data collected by sites over 36 months is presented. Specific examples are provided on clinical areas where a high volume of data has been collected by multiple sites. To evaluate the outcomes of individuals presenting with the same communication or swallowing diagnosis, but with different underlying medical conditions, a clinical area was selected where it can exist due to multiple underlying aetiologies. To evaluate the outcomes of individuals presenting with the same communication or swallowing diagnosis receiving care from different services, a minimum of six completed episodes of care per site for the specified clinical area was required.

Results

Twelve speech and language therapy teams/services from across the UK volunteered to co-produce and pilot the ROOT.

The ROOT facilitates the collection and analysis of outcome data along with other information, such as International Classification of Diseases (ICD-10) codes.²² Two data collection approaches were developed. Clinicians can enter data directly into the ROOT, or, where data is collected in local electronic systems, this can be uploaded to the ROOT in bulk. Sites determined that they would need tailored reports for different purposes and audiences (eg, SLTs, commissioners, service managers, individuals accessing services and the professional body). Thus, different levels and style of analysis were developed with the ability to refine outcomes data through applying a range of filters as required.

Twenty-one services were recruited (table 1). The sites spanned the UK and were predominantly National Health Service funded. The sites contributed data for between 5 and 8780 episodes of care over the time period.

Table 1.

Summary of sites involved in the 36 month data collection period (1 June 2016 to 31 May 2019)

Site	Total episodes of care	Geographical region	Organisational funding	Adults/paediatric/both
Site A	85	Wales	NHS	Both
Site B	151	Northern Ireland	NHS	Both
Site C	5	England	NHS	Adult
Site D	74	England	NHS	Adult
Site E	16	England	Charity	Paediatric
Site F	5648	Scotland	NHS	Adult
Site G	72	England	NHS	Paediatric
Site H	14	England	Independent	Adult
Site I	506	England	NHS	Adult
Site J	6	Northern Ireland	NHS	Both
Site K	161	England	NHS	Adult
Site L	76	England	NHS	Adult
Site M	1510	England	NHS	Adult
Site N	262	Northern Ireland	NHS	Both
Site O	130	England	NHS	Paediatric
Site P	8780	Wales	NHS	Both
Site Q	25	England	Independent	Adult
Site R	11	England	NHS	Paediatric
Site S	25	Northern Ireland	NHS	Both
Site T	174	England	NHS	Both
Site U	134	England	NHS	Both

NHS, National Health Service.

Collectively, they have provided data on 16 356 individuals who have received a total of 17 865 episodes of care for a broad range of communication and swallowing needs (table 2).

Table 2.

Number of episodes of care delivered by speech and language therapists from 21 sites over 36 months (1 June 2016 to 31 May 2019)

TOMs scale	Total number of episodes of care
Dysphagia	11 065
Dysphonia	1543
Aphasia/Dysphasia	1515
Dysarthria	1156
Core scale	673
Phonological disorder	382
Child language impairment	375
Augmentative and alternative communication	336
Dysfluency	295
Learning disability - communication	258
Autistic spectrum disorder	95
Cognition	78
Tracheostomy	49
Laryngectomy	23
Other	22
TOTAL	17 865

‘Other’ includes: Dementia, head injury, neurological disorders (including progressive neurological disorders), musculoskeletal, complex and multiple difficulty, dietetic intervention for undernutrition: Paediatric, dyspraxia - developmental coordination difficulties, hearing therapy/aural rehabilitation.

TOMs, Therapy Outcome Measures.

Across an episode of care, an increase of 0.5 or more on the TOMs is a clinically significant change.¹⁵ Table 3 indicates that 22.9% of individuals did not make a clinically significant improvement on any domain of the TOMs and 26.5% improved in four domains, 16.2% improve in one domain, 18% improve in two domains, 13.8% improve in three domains and 2.6% improve in five domains, but it should be recognised that carer well-being is only recorded when appropriate. Overall, improvement in one or more domains occurred in 77.1% of episodes of care.

Table 3.

Number and percentage of episodes of care in which a clinically significant (positive) change occurred across zero to five domains of the TOMs

	Clinically significant (positive) change in TOMs domains						Total
	Zero	One	Two	Three	Four	Five
Number of episodes of care	4010	2842	3159	2412	4649	457	17 529
Proportion	22.9%	16.2%	18.0%	13.8%	26.5%	2.6%	100.0%

Data from 21 sites collected 1 June 2016 to 31 May 2019 across all TOMs scales, excluding augmentative and alternative communication, reported to one decimal place.

TOMs, Therapy Outcome Measures.

To evaluate the outcomes of individuals presenting with the same communication or swallowing diagnosis, but with different underlying medical conditions, aphasia was identified as a clinical area fulfilling the criteria. Figure 1 compares 469 people with aphasia as a consequence of stroke with 483 people with aphasia associated with other neurological incidents. It indicates a higher proportion of individuals with stroke-related aphasia make clinically significant improvements across each of the domains compared with those with non-stroke-related aphasia.

Figure 1.

Proportion of episodes of care showing clinically significant improvement, maintenance or decline between the start and end of Therapy Outcome Measures ratings for each domain (1a-1e) for 469 individuals with stroke-related aphasia (507 episodes of care) and 483 individuals with non-stroke-related aphasia (531 episodes of care). Data from nine sites collected from 1 June 2016 to 31 May 2019.

To evaluate the outcomes of individuals presenting with the same communication or swallowing diagnosis receiving care from different services, developmental language disorder (DLD) was identified as a clinical area fulfilling the criteria.

Table 4 presents data for individuals with DLD from three sites working with this client group, and the ROOT total for DLD. It indicates that the average change in TOMs domains in the ROOT total is clinically significant across all domains. However, outcomes from individual services are varied.

Table 4.

The average change in TOMs scores, across each domain (impairment, activity, participation, well-being and carer well-being)

Data source	Episodes of care	Average (median) change in TOMs domains
		Impairment	Activity	Participation	Well-being	Carer well-being
ROOT total	473	0.5*	0.5*	0.5*	0.5*	0.5*
Site 1	254	0.5*	0.5*	0.5*	0.3	0.5*
Site 2	20	0.5*	0.3	0.5*	0.5*	−1.0
Site 3	83	1.3*	1.5*	1.0*	1.0*	0.5*

Data from three sites collected from 1 June 2016 to 31 May 2019, reported to one decimal place. An increase of 0.5 or more on the TOMs is a clinically significant change and is marked with an asterisk.

RCSLT, Royal College of Speech and Language Therapists; ROOT, RCSLT Online Outcome Tool; TOMs, Therapy Outcome Measures.

Discussion

The ROOT provides UK SLTs with a means of collecting and analysing outcomes data. Developing a digital solution meeting the requirements of SLTs working across a range of clinical groups, settings and with access to different information systems has been challenging. However, co-producing the ROOT with SLTs and using an Agile approach has been supportive in developing a user-friendly, intuitive tool that can assist SLTs with utilising their outcomes data.

The expanding outcomes data set is supporting the speech and language therapy profession to evaluate its overall impact within the local context. Table 2 highlights the range of clinical areas SLTs support, indicating dysphagia outcomes are most frequently collected. However, the data set will be skewed as individual sites contributed differing amounts of data (table 1) since caseload size was not controlled for, so this is not necessarily the most prevalent disorder encountered by SLTs.

The results show that positive, clinically significant improvement in one or more domains occurs in 77.1% of episodes of care (table 3) and is most often seen in four domains of the TOMs (26.5%). This is not surprising as speech and language therapy is a profession contributing to rehabilitation and enablement, which is broader than reducing the disorder alone^{23 24} and may reflect the holistic and personalised care provided by SLTs. Interventions include providing strategies to improve communication or swallowing, enhance participation socially, educationally and in employment, along with attending to the well-being of the individual and their family. As interventions will not always target all domains of the TOMs, it is not unexpected that there is rarely change in five domains (2.4%). Table 3 also indicates that 22.9% of individuals do not show positive change on the TOMs over an episode of care. Further inspection of the data would provide information on the types of individuals who do not make improvement, including individuals with progressive conditions, for whom maintenance of function or carefully managed decline is the expected outcome.

The ROOT is beginning to provide insight into the impact of speech and language therapy for individuals with different underlying medical conditions (figure 1) and receiving care from different sites (table 4). The data presented illustrates variation in outcomes between aetiology and service delivery. We recognise this is only descriptive and indicative, and requires further investigation in order to establish significance.

Figure 1 illustrates trends in the outcomes for individuals with stroke-related aphasia compared with non-stroke-related aphasia. Across all five domains of the TOMs, a higher proportion of individuals with stroke-related aphasia show improvement post-intervention. Speculation on the reasons for this include better and more immediate access to services (such as dedicated stroke units and rehabilitation teams) and possibly a lower level of complexity of needs (such as those experienced following traumatic brain injury, for instance, cognitive impairments). Only a few small studies have made comparisons between stroke and non-stroke aphasia^{25 26} and this data provides an opportunity to complement these studies with ‘real world’ data.

Outcomes for similar clinical groups can be compared through benchmarking, which ‘as a component of Quality Management, offers a continuous process by which an organisation can measure and compare its outcomes overtime with peer organisations and use the findings to inform management decision making’.²⁷ Table 4 indicates some variation in outcomes between different sites for a given communication or swallowing diagnosis, reflected elsewhere in the literature.^{28 29} The ROOT total shows a clinically significant increase in every domain. Sites 1 and 2 are broadly in line with this average, while Site 3 achieves beyond this. Potentially, this is related to the service provision offered, different referral patterns or variation in the severity of impairment at therapy commencement. The reasons for this variation will need to be explored, which then could be considered by other services to support quality management.

This innovative and ambitious project has demonstrated the utility of ‘Big Data’³⁰ and has equipped the profession with robust data to evidence its impact and use in national influencing. Outcomes data can be valuable in looking at ‘real world’ change across a range of contexts, without applying stringent participant criteria and should be used to complement and facilitate interpretation of the existing evidence base. Nonetheless, we acknowledge there are some methodological limitations. Outcomes are not compared with control groups, nor are variables controlled for within-groups, thus, isolating potential agents of change is limited in comparison to RCTs. Yet, the data is advantageous in other respects. For example, it contains information on individuals that tend to be excluded from RCTs.

While the ROOT has a notable volume of data, wider implementation of the ROOT would increase its validity. The main barrier to implementation has been delays in approval due to uncertainty around the new data protection legislation. To support prospective sites, an Information Governance Pack was developed³¹ summarising key information about the ROOT in relation to data processing, online security and risk mitigation. As more SLT services use the ROOT, we can be more confident in assumptions about the data being representative of the range of services provided by SLTs across different clinical groups and settings.

The data in the ROOT is already beginning to demonstrate its value and case studies are emerging which document use of the ROOT to highlight the impact of speech and language therapy to senior directors, commissioners/funders and evaluate where interventions are having most impact and identify areas for improvement.^32–34 Future work should focus on application of the ROOT data to answer clinical uncertainties and for quality management. Furthermore, as the volume of data increases, it may provide insight into complex clinical questions currently unanswered by the traditional research methods.

This project illustrates that the implementation of a digital solution is about more than introducing new technology. This is part of a larger system change, and working collaboratively to identify what data to gather, how to gather it and make use of the information, along with consideration of the practicalities and minimising the barriers to implementation, is essential. We have shown that our profession can use digital innovation to collectively demonstrate the impact of speech and language therapy, with the potential to inform the way we deliver care and improve outcomes.