Skip to main content
PMC home page
UKPMC Funders Author Manuscripts logoLink to UKPMC Funders Author Manuscripts
. Author manuscript; available in PMC: 2019 May 1.
Published in final edited form as: Child Adolesc Ment Health. 2018 Jan 24;23(4):328–333. doi: 10.1111/camh.12260

Routine Outcome Monitoring in CAMHS: How Can We Enable Implementation in Practice?

S M Waldron 1,3, ME Loades 1,2, L Rogers 3
PMCID: PMC6217995  EMSID: EMS79604  PMID: 30410423

Abstract

Background

Many CAMHS teams across the UK are now required to use Routine Outcome Monitoring (ROM). However, some clinicians hold negative attitudes towards ROM and various practical implementation issues have been identified.

Method

The aim of this study was to explore clinician experiences of using ROM in the context of an ‘enforced’ initial implementation initiative. Twenty clinicians were surveyed at the beginning (T1) and end (T2) of the six-month period in a large UK CAMHS network adopting CYP-IAPT practice. Changes in the use of and attitudes towards ROM were investigated, as well as barriers to implementation.

Results

Overall, a small but significant increase in clinician use of ROM was observed from T1 to T2, but attitudes towards ROM did not change significantly. On the whole, clinicians were more positive than negative about ROM during the implementation period, but key implementation challenges included clinician concerns about the value and (mis)use of ROM data, poor technological support and additional workload demands.

Conclusions

CAMHS clinicians will not necessarily become more receptive to ROM simply as a by-product of being asked to use it more. Seeking clinician feedback at the early stages of implementation can help provide a focus for improvement efforts. Ideas for future research and important limitations of the study are discussed.

Keywords: Psychology, Questionnaires, Rating Scales, Service Development

Introduction

Many CAMHS teams in the UK are now required to use Routine Outcome Monitoring (ROM), defined here as the “detailed evaluation of the impact of treatment on areas of a client’s functioning that are of clinical relevance” (Johnston & Gowers, 2005, p.133). Indeed, the Children and Young People’s Increasing Access to Psychological Therapies transformation (CYP-IAPT), which is a UK systems-level approach to widening access to psychological therapies and mental health treatment, includes ROM as a mandatory element of service provision (Law & Wolpert, 2014). Some of the service user-reported measures within CYP-IAPT focus on symptoms and functioning (e.g., the Revised Child Anxiety and Depression Scale), whilst others focus on goal setting/monitoring (e.g., the Goal Progress Outcome form) and therapeutic alliance (e.g., the Session Rating Scale). A commonly used CYP-IAPT clinician-reported measure is the Current View (Jones et al., 2013), where the clinician estimates change in symptoms and functioning over time.

The rationale for using ROM comes predominantly from adult mental health settings in which a number of randomised controlled trials have demonstrated better mental health outcomes when clinicians are provided with regular service-user feedback (see Gondek, Edbrooke-Childs, Fink, Deighton, & Wolpert, 2016 for a recent review). Researchers have drawn on Feedback Intervention Theory (Kluger & DeNisi, 1996) to explain these results, arguing that clinicians adapt their intervention according to the feedback they receive from their service users (de Jong, 2016). Current UK health policy context, as with many other countries, sees ROM as important for individual care planning and also as valuable at a service level to guide service evaluation and deploy resources as efficiently and effectively as possible (DoH, 2011).

However, various studies have highlighted clinician concerns regarding the value of ROM, as well as practical (e.g. time, cost) and philosophical (e.g. privacy, fear and mistrust) challenges to implementation (Boswell, Kraus, Miller & Lambert, 2015). In the context of child and adolescent mental health, Johnston and Gowers (2005) have reported that staff resistance to ROM is a key barrier. Indeed, Hall et al. (2013) have pointed to an array of clinician concerns, including the time it takes to administer ROM, perceptions of ‘irrelevance’ and concerns with the idea of labeling mental health problems in line with a medical model. This is problematic because stakeholder ‘buy-in’ is key to systemic change (Iles & Sutherland, 2001). For example, when mental health clinicians do not feel that ROM adds value to their practice, implementation tends to be low (de Jong, van Sluis, Nugter, Heiser, & Spinhoven, 2012; Gleacher et al., 2016).

Given these concerns, research is needed to investigate the impact of clinician attitudes and practical issues on the use of ROM in CAMHS teams starting to implement it for the first time. In particular, this needs to be investigated in the context of services being asked to use ROM as mandatory. This is a common scenario in the UK, and as such many CAMHS teams are hesitant and unsure how best to go about implementing an initiative that is unlikely to be popular with all clinicians. Therefore, the main aim of this study was to investigate clinician experiences of using ROM in a large urban CAMHS network implementing it as mandatory for the first time (after receiving CYP-IAPT funding). Although ROM implementation guidance had already been circulated to clinicians in the service, and a local ROM champion had been identified in each of the four network teams, the need for this project arose out of concerns that implementation would be problematic.

This questionnaire-based study will explore changes in the use and attitudes towards ROM, as well as barriers to implementation. The four key hypotheses to be tested were as follows:

  • Clinician use of ROM will increase during the initial implementation period

  • Clinician attitudes towards ROM will improve during the initial implementation period (as a function of using ROM more)

  • Pre-existing clinician attitudes towards ROM will affect use of ROM (e.g., those with a positive predisposition towards ROM will use ROM the most)

  • Key barriers will include practical issues (such as time and technological constraints) and clinician attitudes towards ROM (e.g., concerns about the use and/or value of ROM)

The clinician beliefs about ROM questionnaire developed by James, Elgie, Adams, Henderson and Salkovskis (2015) will be used to measure changes in attitudes towards ROM, and open-text response questions will ask clinicians to share their experiences of using ROM.

Method

Participants

Sixty clinicians across the four CAMHS teams were invited to take part in the survey. Informed consent was sought for all participants and identifiable information has been removed from this publication.

Materials, Procedure and Design

Part 1 of the survey was composed of the twelve items from the James et al. (2015) clinician beliefs about ROM questionnaire found to load onto the ‘positive’ and ‘negative’ attitude subscales. Each item is presented in the form of a statement (e.g., “ROM wastes time in session”) and the respondent is required to use a 5-point Likert scale to record the extent to which they endorse that statement (1 = not at all, 3 = somewhat, 5 = totally). James et al (2015) reported a Cronbach’s alpha of .89 for the negative subscale and .91 for the positive subscale, suggesting good internal consistency. Part 2 of the survey asked clinicians to rate the extent to which they were currently using ROM on a 5-point Likert scale (1 = never, 3 = sometimes, 5 = almost always) and also provide open-text answers to questions about their experiences of using ROM. The same survey was administered at the beginning (T1) and end (T2) of the initial six-month implementation period (with individual clinician responses being matched across the two time points).

Ethical Approval

Ethical approval was granted from the University of Bath Psychology Department ethics committee.

Data Analysis

In order to assess changes in clinician use of and attitudes towards ROM from T1 to T2, Wilcoxon signed rank tests were planned for the individual Likert items (because this should be treated as non-parametric ordinal data) and paired t-tests were planned for the summed subscale scores (because combining individual Likert items creates interval data which should be treated as parametric). In order to assess whether pre-existing clinician use and attitudes towards ROM affected changes in use and attitudes towards ROM, mixed ANOVA and bivariate correlations were planned for. In order to assess barriers to ROM implementation, the open-text data provided by clinicians in response to questions about their experiences of using ROM will be investigated and coded according to themes emerging in the dataset (see further details in Results section). In doing so, the principles of inductive thematic analysis as described by Braun and Clarke (2006) were broadly employed.

Results

Sample Demographics

Twenty clinicians took part in the survey at T1 (30% response rate) and 19 re-participated at T2 (six-months later). This sample included 11 Clinical Psychologists (55% of total sample)1, two Family Therapists, two CAMHS Nurses, two Psychiatrists, one Psychotherapist, one Occupational Therapist and one Primary Mental Health Specialist. The mean age of the clinician survey respondents was 49.1 years (SD = 10.6, range: 30-65), the mean number of years working in CAMHS was 13.8 (SD = 7.6, range: 2-35), 75% of the sample were female and 40% had received some form of CYP-IAPT training.

Use of ROM

In terms of the overall use of ROM, the average 5-point Likert scale scores (1 = never, 3 = sometimes, 5 = almost always) increased from 3.4 (SD = 1.07) at T1 to 3.8 (SD = 1.01) at T2. A non-parametric Wilcoxon signed rank test2 (including continuity correction) revealed this small increase to be significant, Z = 1.99, p = <.05. Table 1 provides an overview of the categorical data. In terms of which ROMs where being used, 70% of the clinicians reported using the RCADS, 55% reported using the Goal Outcome Form, 55% reported using the Session Rating Scale, 20% reported using specific symptom trackers (e.g., RCADS subscales, CRIES-8), 20% reported using the Outcome Rating Scale and 20% reported using the Current View at T1. Usage of each of these measures increased at roughly the same rate from T1 to T2.

Table 1. Use of ROM at beginning (T1) and end (T2) of implementation period.

Never Rarely Sometimes Often Almost always
T1 5% 10% 35% 35% 15%
T2 0% 5% 28% 34% 33%
Open in a new tab

Attitudes towards ROM

Figure 1 provides an overview of the extent to which each of the 12 items taken from the James et al. (2015) ROM beliefs questionnaire were endorsed by the clinicians participating in this study. Cronbach’s alpha values at T1 were calculated as .93 for the full 12 items, .77 for the positive subscale and .72 for the negative subscale, demonstrating good internal consistency. Similar values were found at T2. Non-parametric Wilcoxen signed rank tests computed for each of the individual items revealed no significant changes from T1 to T2. It is worth noting however, that the largest (non-significant) change was the increased endorsement of the item “ROM is another job for clinicians to take on”. Paired parametric t-tests revealed the positive attitudes subscale score to be significantly larger than the negative attitudes subscale score at T1 (means = 3.7 vs 2.7, SDs = 1.1 vs 0.7) and at T2 (means = 3.7 vs 2.8, SDs = 1.0 vs 0.7), t(18) = 2.57, p<.05, and t(18) = 2.69, p<.05. However, neither subscale score changed significantly from T1 to T2 (p>.05), indicating that clinicians did not become more positive or negative about ROM during the implementation period.

Figure 1.

Figure 1

Open in a new tab

Average clinician endorsement of individual statements at the beginning and end of the initial ROM implementation period (error bars represent standard error)

Table 2 provides an overview of how clinicians’ attitudes towards ROM changed from T1 to T2 when broken down by the group of clinicians that reported ‘low’ initial use of ROM (Likert scores of 1, 2 or 3) and the group of clinicians that reported ‘high’ initial use of ROM (Likert scores of 4 or 5). A mixed ANOVA was conducted on these data, with time entered as the within-subject factor (T1 and T2), group entered as the between-subject factor (low initial use of ROM and high initial use of ROM), and clinician attitude towards ROM entered as the dependent variable. Neither a main effect of time, F(1,17) = 0.42, p>.05, nor a time x group interaction was found, F(1,17) = 0.55, p>.05. This suggests that initial use of ROM did not differentially effect changes in attitudes towards ROM over time.

Table 2. Mean changes in attitudes toward ROM broken down by ‘low’ and ‘high’ initial use of ROM (standard deviations in parentheses).

Low initial use of ROM (N=10) High initial use of ROM (N=9)
T1 attitude towards ROM 32.6 (6.2) 46.2 (2.0)
T2 attitude towards ROM 32.7 (7.1) 44.8 (5.1)
Open in a new tab

Table 3 provides an overview of how clinician use of ROM changed from T1 to T2 when broken down by the group of clinicians that reported negative initial attitudes towards ROM (full scale scores <40) and the group of clinicians that reported positive initial attitudes towards ROM (full scale scores >40). A mixed ANOVA was also conducted on these data, with time entered as the within-subject factor (T1 and T2), group entered as the between-subject factor (negative initial attitude of ROM and high initial attitude of ROM), and clinician use of ROM entered as the dependent variable. A main effect of time was found, F(1,17) = 4.30, p<.05, but no time x group interaction was observed, F(1,17) = 0.04, p>.05. This suggests that use of ROM increased to a similar extent for each group of clinicians, with significantly higher use of ROM found at both at T1 and T2 for the group of clinicians with positive initial attitudes towards ROM (p<.05). Finally, as expected, bivariate correlational analyses revealed that clinicians who used ROM more tended to hold more positive beliefs about ROM, and this was true at T1, r(19) =.74, p<.001, and T2, r(19) =.60, p<.05.

Table 3. Mean changes in use of ROM broken down by ‘negative’ and ‘positive’ initial attitudes towards ROM (standard deviations in parentheses).

Negative initial attitude towards ROM (N=8) Positive initial attitude towards ROM (N=11)
T1 use of ROM 2.5 (0.8) 4.1 (0.7)
T2 use of ROM 2.9 (0.6) 4.5 (0.5)
Open in a new tab

Barriers to Implementation

In addition to the quantitate data reported above, the following qualitative data add to our understanding about the barriers experienced in using ROM. Table 4 provides an overview of the themes cited most frequently by the clinicians that took part in the survey. An iterative process of identifying themes and coding passages of text was undertaken by the lead author (SW) and checked by the co-author LR. This analysis was inductive in the sense that the themes were linked to the data and not driven by theoretical interests, and it was semantic and realist in that the surface meanings of the data were used and a simple unidirectional relationship between meaning and language was assumed. These qualitative data indicate that technological support was also a key barrier to implementation of ROM, as was clinician concern about when ROM may not be appropriate and how the data gathered will be used (or ‘misused’) by commissioners.

Table 4. Key Themes Regarding Barriers to using ROM (illustrative quotes in parentheses).

Proportion of clinicians mentioning each theme
Poor IT support
“We need IT support - tablets and a better data input and sharing system” 		35%
Difficulties capturing complexity
“ROM has limited capacity to capture the therapeutic process. It is problem focused rather than holistic, and doesn't capture systemic changes or capacity to reflect/mentalise/regulate” 		35%
Lack of time
“ROM takes up time and adds to the pressure of time in sessions” 		30%
Doesn’t always feel appropriate
“ROM is not always appropriate for crisis or extended assessment” 		20%
Commissioners may misuse the data
“A concern is how the data is extrapolated to reflect a service’s performance, particularly when the goal-based outcomes are person-specific” 		20%
Feels like a top-down directive
“Lets do things that are meaningful, that clinicians feel are useful [instead]” 		15%
Open in a new tab

Discussion

Overall, a small but significant increase in clinician use of ROM was observed from the beginning to the end of the initial ‘enforced’ six-month implementation period. Importantly however, clinician attitudes towards ROM did not change significantly. Therefore, the reason for the behaviour change (increased use of ROM) was more likely to have resulted from management instruction and surveillance (the teams were told that their use of ROM would be evaluated month-by-month against specific targets). Clinicians were more positive than negative about ROM throughout the implementation period, but key implementation challenges raised included concerns about the value and (mis)use of ROM data, poor technological support and the additional workload demands. These challenges are not unique to the CAMHS network under investigation here, and are similar to the issues highlighted by previous research in this field (Boswell et al., 2015; Gleacher et al., 2016; Hall et al., 2013; James et al., 2015; Wolpert, Curtis-Tyler, & Edbrooke-Childs, 2016).

The unchanging attitudes towards ROM suggests that the CAMHS network involved in the current study, and others like it, cannot assume that clinicians will develop more positive attitudes towards ROM simply as a function of being asked to use them more. Consistent with the previous research (James et al., 2015), the current study also found that clinicians using ROM more were also generally more positive about it. However, in the current study clinicians holding negative views about ROM tended to use ROM less, whereas James et al found that negative attitudes were also higher in those using ROM more. The current finding is important because previous research has demonstrated that if mental health clinicians do not fully ‘embrace’ ROM, then the potential for positive impact can be circumvented (Gleacher et al., 2016). At the very least, the inconsistency between the studies demonstrates the need for further robust studies to shed further light on the factors that promote and enable ROM usage by clinicians.

Conceptualised Feedback Intervention Theory (Riemer, Rosof-Williams, & Bickman, 2005) states that clinicians’ belief in the value of service user feedback is key to successful implementation, and De Jong (2016) has recently suggested that changing negative clinician attitudes towards ROM may need to be a first step before implementation. Willis, Deane and Coombs (2009) have demonstrated that video training for clinicians via a workshop exploring the perspectives of service users and clinicians can help improve attitude towards ROM. Similarly, Edbrooke-Childs, Wolpert and Deighton (2016) also recently reported that their use of ‘UPROMISE’ training can improve clinician attitudes towards ROM in CAMHS. Their training, which can be delivered as a single day, or over three days, includes a focus on understanding and challenging personal barriers, understanding how ROM can be useful and meaningful, learning how to collaboratively use ROM, and developing strategies for embedding the use of ROM in practice and supervision.

However, it should be noted that the current study has found that clinicians’ holding a positive attitude to ROM is not sufficient alone to result in the high levels of completion expected of services when ROM implementation is put in place. Therefore, further research should also explore how ROM can be made to work for each professional group, how practical issues can be overcome, and how ROM data can be used positively by commissioners. Douglas, Button and Casey (2016) have emphasised that ROM needs to be integrated with “clinical values and workflow” to be effective. However, this is not easy to achieve in busy clinical environments with multiple competing demands and a variety of different professional identities and backgrounds (Powell & Davies, 2012).

Strengths and Limitations

The main strength of this study is the focus on investigating clinician use and attitudes towards ROM during an initial ‘imposed’ implementation period. However, the most significant limitation is the small and potentially biased sample; of the 60 clinicians invited to take part in the survey only 20 participated and Clinical Psychologists were over-represented. Outcome monitoring and evidence-based practice often form part of Clinical Psychology training, and so Clinical Psychologists may generally hold more positive views about ROM. It is also possible that clinicians with more positive views about ROM were more likely to take part in the survey. Also, limited attention was paid to what Gleacher et al (2016) have called ‘facilitating factors’: that is, factors that help facilitate clinician use of ROM. This study did not specifically ask clinicians what was helping them to use ROM (e.g., leadership, structural support). This might have been of interest to the wider research community.

However, notwithstanding the limitations highlighted above, it can be concluded that CAMHS teams continue to face a number of challenges implementing ROM (particularly clinician concerns about the value of ROM and practical issues). Importantly, it cannot be assumed that these barriers will fade purely as a product of being asked to use ROM more. However, evaluating clinician experiences of using ROM during an initial implementation period can help to identify specific challenges, which can then be translated into improvement ideas.

Practitioner message.

  • Routine Outcome Monitoring (ROM) in child and adolescent mental health settings is believed to be beneficial to individual care planning and treatment provision, as well as at a service-level to ensure that resources are used as efficiently and effectively as possible.

  • Evaluating clinician experiences of using ROM during an initial implementation period can help identify specific challenges, which can be translated into improvement ideas

  • Child and Adolescent mental health services cannot assume that clinician concern and practical issues associated with ROM will reduce purely as a by-product of using ROM more

  • Activities such as training may be needed to support the implementation process.

Acknowledgements

The authors would like to thank the clinicians who gave up valuable time to take part in this study, the anonymous reviewers of this manuscript for their helpful and constructive feedback and suggestions, and also Barbara Hills for helping to turn the key findings into practical action at a service level.

Study funding: This study received no specific funding. Dr Loades is funded by the National Institute for Health Research (Doctoral Research Fellowship, DRF-2016-09-021). This report is independent research. The views expressed in this publication are those of the authors(s) and not necessarily those of the NHS, The National Institute for Health Research or the Department of Health.

Footnotes

Conflicts of interest: LR is an employee of the organisation in which this research was conducted, and a member of one of the teams within the CAMHS network. SW undertook a 6 month training placement within one of the teams within the CAMHS network where this research was conducted during the study period. ML has declared no potential conflicts of interest.

1

Of all of the clinicians expected to start using ROM across the four teams, approximately 30% were Clinical Psychologists. Therefore having 55% in the survey sample indicates an over-representation.

2

In short, a Wilcoxen signed rank test calculates the differences between repeated measurements, ranks according to size of difference, multiplies negative differences by -1, sums the signed ranks, and adds a continuity correction for samples larger than n=10.

References

  1. Batty MJ, Moldavsky M, Foroushani PS, Pass S, Marriott M, Sayal K, Hollis C. Implementing routine outcome measures in child and adolescent mental health services: from present to future practice. Child and Adolescent Mental Health. 2013;18(2):82–87. doi: 10.1111/j.1475-3588.2012.00658.x. [DOI] [PubMed] [Google Scholar]
  2. Boswell JF, Kraus DR, Miller SD, Lambert MJ. Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy research. 2015;25(1):6–19. doi: 10.1080/10503307.2013.817696. [DOI] [PubMed] [Google Scholar]
  3. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative research in psychology. 2006;3(2):77–101. [Google Scholar]
  4. Clark DM. Implementing NICE guidelines for the psychological treatment of depression and anxiety disorders: the IAPT experience. International Review of Psychiatry. 2011;23(4):318–327. doi: 10.3109/09540261.2011.606803. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. de Jong K. Deriving implementation strategies for outcome monitoring feedback from theory, research and practice. Administration and Policy in Mental Health and Mental Health Services Research. 2016;43(3):292–296. doi: 10.1007/s10488-014-0589-6. [DOI] [PubMed] [Google Scholar]
  6. de Jong K, van Sluis P, Nugter MA, Heiser WJ, Spinhoven P. Understanding the differential impact of outcome monitoring: Therapist variables that moderate feedback effects in a randomized clinical trial. Psychotherapy Research. 2012;22(4):464–474. doi: 10.1080/10503307.2012.673023. [DOI] [PubMed] [Google Scholar]
  7. DoH. Talking therapies: A four year plan of action. London: Department of Health; 2011. [Google Scholar]
  8. Douglas S, Button S, Casey SE. Implementing for sustainability: Promoting use of a Measurement Feedback System for innovation and quality improvement. Administration and Policy in Mental Health and Mental Health Services Research. 2016;43(3):286–291. doi: 10.1007/s10488-014-0607-8. [DOI] [PubMed] [Google Scholar]
  9. Edbrooke-Childs J, Wolpert M, Deighton J. Using Patient Reported Outcome Measures to Improve Service Effectiveness (UPROMISE): Training clinicians to Use Outcome Measures in Child Mental Health. Administration and Policy in Mental Health and Mental Health Services Research. 2016;43(3):302–308. doi: 10.1007/s10488-014-0600-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Gleacher AA, Olin SS, Nadeem E, Pollock M, Ringle V, Bickman L, et al. Hoagwood K. Implementing a measurement feedback system in community mental health clinics: A case study of multilevel barriers and facilitators. Administration and Policy in Mental Health and Mental Health Services Research. 2016;43(3):426–440. doi: 10.1007/s10488-015-0642-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Gondek D, Edbrooke-Childs J, Fink E, Deighton J, Wolpert M. Feedback from outcome measures and treatment effectiveness, treatment efficiency, and collaborative practice: A systematic review. Administration and Policy in Mental Health and Mental Health Services Research. 2016;43(3):325–343. doi: 10.1007/s10488-015-0710-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Hall CL, Moldavsky M, Taylor J, Sayal K, Marriott M, Batty MJ, et al. Hollis C. Implementation of routine outcome measurement in child and adolescent mental health services in the United Kingdom: a critical perspective. Eur Child Adolesc Psychiatry. 2013 doi: 10.1007/s00787-013-0454-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Iles V, Sutherland K. Organisational change: A review for health care managers, professionals and researchers. London, UK: Service Delivery and Organisation Research and Development Programme; 2001. [Google Scholar]
  14. James K, Elgie S, Adams J, Henderson T, Salkovskis P. Session-by-session outcome monitoring in CAMHS: clinicians’ beliefs. The Cognitive Behaviour Therapist. 2015;8:e26. [Google Scholar]
  15. Johnston C, Gowers S. Routine outcome measurement: a survey of UK child and adolescent mental health services. Child and Adolescent Mental Health. 2005;10(3):133–139. doi: 10.1111/j.1475-3588.2005.00357.x. [DOI] [PubMed] [Google Scholar]
  16. Jones M, Hopkins K, Kyrke-Smithm R, Davies R, Vostanis P, Wolpert M. Current View Tool Completion Guide. London: CAMHS Press; 2013. [Google Scholar]
  17. Kluger AN, DeNisi A. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin. 1996;119(2):254. [Google Scholar]
  18. Law D, Wolpert M. Guide to using outcomes and feedback tools with children, young people and families. London, UK: CORC; 2014. [Google Scholar]
  19. Mellor-Clark J, Cross S, Macdonald J, Skjulsvik T. Leading horses to water: Lessons from a decade of helping psychological therapy services use routine outcome measurement to improve practice. Administration and Policy in Mental Health and Mental Health Services Research. 2016;43(3):279–285. doi: 10.1007/s10488-014-0587-8. [DOI] [PubMed] [Google Scholar]
  20. Moran P, Kelesidi K, Guglani S, Davidson S, Ford T. What do parents and carers think about routine outcome measures and their use? A focus group study of CAMHS attenders. Clinical Child Psychology and Psychiatry. 2011 doi: 10.1177/1359104510391859. 1359104510391859. [DOI] [PubMed] [Google Scholar]
  21. Powell AE, Davies HT. The struggle to improve patient care in the face of professional boundaries. Social science & medicine. 2012;75(5):807–814. doi: 10.1016/j.socscimed.2012.03.049. [DOI] [PubMed] [Google Scholar]
  22. Riemer M, Rosof-Williams J, Bickman L. Theories related to changing clinician practice. Child and Adolescent Psychiatric Clinics of North America. 2005;14(2):241–254. doi: 10.1016/j.chc.2004.05.002. [DOI] [PubMed] [Google Scholar]
  23. Stasiak K, Parkin A, Seymour F, Lambie I, Crengle S, Pasene-Mizziebo E, Merry S. Measuring outcome in child and adolescent mental health services: Consumers’ views of measures. Clinical child psychology and psychiatry. 2013;18(4):519–535. doi: 10.1177/1359104512460860. [DOI] [PubMed] [Google Scholar]
  24. Thew GR, Fountain L, Salkovskis PM. Service user and clinician perspectives on the use of outcome measures in psychological therapy. The Cognitive Behaviour Therapist. 2015;8:e23. [Google Scholar]
  25. Unsworth G, Cowie H, Green A. Therapists’ and clients’ perceptions of routine outcome measurement in the NHS: A qualitative study. Counselling and Psychotherapy Research. 2012;12(1):71–80. [Google Scholar]
  26. Willis A, Deane FP, Coombs T. Improving clinicians' attitudes toward providing feedback on routine outcome assessments. International Journal of Mental Health Nursing. 2009;18(3):211–215. doi: 10.1111/j.1447-0349.2009.00596.x. [DOI] [PubMed] [Google Scholar]
  27. Wolpert M, Curtis-Tyler K, Edbrooke-Childs J. A qualitative exploration of patient and clinician views on patient reported outcome measures in child mental health and diabetes services. Administration and Policy in Mental Health and Mental Health Services Research. 2016;43(3):309–315. doi: 10.1007/s10488-014-0586-9. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES