Embedding OR modelling as decision support in health capacity planning: insights from an evaluation

ABSTRACT

Literature reviews over five decades have reported the paucity of examples of OR methods being routinely used to support decision-making in health and social care. This paper presents insights from an independent evaluation of a project intended to overcome some of the barriers to implementation by establishing a “community of practice” in Kent (England). The project itself was undertaken by practitioners, and had two main aims: providing training in system dynamics modelling to analysts, and making senior managers aware of the benefits of modelling. The findings largely confirmed previous studies, but also raised issues about style of training delivery and selection of problems to be modelled. Project leaders fully understood the barriers to embedding OR modelling skills, and made considerable efforts to avoid them, but nevertheless the main barrier, pressures on people’s time, remained an obstacle. The paper concludes with general reflections and advice.

1. Introduction

The literature provides extensive evidence that from the 1970s to the present day, OR modelling has not had the same impact in the health and social care sectors as it has had in manufacturing industry, logistics, transportation and defence. Over five decades, a succession of review papers – Tunnicliffe-Wilson (1980), Tunnicliffe Wilson (1981), Jun et al. (1999), Fone et al. (2003), Brailsford et al (2009), and Katsaliaki and Mustafee (2011), Kirchhof and Meseth (2012), Tako and Robinson (2015), Klein and Young (2015), Jahangirian et al. (2015), and (2017)) – all tell the same story: “ … despite a plethora of one-off applications in the academic literature, very few papers report the outcomes of implementation or sustained adoption of these models, and so the value of modelling in health remains an open question” (Brailsford et al., 2013). Over the years several authors have developed guidelines for modellers: Harper (2002) focused on modelling hospitals, Harper and Pitt (2004) described a general framework for increasing the chance of success, and Brailsford et al. (2013) identified a set of barriers and facilitators. The barriers included lack of time and capacity to undertake anything other than urgent tasks, lack of senior management support, lack of trained analyst capability and a whole raft of issues around data availability and quality. In addition to senior management support, facilitators included having an enthusiastic local “champion” (who need not have analytical skills themselves), working on a business-critical and visible problem, and making effective use of any training provided.

This paper presents insights gained from an independent evaluation of a project that set out to overcome some of the above barriers to implementation by establishing a “community of practice” in system dynamics modelling. This project, formally entitled Developing a community of practice around capacity planning for the Kent and Medway Sustainability and Transformation Partnership but referred to in this paper as the Kent CoP, was one of a number of projects funded by The Health Foundation (THF) under its Advancing Applied Analytics programme (https://www.health.org.uk/funding-and-partnerships/programmes/advancing-applied-analytics). The overall objective of the programme was to increase the use of modelling and data science within the UK National Health Service (NHS). Under this scheme, projects could involve academic or commercial organisations but had to be led by an NHS organisation or a Local Authority. An additional eligibility criterion was that applicants had to confirm that they already had access to all the required data.

The paper is structured as follows. Section 2 contains a brief description of the Kent CoP project and its objectives, while Section 3 presents the specific objectives of the evaluation and explains how these are aligned with the broader issues of implementation and adoption discussed in the literature. Section 4 describes the evaluation methodology, which followed a mixed methods approach: the results of the qualitative analysis are presented in Section 5. Section 6 discusses some of the models developed by CoP members. Section 7 presents quantitative results from a survey conducted at the end of the CoP project. Finally, Section 8 reflects on the findings of the evaluation and relates these back to the wider, more fundamental question: what can be done to increase the use of OR modelling by health and social care organisations?

2. The Kent community of practice project

The Kent CoP project ran from January 2018 to April 2019. It was led by Dr Abraham George, Consultant in Public Health at Kent County Council (KCC), in collaboration with the system dynamics modelling consultancy Whole Systems Partnership (WSP), led by Peter Lacey. WSP specialise in healthcare modelling. One of the resources available to the project was the Kent Integrated Dataset (the KID, https://ijpds.org/article/view/427), one of the largest health and care databases in the UK. The KID contains linked pseudonymised patient-level data from a wide number of local service providers: all the acute Trusts (hospitals), Community Mental Health Trusts, 240 primary care practices, social care, and several non-NHS organisations such as Fire & Rescue. At the start of the CoP project all the information governance permissions required to access the KID were in place.

The Kent CoP had two objectives. The first was capacity building: reducing the need for reliance on external consultants by training a cadre of KCC/NHS staff in system dynamics (SD) and the use of Stella (the SD software used by WSP, previously known as iThink). This was to be achieved through a hands-on “learning by doing” process involving the development of a number of Stella models for real-world problems identified by the participants themselves, using data from the KID. Trainees were to be supported by WSP through online Skype webinars and a monthly series of workshops at the KCC head office in Maidstone, where they would all meet face to face. The second objective was awareness raising in senior managers and decision-makers about the potential value of SD modelling for tackling complex, system-wide problems. This would be achieved in part through the specific models being developed, but also through attending the workshops, which were designed so that half the day would be spent on technical modelling issues, and the other half would focus on more strategic issues. The term Community of Practice was used (loosely) to convey the message that meeting both objectives would be required in order for SD modelling to gain traction within KCC and the NHS in Kent. Originally three levels of CoP membership were planned, although in reality the distinction between Associate and Friend was blurred:

• Core: a small number (8–10) of people, who led on, or played a significant role in, a modelling project, doing hands-on modelling and/or playing an active role in all the Workshops;

• Associate: a larger group (15–20) of people who understood the system being modelled and could ask intelligent questions of the model, without necessarily having the technical skills to build a model themselves;

• Friend: a wider, unspecified group of senior leaders who would ultimately use SD model results routinely to inform policy decisions.

Part of the project funding was earmarked for an independent evaluation of the project, to be put out to tender once the project was under way. The contract was awarded to a team from the University of Southampton, with expert advice from MASHnet (the UK Network for Modelling and Simulation in Healthcare, www.mashnet.info). When the evaluation team were appointed, the CoP project had been running for six months.

3. Objectives of the evaluation

The evaluation ran from July 2018 to September 2019. Its objectives (EOs), written into the contract as well as KCC’s original application to THF, were to assess:

Whether, and to what extent, the stated objectives set out in the CoP have been met;

1. Whether, and to what extent, the choice of System Dynamics fits with the challenges being addressed in the CoP;

2. What benefits have accrued to participants in the CoP, their host organisations and the wider system;

3. Whether, and in what way, the CoP should continue beyond the THF funding including resourcing and value for money.

EO4 relates solely to the Kent CoP project and is not discussed in this paper. However, the other three objectives resonate with broader issues concerning the use of OR modelling in healthcare, as they refer to several of the barriers and facilitators identified in the literature in Section 1. The evaluation framework was therefore designed to address these more generic issues, in addition to the CoP-specific objectives the academics were asked to evaluate.

The Southampton team had a combined total of over a hundred years of simulation modelling experience with healthcare organisations; moreover, three of them had previously worked on a similar project, evaluating the use of the simulation tool Scenario Generator for strategic planning in Primary Care Trusts (Brailsford et al., 2013) and were co-investigators in the EPSRC-funded RIGHT project (2007–09), the aim of which was to understand the reasons for the low take-up of modelling and simulation in healthcare compared with other sectors. Their initial feeling, based on this extensive experience, was that the “stated objectives set out in the CoP” were highly ambitious and unlikely to be fully achieved within the project timescale. However, it rapidly became clear that the project leaders recognised that the CoP project was simply a first step towards a major culture change that would take many years to be realised. EO1 was therefore modified to assess the extent to which progress towards those objectives was being made.

4. Methodology

4.1. Overall approach

The approach taken was predominantly qualitative, and involved semi-structured interviews based on the framework described in section 4.2, observations of workshops and meetings, and analysis of the free-text responses to an online survey administered after the end of the project (July 2019). A limited amount of quantitative data on CoP members’ perceptions of their analytical skills and/or understanding of SD before and after joining the CoP was obtained through this online survey. In addition, a light-touch review of the SD models themselves was undertaken. Ethics approval for the study was obtained from the University of Southampton Research Ethics Committee in July 2018.

A two-phase approach was originally planned: an initial phase to assess the status of the CoP after six months and a second phase to evaluate its effectiveness after it has been operating for a full year. Due to participant availability it turned out to be impossible to schedule a second set of face to face interviews, so these were replaced with an online survey emailed to all Phase 1 participants. A feedback session was held early in January 2019, facilitated by Martin Pitt from MASHnet, in which interim findings were presented to George and Lacey, and potential approaches for the remaining months of the project were discussed. Two further interviews with senior decision-makers in KCC were conducted in June 2019, but it was clear that while they were aware of the project and of previous work undertaken in Kent by WSP, their direct engagement with the CoP had been limited.

4.2. Evaluation framework

The framework used was a modified version of the Scenario Generator evaluation framework described in Brailsford et al. (2013), based on Frambach and Schillewaert (2002) multi-level framework. This framework, which recognises that innovation adoption is influenced by both organisational and individual factors, was in turn strongly influenced by Davis’s Technology Assessment Model (TAM) (Davis, 1989) and its extension TAM2 (Venkatesh & Davis, 2000). TAM2 includes additional factors that influence the TAM constructs perceived usefulness and behavioural intentions to use, such as social influences and pressures from colleagues and others to adopt innovations. By definition, social influences are likely to be important in a community of practice. In the case of Scenario Generator these additional factors included relevance to people’s day jobs, whether tangible benefits could be directly attributed to the software, and issues such as whether usage is voluntary or mandatory. The Scenario Generator evaluation framework also included constructs selected from other user acceptance and diffusion models, such as Goodhue and Thompson’s task-technology fit approach (Goodhue & Thompson, 1995), enabling the study to address not only the acceptance of the Scenario Generator software, but also of the method – discrete event simulation – which it implemented. This aspect was particularly important for the Kent CoP evaluation, as we were interested in the appropriateness of system dynamics for tackling the challenges facing KCC. We were not merely evaluating the acceptance of a new software tool, or people’s technical skills, but also the effectiveness of the CoP as a means of developing broader understanding of SD and systems thinking.

The Kent CoP evaluation framework omitted two of the organisation-level constructs in the Scenario Generator evaluation framework (Figure 4 in Brailsford et al. (2013)) as the two factors influencing initial adoption decision by organisation (environmental influences and organisational culture) were not relevant in this case since KCC had already taken the initial adoption decision. However, organisational culture was still represented through the individual level construct internal organisational factors and support from colleagues. The constructs supplier efforts, perceived usefulness and perceived fit were retained as they were directly relevant to all three evaluation objectives. The Kent CoP framework is presented below, showing how each construct relates to the evaluation objectives. The interview guides were structured around these seven constructs and explored the role of the CoP in each area.

4.3. Organisational level

• Supplier efforts. The approach adopted by WSP for providing training, via workshops and online webinars. Relates to the capacity building aspect of EO1.

• Perceived usefulness. Benefits to the organisation, return on investment. Relates to the awareness raising aspect of EO1, and also to EO3: what benefits accrued to KCC as a result of the CoP.

• Perceived fit. Fit with intended tasks & goals. Relates to EO2: is SD a suitable approach to address the challenges faced by KCC. How did people decide what problems to model.

4.4. Individual level

• User’s personal characteristics in terms of background, knowledge and experience. Relates to EO1 (capacity building) by providing a baseline.

• Internal organisational factors and support from colleagues. Relates to both aspects of EO1.

• Social aspects and network effects. Relates to both aspects of EO1: how did belonging to the CoP help individuals develop their skills, and raise awareness in senior managers.

• Individual’s experience with new technology. Relates to EO1: how confident did participants feel about their newly gained skills in using SD, and the individual element of EO3: what benefits accrued to individuals from belonging to the CoP.

Two different interview guides were used for Core members and for Associate/Friends. These can be found in Appendix 1. In summary, Core members were asked about their technical knowledge of SD and their ability to use Stella; whether they knew of, or had ever used, any other analytical modelling approaches; how they had decided which problem to model; what data they used and what issues they had encountered obtaining it; what they saw as the main challenges in learning SD; and what support, from WSP or elsewhere, they had found to be the most helpful. Associates and Friends were asked more general questions exploring their broad understanding of the value of SD to support strategic decision-making: these interviews tended to be more free-ranging than the Core interviews. Both sets of participants were asked about the CoP itself: why they personally had got involved, practical aspects such as the style/format of the workshops and the training provided, and broader questions about their perceptions of the purpose of the CoP and its likely sustainability after the end of the Health Foundation funding.

4.5. Workshop observations

Members of the evaluation team attended four CoP workshops held in Maidstone: 5th July, 3rd September and 29th November2018, and the final Showcase event on 7th May2019. Attendance of CoP members at the workshops understandably varied from month to month, due not only to people’s availability but also to the fact that some workshops focused on specific topics, e.g. workforce planning, and hence attracted people working in that area who were not regular attendees. The team also had informal conversations with CoP members at these workshops.

4.6. Phase 1 interviews

A total of sixteen Phase 1 interviews were conducted by the Southampton team in the autumn of 2018. In July 2018 the evaluation team were provided with a spreadsheet containing the names and email addresses of KCC or NHS staff in Kent and Medway to whom Workshop invitations were sent. The list contained 94 names, although only about half of these were formally designated as CoP members in terms of being Core, Associate or Friends. On the assumption that people who were not thus designated were unlikely to have engaged significantly with the CoP, they were excluded from the list of potential interview participants. This left 41 potential interviewees: eight Core, 14 Associates and 19 Friends. All 41 were emailed, inviting them to be interviewed. Of these 41, 13 agreed to be interviewed: six Core, five Associates and two Friends. In addition to these 13 respondents, George and Lacey were also interviewed, although given their formal roles in the project they were treated differently and the standard interview guide was not used. The majority of interviews were face to face, either in KCC’s head office in Maidstone or the participant’s normal place of work, but a small number were conducted by Skype. Each interview involved at least two members of the research team. Audio recordings of all interviews were made and subsequently transcribed for further analysis.

A textual analysis of all 16 interview transcripts was undertaken. Each transcript was read independently by two researchers. Rather than map responses explicitly against the framework constructs or the evaluation objectives, the researchers identified recurring themes or issues raised by multiple participants. In many cases, of course, these themes directly addressed one or more of the four objectives of the evaluation. This approach had elements in common with the grounded theory approach of Glaser and Strauss (1967) in which themes emerge in a bottom-up fashion from qualitative data. The findings from these pairwise analyses were then discussed by the whole team, and a final set of issues and themes agreed on. These themes form the subsection headings in Section 5. In Section 8 the themes are linked back to the previous literature on adoption.

5. Results of the qualitative analysis

In this section, verbatim quotes from interview participants are shown in italic text.

5.1. Perceptions of the aims and purpose of the CoP

Broadly speaking, these reflected the aims set out in the original grant application form and showed a shared understanding among the majority of interviewees of what the CoP was trying to achieve. Capacity building and awareness raising were mentioned explicitly several times, and not just by the project leadership! All participants recognised that these would take time to be achieved, and there were trade-offs to be made between getting timely results for a live problem while at the same time developing in-house skills on a longer-term basis.

In general, as one would expect, respondents’ perceptions of the purpose and potential value of the CoP were closely (and mainly positively) correlated with their level of involvement in it. One or two Core respondents, mainly analysts directly involved in learning how to use Stella, were slightly less clear about the purpose of the CoP: “I think you sort of understand I’m not totally clear on the Community of Practice as a concept”. Another participant commented ”I like being part of the CoP; I just find it a little bit unstructured at the moment. It’s a shame it’s not more structured and it’s a shame that being part of it isn’t more recognised”.

5.2. Learning arising from CoP membership

Overall, most of the Core respondents who were directly involved in hands-on model-building were very enthusiastic about the personal skills development aspects, and commented positively on the technical support and training provided by WSP: “ I thought initially that it was not something that I would be able to do and then built my kind of understanding, talked to a few people and realized that actually it was”. Most people found learning about Stella as a tool, and more generally about SD as a modelling approach, to be “exciting” and “enjoyable”. Clearly, learning styles differed and while some people would have preferred a more structured, classroom approach to learning (one respondent felt that the broader aim of in-house skills development appeared secondary to building specific models) others were happy with the more problem-focused, learning-by-doing type approach adopted by WSP. Several respondents commented that members should be given some kind of formal certification, as an external recognition of skills acquisition that could put on their CV, and which might also justify spending time out of the office.

The fortnightly online webinars were less successful; information transfer was mostly from WSP to CoP members, rather than CoP members learning from each other. A common observation was that if someone encountered a technical problem with using Stella, their first thought was to ask WSP and not another CoP member, although at the beginning they did ask colleagues: it seemed that as they got more confident, they felt less embarrassed about asking WSP for help. “Probably when I was at the more basic stage of it I did ask XXXX for a couple of tips; now I’d be going to Peter. I’ve got enough of it to go directly to Peter. I’ve only joined probably a couple of the technical Skype calls that run every fortnight; I’d raise it now at one of those or directly with Peter”. Overall, although both Core and Associate members clearly appreciated the value of the CoP for networking, sharing of experience, and learning about the broader aspects of systems thinking as a way of tackling complex problems, the potential for self-learning and mutual support in the CoP was not yet fully realised.

5.3. Choice of SD as a decision support tool

Section 6 describes the specific models developed by CoP members, and the appropriateness of SD for the problems that were selected, but in terms of the interview responses views ranged from sceptical to hugely enthusiastic: “ … it’s far better than the way that we currently do things, which is looking at historic data and trying to extrapolate when things are moving quite swiftly – particularly in Kent and Medway – because the population has changed all the time and growing all the time. So, I think relying on old data is going to be less and less helpful” and “Modelling from a systems perspective could help give us an insight that we wouldn’t otherwise get”. Others were more ambivalent, with comments like “ ... SD is unnecessarily complex” and “ ... systems modelling can be used in different ways but not necessarily for the work that I am doing”.

Some respondents knew of, or had even used, other modelling tools such as Simul8 and Scenario Generator, and all seemed to be aware that “analytics” and “modelling” were umbrella terms that encompassed many other techniques in addition to SD. Broadly speaking the vast majority of respondents agreed that while SD had limitations, it also had the capability to improve decision-making and was often better than current practice, i.e., the use of Excel and standard statistical tools. However, a couple of respondents raised concerns about SD as a methodology (in general): one person commented “ ... as I’ve learned more about it, I think I’m a little bit more sceptical … I don’t think people would statistically validate it or have a closer look at the outputs as they should”.

5.4. Meetings – structure and content

A variety of views were expressed. People agreed there was a value in showcasing the capabilities of modelling to commissioners and other senior decision-makers, and several people said the workshops provided an opportunity to meet people who they would not otherwise have met in the normal scheme of things. Like any “away-day”, the workshops were a chance for people to escape from day-to-day operational pressures and discuss issues from a broader perspective: “ ... one of the good things … is the slightly more cross-organisational learning that does come with some of those Community of Practice meetings; we get to work across, there are other Commissioners and providers there as well; it’s been brilliant”.

Some people would have preferred the meetings to focus entirely on technical training, and since all the workshops took place in Maidstone there were inevitably geographical challenges for people based further away, quite apart from the equally obvious and insuperable issues of busy people finding time in their diaries to attend a whole or half-day meeting. Formal opportunities for people to meet and/or interact with other CoP members outside the Workshop meetings appeared limited, unless they were physically based in the same office and already working together on other projects. Skype (either because of technical issues to do with NHS firewalls, or unsatisfactory connections) was not universally popular, although some found the Skype sessions more useful than the workshops: “I really appreciated the Skype calls actually. I think they’re really good and really directed and, yeah, [unlike the workshops] I felt like that was really valuable and they did technical calls, bi-weekly technical calls. I think they’re quite a good thing to go to or listen in on”. The conclusion is that you can’t please all the people all the time!

5.5. Engagement with the CoP

Understandably, this was hugely variable. Relatively few people on the extensive mailing list of people who were invited to meetings attended regularly or were actively engaged in building models. The reasons respondents gave for lack of engagement were exactly what one would have expected: time pressures, logistical difficulty in attending meetings, and lack of support from immediate line managers to allow people to take time out from their day job for something they perceived as non-essential: “I think my barrier on it, one of my main reflections of it, is about – I suspect everyone talks about time availability. Particularly in a more senior strategic role as I was although it was interesting it had, by the nature of it … it was slightly more peripheral to what I was doing”. Despite these challenges, a small number of Core members were clearly highly engaged: “I also encourage modelling to be written into all of the job descriptions of the public health specialists” and “I love the idea that we can work together as a group and share knowledge” and one participant commented that KCC was “ … as a council quite committed around learning and development. And I’ve never had any difficulty making time for training and development”.

5.6. The role of the Kent integrated dataset

As the evaluation team had prior knowledge of the KID and knew it to be an impressive resource, they had approached the topic of data in the interviews with a high degree of optimism, even though their previous experience, and the academic literature on getting models used in the NHS, indicated that issues with data availability were often a major barrier to successful implementation in practice. In the case of the CoP, the expectation was that the KID had enabled respondents to overcome many of the barriers reported in the literature, but sadly, and very surprisingly, this proved not to be the case. Despite its obvious enormous potential for modelling, several respondents reported they had encountered technical issues with accessing and using the KID, or found that relevant data for the topic they had selected for modelling was not there. “I haven’t relied on the KID at all actually for it. So, the data I’m using for it is … so, some of it is from published data, so prevalence estimates in the JSNA or just other data sources like that … ”

Like many broadly similar datasets in the UK, the KID was designed to serve multiple purposes: real-time use in clinical practice, epidemiological research, and population health planning. This flexibility is not only a huge benefit but is essential in order to gain the required buy-in from the organisations that provide the data. However, like its counterparts in other regions, this multi-purpose design brings its own complications and there are still issues to be resolved. One respondent said “I’ve been involved and aware of the KID for several years so I’m aware of the value that it would be able to bring, and I can definitely see how it would be able to add some value to it. I guess that I have a slight just general kind of frustration with the KID in that … every time I needed something from the KID it couldn’t do it”. The evaluation team shared this frustration. While firmly believing that models using KID data will be used to support future evidence-based decision-making in Kent, they were forced to conclude that the findings for the CoP reflected the general findings in the academic literature.

5.7. The role of WSP (and consultancies in general)

The role of WSP in the CoP project raised some interesting questions about the role of consultancies in promoting skills transfer within public sector organisations. WSP, who as noted above had worked in Kent prior to the CoP project, has a long track record of working collaboratively with public sector organisations and training client staff in the use of Stella. Indeed, several respondents commented that WSP took a different approach to other consulting firms: “ … the thing about a conventional consultancy relationship might be that you wouldn’t get much, if you like, transfer of expertise”.

Public sector organisations bring in external consultants for a variety of reasons. One respondent felt that any future work done in-house by NHS/KCC analysts, no matter how advanced the model that was developed, would never be able to carry the same “kite mark” of credibility as work (potentially of lower quality) done by big name consultancies: “And I think a lot of the time that’s why Trusts, if they’ve got a project coming up, will go to a consultancy company that allegedly has this expertise … because it’s easier, you can pay someone, it’s a short-term cost and it will go away after a set period of time and you’ve got a name stamped on it so, you know, it’s XXXX or whatever”.

One of the senior managers interviewed in June 2019 commented: “I’m not clear whether you would actually need to build an internal capability for this, the reason I say that is I don’t know whether there is sufficient work to justify an internal capability on it. So, I think ... if we can be an intelligent customer it helps us to design and commission and manage these pieces of work” Clearly, this interviewee had a good appreciation of the value of SD modelling and was keen for Kent to use it, but remained unsure whether the benefit justified the cost of developing in-house capability.

5.8. The “champion” role

The academic literature on adoption of OR modelling in the NHS emphasises the importance of a champion, an influential and well-networked person within the organisation who is not necessarily technically skilled at modelling themselves but fervently believes that it is useful, and (crucially) is trusted by colleagues as “one of us” and is able to persuade others of its value. In the CoP George exemplified this role par excellence and was clearly the driving force behind the project. “Abraham’s always had this vision of developing a kind of … the … in house ability of the system being able to use kind of system dynamics in order to improve planning and he’s had this vision for many years”. Moreover, he was not the only KCC employee who is a massive enthusiast for SD modelling: “Well, it’s a no-brainer to me. It makes absolute sense that we need to model what’s going to happen in the future; we can’t continuously look back at old data. That is the way of the world. Every other massive organisation across the globe models what’s going to happen. Facebook and Google they’re not looking – ‘Oh, what did people do in 1999? Let’s extrapolate’. Of course they’re not because the future is not even going to look like that”.

6. The SD models developed during the project

Sections 6.1–6.3 describe three models built entirely by CoP members and selected by the CoP leaders for independent review. Given that the main purpose of these models was educational, i.e., they were essentially a device to enable people to learn the skills of Stella modelling, the evaluators had no expectation that the models would look professional, be highly complex, or already have been used in practice to inform real-world decisions. The models to be reviewed were for Neurodevelopmental Care, Tier 4 Autism Placements and Critical Care Beds. The first two were among several models developed for Children and Adolescent Mental Health Services (CAMHS) as part of the CoP activities. The CAMHS models also included a complex system-wide population-level model, built with considerable input from WSP. This model demonstrated a genuine appreciation of the value of SD for tackling cross-organisational issues, and we understand that it has been used to inform management decision-making.

6.1. Neurodevelopmental care

This model examined ways in which the waiting list size for one single specialist service could be reduced. The effects of three interventions on the performance of the system were modelled: reduction of referrals, use of a prescribing nurse to increase the capacity of the service, and measures to support people while they were still on the waiting list. The model contained a number of assumptions about the proportions of patients who follow the different paths through the system, for which data did not appear to be available. The modelled system was relatively simple, and since accumulation and feedback were not major factors and the proposed interventions did not appear to conflict or interact, arguably the problem could have been modelled in a spreadsheet. Having said this, the modeller had no previous knowledge of Stella or of SD and the model definitely provided evidence of learning and skills transfer.

6.2. Tier 4 Autism placements

This model analysed how hospital admissions among children with a learning disability and/or autism could be reduced (or avoided) by early interventions. This model was slightly more complex than the Neurodevelopmental Care model, and included the effects of, and interactions between, various community-based interventions. In theory SD is an appropriate method to capture such interactions, but since only ten inpatient beds were available for such placements random fluctuations in demand are likely to have a huge impact on the system. SD is not well suited to capture such fluctuations, and arguably discrete event simulation (DES) would have been a more natural approach for this problem. However, as a learning device the model had clearly achieved its purpose: it was very competently constructed and an impressive piece of work for a novice modeller.

6.3. Critical care beds

This model addressed the utilisation of critical care beds in three different hospitals in East Kent, and examined how strategies such as early discharge could impact occupancy rates. Problems arise in the real world system due to variability and uncertainty in terms of patient arrival rates and lengths of stay. This is a classic capacity-constrained queuing system, but the model contained no queues, no capacity limits and no variability. The academics had considerable reservations about the suitability of SD for this particular problem. It transpired that the vast majority of the Stella modelling and all the (very impressive) interface development was done by WSP; the CoP member involved was mainly involved in data collection and stakeholder engagement. This model was by far the largest and most complex of all the models developed. Undoubtedly, it was a valiant attempt to make Stella do something it was not originally designed to do and clearly a large amount of work had gone into data collection and analysis, as well as stakeholder engagement. Nevertheless, it was unclear whether any genuine learning about the concepts, methods or benefits of SD was achieved in the process.

6.4. Summary

The models that appeared to demonstrate the greatest learning by CoP members, not only about the technicalities of using Stella but also about the concepts of SD in general, were the models developed for CAMHS. It was evident that the two CoP members involved, who work together on a daily basis, had gained a huge amount from the CoP project. SD had truly become socialised as part of their daily work and they supported each other. It appeared that as long as they could still ask WSP for technical advice when they got really stuck, they did not need the wider virtual support provided by the CoP and had developed their own “Partnership of Practice”.

7. Online survey

This was implemented using the University of Southampton’s web-based survey tool iSurvey. The survey was anonymous, although participants had the option of leaving their name. In total ten people submitted responses, although one had only partially completed the survey and two identified themselves as WSP employees. Since several people chose to remain anonymous, it was not possible to match their responses to their previous interviews. The overall aim was to explore people’s opinions after the project had formally ended. The survey questions, some of which were quantitative and some of which were free text, are listed in Appendix 2. Some responses to the free text questions simply reiterated findings from the Phase 1 interviews, but others were more reflective.

Most people felt they now had a good understanding of the principles of SD and its benefits, but were less confident about actually using Stella without support from WSP: “I now have basic understanding, and would feel confident to use this to build basic models if I had some support from others in the group initially”. Overall, people felt the CoP had met both objectives (skills development and awareness raising), even if only partially: “The understanding and skills development I think is still restricted to a small group of analysts” and felt that in order for modelling to become fully embedded to support decision-making, a dedicated team of modellers would be required, with wider senior management engagement and formal recognition of modelling skills.

Table 1 . Survey responses (numerical data)

Table 1.

Contains the quantitative data from the seven respondents who were not WSP employees. One respondent appeared particularly lacking in confidence in all areas, but was something of an outlier.

On a scale of 1 to 10, where 1 = not at all and 10 = very confident, how would you rate your own ability to:-								Mean
Develop a model from scratch without any help	2	7	4	4	3	7	5	4.57
Develop a model from scratch with support from a colleague	2	8	4	6	7	8	6	5.86
Develop a model from scratch with support from WSP	1	10	10	7	9	10	9	8.00
Modify the structure of a model you’d previously worked on	1	9	7	4	5	8	7	5.86
Make minor changes, e.g., change the initial value of a stock, in a model you’d previously worked on	2	10	10	6	5	10	10	7.57
Explain the basic concepts of stocks and flows to a lay person	6	10	10	6	6	10	9	8.14
Describe the benefits of modelling to a senior manager	5	10	8	7	10	9	9	8.29
To what extent do you now feel in a better position to see the value of SD modelling in helping you to address real-world problems you face in your work?	3	9	6	6	10	10	9	7.57

8. Discussion

The first subsection summarises the evaluators’ conclusions regarding the three specific objectives of the Kent CoP evaluation, as set out in Section 3. Several issues arising from these findings, relating to broader issues of adoption and implementation raised in previous studies in the literature, are then discussed. The section concludes with a set of insights gained and a discussion of their wider relevance.

8.1. Objectives of the evaluation

The first objective (EO1) was to assess whether, and to what extent, the stated objectives set out in the CoP had been met. In terms of capacity-building, there was indisputable evidence in the data that a (relatively small) number of CoP members had acquired technical skills in using Stella, although of course it would be unrealistic to expect that they were now sufficiently expert to enable KCC or other local partners to dispense completely with the use of external consultants in future. The success of the CoP in raising awareness in senior managers and decision-makers about the potential value of SD was harder to judge, since the influence of the CoP was more subtle. Nevertheless, there was clear evidence that a number of respondents now had a heightened awareness of the benefits of modelling, even if they did not demonstrate a particularly good understanding of systems thinking (as normally understood).

There is no such thing as “an SD problem” and there are no problems for which SD should never be used; even the critical care beds problem would have had some aspects where SD could be useful. Having said this, we may speculate that had DES had been used for this stochastic resource-constrained queuing system, the model might have been of more immediate practical use to inform planning decisions, and hence more might have been achieved in terms of awareness raising.

The second objective (EO2) was to assess whether, and to what extent, the choice of System Dynamics fits with the challenges being addressed in the CoP. This was a matter of judgement. Based on their experience, the academics felt that SD was undoubtedly an extremely good fit for the wider strategic issues facing decision-makers in Kent, exemplified by the system-wide CAMHS model mentioned briefly at the start of section 6. SD did not seem to be an obvious choice for the three problems described later on in that section, but the main purpose of these models was to train staff in using Stella. This raised some interesting questions, discussed in section 8.2 below.

The third objective (EO3) was to assess what benefits have accrued to participants in the CoP, their host organisations and the wider system. In addition to gaining technical modelling skills, the evidence showed that people had clearly benefited from opportunities to network – often with those beyond the usual boundaries of their own organisations – and had gained an appreciation of a more systemic perspective to the specific problems that they shared, or with which they could readily empathise. Changes in perspective, rather like changes in organisational culture more generally, are likely to benefit both the host organisation and the wider system but are inevitably slow to take effect, and such benefits are often notoriously difficult to quantify.

8.2. Reflections on capacity building

An issue relating to EO2, and closely aligned to the framework construct perceived fit with intended tasks & goals, was how people chose which problem to model. Trainees were undoubtedly given excellent technical support with Stella, but were seemingly left to their own devices when it came to selecting what to model. It was entirely understandable that the people who were involved in hands-on model-building (who were mainly analysts) chose operational problems, in familiar systems related to their day jobs, rather than the more strategic and complex population-level problems to which SD is generally better suited. As noted in section 1, the literature on adoption emphasises the importance of working on a business-critical and visible problem. Therefore, allowing people to select a problem that really mattered to them and then letting them learn the technical aspects of Stella through trying to model it without being overly interfering (or worse, discouraging) was arguably a valid approach.

However, the CoP trainees were not taught about problem structuring or conceptual modelling, or offered guidance about what attributes of a problem situation might suggest that SD was (or was not) an appropriate modelling approach. It is reasonable to speculate that had the trainers been academics rather than consultants, they would have designed a structured learning programme using material they teach to university students, including an introduction to the basics of SD modelling and the development of “toy” models in Stella. Only after this would the trainees have started (been allowed to start) building real-world models themselves, and academic trainers might well have intervened (interfered) a lot more in the problem selection and conceptual modelling phases. It is far from obvious that this didactic approach would have been any more successful with NHS staff or Local Authority employees: this is an interesting area for further research. Previous empirical research into how learners engage with modelling, for example, Powell and Willemain (2007), who investigated how novices develop models, and Tako and Robinson (2009), who compared novice user perceptions of SD and DES, used MBA students as participants. MBA students have several years’ business experience and therefore have some characteristics in common with CoP members, but – crucially – unlike the CoP members, the participants in these two earlier studies had all recently received formal academic tuition in modelling as part of their MBA programme.

8.3. Reflections on data

The Health Foundation has funded around 50 projects through its Advancing Applied Analytics programme. Collectively, these projects have undoubtedly had significant impact. Nevertheless, despite the eligibility criterion requiring applicants to demonstrate that the necessary data are available and information governance agreements are in place, many of the AAA projects have reported issues with data availability. In most cases problems have been overcome through ingenuity and good will, and each individual project represents a step along the way towards a future halcyon state where the massive quantity of routine data collected by the NHS and local authorities can be fully exploited for the benefit of all.

In the case of the Kent CoP the proposed data source was the KID, which considerably exceeded the criteria for funding even though the specific problems to be modelled were not known at the time of application. Unfortunately, as discussed in Section 5.6, several CoP participants reported issues with data availability. The evaluation team did not have access to the KID and therefore it is possible that people had unrealistic expectations of the KID, or that some of the required data were actually in the KID but not in a format suitable for modelling that particular (operational) problem. However, the problems encountered were depressingly familiar, and were typical of issues with data described in the academic and grey literature. For example, Tako and Robinson (2015) reported that 89% of respondents agreed or strongly agreed with the statement that it is more difficult to access data in healthcare than in other settings.

There is a symbiotic relationship between modelling and data. In the early stages of model development – mapping the system and building a conceptual model – the modeller gains a lot of insight into what data are needed. If existing datasets are unable to provide this information, the modeller has not only inadvertently exposed these gaps but can also articulate the benefits of collecting that data. Continuous quality improvement in data collection is a by-product of the modelling process, and recommendations about what data could be collected (in addition to routine performance data), leading to better service delivery and ultimately patient benefit, are often an important output of modelling projects.

8.4. Reflections on training

It was evident from participants’ responses that different people had different learning styles, and while some would have preferred a traditional classroom approach, this would not have worked for everyone. From a capacity building perspective, a flexible approach to skills development that allows trainees to learn in their own preferred way is probably the most likely to succeed, but would be labour-intensive for the trainers and would require a readiness to adapt to different learning styles. It was also clear from participants’ comments that some kind of formal accreditation would enable trainees to justify time out of the office to their line managers and would demonstrate that modelling skills are recognised and valued by their organisation. Initiatives such as the Certified Analytics Professional scheme administered by INFORMS (the American OR Society), and also available through the UK OR Society, could be useful here.

8.5. Overall conclusions and lessons learned

We conclude with some general points highlighted by the study about what measures could increase the use of OR modelling by health and social care organisations.

• Care needs to be taken in assuring a realistic match between the chosen OR approach and the characteristics of the problem(s) faced by the organisation. We discuss in section 8.1 and 8.2, for example, the appropriateness of SD in this case. Assuring a familiarity with the wide range of OR approaches available, and their relative strengths and weakness, might increase the successful outcomes that modelling ought to facilitate.

• Awareness of the different (and sometimes conflicting) objectives that participants might have in learning modelling skills. It was clear that, at the operational level, some participants seized the opportunity to learn more about how modelling might help them, not only in terms of finding ways to address persistent operational problems, but also in acquiring skills that could enhance their career prospects. Actively seeking out formal accreditation opportunities for such participants (see 8.4) is one action that healthcare organisations could pursue.

• An awareness of the type of data needed, and of its availability and ease of access. Data access is of course one of the main barriers identified in the literature. In the case of the Kent CoP, it was clear that those wishing to use the KID faced a number of challenges (see 5.6 and 8.3), often heightened by the nature of the data itself. Nevertheless the KID demonstrates the enormous potential value of linked datasets for modelling to support planning decisions, as well as real-time clinical care and academic research.

• The importance of networks. Whilst this lesson is not unique to healthcare organisations, the healthcare community is in a strong position to recognise that partnership beyond traditional organisational boundaries can yield benefits. This study showed that a Community of Practice approach can provide opportunities for nurturing such networks.

• Busy people often (and understandably) cannot find the time to engage fully with the modelling process. This issue is evident less at an operational level, where the building of a model might be seen as a route to the resolution of a specific issue, but more at a strategic level, where enforced breaks in continuity in participation in the (strategic) aspects of the modelling process might lead to it becoming viewed as nonessential, as attention shifts towards other issues.

The twin objectives of in-house capacity building and awareness raising in senior managers are both essential components of any initiative designed to improve decision-making through the use of some new technology without buying in expertise from outside. Unfortunately, while both objectives need to be achieved for such initiatives to become sustainable, they operate on different timescales; capacity building is technical and is relatively quick and easy to achieve provided sufficient resource is dedicated to it, whereas awareness raising involves culture change and is far more challenging.

Funding Statement

This work was supported by The Health Foundation.

Disclosure statement

No potential conflict of interest was reported by the author(s).

A. Membership history

• How did you learn about the existence of the CoP?

• What do you consider the primary purpose of the CoP to be?

• How long have you been a member of the Cop?

• What were the main reasons that encouraged you to join the CoP?

• What role (if any) did your organisation play in encouraging you to join?

• Have you actively encouraged others to join?

• Are you a member of any other CoPs, either related to healthcare or related to other aspects of your work?

B. Knowledge and experience of Systems Dynamics (SD)

On a scale of 1 to 10, where 1 = None and 10 = Lots, describe … .

• What is your own knowledge of simulation/modelling in health care?

• What is your own experience of simulation/modelling in health care?

• What is your prior knowledge of using SD (prior to joining the CoP)?

• How many different models have you built using SD?

  • What kinds of decision do you use SD to support? E.g.,
  • Testing the assumptions in strategic plans.
  • Modelling impact of changing population
  • Assessing the feasibility of new models of care (e.g., care closer to home, new urgent care provision and community-based services)
  • Workforce planning
  • Other

• Do you find SD tools intuitive and easy to use?

C. Knowledge of other types of Decision Support Software

• What other standard software do you use in your work and for what purpose? e.g., Excel spreadsheet modelling for simple forecasting, SPSS for correlation analysis …

• What other specially designed software tools do you use, if any, and for what purpose?

• If yes to either/both of the above two questions, what role does SD play, either as a substitute for or complement of these tools?

• How does SD compare to other software (e.g., Excel) you have used in terms of ease of use?

• Do you use the CoP to support you in your use of other software decision-support tools (not just SD)?

D. Training and Support

• Do you see the CoP as primarily a vehicle for training, support, or both equally?

• If you could give one piece of advice to a new user of SD which would help them get started, what would that be?

• If you had to do a one minute “elevator pitch” for using SD in healthcare, what would you say was the best thing about it?

E. Inputs and Outputs

• What input data, if any, do you find difficult to obtain?

• How did you get this data?

• Was data readily available or did you need to chase people for it?

• Could you get accurate data or were they “expert opinions”?

• In terms of outputs:

  • What does your organisation find (most) useful?
  • What do you find hard to interpret/communicate to your boss/colleagues?
  • What do you think could be left out?

• When developing a model, what features of SD do you find the most useful?

• How long does it take you to build a model?

• What takes the most time?

• How could this be made easier?

G Others

• What contribution do you think tools like SD make to the management and decision making process in your organisation?

• Overall, do you think SD modelling improves decision-making in your organisation, and if so, in what ways?

• Do you believe that membership of the CoP has enhanced your knowledge of modelling/simulation? In what ways?

• Are there any activities that the CoP currently does not do, or does not do sufficiently, that would enhance your use of SD modelling?

• Are there any changes that you would like to see made to the way in which the CoP currently operates?

Template used for Friends and Associates

A. Membership history

• How did you learn about the existence of the CoP?

• What do you consider the primary purpose of the CoP to be, from your perspective?

• How long have you been an associate member of the CoP?

• As an Associate/Friend, what do you think might be the most important or influential input you can make to the CoP?

• What were the main reasons that encouraged you to join the CoP?

• What role (if any) did your organisation play in encouraging you to join?

• Have you actively encouraged others to join, either as core members or associates?

• Are you a member of any other CoPs, either related to healthcare or related to other aspects of your work?

B. Knowledge and experience of Systems Dynamics (SD)

On a scale of 1 to 10, where 1 = None and 10 = Lots, describe … .

• What is your own knowledge of simulation/modelling in health care?

• What is your own experience of simulation/modelling in health care?

• What is your prior knowledge of using SD (prior to joining the CoP)?

C. Knowledge of other types of Decision Support Software

• What other standard software do you use in your work and for what purpose? e.g., Excel spreadsheet modelling for simple forecasting, SPSS for correlation analysis

• What other specially designed software tools do you use, if any, and for what purpose?

• If yes to either/both of the above two questions, what role does SD play, either as a substitute for or complement of these tools?

• How does SD compare to other software (e.g., Excel) you have used in terms of ease of use?

• Do you use the CoP to support you in your use of other software decision-support tools (not just SD)?

D. Engagement with the CoP

• Do you see the CoP as primarily a vehicle for gaining general knowledge about systems modelling, or more specifically for, say, training or support?

• If you could give one piece of advice to a new associate member of the CoP which would encourage them to contribute to the CoP, what would that be?

• What specific inputs, if any, do you feel you have made to the CoP, either to the way it operates or to the content/direction of the modelling process?

• What, if any, are the inhibiting factors in your involvement with the CoP?

• If you had to do a one minute “elevator pitch” for becoming an Associate member of the CoP, what would you say was the best thing about it? And the worst thing?

E. Others

• What contribution do you think modelling tools like SD make to the management and decision making process in your organisation?

• Overall, do you think SD modelling improves decision-making in your organisation, and if so, in what ways?

• Do you believe that membership of the CoP has enhanced your knowledge of modelling/simulation? In what ways?

• Are there any activities that the CoP currently does not do, or does not do sufficiently, that would make your job easier?

• Are there any changes that you would like to see made to the way in which the CoP currently operates?

• Does your experience with interacting with the CoP give you more confidence with using new software?

• What are the main pressures facing your organisation? Probe for answers, e.g., external pressures, time, money, politics

• What are the main pressures on you personally? E.g., time, balance between firefighting/short term operational work and strategic thinking, lack of support from senior management

• What insights, if any, has your membership of the CoP provided about these pressures, and how modelling might address them?

Online survey questions

1. From a general perspective, what do you think were the objectives of the CoP as a whole?

2. To what extent do you feel they have been achieved?

3. What were you personally expecting to get from the CoP?

4. Please describe the extent to which your personal expectations have, or have not, been met.

5. What, for you, has been the best thing about the CoP.

6. and the worst?

7. What was your knowledge of using SD prior to joining the CoP, and to what extent has this changed?

8. If you were directly involved in developing any models during the project:

   1. Please state which and describe your involvement
   2. Who decided what problem you were going to model and why do you think that particular problem was chosen?
   3. What were the main challenges you encountered and how were these overcome?
9. On a scale of 1 to 10, where 1 = not at all and 10 = very confident, how would you rate your own ability to:

   1. Develop a model from scratch without any help
   2. Develop a model from scratch with support from a colleague
   3. Develop a model from scratch with support from WSP
   4. Modify the structure of a model you’d previously worked on
   5. Make minor changes, e.g., change the initial value of a stock, in a model you’d previously worked on
   6. Explain the basic concepts of stocks and flows to a lay person

10. Describe the benefits of modelling to a senior manager

11. To what extent do you now feel in a better position to see the value of SD modelling in helping you to address real-world problems you face in your work?

12. To your knowledge, which of the CoP models were used to inform a real-world decision, and how?

13. How likely are you to use SD modelling in the foreseeable future, and for what?

14. To your knowledge, which of the CoP models are still being worked on and/or used, and for what?

15. Are you aware of any new models being developed since the official end of the project?

16. How has the CoP enabled you to make useful contacts, and are these continuing?

17. In your view, what would it take for modelling to become fully embedded within your organisation to support decision-making?

18. Are there any further points, not covered by the questions above, that you would like to add?

19. Your answers are anonymous. You can optionally give us your name and contact details so that we can contact you for any clarifications.