Skip to main content
BMJ Open Access logoLink to BMJ Open Access
. 2017 Jan 6;26(8):641–652. doi: 10.1136/bmjqs-2016-005636

Combining qualitative and quantitative operational research methods to inform quality improvement in pathways that span multiple settings

Sonya Crowe 1, Katherine Brown 2, Jenifer Tregay 2, Jo Wray 2, Rachel Knowles 3, Deborah A Ridout 3, Catherine Bull 2, Martin Utley 1
PMCID: PMC5537516  PMID: 28062603

Abstract

Background

Improving integration and continuity of care across sectors within resource constraints is a priority in many health systems. Qualitative operational research methods of problem structuring have been used to address quality improvement in services involving multiple sectors but not in combination with quantitative operational research methods that enable targeting of interventions according to patient risk. We aimed to combine these methods to augment and inform an improvement initiative concerning infants with congenital heart disease (CHD) whose complex care pathway spans multiple sectors.

Methods

Soft systems methodology was used to consider systematically changes to services from the perspectives of community, primary, secondary and tertiary care professionals and a patient group, incorporating relevant evidence. Classification and regression tree (CART) analysis of national audit datasets was conducted along with data visualisation designed to inform service improvement within the context of limited resources.

Results

A ‘Rich Picture’ was developed capturing the main features of services for infants with CHD pertinent to service improvement. This was used, along with a graphical summary of the CART analysis, to guide discussions about targeting interventions at specific patient risk groups. Agreement was reached across representatives of relevant health professions and patients on a coherent set of targeted recommendations for quality improvement. These fed into national decisions about service provision and commissioning.

Conclusions

When tackling complex problems in service provision across multiple settings, it is important to acknowledge and work with multiple perspectives systematically and to consider targeting service improvements in response to confined resources. Our research demonstrates that applying a combination of qualitative and quantitative operational research methods is one approach to doing so that warrants further consideration.

Keywords: Quality improvement methodologies, Healthcare quality improvement, Implementation science

Introduction

Improving continuity1–8 and integration9–12 of care across sectors is a priority in many health systems. Increasingly, patients are discharged to community care as early as clinically appropriate to alleviate pressure on hospitals and in response to patient preferences for receiving care closer to home.13 14 However, it is challenging to design and assure the quality of services that are delivered across multiple organisations, not least because responsibility is dispersed among health professional groups that have different priorities and perspectives.

Literature on the development of complex interventions15–17 and guidance on service design18 stress the importance of evidence-informed approaches to concentrate attention and scarce resources on changes most likely to be effective.19–21 Yet, the evidence to support service change may be incomplete and comprise disparate quantitative and qualitative data as well as tacit knowledge.22–26 Formal group consensus methods such as Delphi and nominal group technique have been used extensively to incorporate the collective tacit knowledge of experts in the formulation of clinical practice guidelines and selection of outcome measures and quality indicators.27–30 However, such approaches are designed to obtain consensus regarding specific issues and are less readily applicable in group processes aiming to characterise complex service problems and to reach decisions about how to tackle these in ways that explicitly accommodate diverse perspectives and motivations.31

Operational research (OR) approaches to structured group decision-making are distinctive in their use of formal models to represent the problem in a manner that is amenable to analysis and manipulation.32 Formal models need not be quantitative, and a class of qualitative problem-structuring methods within the ‘soft’ (interpretivist) OR paradigm exists to help groups explore and address complex problems.33 These approaches permit pragmatic partial or local improvements to be agreed without requiring consensus among different interests on an overall solution.34 They include decision-conferencing,35 facilitative modelling,36 cognitive mapping,37 strategic options development and analysis,37 and strategic choice approach.38 The most relevant for system redesign is soft systems methodology, designed to tackle complex issues through systematic learning about the problem, decision processes and levers of change.39 Problem-structuring methods have predominantly been used to address issues in single organisations rather than those that span organisations, the latter being more challenging given the diffuse decision-making and greater diversity of working practices and goals.40 Recent notable exceptions include the use of soft systems methodology to assess existing provision and identify improvement strategies for children's mental health services41 and stroke rehabilitation.42

Quantitative techniques within the ‘hard’ (positivist) OR paradigm (eg, queuing theory, optimisation, simulation and statistical analysis43–45) support decision-making by using reductionist models to quantify the potential impact of proposed actions. Although the purpose of quantitative analyses within OR is to inform decisions and improvement rather than to generate knowledge, the implementation of quantitative OR methods is low in healthcare.44 46–48 One possible explanation is that reductionist approaches used in isolation capture insufficient understanding of the nature and context of complex issues to give relevant and practical findings and fail to secure buy-in and acceptance from stakeholders.49 It has been argued that combining quantitative and qualitative OR methods can enhance the likelihood of beneficial adoption.49 50 Studies using qualitative and quantitative OR methods in healthcare applications have predominantly taken the form of problem-structuring followed by simulation modelling,36 51 52 and have largely been confined to single organisation settings.53

We set out to combine soft systems methodology and quantitative OR methods to facilitate and inform an improvement initiative concerning a complex care pathway spanning multiple sectors. Our approach was designed to support a stakeholder group in developing and agreeing targeted, evidence-informed recommendations for service improvement within a case study provided by an existing research project. Following Eden54 and others, we judged the effectiveness of our approach by the material impact it had on the process of producing the recommendations rather than the eventual implementation or system outcomes in the much longer term.55 We conclude by reflecting on the potential for complementary qualitative and quantitative OR methods to augment service design and quality improvement work involving services that span multiple sectors and professions.

Methods

Case study setting

Our case study setting was a UK research project investigating the outcomes and support services for infants discharged following intervention for congenital heart disease (CHD) using qualitative data regarding the experiences of families, health professionals and helpline staff and quantitative national audit data (see box 1 for further details).56–61 OR did not feature in the original research design, but was added to inform the process of analysing and using the data obtained to develop recommendations for improving services.

Box 1 Services for congenital heart disease and the programme of research in our case study.

Services for congenital heart disease in the UK

In the UK, paediatric cardiac surgery for congenital heart disease (CHD) is commissioned nationally and delivered in specialist tertiary centres. Following surgical or catheter intervention in these centres, patients are discharged home, sometimes via their secondary care hospital team. Children continue to receive follow-up outpatient care at the tertiary centre, and potentially in outreach clinics locally. Some may receive home monitoring facilitated by community nurses and/or specialist cardiac nurses at the tertiary centre. All infants and their parents have access to a local general practitioner (GP) and a health visitor (nurses or midwives with additional training as specialist community public health nurses). The local services are primarily commissioned through Clinical Commissioning Groups (CCGs).

The programme of research in our case study

The number and complexity of infants with CHD surviving intervention and requiring care in the community following discharge from the specialist surgical centres are increasing.62 A 2-year multidisciplinary programme of research was conducted in response to concerns over mortality levels in this patient population, variability in the provision of services at and following discharge, and barriers to care experienced by some families. It included conducting:

  • A systematic review of potential risk factors for unexpected deaths and emergency readmissions following hospital discharge;56

  • Statistical analyses of national CHD and paediatric intensive care audit datasets to develop a risk model of death or emergency readmission to intensive care in the year following hospital discharge;57

  • Interviews regarding care at or following discharge with: 20 parents of children who had either died following discharge or had been readmitted as an emergency to intensive care; 38 health professionals (from primary, secondary and tertiary organisations) and 10 helpline staff based at charitable organisations;58 60

  • An online discussion forum with parents of children with CHD regarding their experiences accessing support at or following discharge from a specialist hospital.

A key aim of the research was to use the information being generated to identify ways to improve the provision of care and support for this patient population in the discharge and postdischarge period.61

Methodological approach: combining qualitative and quantitative OR methods

Our approach was tailored to the multiple aspects of the problem and the multiple phases of OR involvement.50 63 Services for infants following discharge involve health professionals from many different organisations and backgrounds, so it was important to understand and incorporate a diversity of perspectives in order to enhance the feasibility and acceptability of the findings. Soft systems methodology (SSM)64 65 was deemed the most appropriate choice of problem-structuring method, given its focus on acknowledging and engaging multiple perspectives in developing interventions.33 66 It was also apparent that resource availability would constrain improvements, and so we explored how recommendations might be targeted at specific patient groups. Classification and regression tree (CART) analysis67 was selected as an appropriate quantitative OR technique for identifying patient groups with different risk profiles.

These complementary OR methods were interwoven,53 with the initial focus on designing and conducting data analysis to inform subsequent stakeholder decision-making (drawing on SSM to analyse qualitative data and CART to analyse quantitative data). SSM also provided an overarching framework for the stakeholder group's decision-making process, including a facilitated workshop at the end of the project. The CART analysis informed the participative decision-making in the final workshop through a graphical representation designed for that purpose.

Qualitative OR methods

In our case study, the key stakeholders were as follows:

An Advisory Group comprising patient group representatives; health professionals from three tertiary cardiac centres; representatives from primary and secondary care; academics in psychology, statistics, epidemiology and OR;

A Working Group convened to propose recommendations for service improvements, which comprised selected members of the Advisory Group and invited additional representatives from the tertiary, community and charitable sectors;

Participants of a facilitated workshop for parents of infants who had undergone intervention and subsequently either died after discharge or required emergency readmission to intensive care.

Box 2 outlines the principles by which problematic situations are conceived and approached in SSM, as described originally by Checkland and Poulter.65

Box 2. The principles of soft systems methodology (SSM).

Adapted from Checkland and Poulter.65

Principle 1 SSM is concerned with real-world ‘problematic situations’, that is, real situations which someone thinks need attention and action.

Principle 2 All thinking and talking about problematic situations will be conditioned by the worldviews of the people doing the thinking and talking.

Principle 3 Every problematic situation will contain people trying to act purposefully, with intent. This means that models of purposeful activity, in the form of systems models built to express a particular worldview, can be used as devices to explore the qualities and characteristics of any problematic situation.

Principle 4 Discussion and debate about such a situation can be structured by using the models in Principle 3 as a source of questions to ask about the situation.

Principle 5 Acting to improve a problematic situation entails finding, in the course of the discussion/debate in Principle 4, accommodations among different worldviews, that is, finding a version of the situation that people with different worldviews can live with.

Principle 6 The inquiry created by Principles 1–5 is, in principle, a never-ending process of learning.

Principle 7 Explicit organisation of the process which embodies Principles 1–6 enables and embodies conscious critical reflection about both the situation itself and also the thinking about it.

These principles underpinned the four defining actions of the SSM approach described below as undertaken in our case study. In line with standard SSM practice, these actions were conducted iteratively in a non-linear process.

SSM action 1—finding out about the situation (using Rich Pictures)

We first developed an account of the problem in the form of a ‘Rich Picture’ that captured key features of the services (eg, the people, processes, places, relationships and viewpoints involved), perceived issues (eg, barriers to accessing care) and the characteristics of possible improvements from a ‘systems thinking’ perspective.68 Our initial Rich Picture was based on narratives from interviews conducted in the case study research project (see box 1). We then added information relating to risk factors and important features of heterogeneity in this patient population that emerged from the systematic review and analyses of national audit datasets (see box 1). This version of the Rich Picture was used within a facilitated workshop of interviewed parents to share findings and explore potential service improvements from their perspectives. The Rich Picture was then augmented with learning from this workshop as well as evidence-informed suggestions for service improvements generated through a facilitated process by the Advisory Group. At this stage, prompts for guiding further discussions about quality improvement were explicitly added (eg, ‘which patients are prioritised for each intervention?’). The augmented Rich Picture was used to familiarise members of the Working Group with the features of the problem being addressed, and additional contributions from their perspectives were captured.

Another component of action 1 was the familiarisation of the operational researcher conducting the SSM (SC) with the problem setting in advance of, and continuing alongside, the group activity of developing the Rich Picture.

SSM action 2—building conceptual models (Root Definitions and Activity Diagrams)

The process of generating a Rich Picture highlighted the different motivations, priorities and constraints of health professionals across the organisations involved. These constitute different ‘worldviews’, which are important to articulate as part of the process of learning about and improving the situation. This involved constructing a structured statement called a ‘Root Definition’ for each of the worldviews identified from the interviews with health professionals. We used the SSM mnemonic ‘CATWOE’ as a way of systematically including information in each Root Definition about:

Customers (eg, infants undergoing cardiac intervention);

Actors (eg, health professionals);

Transformation process (eg, clinical content of the service);

Worldview;

Owners (eg, national specialist commissioning);

Environmental constraints (eg, shortage of time for staff).

Conceptual models called ‘Activity Diagrams’ were also constructed, each comprising a set of linked purposeful activities that encapsulate a particular worldview. Such models are not intended to describe completely the real world (since each is based only on a single worldview), but rather are devices to explore potential improvements in an organised way. Activity Diagrams for each worldview were initially constructed using the Root Definitions and narratives expressed in health professional interviews and then checked for face validity with the members of the Working Group.

SSM actions 3 and 4—using conceptual models to question the situation and defining action to improve

An operational researcher (SC) used the conceptual models to develop questions exploring where changes to services could be made and what the implications might be from different perspectives. Each of the conceptual models and their related questions were then discussed individually with a relevant health professional from the expert Working Group. In a parallel process, draft evidence-informed service improvements were generated by constructing a hyperframework of qualitative analyses, identifying archetypal service problems, and linking these to candidate recommendations using data generated in the research project (as described in ref. 61). As part of the SSM approach, a final workshop with the entire Working Group was convened in which the operational researcher facilitated a structured discussion about the feasibility and acceptability of the drafted service improvements from different worldviews, informed by learning captured in the Rich Picture, conceptual models and individual conversations about potentially conflicting viewpoints or processes. The aim of the workshop was to agree upon a set of recommendations for service improvements with broad stakeholder endorsement.

Quantitative OR methods

The research project in our case study included the development of a risk model for late adverse events (death or emergency readmission to intensive care in the year following hospital discharge) using linked national datasets.53 This was effective in generating knowledge about patient-level factors independently associated with these outcomes, but did not provide concrete information to guide the clinical community in improving services for this patient population. We therefore used complementary CART analysis designed to identify subgroups of patients that might benefit from different interventions because of the differing nature and scale of their risk. This resulted in the division of the population into six groups, each defined by a combination of risk factors known at the point of discharge such as non-cardiac comorbidity, ‘high-risk’ primary cardiac diagnosis and long length of stay in hospital. Details regarding the methods of analysis and the characteristics of these groups are presented elsewhere57; here, we focus on the process by which it fed into the development of the recommendations.

Specifically, a visual representation of the analysis was designed to illustrate, in an intuitive and engaging manner, the relative size of the patient groups alongside their risks of adverse outcome to inform the Working Group's thinking about how limited resources might be targeted most effectively. This visual representation was integrated within the SSM approach at the final workshop, in which the operational researcher facilitated the Working Group in considering the visual aid, the characteristics of the patient groups, the nature of the potential service improvements and findings from qualitative research where this provided additional information not captured in routinely collected national audit data.

Study ethics

The collection and analysis of quantitative and interview data and the generation of recommendations were conducted as part of a research project funded by the National Institute for Health Research, which had Research Ethics Committee approval. All interviewees provided informed consent.

Results

Qualitative OR

Figure 1 shows the Rich Picture, which was printed as posters approximately 85×60 cm. It captures key features of the entire patient journey including the different sectors that provide services, the health professionals involved and the interactions and flows of information between them. The family appear with their baby in all of the settings, and as they move through their journey, they express different concerns and issues that need addressing. Blue text highlights any information relating to risk factors, important features of heterogeneity in this patient population and prompts for guiding further discussions about quality improvement. The word ‘communication’ is highlighted in large red letters to reflect the widely held view that poor communication between health professionals and with families was a major system-wide problem. The ‘overseeing eye’ in the top right corner reflects one of the key concerns of the families, namely the lack of a single point of contact with responsibility for coordinating the entire care journey, which they identified as an important area for improvement. A separate cloud bubble shows aspects of the ‘wider context’ identified by stakeholders as affecting the situation and potential improvements, such as a national review of services for CHD being conducted at the time.

Figure 1.

Figure 1

Rich Picture generated in case study. The Rich Picture that was generated using a combination of interview data,53–55 data analysis57 and further engagement with stakeholders, including families and health professionals from different health service settings. PEC, paediatrician with expertise in cardiology; GPs, general practitioners; CLN, cardiac liaison nurse; MDT, multidisciplinary team; LOS, length of stay; HLHS, hypoplastic left heart syndrome; UVH, univentricular heart; PA, pulmonary atresia; INR, international normalised ratio (blood-monitoring machine); EWS, early warning system; S&S, safe and sustainable review; BCCA, British Congenital Cardiac Association.

Five key worldviews were identified and encapsulated in conceptual models corresponding to the services provided by: specialist children's cardiac centres; local hospitals; GPs; community nursing; health visitors. Figure 2 shows an example of a CATWOE and a Root Definition developed from the perspective of community nursing. These underpin the Activity Diagram for community nursing shown in figure 3, which shows the linked set of purposeful activities required to deliver the service described by the Root Definition. The five main activities are in bold, with related subactivities feeding into each of these. The activities in grey are external prompts that initiate the community nursing service. The full set of Root Definitions and Activity Diagrams are available as online supplementary material.

Figure 2.

Figure 2

CATWOE and Root Definition for community nursing. These examples of a CATWOE and Root Definition are from the perspective of community nursing (one of the five ‘world views’ we explored) and were developed from the narratives expressed in interviews with community nurses. These underpin the Activity Diagram shown in figure 3. CCG, Clinical Commissioning Group. CATWOE, mnemonic for systematically including information about customers, actors, transformation process, worldview, owners and environmental constraints.

Figure 3.

Figure 3

Activity Diagram for community nursing. This shows the linked set of purposeful activities required to deliver the community nursing services described in the Root Definition (figure 2). The five main activities are in bold, and the related subactivities feed into them. Activities in grey initiate the community nursing service.

supplementary methods

bmjqs-2016-005636supp001.pdf (743.1KB, pdf)

Quantitative OR

Figure 4 shows the graphical data summary that informed the process of prioritising service improvements. It shows the proportion of patients in each of the six groups and the proportion of adverse events among them, with groups ordered from left to right in decreasing risk of adverse events. When used in the workshop, a detailed key showed the combinations of risk factors known at the point of discharge that defined each group (see ref. 57 for further details).

Figure 4.

Figure 4

Graphical data summary used to inform prioritisation of service improvements. A graphical data summary developed to inform the decision process of prioritising service improvements. The patient groups were developed using classification and regression tree analysis of linked national UK congenital heart disease and paediatric intensive care audit datasets (see57 for details). Groups are ordered from left to right in decreasing risk of adverse events (death or emergency readmission to paediatric intensive care within a year after discharge from infant cardiac surgery). This graph was used in a workshop to guide the Working Group's discussions about targeting interventions at patient groups. When used in the workshop, a detailed key was included to describe the specific clinical features of each group (see57 for further details).

The Working Group used the graph to recommend multidisciplinary care teams only for children with long-term complex needs in addition to their cardiac diagnosis (patient groups 1, 2 and 5). It was also decided that formal home-monitoring programmes (surveillance at home involving saturation and weight monitoring) were only required from a medical perspective for babies with certain high-risk cardiac diagnoses (patient group 3). This was considered a small enough group to make this intervention, expensive at patient level, affordable. Another potential intervention with considerable resource implications was ‘step-down care’ in which patients are discharged from the specialist cardiac centre to their local hospital prior to going home. The Working Group's decision to target this at patient groups 1–4 was informed by recognising from figure 4 that over half of all adverse events were occurring in this 21% of patients.

As an aside, it is worth noting that qualitative evidence from the research study also fed into the decision-making process. For example, it was recommended that non-English speaking families and families with learning difficulties would benefit from more intensive provision of predischarge training and information, because findings from interviews with health professionals suggest they experience greater difficulties in accessing support.58 59

Combined OR

In the final workshop, the operational researcher drew on learning from the Rich Picture and conceptual models (Activity Diagrams) to guide structured decision-making about feasible and acceptable service improvements, and on the data visualisation to facilitate a process of targeting these at patient groups. The qualitative and quantitative OR approaches were therefore both integral to the decision-making process, and complementary in that neither approach could have fulfilled the role of the other. At the end of the workshop, the Working Group had agreed on a coherent set of targeted recommendations for service improvement deemed acceptable from each professional perspective as well as desirable and feasible from a system perspective (see ref. 61 for details of the recommendations themselves).

Discussion

Meaning of findings

In our case study, soft systems methodology provided a systematic process to explore the need for, and potential implications of, changes to services for infants with CHD. A Rich Picture was a source of shared learning and engagement for a broad range of stakeholders involved in the complex care pathway. It informed, and was used alongside, conceptual models that the operational researcher used to guide the systematic consideration of potential service change by an expert group. The design and graphical representation of quantitative data analysis helped stakeholders to consider which patient groups to target with which interventions, given resource constraints. The combination of qualitative and quantitative OR methods contributed to the establishment of a coproduced agenda for service improvement.61

Study findings (including the Rich Picture and graphical data summary) have been shared with wider interested parties and fed directly into a consultation on standards for CHD services by National Health Service England, and their commissioning of these through service specifications and a Quality Dashboard.69 The feasibility of implementing recommendations was a guiding consideration in this work, which we anticipate has enhanced the prospects of uptake. For example, perspectives from all of the services involved regarding issues of feasibility and acceptability were elicited, and notions of prioritisation with respect to limited resources (financial as well as time) were incorporated. The process resulted in a coherent set of endorsed recommendations acceptable to all parties (with acceptability determined via email following the workshop).

Strengths and weaknesses

The OR techniques described in this article augmented an existing 2-year funded programme of research to which a number of researchers, health professionals and families had already committed their time. This provided a focal point for applying OR iteratively and coherently over time within an existing decision process. In particular, the OR work benefited from evidence that was specifically developed with service improvement in mind during the case study research project.51–55 Our conceptual models focused on different worldviews with regard to the service providers, while the Rich Picture was structured around the patient journey and family perspective: the family and their baby are the only people who experience all parts of the system, and so the system-wide perspective that OR brought naturally chimed with a patient-centred view. No non-English speaking families participated in the facilitated workshop, but recommendations relating specifically to non-English speaking families were informed by the qualitative evidence from the case study research project.

The OR approach in this case study was effective in the sense that it had a material impact on the process of developing recommendations. This impact was most tangible in the targeting of recommendations at patient risk groups defined by the quantitative OR analysis. As others have noted,55 it is more challenging to isolate the specific influences of problem-structuring methods, as they are complex social interventions. Our own reflection is that, in the absence of an articulated process for considering service improvement systematically in the original research proposal, the SSM was pivotal in providing an overarching framework for doing so. Given the broad endorsement of the ensuing recommendations and their impact on national service standards and commissioning in this area, we infer a degree of effectiveness in the delivery of the SSM framework.

A more comprehensive evaluation of the combined methods would be very instructive. While systematic, there is an element of craft skill involved in conducting the analyses; so, it would also be interesting to probe the extent to which the effectiveness was a result of the OR methods per se, as opposed to personal attributes of the practitioner, the dynamics of the project team or the particular context of the area of application. The roles of these influences may need to be better understood for OR methods to be applied effectively at scale.

We also note that the focus of OR in this research, and our notion of its effectiveness, was its contribution to the robustness of evidence-informed recommendations for service change rather than any downstream impact of the recommendations.

Implications for policy-makers and clinicians

The combination of qualitative and quantitative OR analyses helped to generate findings of direct practical relevance to clinicians, policy-makers and commissioners regarding a complex service spanning multiple organisations.

OR is a problem-focused discipline in which method selection is guided by the features of the particular problem and decision processes at hand, with flexibility to customise the blend of different quantitative and qualitative approaches. Our case study demonstrates that this warrants further consideration as an approach, for example, by bodies such as the Centre for Clinical Practice at the National Institute for Health and Care Excellence, which is responsible for developing guidance focused on the organisation and delivery of UK healthcare services.

Future work

The recommendations for improving services for infant CHD included suggestions for further research, implementation and evaluation (see61). More broadly, further research is required to understand how and in what circumstances similar combinations of OR methods could be used effectively in other areas of healthcare,53 and whether there are viable metrics to assess and inform the support for service change. This could build on recent attempts within the OR literature to develop frameworks for evaluating problem-structuring methods (eg 55) and mixed OR methods within decision support.50

Further work to understand how the social sciences (which generate deep understanding) and OR (which tailors understanding to inform decision processes) could be used in a complementary multidisciplinary approach towards quality improvement would also be beneficial.

Acknowledgments

The authors thank all of the parents and health professionals who contributed to the workshops and shared their perspectives. SC would also like to thank Jess Flood-Paddock for her artistic contribution to the Rich Picture.

Footnotes

Contributors: SC contributed to the design and conception of the study, the analysis and interpretation of data, and drafting of the work. KB, JT, JW, RK, DAR and CB contributed to the interpretation of data. MU contributed to the design of the study and drafting of the work. All authors revised the work critically for important intellectual content. SC is the guarantor of the paper and takes responsibility for the integrity of the work as a whole, from inception to published article.

Funding: SC was supported by the Health Foundation, an independent charity working to continuously improve the quality of healthcare in the UK. KB, JT, JW, RK, DAR and CB were funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme (Project No: 10/2002/29). The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the NIHR HS&DR programme or the Department of Health. MU was supported by the NIHR Collaboration for Leadership in Applied Health Research and Care (CLAHRC) North Thames at Bart's Health NHS Trust. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Competing interests: KB declares participation on the steering committee of the UK National Congenital Heart Diseases Audit.

Provenance and peer review: Not commissioned; externally peer reviewed.

References

  • 1.Schoen C, Osborn R, Squires D, et al. . New 2011 survey of patients with complex care needs in eleven countries finds that care is often poorly coordinated. Health Aff (Millwood) 2011;30:2437–48. 10.1377/hlthaff.2011.0923 [DOI] [PubMed] [Google Scholar]
  • 2.Göbel B, Zwart D, Hesselink G, et al. . Stakeholder perspectives on handovers between hospital staff and general practitioners: an evaluation through the microsystems lens. BMJ Qual Saf 2012;21(Suppl 1):i106–13. 10.1136/bmjqs-2012-001192 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Johnson JK, Farnan JM, Barach P, et al. . Searching for the missing pieces between the hospital and primary care: mapping the patient process during care transitions. BMJ Qual Saf 2012;21(Suppl 1):i97–105. [DOI] [PubMed] [Google Scholar]
  • 4.Romagnoli KM, Handler SM, Ligons FM, et al. . Home-care nurses’ perceptions of unmet information needs and communication difficulties of older patients in the immediate post-hospital discharge period. BMJ Qual Saf 2013;22:324–32. 10.1136/bmjqs-2012-001207 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Jones CD, Vu MB, O'Donnell CM, et al. . A failure to communicate: a qualitative exploration of care coordination between hospitalists and primary care providers around patient hospitalizations. J Gen Intern Med 2014;30:417–24. 10.1007/s11606-014-3056-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Kripalani S, LeFevre F, Phillips CO, et al. . Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care. JAMA 2007;297:831 10.1001/jama.297.8.831 [DOI] [PubMed] [Google Scholar]
  • 7.Davis MM, Devoe M, Kansagara D, et al. ‘Did I do as best as the system would let me?’ Healthcare professional views on hospital to home care transitions. J Gen Intern Med 2012;27:1649–56. 10.1007/s11606-012-2169-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Chen AH, Yee HF. Improving the primary care–specialty care interface: getting from here to there. Arch Intern Med 2009;169:1024 10.1001/archinternmed.2009.140 [DOI] [PubMed] [Google Scholar]
  • 9.Godlee F. Integrated care is what we all want. BMJ 2012;344:e3959 10.1136/bmj.e3959 [DOI] [Google Scholar]
  • 10.Fulop N, Mowlem A, Edwards N. Building integrated care: lessons from the UK and elsewhere. London: NHS Confederation, 2005. http://www.nhsconfed.org/~/media/Confederation/Files/Publications/Documents/Building%20integrated%20care.pdf (accessed 17 Jul 2015). [Google Scholar]
  • 11.Iacobucci G. NHS England announces 29 sites to spearhead integrated care models. BMJ 2015;350:h1362 10.1136/bmj.h1362 [DOI] [PubMed] [Google Scholar]
  • 12.Vize R. Integrated care: a story of hard won success. BMJ 2012;344:e3529 10.1136/bmj.e3529 [DOI] [PubMed] [Google Scholar]
  • 13.Royal College of Physicians. Care closer to home: narrative report London: Royal College of Physicians, 2012. http://www.rcplondon.ac.uk/resources/care-closer-home (accessed 17 Jul 2015). [Google Scholar]
  • 14.Royal College of Nursing. Moving care to the community: an international perspective. London: Royal College of Nursing, 2013. http://www.rcn.org.uk/__data/assets/pdf_file/0006/523068/12.13_Moving_care_to_the_community_an_international_perspective.pdf (accessed 17 Jul 2015). [Google Scholar]
  • 15.Craig P, Dieppe P, Macintyre S, et al. . Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655 10.1136/bmj.a1655 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Campbell NC, Murray E, Darbyshire J, et al. . Designing and evaluating complex interventions to improve health care. BMJ 2007;334:455–9. 10.1136/bmj.39108.379965.BE [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Medical Research Council. Developing and evaluating complex interventions: new guidance. London: MRC, 2008. http://www.mrc.ac.uk/complexinterventionsguidance. (accessed 17 Jul 2015). [Google Scholar]
  • 18.The Centre for Clinical Practice at NICE . Interim methods guide for developing service guidance 2014: Process and methods guide. http://www.nice.org.uk/article/pmg8 (accessed 17 Jul 2015).
  • 19.Naylor MD, Kurtzman ET, Grabowski DC, et al. . Unintended consequences of steps to cut readmissions and reform payment may threaten care of vulnerable older adults. Health Aff (Millwood) 2012;31:1623–32. 10.1377/hlthaff.2012.0110 [DOI] [PubMed] [Google Scholar]
  • 20.Roland M, Abel G. Reducing emergency admissions: are we on the right track? BMJ 2012;345:e6017. [DOI] [PubMed] [Google Scholar]
  • 21.Dixon-Woods M, McNicol S, Martin G. Ten challenges in improving quality in healthcare: lessons from the Health Foundation's programme evaluations and relevant literature. BMJ Qual Saf 2012;21:876–84. 10.1136/bmjqs-2011-000760 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.O'Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy 2008;13:92–8. 10.1258/jhsrp.2007.007074 [DOI] [PubMed] [Google Scholar]
  • 23.Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy 2005;10(Suppl 1):6–20. 10.1258/1355819054308576 [DOI] [PubMed] [Google Scholar]
  • 24.Brown C, Hofer T, Johal A, et al. . An epistemology of patient safety research: a framework for study design and interpretation (Part 1). Conceptualising and developing interventions. Qual Saf Health Care 2008;17:158–62. 10.1136/qshc.2007.023630 [DOI] [PubMed] [Google Scholar]
  • 25.Brown C, Lilford R. Evaluating service delivery interventions to enhance patient safety. BMJ 2008;337(:a2764. [DOI] [PubMed] [Google Scholar]
  • 26.Shaw S, Rosen R, Rumbold B. . What is integrated care? An overview of integrated care in the NHS. Nuffield Trust 2011. http://www.nuffieldtrust.org.uk/sites/files/nuffield/publication/what_is_integrated_care_research_report_june11_0.pdf. (accessed 24 Jul 2015).
  • 27.Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995;311:376–80. 10.1136/bmj.311.7001.376 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Murphy MK, Black NA, Lamping DL, et al. . Consensus development methods, and their use in clinical guideline development. Health Technol Assess Winch Engl 1998;2:i–iv, 1–88. [PubMed] [Google Scholar]
  • 29.Campbell SM, Braspenning J, Hutchinson A, et al. . Research methods used in developing and applying quality indicators in primary care. BMJ 2003;326:816–19. 10.1136/bmj.326.7393.816 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLOS Med 2011;8:e1000393 10.1371/journal.pmed.1000393 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Foth T, Efstathiou N, Vanderspank-Wright B, et al. . The use of Delphi and Nominal Group Technique in nursing education: a review. Int J Nurs Stud 2016;60:112–20. 10.1016/j.ijnurstu.2016.04.015 [DOI] [PubMed] [Google Scholar]
  • 32.Franco LA, Montibeller G. Facilitated modelling in operational research. Eur J Oper Res 2010;205:489–500. 10.1016/j.ejor.2009.09.030 [DOI] [Google Scholar]
  • 33.Rosenhead J, Mingers J. Rational analysis for a problematic world revisited: problem structuring methods for complexity, uncertainty and conflict. 2nd edn John Wiley & Sons, 2001. [Google Scholar]
  • 34.Mingers J, Rosenhead J. Problem structuring methods in action. Eur J Oper Res 2004;152:530–54. 10.1016/S0377-2217(03)00056-0 [DOI] [Google Scholar]
  • 35.Phillips L, Bana e Costa C. Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing. Ann Oper Res 2007;154:51–68. 10.1007/s10479-007-0183-3 [DOI] [Google Scholar]
  • 36.Kotiadis K, Tako AA, Vasilakis C. A participative and facilitative conceptual modelling framework for discrete event simulation studies in healthcare. J Oper Res Soc 2013;65:197–213. 10.1057/jors.2012.176 [DOI] [Google Scholar]
  • 37.Ackermann F, Eden C. Strategic options development and analysis. In: Reynolds M, Holwell S, eds. Systems approaches to managing change: a practical guide. Springer; 2010:135–90. [Google Scholar]
  • 38.Friend J, Hickling A. Planning under pressure. Routledge, 2012:413. [Google Scholar]
  • 39.Checkland P, Poulter J. Soft systems methodology. In: Reynolds M, Holwell S, eds. Systems approaches to managing change: a practical guide. Springer; 2010:191–242. [Google Scholar]
  • 40.Alberto Franco L. Problem structuring methods as intervention tools: reflections from their use with multi-organisational teams. Omega 2009;37:193–203. 10.1016/j.omega.2006.08.001 [DOI] [Google Scholar]
  • 41.Vandenbroeck P, Dechenne R, Becher K, et al. . Recommendations for the organization of mental health services for children and adolescents in Belgium: use of the soft systems methodology. Health Policy Amst Neth 2014;114:263–8. 10.1016/j.healthpol.2013.07.009 [DOI] [PubMed] [Google Scholar]
  • 42.Sinclair E, Radford K, Grant M, et al. . Developing stroke-specific vocational rehabilitation: a soft systems analysis of current service provision. Disabil Rehabil. 2014;36:409–17. 10.3109/09638288.2013.793410 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Hulshof PJH, Kortbeek N, Boucherie RJ, et al. . Taxonomic classification of planning decisions in health care: a structured review of the state of the art in OR/MS. Health Syst 2012;1:129–75. 10.1057/hs.2012.18 [DOI] [Google Scholar]
  • 44.Brailsford SC, Harper PR, Patel B, et al. . An analysis of the academic literature on simulation and modelling in health care. J Simul 2009;3:130–40. 10.1057/jos.2009.10 [DOI] [Google Scholar]
  • 45.Pitt M, Monks T, Crowe S, et al. . Systems modelling and simulation in health service design, delivery and decision making. BMJ Qual Saf 2016;25:38–45. 10.1136/bmjqs-2015-004430 [DOI] [PubMed] [Google Scholar]
  • 46.Fone D, Hollinghurst S, Temple M, et al. . Systematic review of the use and value of computer simulation modelling in population health and health care delivery. J Public Health 2003;25:325–35. 10.1093/pubmed/fdg075 [DOI] [PubMed] [Google Scholar]
  • 47.Harper PR, Pitt MA. On the challenges of healthcare modelling and a proposed project life cycle for successful implementation. J Oper Res Soc 2004;55:657–61. 10.1057/palgrave.jors.2601719 [DOI] [Google Scholar]
  • 48.Eldabi T, Paul RJ, Young T. Simulation modelling in healthcare: reviewing legacies and investigating futures. J Oper Res Soc 2007;58:262 10.1057/palgrave.jors.2602222 [DOI] [Google Scholar]
  • 49.Sachdeva R, Williams T, Quigley J. Mixing methodologies to enhance the implementation of healthcare operational research. J Oper Res Soc 2007;58:159 10.1057/palgrave.jors.2602293 [DOI] [Google Scholar]
  • 50.Franco LA, Lord E. Understanding multi-methodology: Evaluating the perceived impact of mixing methods for group budgetary decisions. Omega 2011;39:362–72. 10.1016/j.omega.2010.06.008 [DOI] [Google Scholar]
  • 51.Tako AA, Kotiadis K. PartiSim: A multi-methodology framework to support facilitated simulation modelling in healthcare. Eur J Oper Res 2015;244:555–64. 10.1016/j.ejor.2015.01.046 [DOI] [Google Scholar]
  • 52.Holm LB, Dahl FA, Barra M. Towards a multimethodology in health care—synergies between Soft Systems Methodology and Discrete Event Simulation. Health Syst. 2012;2:11–23. 10.1057/hs.2012.21 [DOI] [Google Scholar]
  • 53.Howick S, Ackermann F. Mixing OR methods in practice: Past, present and future directions. Eur J Oper Res 2011;215:503–11. 10.1016/j.ejor.2011.03.013 [DOI] [Google Scholar]
  • 54.Eden C. Problem Construction and the Influence of O.R. Interfaces 1982;12:50–60. 10.1287/inte.12.2.50 [DOI] [Google Scholar]
  • 55.Midgley G, Cavana RY, Brocklesby J, et al. . Towards a new framework for evaluating systemic problem structuring methods. Eur J Oper Res 2013;229:143–54. 10.1016/j.ejor.2013.01.047 [DOI] [Google Scholar]
  • 56.Tregay J, Wray J, Bull C, et al. . Unexpected deaths and unplanned re-admissions in infants discharged home after cardiac surgery: a systematic review of potential risk factors. Cardiol Young 2015;25:839–52. 10.1017/S1047951114002492 [DOI] [PubMed] [Google Scholar]
  • 57.Crowe S, Ridout D, Knowles R, et al. . Death and emergency readmission of infants discharged after interventions for Congenital Heart Disease: a national study of 7643 infants to inform service improvement. J Am Heart Assoc 2016;5:e003369 10.1161/JAHA.116.003369 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58.Tregay J, Wray J, Crowe S, et al. . Going home after infant cardiac surgery: a UK qualitative study. Arch Dis Child 2016; 2016;101:320–5. 10.1136/archdischild-2015-308827 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Tregay J, Brown KL, Crowe S, et al. . Signs of deterioration in infants discharged home following congenital heart surgery in the first year of life: a qualitative study. Arch Dis Child 2016. t;101:902–8. 10.1136/archdischild-2014-308092 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Tregay J, Brown K, Crowe S, et al. ‘I was so worried about every drop of milk’—feeding problems at home are a significant concern for parents after major heart surgery in infancy. Matern Child Nutr 2016; http://dx.doi.org/10.1111/mcn.12302 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Crowe S, Knowles RL, Wray J, et al. . Identifying improvements to complex pathways: evidence synthesis and stakeholder engagement in infant congenital heart disease. BMJ Open 2016;6:e010363; 10.1136/bmjopen-2015-010363 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62.Brown K, Crowe S, Franklin R, et al. . Trends in 30-day mortality rate and case mix for paediatric cardiac surgery in the UK between 2000 and 2010. Open Heart 2015;2: 1e000157 10.1136/openhrt-2014-000157 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 63.Mingers J. Variety is the spice of life: combining soft and hard OR/MS methods. Int Trans Oper Res 2000;7:673–91. 10.1111/j.1475-3995.2000.tb00224.x [DOI] [Google Scholar]
  • 64.Checkland P. Soft systems methodology: a thirty year retrospective. Syst Res Behav Sci 2000;17:S11–58. [DOI] [Google Scholar]
  • 65.Checkland P, Poulter J. Learning for action: a short definitive account of Soft Systems Methodology and its use for practitioners, teachers and students. 1st edn Hoboken, NJ: John Wiley & Sons, 2007. [Google Scholar]
  • 66.Heyrani A, Maleki M, Marnani AB, et al. . Clinical governance implementation in a selected teaching emergency department: a systems approach. Implement Sci 2012;7:84 10.1186/1748-5908-7-84 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 67.Breiman L, Friedman J, Olshen R, et al. . CART: Classification and regression trees. Wadsworth Belmont, CA, 1983;156. [Google Scholar]
  • 68.Checkland P. Systems thinking, systems practice: includes a 30 year retrospective. New Ed edition Chichester; NY: John Wiley & Sons, 1999. [Google Scholar]
  • 69.Georgieva I, Thompson I, Triggs H, et al. Consultation on draft standards and service specifications for congenital heart disease services—A report to NHS England. Dialogue by Design 2015. http://www.dialoguebydesign.co.uk/project/nhs-england-consultation-draft-standards-service-specifications-congenital-heart-disease-services/ (accessed 26 Jul 2015).

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

supplementary methods

bmjqs-2016-005636supp001.pdf (743.1KB, pdf)


Articles from BMJ Quality & Safety are provided here courtesy of BMJ Publishing Group

RESOURCES