Realist evaluation of a complex integrated care programme: protocol for a mixed methods study

Abstract

Introduction

The lack of understanding of how complex integrated care programmes achieve their outcomes due to the lack of acceptable methods leads to difficulties in the development, implementation, adaptation and scaling up of similar interventions. In this study, we evaluate an integrated care network, the National University Health System (NUHS) Regional Health System (RHS), consisting of acute hospitals, step down care, primary care providers, social services and community partners using a theory-driven realist evaluation approach. This study aims to examine how and for whom the NUHS-RHS works to improve healthcare utilisations, outcomes, care experiences and reduce healthcare costs. By using a realist approach that balances the needs of context-specific evaluation with international comparability, this study carries the potential to address current research gaps.

Methods and analysis

This evaluation will be conducted in three research phases: (1) development of initial programme theory (IPT) underlying the NUHS-RHS; (2) testing of programme theory using empirical data; and (3) refinement of IPT. IPT was elicited and developed through reviews of programme documents, informal discussions and in-depth interviews with relevant stakeholders. Then, a convergent parallel mixed method study will be conducted to assess context (C), mechanisms (M) and outcomes (O) to test the IPT. Findings will then be analysed according to the realist evaluation formula of CMO in which findings on the context, mechanisms will be used to explain the outcomes. Finally, based on findings gathered, IPT will be refined to highlight how to improve the NUHS-RHS by detailing what works (outcome), as well as how (mechanisms) and under what conditions (context).

Ethics and dissemination

The National Healthcare Group, Singapore, Domain Specific Review Board reviewed and approved this study protocol. Study results will be published in international peer-reviewed journals and presented at conferences and internally to NUHS-RHS and Ministry of Health, Singapore.

Strengths and limitations of this study.

• This study is one of the first studies to holistically evaluate a new integrated care network in the context of Southeast Asia.

• Using a theory-driven realist evaluation approach, findings from this study are expected to generate contextually relevant evidence for improving efficiency and effectiveness of integrated care as well as informing policy decisions in Singapore and similar health systems.

• The use of mixed methods in this study allows us to draw on the strengths of both qualitative and quantitative methods, enhancing the credibility of the evaluation findings and allows for generation of in-depth insights.

• Inclusion of relevant stakeholders in the design of the evaluation provides confidence that the evaluation efforts are relevant and findings are likely to be adopted for improvement in current models and for development of future initiatives.

• Realist evaluation and mixed methods design are relatively new methods in comparison with the randomised controlled trials (RCTs) commonly used in the evaluation of medical interventions. Therefore, this study design may draw criticism for not being as rigorous as the typical RCT.

Introduction

Across the world, rapidly ageing populations and increasing demand for healthcare services call for a paradigm shift from disease-centred healthcare provision to a more holistic, people-centred care.¹ People-centred care involves placing people and communities at the focus of health systems, not diseases. It highlights the importance of empowering people to take charge of their health, takes into account holistic needs of a person and seeks to provide quality care timely and appropriately.² Integrated care supports people-centred care by coordinating healthcare services in a way that ensures people receive a continuum of care at the different levels and sites of care within the health system, according to their needs. The WHO’s framework of people-centred integrated care describes the complexity of such interventions and the need to involve various stakeholders in their execution. Complex multicomponent delivery strategies are also typically recommended.¹

As characterised by Valentijn et al³, care integration can take place on many dimensions: at the micro-level clinical level, the meso-level professional or organisational level and the macro or systemic level. The types of interventions that qualify under the wide umbrella of ‘integrated care’ vary tremendously, as do the settings in which they are introduced and their attendant benefits and costs. Moreover, introducing integrated care often calls for complex, multicomponent programmes that involve many stakeholders, sometimes with conflicting perspectives and interests. When implemented successfully, integrated people-centred health services can be more effective, cost less, improve patient engagement and better prepared to respond to increasing demands for healthcare services.⁴

However, despite these growing needs and the availability of basic design principles related to integrated care, the development and implementation of such care models remains challenging. In their systematic review, Suter et al identified 10 universal principles for successful health systems integration including: (1) comprehensive services across the continuum of care, (2) person-centredness, (3) collaboration between organisations, (4) standardised care delivery through interprofessional teams, (5) performance management, (6) information systems, (7) organisational culture, (8) professional integration, (9) good governance and (10) financial management.⁵ Furthermore, it is acknowledged that successful implementation of integrated care requires an effective composition of interventions at the micro-level, meso-level and macro-level.^{6 7} However, the multifaceted interplay between the building blocks of integrated care and the influence of various contextual factors on outcomes render the development of a simple and standardised implementation model impossible.⁸ In every instance, there is a need for healthcare providers and organisations to understand which care models work, for whom and how they work in the unique setting in which integrated care is planned for a particular population so as to facilitate implementation and ensure longer term sustainability.⁹

Current evaluations such as the ‘gold-standard’ randomised controlled trial (RCT) design rarely adequately or even explicitly address the context-specific drivers behind implementation outcomes and their relationship to the underlying programme theory, making it difficult to interpret their findings in light of other programmes in different settings. As a result, few evaluation strategies are widely accepted as appropriate, and to date, the net benefit of integrated care interventions and understanding of how variable outcomes are achieved remains empirically uncertain. It is therefore essential to develop comprehensive, rigorous and practical methods to evaluate people-centred integrated care programmes, to inform the selection of effective and efficient interventions and to facilitate improvement and scaling-up. In the evaluation of such complex interventions, the Medical Research Council (MRC) argues for the importance of process evaluation in conjunction with outcome evaluation, to account for variability in implementation.¹⁰ The MRC’s process evaluation framework guides evaluators to understand the implementation processes (what is implemented and how), mechanisms of intervention (how the delivery of the intervention produces change) and contextual factors that affect implementation and outcomes.¹¹

Singapore is typical of many other countries with a rapidly ageing population that is accompanied by an increasing prevalence of chronic and complex illnesses.¹² Designed with an emphasis on providing episodic care within acute hospitals in a largely disease-specific manner and controlling infectious disease in a young population, Singapore’s health system now faces the challenge of ensuring appropriate care and long-term fiscal sustainability for a long-lived population at increasing risk of multiple chronic diseases. In addition, having already reached the highest levels of efficiency in the world,¹³ simple or easy fixes to increase performance while keeping costs low are increasingly rare. A radical change in vision and strategy is therefore needed, requiring complex systems-level interventions that bring about changes of organisational, policy, power and financing structures and break down existing siloes within the healthcare system to refocus on prevention, primary care and community-based management. Such interventions will need to be built on a strong foundation of integrated care.

In 2012, the Ministry of Health (MOH) Singapore launched a major initiative to reorganise healthcare at the national level into six clusters or regional health systems (RHSs) to foster care integration.¹⁴ This was recently reorganised in 2017 into three integrated clusters to better meet future healthcare needs.¹⁵ Every RHS comprises of a network each led by a major public hospital working in close partnership with healthcare providers (primary care providers, community hospitals, nursing homes, home care and day rehabilitation providers) and social care providers (including senior activity centres, grassroots organisations and social service offices) within the same geographical region. Each RHS has the mandate and funding support to design and implement integrated care programmes that leverage this network to provide healthcare beyond the hospital to the community, value-driven healthcare and holistic care across the entire care continuum in a cost-effective way.¹⁶ To support this common vision, every RHS was tasked to implement programmes identified by the MOH to be of priority and held accountable for same outcomes jointly agreed on between the MOH and the RHSs throughout the country. Following this lead, each RHS implemented the priority programmes tailored to the unique needs and demographic of the population in which it serves as well as the different strengths of community partners within its network. Other programmes unique to respective RHS were also developed and implemented accordingly.

In this study, we propose to take advantage of a unique opportunity to document and evaluate the formation and implementation of a multipillared, regional integrated care network. This evaluation study began in June 2016 at the National University Hospital System (NUHS)-RHS, the RHS at the western part of Singapore. This study aims to examine how and for whom the NUHS-RHS, as an integrated care network, works from healthcare providers’ and healthcare users’ perspectives to improve healthcare utilisation, health outcomes and care experiences as well as to reduce healthcare costs.

Methods and analysis

Considering the developmental nature of the NUHS-RHS, this evaluation is designed primarily as a formative evaluation that incorporates outcome valuation principles.¹⁷ The evaluation findings will be used to facilitate modifications to existing NUHS-RHS programmes as well as to provide evidence to support the increased likelihood of future success.

The evaluation will be conducted using a realist evaluation approach,¹⁸ which seeks to test and refine the programme theory while assessing whether and how the programme succeeds in the local setting, in order to generate important insights for Singapore and for the wider field of integrated care research. Realist evaluation is increasingly applied in the evaluation of complex healthcare interventions as it seeks to provide a more explicit and in-depth understanding of what works, for whom and in what circumstances and has been recommended for the evaluation of integrated care interventions.^{19 20} It is a theory-driven approach in which interventions are assumed to be based on theories but are also active, flexible to changes and embedded in a social reality that influences how the intervention is implemented and how various actors in that reality respond to it.¹⁸ A realist programme theory specifies which outcomes are linked to the intervention and what mechanisms generate the outcomes and what features of the context affect them. The context–mechanism–outcome (CMO) configuration is used as the main structure for analysis, with the goal of identifying both mechanisms (what and how components of interventions result in changes) and contextual factors (features of the conditions that influence the mechanisms of interventions) are associated with variation in outcomes.^{21 22} Pawson and Tilley²¹ argue that an intervention can only achieve successful outcomes if the appropriate ideas are applied to the right context with appropriate social and cultural conditions. A realist evaluation therefore includes a theory-driven formative evaluation,²³ process evaluation¹⁰ as well as outcomes evaluation and avoids the rigorously successionist format of experimental design.

This evaluation will be conducted in three research phases according to the framework for realist evaluation outlined by Pawson and Tilley (figure 1)²¹: (1) development of initial programme theory (IPT) underlying the NUHS-RHS; (2) testing of programme theory using empirical data; and (3) refinement of IPT.

Figure 1.

Realist evaluation processes and research phases according to Pawson an d Tilley. CMO, context, mechanism and outcome; NUHS-RHS, National University Hospital System-Regional Health System.

Phase 1: development of IPT

This evaluation began with first eliciting the IPT which subsequently formed the basis of the evaluation, focus the evaluation activities and determined the appropriate study design, data collection and analysis methods. First, program-related documents of the NUHS-RHS that describe the rationales, components of interventions and programme protocols were reviewed to identify the underlying assumptions about how the respective programmes are expected to work to achieve their intended outcomes. Ethnographic observations were then conducted to verify and obtain additional information to comprehensively describe the theory of change underlying the NUHS-RHS programmes.

Subsequently, key stakeholders involved in the planning, development and implementation of the NUHS-RHS programmes were engaged through group discussions to identify: (1) factors that influence the implementation of the programmes, (2) initial observations on how and for whom the programmes work, (3) objectives of evaluation and (4) outcomes that are relevant to the stakeholders. In collaboration with the RHS strategic planning office, a logic model (figure 2), a tool that describes logical linkages among programme resources, activities and intended outputs, audiences and short-term, intermediate-term and long-term outcomes²⁴ related to the establishment of NUHS-RHS was drafted to depict IPT underlying the NUHS-RHS. To ensure the accuracy of the IPT, the logic model was circulated to stakeholders who were previously engaged for further inputs. After which, refinement to the IPT was made.

Figure 2.

Logic model of NUHS-RHS. NUHS-RHS, National University Hospital System-Regional Health System.

As illustrated under activities on figure 2, programmes (P1–6) describes the MOH priority programmes implemented nationally. The NUHS-RHS Integrated Interventions and Care Extension (P1) programme aims to help patients with complex conditions who were admitted at least three or more times a year through a holistic case management. NUHS Transition Care Programme (P2) was implemented to enable patients to transit smoothly from hospital to homes through a multidisciplinary team to ensure continuity of care. Appropriate Sitting of Care (P3 and P4) aims to ensure appropriate sitting of patients who are medically stable and deemed to not required specialist care from acute hospital to the community through partnership with primary and community care providers. Health Promotion and Disease Management (P5) focuses on health education, early detection of chronic diseases and timely interventions to prevent development of chronic illnesses as well as slow down the deterioration of diseases. Strengthening of primary and community care within the NUHS-RHS (P6) is achieved through cross deployment of skilled manpower from the acute hospital to the partnering primary and community care providers so as to improve the capability and confidence of the partners of the NUHS-RHS. Leveraging on telehealth and IT enablers, the Carehub aims to provide a single point of contact through individual’s continuum of care posthospital discharge. The Primary Care Network brings together a group of family physicians and community partners to create a support network for patients with chronic medical conditions in the community.²⁵

Phase 2: testing of programme theory using empirical data

Prioritisation of NUHS-RHS programmes for testing of programme theory

As the number of programmes undertaken under the purview of the NUHS-RHS is large, programmes were selected for the testing of the IPT in consultation with stakeholders including healthcare providers and administrators, based on: (1) maturity of programme, (2) urgency for evaluation, (3) complexity of programmes and (4) scientific importance.

Programme maturity was assessed based on the duration of programme implementation. A programme that had been implemented for a longer duration was considered to more mature and given higher priority than one which was recently implemented. Programmes nearing the end of their funding period were also given higher priority for evaluation, as evaluation was essential to inform the future development of the programme. Complexity was determined by assessing the number of dimensions of integrated care as defined previously (clinical, professional, organisational, systematic, normative and functional integration).³ A programme that set out to integrate care in more dimensions was considered to be more complex than one which integrated care in fewer dimensions.

Finally, scientific importance was examined by a knowledge gap analysis conducted by reviewing available published articles relevant to these programmes on PubMed. A programme with the least number of relevant articles was considered to be of greatest scientific importance. According to these criteria, MN and HJMV ranked various NUHS-RHS programmes as low, medium and high priority for evaluation. Through the priority-setting process, four programmes (P1, P2, P3 and P4) were identified to be the best candidate programmes for evaluation.

Study participants

Two distinct groups of study participants—healthcare providers/managers and healthcare users—are selected based on defined criteria in this study. Healthcare providers/managers who are a part of the governance, planning and implementation of the programmes are invited to take part in structured interviews and to complete the online Rainbow Model of Integrated Care (RMIC) survey,²⁶ while healthcare users, including patients or proxies (caregivers), are invited for structured interviews or prospective surveys over three time points. Interactions between healthcare providers and healthcare users will be observed.

Study design

Given the complexity of integrated care programmes being evaluated and a range of perspectives in which this study tries to capture, a convergent parallel mixed methods study will be undertaken to test the IPT developed in phase 1. Using the convergent parallel strategy, both quantitative and qualitative data will be collected concurrently. Components will be given equal weight, and two datasets will be analysed, compared and merged through iterative cycles of validation and confirmation of findings.²⁷

Evaluation of implementation, context and mechanisms

Implementation fidelity, context and mechanisms that underpin the NUHS-RHS as a whole will be assessed according to the modified version of the Conceptual Framework of Implementation Fidelity (CFIF).²⁸ The CFIF provides a framework to assess content, frequency and dose of programme implementation and the moderating factors affecting implementation fidelity of complex interventions. Moderating factors including participants’ responsiveness, intervention complexity, comprehensiveness of policy description, strategies to facilitate implementation, quality of delivery, recruitment and context will be assessed. Data will be gathered through reviews of programme documents, medical records, structured interviews healthcare providers and observations of actual delivery of interventions as described in table 1.

Table 1.

Evaluation of implementation fidelity, context and mechanism underlying NUHS-RHS programmes

	Specific component	Data sources
Adherence	Content	Ethnographic observation, medical records review and interview with programme team members
	Frequency/duration	Medical records review, interview with programme team members and healthcare users
	Coverage	Programme databases, interview with programme team members
Moderating factors	Participant responsiveness	Ethnographic observation, interview with programme team members and healthcare users and possibly medical records review (if recorded)
	Intervention complexity	Programme documents (protocols and guidelines)
	Comprehensiveness of policy description	Programme documents (protocols and guidelines)
	Strategies to facilitate implementation	Interview with programme team members
	Quality of delivery	Interview with healthcare users
	Recruitment	Programme database, interview with programme team members
	Context	Interview with programme team members

NUHS-RHS, National University Hospital System-Regional Health System.

Programme documents including guidelines and databases will be reviewed by evaluators. Programme guidelines will be examined to provide an understanding of the planned activities for each programme and the proposed mechanisms of achieving targeted outcomes. The number of eligible patients referred, response rate, reasons for non-enrolment into the programmes, frequency of programme delivery as well as duration of enrolment will be retrieved from the existing programme databases to determine the coverage, frequency and duration of respective programmes. Furthermore, medical records of patients enrolled into the programmes will be reviewed to provide a comprehensive picture of interactions between healthcare providers and users throughout a patient’s healthcare journey from enrolment to discharge. Medical records will be proportionately sampled based on the number of healthcare providers that are able to provide referrals within each programme and the time point at which patients are enrolled into the programmes.

Healthcare providers (physicians, nurses and other allied health professionals) and healthcare managers (care coordinators and healthcare administrators) who are involved in the planning, development and implementation of the four programmes are invited to participate in face-to-face structured interviews. Written informed consent and demographic information including their age, gender, duration of involvement and roles will be collected prior to the interviews. Guided by the CFIF, content of interventions and moderating factors, which may have contributed or hampered the implementation of the interventions, will be assessed during interviews. All interviews will be audio-recorded, transcribed verbatim and coded thematically.

Two sessions of ethnographic observations (induction and follow-up) for each study participant will be randomly conducted at patients’ homes and/or through phone calls depending on where and how the interventions are provided. We expect to observe two sessions each for approximately 30 participants from each programme and informed consent will be obtained from participants. Content of care delivery, interactions between healthcare professionals and users together with patients’ responses to the service will be observed. For each session, field notes will be taken and findings will be written in narratives and analysed thematically.

Data analysis

Data from the various sources will be given equal weightage and will subsequently be integrated at data analysis stage guided by the modified version of the CFIF using the triangulation protocol methodology.²⁹ The extent of which the intervention components were implemented will be rated on a 5-point Likert scale ranging from never to always by two independent evaluators. Potential moderating factors that affect the implementation of the programmes will be assessed by examining participant responsiveness, comprehensiveness of protocols/guidelines, support available to facilitate recruitment and implementation, quality of service delivery and context (patient, professionals, organisational, political and economic factors). Analysis will also take into account other emergent themes not defined by the CFIF.

Evaluation of outcomes

Healthcare utilisation and health outcomes

Healthcare utilisation, health outcomes and patient-reported experiences and outcomes associated with the NUHS-RHS programmes will be assessed through review of clinical indicators, structured interviews and administration of surveys as described below to identify alignment with the Chronic Care Model (CCM).³⁰ The CCM identifies the community, the health system, self-management support, delivery system design, decision support and clinical information systems as key elements of a health system that are essential to provide good quality chronic care.³⁰

The effect of the programme on healthcare utilisation will be measured by analysing the number of admissions, emergency department (ED) attendance, number of specialist outpatient clinic (SOC) attendances and average bed days for patients enrolled from the start of the programmes. To capture the potentially evolving nature of the programmes and their effects, data will be collected over various time points with respect to their respective date of enrolment programme: 12 months before (T-12), 9 months before (T-9), 6 months before (T-6), 3 months before (T-3), date of enrolment (T0), 3 months after (T3), 6 months after (T6), 9 months after (T9), 12 months after (T12) and 15 months after (T15). Demographic data (gender, age, race, comorbidity and socioeconomic status) will also be retrieved for all patients.

Clinical indicators at various time points for all patients who are enrolled and who declined will be extracted from the existing NUHS-RHS databases. In reference to the point of patients’ enrolment into the programmes, clinical indicators at T-12, T-9, T-6, T-3, T0, T3, T6, T9, T12 and T15 will be extracted.

The effect of the programmes on patient-reported outcomes, patients’ quality of life, care experience and self-efficacy will be assessed using validated survey instruments over three time points: at the starting point, midway and the end of the programme. Quality of life of patients will be assessed using the EuroQol-5 Dimension (EQ-5D) instrument. The EQ-5D consists of five items, which recognises the fundamental importance of independent physical, emotional and social functioning, as part of a more holistic view of health. It has been validated in Singapore and is a common measure used to assess quality of life as the result of healthcare interventions.³¹ Patients’ perception of the quality of care received will be examined using the Patient Assessment of Care for Chronic Conditions (PACIC).³² The PACIC seeks to understand the frequency with which various aspects of care patients with chronic conditions received from their healthcare providers aligned with components of the CCM. The healthcare providers include regular doctors, nurse, care coordinators, allied health providers and others. PACIC has been widely validated and used at various healthcare settings across the world.^33–36

Patients’ experience of care continuity will be assessed using the CCAENA (continuity of care between care levels) questionnaire. The CCAENA consists of 29 items and assesses continuity of care across different levels from the patients’ perspectives.³⁷ The components covered by the CCAENA include the relationships between patient and primary care providers and between specialists and primary care providers. Finally, self-efficacy is measured using the Patient Activation Measure, which consists of 13 items, and will be used to measure patient knowledge, skill and confidence for self-management.³⁸

Complementary to the quantitative analysis of outcomes, a more in-depth understanding of healthcare users’ perception of healthcare quality, including why and how it has worked, or did not work, will be explored qualitatively through interviews with healthcare users. Healthcare users including patients and their family members who have had experiences with the programmes for at least 3 months are invited to take part in face-to-face interviews. Interviews are structured according to the programme logic model and key components of the CCM.³⁰ Perception of the level of care integration as well as recommendations for improvement to the programmes will also be elicited during the interviews. Potential participants will be recruited through care coordinators and will be screened against the selection criteria by the study team prior to the interviews. Only those who are eligible and are willing to be audio recorded will be interviewed. Written informed consent and demographic information will be attained from the participants before proceeding with the interviews. Interviews will be conducted in English, Chinese or Malay at the patients’ homes or at NUHS. Notes will be taken during observations, and all interviews are audio recorded, transcribed verbatim and will be imported to Atlas.ti V.7.0 for analysis.

Data analysis

An interrupted time series analysis (ITSA) will be conducted for all programmes to examine the effects of the NUHS-RHS programmes on healthcare utilisation, health outcomes and care experiences. In addition, difference in difference (DID) analysis will be conducted for those with comparison groups.^{39 40} Proposed to be more flexible compared with the traditional RCTs analysis, ITSA is conducted to estimate the changes associated with the introduction of interventions over time. To estimate changes in the above outcomes, time series regression that includes data from patients enrolled into the programmes 1 year prior to and after referral to the programmes will also be run. The model will have three main parameters: (1) estimation of the annual trend in healthcare utilisation (ie, hospital admission, SOC attendances, ED attendances and average bed days) and clinical indicators 1 year before until the introduction of the programmes; (2) estimation of changes in all outcomes associated with introduction of the programmes; and (3) the trend change in outcomes associated with each year. For each outcome, we will apply a generalised estimating equation model clustering by the combination of age, sex and comorbidity score with robust standard errors and an autoregressive correlation matrix. We will report rate ratios calculated from the parameter estimations, which represent population-averaged rates. DID analysis will subsequently be performed to estimate the impact of the programmes by comparing differential effects in the outcomes between intervention and comparison groups. Sensitivity analyses will be performed, and confounding effects will be accounted for.

Observation notes and interview transcripts will be thematically coded by two independent researchers using a two-step approach according to the integrated approach as described by Bradley et al.⁴¹ We have selected this approach as this study adopts an existing CCM framework, but we would like to ensure completeness of findings by assuring other emergent themes not previously described in the CCM are also considered. The first step consists of a deductive analysis, where units of data are coded according to the CCM. This will be followed by an inductive analysis, where new themes or unexpected findings are elicited through coding and categorising. If the interviews were conducted in other languages beside English, these will be transcribed, and the analysis will be conducted by a researcher that is fluent in that language. The relevant quotes representative of the analysis will then be translated into English. Subsequently, data collected through prospective surveys and interviews with healthcare users will be merged using the CCM as a framework.

Care integration

The extent of care integration within the NUHS-RHS will be assessed through the administration of the RMIC measurement tool (MT) adapted to Singapore.^{26 42} Tested and validated in Singapore, the RMIC-MT was developed to measure the extent of care integration from the healthcare providers’ perspectives. It consists of 62 items grouped into eight factors of integrated care related to person-focused and population-based care, clinical integration, professional, organisational, systematic, functional and normative integration. The RMIC-MT uses a 4-point Likert scale ranging from ‘never’ to ‘all the time’ and an additional option of ‘not sure/don’t know’.²⁶

Data analysis

Incomplete RMIC-MT will be excluded from analysis. The ‘not sure/don’t know’ option will be considered as missing data and will be imputed with median score of each item. The RMIC-MT will be scored as described previously by Nurjono et al in which the average score for respective dimensions of RMIC and overall care integration will be computed. A higher score on the RMIC-MT is considered to reflect a greater extent of care integration. The RMIC-MT was developed through a systematic literature review and validated by international Delphi panels.³ It describes six dimensions of care integration (clinical, professional, organisational, functional and normative integration) across different levels within a health system needed to provide a continuous, comprehensive and coordinated delivery of services to the individual and population. Furthermore, the degree of integrated care within NUHS will be compared with the degree of integrated care in other health systems (Netherlands and Australia) in which the RMIC-MT has been administered.

Healthcare cost

To examine the effect of NUHS-RHS programmes on healthcare costs, six categories of costs including (1) development costs, (2) programme implementation costs, (iii) healthcare utilisation costs (inpatient, outpatient and community), (4) productivity costs incurred to patients and/or caregivers, (5) travel costs borne by patients for travelling to receive care and (6) caregiving cost will be collected.⁴³ Cost categories 1–3 are considered related to the healthcare system whereas categories 4–6 take into account costs from the societal perspective.

Development costs include costs incurred during the development of the NUHS-RHS and their respective programmes. The programme implementation costs take into account costs related to the implementation of the programmes including the manpower costs, travel costs, costs of equipment and costs of materials used for patient and caregiver training and costs associated with multidisciplinary team meetings. The development and implementation costs will be systematically collected using WHO’s CostIt instrument⁴⁴ and will be divided by the number of clients included into the programmes to obtain per patient cost.

The cost of healthcare utilisations (hospital inpatient, outpatient services, primary and community care services), productivity costs (estimated based on absence from paid employment due to illness or providing care), travel costs (distance to healthcare providers and expenses incurred related to providing care to enrolled patients) and caregiving costs (costs related to the hire of domestic helpers, costs related to improvement in home environment and costs of caregiver training) will be collected using routinely collected hospitals data and a cost questionnaire administered to participants at T0 and T12 where each participant will be asked to recall his/her expenses 3 months prior to the time of the interview.

Data analysis

Differences in costs by category before and after enrolment will be analysed using paired Wilcoxon tests. The difference will be considered statistically significant if the p value is <0.05. Paired analysis will be adjusted by age, gender and number of comorbidities at baseline. To estimate the cost-effectiveness of the right-site care programmes, an incremental cost-effectiveness ratio (ICER) will be computed using changes in quality-adjusted life-years (QALYs) attributable to the programme as the primary measure of programme effectiveness, divided by changes in costs measured both from the health system and societal perspectives. The programme will be considered cost-effective if the ICER is lower than Singapore-specific thresholds. Currently, in the absence of a formal nationally accepted threshold for societal costs, we will adapt a default threshold consistent with WHO Choosing Interventions that are Cost-Effective (CHOICE) benchmarks of approximately $60–75 000 per QALY.⁴⁵ The appropriate cost-effectiveness thresholds will be reviewed and updated to be consistent with national best practice at the time of completion of the analysis.

Phase 3: refinement of IPT

Finally, to provide an overall evaluation of the NUHS-RHS taking into consideration various perspectives, qualitative and quantitative data obtained from various stakeholders including the programme team members and patients/proxies will be integrated through the process of triangulation at the data interpretation stage when both quantitative and qualitative data have been analysed separately.²⁹ The triangulation protocol will be adopted to guide data integration by first producing a convergence coding matrix according to the guiding conceptual frameworks to display findings emerging from each component followed by consideration of where there is agreement, partial agreement, silence or dissonance between findings from different data sources. Assessment of the fit of data integration will be conducted by examining the coherence of findings from various methods used, as suggested by Fetters et al.⁴⁶ Data on CMO will be gathered and analysed guided by respective frameworks as described in phase 2. These will then be linked according to the realist evaluation CMO formula in which findings on context and mechanisms will used to explain outcomes observed. A few potential CMO configurations will then be proposed and discussed (for validation purposes) through 2–3 focus group discussions comprising of 8–10 different stakeholders each. After which, the IPT will be refined to highlight how to improve the NUHS-RHS by detailing what works (outcome), as well as how (mechanisms) and under what conditions (context).

Ethics and dissemination

Data collection for this study was started in June 2016 and is expected to continue until June 2019. The National Healthcare Group, Singapore, Domain Specific Review Board reviewed and approved this study protocol (DSRB Ref: 2016/00410 and 2016/00914). Written informed consent forms are obtained from participants (healthcare providers and users) included for interviews and surveys. To maintain confidentiality of the research participants, identifiable information obtained from research participants are kept securely under password protection. A unique respondent identification is assigned to each study participant so that data can be processed anonymously. Waiver of informed consent was obtained for analysis of retrospective data collected as part of the hospital and MOH routine data collection, and no identifiable data are known by the researchers. Study results will be written up and published in international peer-reviewed journals and will be presented at national and international conferences and internally at NUHS-RHS and the MOH, Singapore.

Mitigation of potential risks and limitations

While realist evaluation approach and mixed methods design have been advocated for the evaluation of complex healthcare interventions, they are relatively new in comparison with the RCTs commonly used in the evaluation of medical interventions. Due to the lack of familiarity on the use of realist evaluation and mixed methods in the field of health and healthcare research, this study design may draw criticism for not being as rigorous as the typical RCT. Furthermore, we acknowledge that the use of mixed methods design also increases complexity. Nonetheless, we strive to reach optimal integration of data at multiple levels—study design, methods, interpretation and reporting—using the convergent parallel mixed methods approach that connects and merges methods and findings.

Furthermore, as participation in the evaluation efforts is voluntary and only individuals who are able or have proxies who are able to provide informed consent are included in this evaluation effort, we also recognise that selection bias may be introduced in this study. To account for selection bias, demographic information including age, gender, ethnicity and role (for healthcare providers) are collected and will be compared between responders and non-responders. Given the prospective nature of some of our data collection methods, difficulties in following up with respondents are also anticipated. To minimise the numbers lost to follow-up, research appointments are scheduled at participants’ time and place of convenience in collaboration with participants’ care coordinators. In the event when loss to follow-up is inevitable due to death, survival analysis will be conducted to account for the missing data.

In addition, it must be acknowledged that as much as it was intended for the delivery of care to be consistent, variability in the extent of how care was provided exists, especially across different disease-specific subprogrammes and also within individual subprogramme. To account for such variability, subgroup analysis will be conducted, and adjustments based on the number of care providers within a subprogramme will also be made.

Discussion

Singapore is in the early phase of developing new integrated care models. Notwithstanding the scope of more general design principles of integrated care, development and implementation of integrated care require adaptation to the context, needs of the local population and the capacity and capability of local healthcare providers. To the best of our knowledge, this study provides the first opportunity to holistically evaluate the NUHS-RHS, and one of the first to conduct a comprehensive evaluation of a new integrated care network in the context of Southeast Asia using a realist evaluation approach.

This study carries significant implications for the health system and society in Singapore as well as for the field of integrated care internationally. By ensuring the rigour of this evaluation and adopting a realist approach, findings from this study are expected to generate contextually relevant evidence for improving efficiency and effectiveness of the NUHS-RHS as it develops and also to inform future initiatives of the NUHS. It is also expected to yield policy-relevant insights for national-level decision makers as they continue to monitor and refine the RHS model for Singapore over the coming years. In designing this evaluation, stakeholders including programme team members, healthcare managers and policymakers were engaged to help evaluators understand the context in which the programmes were implemented, how interventions have been delivered, to define objectives, evaluation questions and outcomes that are relevant to the stakeholders and to identify constraints to the feasibility of implementation of the evaluation itself. Various potential pitfalls in evaluation design and implementation have been collectively identified, and risk management strategies were put into place wherever possible. This provides confidence that the evaluation efforts are rigorous, relevant and has the potential to provide insights for programme improvement as well as provide evidence to support policy decisions to scale-up/down activities within NUHS. Engagement of stakeholders early in the evaluation efforts is also expected to increase the likelihood that recommendations will be adopted to improve the relevant components of the existing care models and define future new care models. However, the involvement of stakeholders may exert pressure to act on preliminary findings. Understanding these dynamics will also be of interest as part of the evaluation itself.

Like NUHS, other integrated care networks within Singapore and abroad are experimenting with new integrated care models. However, the findings from existing research of the (cost-)effectiveness of integrated care are to a major extent inconsistent because of the variations in the strategic outcomes, methods of implementation, contexts (ie, system and policy) and/or applied evaluation measures. Examples include next-generation accountable care organisations in the USA,⁴⁷ integrated care pioneers and vanguard sites in England,^{48 49} population health management pilots in the Netherlands,⁵⁰ integrated care pilots in Belgium⁵¹ and the integrated care demonstrators in Australia.⁵² This study provides an opportunity to fill some of the gaps in current research by evaluating the NUHS-RHS and its constituent programmes using a rigorous and comprehensive design that balances the needs of context-specific evaluation with international comparability. Wherever possible, steps such as using internationally validated instruments for evaluation of chronic disease are taken, to allow for meaningful international comparisons that can increase collective knowledge about the restructuring of CCMs towards advancing integrated care.