Skip to main content
PMC home page
Wellcome Open Research logoLink to Wellcome Open Research
. 2023 Nov 1;8:29. Originally published 2023 Jan 19. [Version 3] doi: 10.12688/wellcomeopenres.18710.3

Mixed methods study protocol for combining stakeholder-led rapid evaluation with near real-time continuous registry data to facilitate evaluations of quality of care in intensive care units

The Collaboration for Research, Implementation and Training in Critical Care in Asia and Africa (CCAA), Aasiyah Rashan 1, Abi Beane 2,3,4,a, Aniruddha Ghose 5, Arjen M Dondorp 3,4,6, Arthur Kwizera 7, Bharath Kumar Tirupakuzhi Vijayaraghavan 8, Bruce Biccard 9, Cassia Righy 10, C Louise Thwaites 6,11, Christopher Pell 4, Cornelius Sendagire 12,13, David Thomson 9, Dilanthi Gamage Done 14,15, Diptesh Aryal 3,13,16, Duncan Wagstaff 9,17, Farah Nadia 18, Giovanni Putoto 19, Hem Panaru 16, Ishara Udayanga 14, John Amuasi 20, Jorge Salluh 13, Krishna Gokhale 15, Krishnarajah Nirantharakumar 15, Luigi Pisani 3, Madiha Hashmi 3,21, Marcus Schultz 22, Maryam Shamal Ghalib 23, Mavuto Mukaka 3,6, Mohammed Basri Mat-Nor 18, Moses Siaw-frimpong 24, Rajendra Surenthirakumaran 25, Rashan Haniffa 2,3,14, Ronnie P Kaddu 26, Snehal Pinto Pereira 27, Srinivas Murthy 28, Steve Harris 29, Suneetha Ramani Moonesinghe 27, Sutharshan Vengadasalam 30, Swagata Tripathy 2,31, Tiffany E Gooden 15, Timo Tolppa 14, Vrindha Pari 32, Wangari Waweru-Siika 26, Yen Lam Minh 11
PMCID: PMC10638482  PMID: 37954925

Version Changes

Revised. Amendments from Version 2

No major changes have been made to this manuscript. As a response to reviewer comments, the following minor changes were made:

  1. We moved text from 'study design' to 'data collection' to make the methods described easier to follow and understand.

  2. We added text to clarify what data the registry holds, how care processes will be chosen for the process mapping and how the MUSIQ 2 calculator will be used.

  3. We made minor changes to text in the introduction and discussion regarding the potential impact of this research, and the benefits of using the described participatory research methods and patient-centred stakeholder-led evaluation for ICU care in LMICs.

Abstract

Background

Improved access to healthcare in low- and middle-income countries (LMICs) has not equated to improved health outcomes. Absence or unsustained quality of care is partly to blame. Improving outcomes in intensive care units (ICUs) requires delivery of complex interventions by multiple specialties working in concert, and the simultaneous prevention of avoidable harms associated with the illness and the treatment interventions. Therefore, successful design and implementation of improvement interventions requires understanding of the behavioural, organisational, and external factors that determine care delivery and the likelihood of achieving sustained improvement. We aim to identify care processes that contribute to suboptimal clinical outcomes in ICUs located in LMICs and to establish barriers and enablers for improving the care processes.

Methods

Using rapid evaluation methods, we will use four data collection methods: 1) registry embedded indicators to assess quality of care processes and their associated outcomes; 2) process mapping to provide a preliminary framework to understand gaps between current and desired care practices; 3) structured observations of processes of interest identified from the process mapping and; 4) focus group discussions with stakeholders to identify barriers and enablers influencing the gap between current and desired care practices. We will also collect self-assessments of readiness for quality improvement. Data collection and analysis will be led by local stakeholders, performed in parallel and through an iterative process across eight countries: Kenya, India, Malaysia, Nepal, Pakistan, South Africa, Uganda and Vietnam.

Conclusions

The results of our study will provide essential information on where and how care processes can be improved to facilitate better quality of care to critically ill patients in LMICs; thus, reduce preventable mortality and morbidity in ICUs. Furthermore, understanding the rapid evaluation methods that will be used for this study will allow other researchers and healthcare professionals to carry out similar research in ICUs and other health services.

Keywords: rapid evaluation, quality of care, intensive care, critical illness, low- and middle-income countries, learning health systems

Introduction

An estimated five million deaths per year worldwide could be avoided by improving the quality of the delivery of healthcare 1 . The Institute of Medicine 2 defines quality of healthcare as “the degree to which health care services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge". Quality of care is reflected in structures of healthcare services, patient-level processes undertaken, and the outcomes of healthcare interactions 3, 4 . Recommendations from the Lancet High Quality Health Systems (HQSS) report called for greater investment in systems and processes that strengthen evaluation and improvement of care in low and lower middle income countries (LMICs), and that these systems should reflect and be sensitive to the diverse needs of communities they serve 4 .

Critical care encompasses healthcare provided to patients with, or at risk of, immediately life-threatening, potentially reversible conditions, irrespective of age, diagnosis, specific patient group or location 5 . Improving the safety and effectiveness of care for critically ill patients has the potential to substantially reduce preventable mortality 1 . Efforts to improve the quality of critical care 6 in intensive care units (ICUs) internationally, have increasingly focused on ameliorating risk of complications associated with both the critical illness itself, and the healthcare interventions delivered to treat it. These efforts have included minimising duration of invasive therapies, optimising infection control practices and increasing adherence to antimicrobial stewardship practices. Whilst each of these interventions has an established evidence base of improving outcomes across the heterogeneity of critical illness 7 , such interventions are often poorly implemented and, or consistently adopted in resource constrained healthcare institutions; notably concentrated in LMICs. In addition, patient-centred care and engagement with families and patients to share experiences of critical care is often overlooked despite being of itself, associated with improved outcomes 8 .

Healthcare providers, including the National Health Service in the UK, have long relied on models of healthcare evaluation including Donabedian’s which measures care provision through the framework of structures, processes and outcomes 9 . Classically, critical care services have evaluated and benchmarked care using comparisons of this framework between different hospitals (often annually) and within the same institutions over time (often years) 10, 11 . However, such methods of evaluation rarely provide the granularity of information or the specific contextual factors needed to determine the reasons for poor care and identify opportunities for improvement. Systematic improvement in quality requires identification of the gaps in care (i.e. the differences between care as intended and care as delivered) and an understanding of the underlying determinants. Furthermore, effectiveness of current care delivery, and the likely success of any quality improvement interventions are dependent on the ability to positively and sustainably influence the behaviour of relevant stakeholders 12 .

Quality of care and success of interventions to improve care is directly attributable to the behaviour of healthcare providers, organisational culture within hospital settings and external factors affecting healthcare accountability 3, 12, 13 . In LMICs 14 , further exploration is needed to understand what factors determine healthcare provider behaviours, organisational cultures and patient and public expectations. Acquiring a better understanding of these determinants is essential to inform the design 3, 12 and increase the effectiveness of interventions targeted at improving care both in individual ICUs, and more widely within a healthcare system 15 .

This project aims to leverage the system infrastructure and community of practice established by The Collaboration for Research, Implementation and Training in Critical Care in Asia and Africa (CCAA) to: evaluate the quality of existing care processes in the ICU; identify individual, team and organisational factors determining current care delivery; and assess the likely influence of these factors on future improvement interventions 3 . This protocol describes the novel methods proposed to undertake this multi-layered, multi-centre evaluation.

Protocol

Study design

This is a multi-centre mixed methods rapid evaluation comprising: registry-enabled assessment of care quality using selected process and outcome metrics, stakeholder-led rapid evaluation of the organisational and contextual factors influencing care (process mapping, structured observations and focus group discussions); and an assessment of local quality improvement capabilities ( Figure 1). Together these methods will provide a replicable, comparable and context-specific evaluation of the quality of care processes and their associated outcomes. This evaluation will help stakeholders identify and understand the underlying factors enabling or impeding the delivery and improvement of high-quality care within their departments.

Figure 1. Schema of activities throughout the project.

Figure 1.

Open in a new tab

Rapid evaluation (RE) methods 25, 26 were chosen for their ability to empower healthcare stakeholders to identify and understand the determinants of existing practice within their own setting 15 . Process mapping 27 conducted by clinical teams, with participation where possible from patients and care receiver representations, will provide a preliminary framework 28 to understand gaps between care as intended and care delivered. This information will inform the scope, location and timing of structured observations 25 and topics for focus group discussions 29, 30 .

Structured observations of care will enable the study team to document and describe team and patient-provider interactions relevant to the care process(es) under evaluation. These observations will enable the teams to identify practices of communication, team working and environmental factors (space, location etc) that may affect care delivery.

Focus group discussions, conducted in parallel to the observations, will be used to further explore and enrich the team’s understanding of the care process(es) of interest from the participants’ view, including their perceptions of barriers to, and enablers of, effective care delivery.

Assessment of organisational readiness for care improvement. The findings of this multi-centre international evaluation of care will be used to inform future co-design of a toolkit for quality improvement in ICUs. As such, the evaluation will include a replicable assessment of the readiness of participating ICU teams and their institutions to undertake quality improvement activities. The international-validated Model for Understanding Success in Quality (MUSIQ 2) 31 calculator designed to identify likelihood of success of future improvement interventions, will be used to gather individual experiences of quality improvement, the site-level data needed to create a shared understanding of local practice and identify local quality improvement capabilities.

Setting

The CCAA is supported by a Wellcome Innovations Flagship Programme and UKRI/MRC award, established in 2020, and currently funded until January 2026. The CCAA’s aim is to establish a community of practice equipped with the infrastructure required for a learning health system capable of providing continuous reliable service evaluation, supporting measurable care improvement and facilitating high quality clinical research which translates to practice change. At its core is the ICU digital registry platform which supports a distributed network of clinician-led registries across 17 LMICs, which capture data on case mix, population characteristics, organisational features, care processes and clinical outcomes contemporaneous to the delivery of clinical care for all ICU (and some emergency, perioperative and acute medicine) encounters. The harmonisable data provides near real-time information for service evaluation, clinical research and contextualised sustainable improvements in care delivery ( Figure 2). The registry data set, data collection methods, data quality assurance processes and research impact are described elsewhere 16, 32– 37 .

Figure 2. Process evaluation p-charts of Richmond Agitation-Sedation Scale (RASS) rates.

Figure 2.

Open in a new tab

The blue line and shaded 95% CI area (left y-axis) display the daily percentage of eligible patients who had a target RASS set and the bars (right y-axis) represent the number of eligible patients. Patients are eligible if they are mechanically ventilated on that day.

CCAA collaborators from ICUs in seven country networks have self-selected to participate in this project. The self-selected countries are Kenya (CCSK), Ghana (KCCR), India (IRIS), Malaysia, Nepal (NICRF), Vietnam and South Africa. Each national network will identify three to six ICUs that have ambition to use their registry for regional and international care quality evaluation, build capacity for health systems research methods and participate in quality improvement interventions. Each participating ICU will also have demonstrated ability to use the ICU registry platform for near real-time daily data collection of ICU encounters inclusive of over 95% of all admissions 16 . Pluralistic health systems co-exist in each of these countries, whereby private, government and non-governmental organization providers offer critical care services within secondary and tertiary facilities. The CCAA is non-discriminatory in its inclusion of institutions and no representation of diversity of providers is being sought. All country networks and ICUs participate by choice and no financial incentives are used. Factors including care provision and financial models of care may influence organisations’ care quality and ability to engage in quality improvement 38 . These will be captured and quantified during the project using the mixed methods described above and a future network-wide process evaluation.

Study team, participants, sample size and recruitment

Study team. Each site will have an ICU quality team consisting of (as a minimum), an ICU nurse, a senior doctor and a data collector. Each ICU team will be supported by their existing national registry coordination team (who include a national lead, coordinator and research assistant). Working alongside these site level and national teams are the already established registry implementation team which includes data scientists, statisticians and clinician researchers with a track record in mixed methods evaluation and service improvement. Together they will provide continued support to the ICU quality team in methods, statistical analysis and overall study conduct. This support is provided where possible remotely, using the Zoom (Zoom Video Communications Inc, San Jose, CA, USA) conferencing application.

Study participants. A wide range of ICU stakeholders will be invited to participate. This will include patients and carers in addition to those responsible for both the strategic service development and delivery of clinical care. We will aim to have balanced representation from each of the stakeholder groups 39 ; however, we recognise the possible limitation of scope from the ICUs in healthcare settings, whereby allied clinical disciplines (e.g. microbiologists, pharmacists, and dieticians) may be limited or poorly represented. Invitation to participate in the project will be sought through the national registry leads and the ICU quality teams.

Stakeholders invited to participate (‘participants’) in the rapid evaluation will have study information made available prior to participation 17 . Participants will have an opportunity to review the participant information sheet a minimum of 72 hours prior to participation to allow sufficient time for them to consider and seek advice from the ICU or national team or other independent parties. All data collection will be in the language commonly used to deliver healthcare in the setting.

Sample size and recruitment. Sample size will be guided by similar studies and based on achieving sufficient data to explore a range of stakeholder perspectives to understand the factors affecting the specific process of care under evaluation. The combination of purposive sampling of key stakeholders 25 , including patients, the focused scope of inquiry 40 defined by process mapping, and flexible, rapid iterations of data collection and analysis conducted in parallel by teams (a feature of RE method design 25 ) means that the research question may be addressed with estimated 2–3 process maps, 2–3 observations and 2–3 focus groups per ICU 41 .

Data collection

The sequence and timings of the five discrete but complementary sources of data are described in Figure 1.

Registry enabled continuous evaluation of care . The components of a learning health system (continuous data for evaluation and new knowledge generation, rapidly integrated into practice) already established by the CCAA will be enhanced by the rapid evaluation, facilitating future practice improvement. The CCAA already has an ICU registry with an established core data set which enables comparable description of case mix, severity of illness and benchmarking of clinical outcomes; the details of the core dataset are described elsewhere. Additionally, the CCAA recently completed a scoping review of ICU quality metrics used internationally and undertook a four round modified Rand Delphi study to identify a set of indicators for evaluation 21 . Indicators were assessed for their feasibility, reliability, validity to predict outcome and sensitivity to change. They have been classified according to the High Quality Health Systems Framework 4 : foundations (encompassing human resources, governance structures, accessibility and tools), quality impacts (including clinical and economic outcomes) and care processes (descriptions of care and systems as well as user-experience). These indicators were then prioritised and defined for implementation through the registry. The care processes associated with these priorities are already endorsed by healthcare policy makers in all but the most resource constrained health systems in LMICs 22 .

The selected metrics are described in Table 1 and reflect previously identified research priorities for improvement: reducing avoidable harms; improving delivery of interventions and processes already proved to improve outcomes and measuring and improving patient-centred outcomes 21 . Avoidable harms include deep venous thrombosis, stress ulcer, ICU delirium, neuromuscular decline, pressure injury and healthcare-acquired infections. These harms are all associated with prolonged organ support including mechanical ventilation 8 . The associated daily care processes and interventions proven to ameliorate these harms include optimising sedation and pain management using objective assessment tools (RASS, CPOT); daily assessment of readiness to wake (SAT), assessment of ability to breath spontaneously (SBT) and passive or active mobilization 23 . Foundations of care, for example nurse:patient ratios, have also been shown to influence these harms. In addition, engagement of family members in daily care in the ICU and their involvement in interventions including respiratory physiotherapy and mobilisation has been demonstrated to improve not only the effectiveness of the interventions, but also to increase both provider and patient compliance, and promote better experience for families and patients 8 . The feasibility of these measures (three foundation, eight quality impacts and eight care processes) and their definitions 17 are currently being assessed 9, 18 for feasibility of collection (using the Khan quality framework assessment criteria for conformance and completeness) 19 in pilot ICUs in collaborating registries including IRIS (Indian Registry of IntenSive care), NICRF (Nepal Intensive Care Research Foundation), CCSK (Critical Care Society of Kenya) and South Africa.

Table 1. Selected measures for registry enabled evaluation.

Foundations 1. Nursing staff to patient ratio
2. Intensivist staffing to bed ratio
3. ICU medical night coverage
Quality impacts 1. Antimicrobial usage, (days of therapy, duration of empirical antimicrobial use)
2. Incidence of ICU-acquired drug resistant organism of interest (DRI)
3. Incidence of HAI (Central Venous Catheter Associated Infection, Catheter Associated Urinary Tract Infections & Infection-related Ventilator-Associated Complication (IVAC))
4. Incidence of unplanned ICU discharge due to financial constraints
5. Unplanned readmission to ICU
6. Standardised mortality rate (ICU & hospital)
7. Length of stay (ICU & hospital)
8. Quality of life at 30 days post ICU
Care processes 1. Venous thromboembolism prophylaxis
2. Duration of mechanical ventilation
3. RASS score (target and actual)
4. Stress ulcer prophylaxis
5. Spontaneous awakening trial
6. Spontaneous breathing trial
7. Incidence of new pressure sores
8. In-bed mobilisation
Open in a new tab

[i] ICU: intensive care units; HAI: hospital acquired infection; RASS: Richmond Agitation-Sedation Scale 20

Table 1 data are captured each day, extracted from patient charts or directly observed by trained data collectors and entered to the registry platform daily during ICU stay. A comprehensive field specification and data collection guide are made available to all ICU teams through the platform and 24-hour online support is available. Data collectors are trained using already published methods. Weekly follow up meetings with the national registry teams and the site quality teams will provide feedback and troubleshooting for data collection and data quality during weeks 1–4, thereafter monthly meetings using a published quality assurance framework 16 . Census checks with independent admission data are used to monitor cohort inclusion weekly.

Rapid evaluation . Process mapping will be locally-led by the ICU quality teams, with the support of the national registry teams who have experience in rapid evaluation methods, and remote support will be provided by experienced researchers in the implementation team. Process mapping will be conducted in months 2–3 of the evaluation and will inform the priorities for observations and focus group discussions. We will use a general process mapping technique rather than one allied to a specific improvement methodology 42 . Stakeholders will be asked to map out the discrete activities (tasks) associated with locally-selected care processes from the list in Table 1, identify the actors and equipment involved, the interactions, decision making and who are the decision makers, for care as intended, and for care as it is actually delivered. A mapping guide 17 will be used as a template. The mappings will provide a structured, visual representation of each care process which will identify deviations from intended practice and inefficiencies in existing care processes.

Structured observations will be conducted by the ICU quality team, in the clinical area relevant to the processes of interest identified through the process mapping (months 3–6). Data from the observations will be collected using a structured observation guide 17 to ensure consistency across study teams and sites. Observations will usually take place within the ICU but may on occasion extend to other locations in local pathways for critically ill patients such as emergency departments and inpatient wards.

Focus group discussions will be conducted by the national registry teams familiar with the context but not directly responsible for care, with support of experienced researchers. Separate focus group discussions will be conducted for healthcare providers and patients/carers so as to limit the impacts of social hierarchies of healthcare within the communities participating 29, 30 . Focus group discussions will be conducted with between six and eight stakeholders at any one discussion. A template to guide discussions has been designed 17 but may be iteratively modified based on the findings of the process mapping and structured observations.

Assessment of organisational readiness . The national registry teams will work in partnership with the ICU quality teams to complete an assessment of readiness for quality improvement using the MUSIQ 2 calculator 31 . ICU quality teams will have an orientation session to the tool led by the national leads, and then complete the MUSIQ 2 calculator online. The MUSIQ 2 will be assessed at the start of the evaluation period (month 1) to help the ICU teams to identify the existing contextual factors which may promote or inhibit the success of the quality evaluation and future improvement initiatives. Scores will be reviewed together with the research team to identify opportunities where the CCAA infrastructure could be leveraged to improve quality improvement capability, for example capacity for quality improvement, data infrastructure, workforce focus and resources for quality improvement. The MUSIQ 2 will be reassessed at month six to describe changes in the micro, environmental and organisational factors as a result of engagement in the evaluation. Scores will be collated and stored on an electronic shared drive for each ICU. Scores will be validated by the registry implementation team for consistency.

Data analysis and data management

Data collection and iterative analysis will occur in parallel 25 . Discovery of information will be a reflexive process in which local knowledge is reconstructed through a cycle of data collection, analysis and planning what to examine next 25 . Given the evaluations will in part be conducted by the ICU quality teams directly responsible for care, the registry implementation team will facilitate debriefing sessions following each data collection procedure to ensure internal biases (present as a result of priori knowledge of the subject area) are discussed and resolved as data collection/analysis continues and conclusions are drawn 43 .

Data pertaining to case mix and demographics will be reported using standard descriptive statistics. Diagnosis will be classified from SNOMED CT and mapped to APACHE IV 44 . Risk adjusted outcomes and predicted mortality will be determined using standardised mortality ratio (SMR). Observed mortality is defined as the percentage of ICU patients who die within hospital (same encounter) as a proportion of all ICU admissions. Observed ICU mortality represents the numerator for risk-adjusted ICU mortality (SMR). The ratio between the observed number of deaths and the predicted number of deaths for the case mix of each ICU, computed by indirect standardisation. Predicted mortality will be determined using APACHE II or e-TropICS (a priori selected by the contributing registry) 45 . Compliance to process measures will be reported for individual patients based on eligibility each day during the ICU encounter. Composite measures of outcome or event indicators will be calculated as per their published and a priori chosen definitions, using data captured either daily, or by event, as appropriate.

Data arising from the process mapping, observations and focus group discussions will be triangulated 25 and analysed using an interpretive analysis approach 46 . Data will be deconstructed and barriers and facilitators to care quality coded using the Consolidated Framework for Implementation Research (CFIR) framework 47 . The CFIR was chosen for its ability to facilitate exploration of the individual, and team characteristics, and the in ICU organisational and external factors that promote and inhibit the routine incorporation of interventions into everyday clinical practice 28, 47, 48 . The findings of each round of analysis will be reconstructed using a Rapid Assessment Process (RAP) sheet 17 , to repackage the different categories and discover the high level themes that cross cut different care processes within and between individual ICUs. The validity of findings will be discussed and checked with process mapping, observation and focus group participants at every site 49 .

The Zoom conferencing application will be used to automatically save the audio files (as a MP4 file) from the process mapping exercises to a project-designated, password-protected and automatically backed-up shared drive storage space. The Zoom conferencing application will be used as set out by the University of Oxford ‘Guidelines for using Zoom’ 50 . The titles of video files will not include a participant’s name, but rather the date of the interview and a site code.

Data management will be overseen by the investigators (DW and AB) and the wider CCAA project team. Data from the process mapping, observations and focus group discussions will be captured digitally and identifiers will be removed. RAP sheets will contain anonymised data only and will also be completed digitally. All data will be retained for five years after the publication of the results. All data will be held on a project-designated, password-protected and automatically backed-up shared drive storage space. All data in the UK will be managed in accordance with EU General Data Protection Regulation (GDPR) 51 and if outside the UK, in line with country-specific data protection regulations.

Ethical and regulatory considerations

The focus of the research is on improvement of healthcare services and is not of a sensitive nature, and thus unlikely to evoke feelings of discomfort or emotional distress for participants. The project will be conducted in accordance with relevant national and international guidance and regulations, including the Global Code of Conduct for Research in Resource-Poor Settings 52 . To ensure that the project is conducted in an ethical manner, this protocol has been submitted to the Oxford Tropical Research Ethics Committee (OxTREC) 53 . National registry leads in each collaborating country will be responsible for coordinating with their institutional or institute review boards for relevant approvals. All participants will be given a participant information sheet prior to providing written informed consent.

Public engagement & involvement. National leads were consulted in the design of this project. There is existing literature to support the use of the methods proposed for research inclusive of patients including within the populations considered in this context 14, 26, 54 .

The registry reports were co-designed and developed with national registry leads and piloted for feedback with the multidisciplinary teams in Kenya, Nepal and India. The research team members are currently being trained in process mapping, observations and focus group discussions by investigators (DW and AB) via video conferencing and using a purpose-build quality improvement resource platform made freely available to all CCAA members 55 .

Quality improvement initiatives designed to improve care for patients and the public will be explored in subsequent research. The findings of this project will be accessible to patients and the public via the MORU Tropical Health Network website ( www.tropmedres.ac).

Dissemination

The findings will be used to directly inform the development of a toolbox for implementation of quality improvement interventions in LMICs led by clinician- researchers. The findings will be developed into country-specific manuscripts for publication and also shared across the CCAA collaborating countries. A report will be developed for the funders, Wellcome and UKRI/ MRC. Findings will be published as academic publications as open access and presented at academic conferences.

The country network teams with the support of the lead investigators (DW and AB) will lead writing and reviewing of manuscripts, abstracts and any other publications arising from the overall project. These will be equitably published in academic, peer-reviewed literature as open-access and will offer practical learning for others seeking to utilise similar methods in healthcare institutions worldwide for service improvement. Authorship will be based on the set of criteria outlined by the journal and where possible follow the CredIT Taxonomy 56 , and will acknowledge that this work is on behalf of the collaborating clinicians, patients and families representing healthcare services within the CCAA. The project results will also be published online ahead of peer review using a free access preprint platform in response to the global academic movement to increase equity and access to healthcare research.

Study status

The study started in August 2022. Due to staggered start-times between countries, data collection will continue until August 2023. No country has yet completed data collection.

Discussion

Measurement of key quality indicators, which include patient-centred outcomes, to drive forward improvement is increasingly being promoted as part of benchmarking healthcare quality. Continuous evaluation (driven by data generated each day during routine care delivery) undertaken contemporaneously to stakeholder-led exploration of organisational cultures to inform improvement interventions provides the potential to accelerate service improvement. In this study, we aim to simultaneously lay the foundations for a culture of healthcare improvement and establish capacity for future institution-led research through which healthcare providers, families and patients reflect and appraise current practice, to identify problems and seek possible solutions. As a result, this research has the potential to inform sustainable and effective ICU quality improvement in LMICs for both clinical outcomes and patient-family experiences.

Rapid evaluations (REs) can be characterised as: an intensive, team-based investigation that uses multiple methods of data collection; having an iterative process for collection and analysis; and following the principles of participatory action in order to quickly develop a holistic understanding of a programme from the perspective of key stakeholders, providing a potentially effective methods for institution-led but scalable improvement 25 . Stakeholder-led rapid research can provide timely insight into specific, complex processes and systems from locally-defined perspectives, which if attempted using more traditional ethnographic methods may take many months before conclusions are made, risking disengagement of stakeholders and delaying service improvement 57 . RE methods, increasingly used in healthcare 58 are amenable to enabling involvement of patient and public stakeholders, representation from whom is often absent in research and which is essential if healthcare providers are to better understand the impact of critical illness on individual families and on wider social and economic population metrics 4 . Attaining understanding of the context 13, 26 in which care is delivered allows tailoring and embedding of any subsequent quality improvement interventions with increased opportunity for their success 13 .

We anticipate that the combination of traditional benchmarking data from the established near real-time clinical registries together with the qualitative approaches of RE which stem from disciplines including anthropology and business, will enable understanding of existing ICU care processes and how organisational factors and health system structures may influence quality of care both at facility level and in relation to individual patient outcomes. For example, objective replicable daily time series data on the completeness of assessment of sedation use, and readiness to breath spontaneously, in each eligible patient, along with daily changes in ICU case mix, acuity, turnover and staffing numbers, will enable stakeholders to unpack (using iterative cycles of analysis and feedback) how internal and external factors affect aggregate and individual quality of individual care processes and their impact on patient outcomes. The multi-dimensional registry data will be fed back in parallel to ongoing evaluation, thereby providing a reliable and replicable measure of process change over time as future improvement interventions are implemented and evaluated for impact 59 .

The findings of the project will directly inform subsequent local projects aimed at improving patient care and the development of the CCAA registry to enable effective utilisation of data to drive quality improvement for stakeholders and their ICUs. By understanding organisational readiness for improvement, we will enable stakeholders to plan interventions best suited to their local cultures, needs and resources. Further to this, given this project’s participatory nature, and having been co-designed by clinical stakeholders, novice researchers and clinicians will be exposed to new methods. We anticipate this approach will facilitate the building research and quality improvement capacity in the CCAA which will extend beyond this project.

Acknowledgments

We would like to acknowledge the patients, family members and healthcare providers who already participate in the co-design and development of this study.

Funding Statement

This study was funded by a Wellcome Innovations Flagship Programme grant and UKRI/MRC award (Wellcome grant number: 215522, <a href=https://doi.org/10.35802/215522>https://doi.org/10.35802/215522</a>, UKRI/ MRC grant number: MR/V030884/1). They had no role in the design, analysis or reporting of this protocol.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

[version 3; peer review: 2 approved]

Data availability

Underlying data

No underlying data are associated with this article.

Extended data

Figshare: Supplementary materials for 'Mixed methods study protocol for combining stakeholder-led rapid evaluation with near real-time continuous registry data to facilitate evaluations of quality of care in intensive care units'. https://doi.org/10.6084/m9.figshare.21763325 17

This project contains the following extended data:

  • -

    File 1: Data Completion Guide

  • -

    File 2: Mapping Session Guide

  • -

    File 3: Focus Group Discussion Topic Guides

  • -

    File 4: Invitation Email to CCAA Country Site Leads

  • -

    File 5: Participant Information Sheet: Observations

  • -

    File 6: Participant Information Sheet: Interviews

  • -

    File 7: Observation Template Sheet

  • -

    File 8: Rapid Assessment Process (RAP) Sheet

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

References

  • 1. Kruk ME, Gage AD, Joseph NT, et al. : Mortality due to low-quality health systems in the universal health coverage era: a systematic analysis of amenable deaths in 137 countries. Lancet. 2018;392(10160):2203–12. 10.1016/S0140-6736(18)31668-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. Agency for Healthcare Research and Quality: Understanding Quality Measurement. 2020; [Accessed 13th April 2021]. Reference Source
  • 3. Hanefeld J, Powell-Jackson T, Balabanova D: Understanding and measuring quality of care: dealing with complexity. Bull World Health Organ. 2017;95(5):368–374. 10.2471/BLT.16.179309 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Kruk ME, Gage AD, Arsenault C, et al. : High-quality health systems in the Sustainable Development Goals era: time for a revolution. Lancet Glob Health. 2018;6(11):e1196–e252. 10.1016/S2214-109X(18)30386-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Schell CO, Gerdin Wärnberg M, Hvarfner A, et al. : The global need for essential emergency and critical care. Crit Care. 2018;22(1): 284. 10.1186/s13054-018-2219-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Jawad I, Rashan S, Sigera C, et al. : A scoping review of registry captured indicators for evaluating quality of critical care in ICU. J Intensive Care. 2021;9(1): 48. 10.1186/s40560-021-00556-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Edwardson S, Cairns C: Nosocomial infections in the ICU. Anaesth Intensive Care Med. 2019;20:14–8. [Google Scholar]
  • 8. Marra A, Ely EW, Pandharipande PP, et al. : The ABCDEF bundle in critical care. Crit Care Clin. 2017;33(2):225–43. 10.1016/j.ccc.2016.12.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. NHS England and NHS Improvement: Online library of quality, service improvement and redesign tools: A model for measuring quality care. [Google Scholar]
  • 10. Salluh JI, Chiche JD, Reis CE, et al. : New perspectives to improve critical care benchmarking. Ann Intensive Care. 2018;8(1): 17. 10.1186/s13613-018-0363-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Salluh JI, Soares MF, Keegan MT: Understanding intensive care unit benchmarking. Intensive Care Med. 2017;43(11):1703–7. 10.1007/s00134-017-4760-x [DOI] [PubMed] [Google Scholar]
  • 12. Zamboni K, Baker U, Tyagi M, et al. : How and under what circumstances do quality improvement collaboratives lead to better outcomes? A systematic review. Implement Sci. 2020;15(1): 27. 10.1186/s13012-020-0978-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Tarrant C, Colman AM, Jenkins DR, et al. : Drivers of broad-spectrum antibiotic overuse across diverse hospital contexts—A qualitative study of prescribers in the UK, Sri Lanka and South Africa. Antibiotics (Basel). 2021;10(1):94. 10.3390/antibiotics10010094 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Pieris L, Sigera PC, De Silva AP, et al. : Experiences of ICU survivors in a low middle income country- a multicenter study. BMC Anesthesiol. 2018;18(1): 30. 10.1186/s12871-018-0494-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Flottorp SA, Oxman AD, Krause J, et al. : A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8: 35. 10.1186/1748-5908-8-35 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Collaboration for Research Implementation and Training in Critical Care Asia Investigators, . Pisani L, Rashan T, et al. : Performance evaluation of a multinational data platform for critical care in Asia [version 2; peer review: 2 approved]. Wellcome Open Res. 2022;6:251. 10.12688/wellcomeopenres.17122.2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Rashan A, Beane A, Ghose A, et al. : Supplementary materials for 'Mixed methods study protocol for combining stakeholder-led rapid evaluation with near real-time continuous registry data to facilitate evaluations of quality of care in intensive care units'.[Dataset] Figshare. 2022. 10.6084/m9.figshare.21763325 [DOI] [PMC free article] [PubMed]
  • 18. World Health Organisation: Global antimicrobial resistance and use surveillance system (GLASS). [Accessed 12/09/2022]. Reference Source
  • 19. Kahn MG, Callahan TJ, Barnard J, et al. : A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. EGEMS (Wash DC). 2016;4(1):1244. 10.13063/2327-9214.1244 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Sessler CN, Gosnell MS, Grap MJ, et al. : The Richmond Agitation-Sedation Scale: validity and reliability in adult intensive care unit patients. Am J Respir Crit Care Med. 2002;166(10):1338–44. 10.1164/rccm.2107138 [DOI] [PubMed] [Google Scholar]
  • 21. Pari V, Collaboration for Research Implementation, Training in Critical Care, et al. : Development of a quality indicator set to measure and improve quality of ICU care in low- and middle-income countries. Intensive Care Med. 2022;48(11):1551–1562. 10.1007/s00134-022-06818-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22. Shah N, Bhagwanjee S, Diaz J, et al. : A roadmap for acute care training of frontline healthcare workers in LMICs. J Crit Care. 2017;41:313–7. 10.1016/j.jcrc.2017.07.028 [DOI] [PubMed] [Google Scholar]
  • 23. Kotfis K, Zegan-Barańska M, Szydłowski Ł, et al. : Methods of pain assessment in adult intensive care unit patients - Polish version of the CPOT (Critical Care Pain Observation Tool) and BPS (Behavioral Pain Scale). Anaesthesiol Intensive Ther. 2017;49(1):66–72. 10.5603/AIT.2017.0010 [DOI] [PubMed] [Google Scholar]
  • 24. Ely E, Gautam S, Margolin R, et al. : The impact of delirium in the intensive care unit on hospital length of stay. Intensive Care Med. 2001;27(12):1892–900. 10.1007/s00134-001-1132-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25. The International Training and Education Center for Health: Rapid Evaluation: A Technical Implementation Guide. Seattle, Washington: University of Washington, University of California San Francisco,2008. [Google Scholar]
  • 26. Vindrola-Padros C, Pape T, Utley M, et al. : The role of embedded research in quality improvement: a narrative review. BMJ Qual Saf. 2017;26(1):70–80. 10.1136/bmjqs-2015-004877 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27. Antonacci G, Lennox L, Barlow J, et al. : Process mapping in healthcare: a systematic review. BMC Health Serv Res. 2021;21(1): 342. 10.1186/s12913-021-06254-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28. Gale NK, Heath G, Cameron E, et al. : Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13:117–8. 10.1186/1471-2288-13-117 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Carpiano RM: Come take a walk with me: the "go-along" interview as a novel method for studying the implications of place for health and well-being. Health Place. 2009;15(1):263–72. 10.1016/j.healthplace.2008.05.003 [DOI] [PubMed] [Google Scholar]
  • 30. Evans J, Jones P: The walking interview: Methodology, mobility and place. Appl Geogr. 2011;31(2):849–58. 10.1016/j.apgeog.2010.09.005 [DOI] [Google Scholar]
  • 31. Kaplan HC, Provost LP, Froehle CM, et al. : The Model for Understanding Success in Quality (MUSIQ): building a theory of context in healthcare quality improvement. BMJ Qual Saf. 2012;21(1):13–20. 10.1136/bmjqs-2011-000010 [DOI] [PubMed] [Google Scholar]
  • 32. Crit Care Asia: Establishing a critical care network in Asia to improve care for critically ill patients in low- and middle-income countries. Crit Care. 2020;24(1): 608. 10.1186/s13054-020-03321-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33. Aryal D, Beane A, Dondorp AM, et al. : Operationalisation of the Randomized Embedded Multifactorial Adaptive Platform for COVID-19 trials in a low and lower-middle income critical care learning health system. [version 1; peer review: 3 approved]. Wellcome Open Res. 2021;6:14. 10.12688/wellcomeopenres.16486.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34. Beane A, De Silva AP, Athapattu PL, et al. : Addressing the information deficit in global health: lessons from a digital acute care platform in Sri Lanka. BMJ Glob Health. 2019;4(1):e001134. 10.1136/bmjgh-2018-001134 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35. Adhikari NKJ, Arali R, Attanayake U, et al. : Implementing an intensive care registry in India: preliminary results of the case-mix program and an opportunity for quality improvement and research [version 2; peer review: 2 approved]. Wellcome Open Res. 2020;5:182. 10.12688/wellcomeopenres.16152.2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36. Asia CRIT Care, . Hashmi M, Beane A, et al. : Leveraging a Cloud-Based Critical Care Registry for COVID-19 Pandemic Surveillance and Research in Low- and Middle-Income Countries. JMIR Public Health Surveill. 2020;6(4):e21939. 10.2196/21939 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37. REMAP-CAP: REMAP-CAP results. 2021; [Accessed 30th June 2022]. Reference Source
  • 38. Shen YC: The effect of financial pressure on the quality of care in hospitals. J Health Econ. 2003;22(2):243–69. 10.1016/S0167-6296(02)00124-8 [DOI] [PubMed] [Google Scholar]
  • 39. Campbell S, Greenwood M, Prior S, et al. : Purposive sampling: complex or simple? Research case examples. J Res Nurs. 2020;25(8):652–61. 10.1177/1744987120927206 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Malterud K, Siersma VD, Guassora AD: Sample size in qualitative interview studies: guided by information power. Qual Health Res. 2016;26(13):1753–60. 10.1177/1049732315617444 [DOI] [PubMed] [Google Scholar]
  • 41. Anker M, Guidotti RJ, Orzeszyna S, et al. : Rapid evaluation methods (REM) of health services performance: methodological observations. Bull World Health Organ. 1993;71(1):15–21. [PMC free article] [PubMed] [Google Scholar]
  • 42. Vindrola-Padros C, Johnson GA: Rapid Techniques in Qualitative Research: A Critical Review of the Literature. Qual Health Res. 2020;30(10):1596–1604. 10.1177/1049732320921835 [DOI] [PubMed] [Google Scholar]
  • 43. Olmos-Vega FM, Stalmeijer RE, Varpio L, et al. : A practical guide to reflexivity in qualitative research: AMEE Guide No. 149. Med Teach. 2022;1–11. 10.1080/0142159X.2022.2057287 [DOI] [PubMed] [Google Scholar]
  • 44. Bakhshi-Raiez F, Cornet R, De Keizer N: Cross-mapping APACHE IV "reasons for intensive care admission" classification to SNOMED CT. Stud Health Technol Inform. 2008;136:779–84. [PubMed] [Google Scholar]
  • 45. Tirupakuzhi Vijayaraghavan BK, Priyadarshini D, Rashan A, et al. : Validation of a simplified risk prediction model using a cloud based critical care registry in a lower-middle income country. PLoS One. 2020;15(12):e0244989. 10.1371/journal.pone.0244989 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46. Creswell JW, Creswell JD: Research design: Qualitative, quantitative, and mixed methods approaches.Sage publications; 2017. Reference Source [Google Scholar]
  • 47. CFIR Research Team-Center for Clinical Management Research: Consolidated Framework for Implementation Research. 2022; [Accessed 30th June 2022]. Reference Source
  • 48. Smith J, Firth J: Qualitative data analysis: the framework approach. Nurse Res. 2011;18(2):52–62. [DOI] [PubMed] [Google Scholar]
  • 49. Strauss ME, Smith GT: Construct validity: Advances in theory and methodology. Annu Rev Clin Psychol. 2009;5:1–25. 10.1146/annurev.clinpsy.032408.153639 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50. Graham Ingram JB, Pinhorn M: Guidelines for using Zoom. 2022; [Accessed 30th June 2022]. Reference Source
  • 51. Voigt P, Von dem Bussche A: The EU general data protection regulation (GDPR): A Practical Guide. 1st Ed Cham: Springer International Publishin. 2017;10:10–5555. 10.1007/978-3-319-57959-7 [DOI] [Google Scholar]
  • 52. TRUST: Global Code of Conduct for Research in Resource-Poor Settings. 2018; [Accessed 30th June 2022]. Reference Source
  • 53. Oxford Tropical Research Ethics Committee (OxTREC): Reseasrch support: Facilitating world-class research: Governance & integrity. 2021; [Accessed 30th June 2022]. Reference Source
  • 54. Sheron VA, Shanmugathas S, Gooden TE, et al. : Healthcare provider and patient perspectives on access to and management of atrial fibrillation in the Northern Province, Sri Lanka: A rapid evaluation of barriers and facilitators to care. 2022. 10.21203/rs.3.rs-1535559/v1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55. Articulate Global: Rise. 2022; [Accessed 30th June 2022]. Reference Source
  • 56. CRediT: Contributor Roles Taxonomy. 2022; [Accessed 30th June 2022]. Reference Source
  • 57. Collet JP, Skippen PW, Mosavianpour MK, et al. : Engaging pediatric intensive care unit (PICU) clinical staff to lead practice improvement: the PICU participatory action research project (PICU-PAR). Implement Sci. 2014;9: 6. 10.1186/1748-5908-9-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58. Vindrola-Padros C, Vindrola-Padros B: Quick and dirty? A systematic review of the use of rapid ethnographies in healthcare organisation and delivery. BMJ Qual Saf. 2018;27(4):321–30. 10.1136/bmjqs-2017-007226 [DOI] [PubMed] [Google Scholar]
  • 59. Jamtvedt G, Flottorp S, Ivers N: Audit and Feedback as a Quality Strategy.Copenhagen, Denmark: European Observatory on Health Systems and Policies; 2019. Reference Source [Google Scholar]
Wellcome Open Res. 2023 Nov 10. doi: 10.21956/wellcomeopenres.22532.r69493

Reviewer response for version 3

Heather Schacht Reisinger 1

Thank you for your thorough response to my review. I'm looking forward to reading more about this study in the future.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Ethnographic and qualitative methods, implementation science, telemedicine. critical care, mental health, US veterans

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Wellcome Open Res. 2023 Sep 29. doi: 10.21956/wellcomeopenres.21150.r66196

Reviewer response for version 2

Heather Schacht Reisinger 1

This manuscript provides a description of an ongoing study on quality improvement in ICUs in LMICs. The study protocol is impressive and could make major contributions on the level of global impact in ICU care in LMICs. The multi-level methods are well-described and there is strong integration of various stakeholders, including patients and caregivers.

My major recommendation is to move the first four subsections under Protocol (Registry enabled…, The selection of measures…, Rapid evaluation, and Assessment of organizational readiness…) toward the end or integrated into the relevant methods into each section (i.e., general overview immediately followed by details of how that element of the evaluations is going to be conducted). Having the overview sections before the details of the protocol led me to ask a lot of questions about how the elements of each of the sections were being carried out. (I had a lot more critiques as I was reviewing the manuscript that I later deleted because it was there, just not where I was expecting it.)

Registry and Selection of Measures:

The set of agile ICU registry platform is impressive, but a few things are unclear from this section of the evaluation. First, is there a core dataset (enabling evaluation of case mix and risk stratified clinical outcomes) AND a new addition of embedded metric(s) to enable daily data collection? Are only the new embedded metrics found in Table 1 or is that a comprehensive set? Also, I am confused by the “recently added the metric for enabling evaluation of care embedded for daily data collection (Table 1)”? It appears there are multiple metric s in Table 1. What does “embedded daily data collection” mean? I think this could be cleared up by integrating the section and the details of the methods.

Rapid Evaluation:

The methods for rapid evaluation are well described. One question I had was which clinical areas will the process maps focus on (e.g., sepsis protocol, vent management, etc.)? Is this pre-determined or will each ICU make their own determination of processes to map?

Organizational readiness:

It would be helpful to know how MUSIQ 2 and organizational readiness for care improvement will be integrated into the other components of the evaluation beyond future planning.

Discussion:

Why was the "learning by doing model" introduced in the Discussion? I’m not sure it added anything to the manuscript to bring it up at the end. If it helped in developing the protocol, it would be helpful to have it at the beginning of the manuscript and more fully described. The discussion focuses a lot on participatory research, patient-centeredness, and stakeholder-led evaluation. While this is part of the work described in the protocol, it was not a key theme in until the Discussion. It may not be needed in the Discussion or it should be highlighted more in the Introduction and Abstract. In contrast to this comment, the Discussion should have more on the impact on global ICU quality improvement that is expected from this protocol and the future impacts. This is how the manuscript was framed in the beginning and it did not return enough to this theme and impact at the end. A minor comment: What was meant by “anthropological methods…may disenfranchise stakeholders from accepting findings, or using them to positive influence service improvement.” This is counter to a lot of anthropological research currently being conducted.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Ethnographic and qualitative methods, implementation science, telemedicine. critical care, mental health, US veterans

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Wellcome Open Res. 2023 Oct 24.
Tiffany Gooden 1

Dear Dr Heather Reisinger, Thank you for taking the time to review our manuscript. We hope you find our responses to your comments below and the corresponding edits to the manuscript has elevated our protocol for approval.  

  1. Reorganisation of subsections: Thank you for this suggestion, we have now moved text around within the protocol methods whereby much of the text that was previously under ‘study design’ is now incorporated into the text under ‘data collection’. We hope this makes the description of the methods easier to follow.

  2. Registry and selection of measures: Table 1 presents the metrics only for evaluation of care; the registry also comprises a core dataset to evaluate case mix and risk stratification. The core dataset is described in detail within a separate publication. We have amended the text to make this clearer. It now reads as follows:

    “The components of a learning health system (continuous data for evaluation and new knowledge generation, rapidly integrated into practice) already established by the CCAA will be enhanced by the rapid evaluation, facilitating future practice improvement. The CCAA already has an ICU registry with an established core data set which enables comparable description of case mix, severity of illness and benchmarking of clinical outcomes; the details of the core dataset are described elsewhere 16. Additionally, the CCAA recently completed a scoping review of ICU quality metrics used internationally and undertook a four round modified Rand Delphi study to identify a set of indicators for evaluation 21. Indicators were assessed for their feasibility, reliability, validity to predict outcome and sensitivity to change. They have been classified according to the High Quality Health Systems Framework 4: foundations (encompassing human resources, governance structures, accessibility and tools), quality impacts (including clinical and economic outcomes) and care processes (descriptions of care and systems as well as user-experience). These indicators were then prioritised and defined for implementation through the registry. The care processes associated with these priorities are already endorsed by healthcare policy makers in all but the most resource constrained health systems in LMICs 22. The selected metrics are listed in Table 1 and reflect previously identified research priorities for improvement: reducing avoidable harms; improving delivery of interventions and processes already proved to improve outcomes and measuring and improving patient-centred outcomes 21. Avoidable harms include deep venous thrombosis, stress ulcer, ICU delirium, neuromuscular decline, pressure injury and healthcare-acquired infections. These harms are all associated with prolonged organ support including mechanical ventilation 8. The associated daily care processes and interventions proven to ameliorate these harms include optimising sedation and pain management using objective assessment tools (RASS, CPOT); daily assessment of readiness to wake (SAT), assessment of ability to breath spontaneously (SBT) and passive or active mobilization 23. Foundations of care, for example nurse:patient ratios, have also been shown to influence these harms. In addition, engagement of family members in daily care in the ICU and their involvement in interventions including respiratory physiotherapy and mobilisation has been demonstrated to improve not only the effectiveness of the interventions, but also to increase both provider and patient compliance, and promote better experience for families and patients 8. The feasibility of these measures (three foundation, eight quality impacts and eight care processes) and their definitions 17 are currently being assessed 9, 18 for feasibility of collection (using the Khan quality framework assessment criteria for conformance and completeness) 19 in pilot ICUs in collaborating registries including IRIS (Indian Registry of IntenSive care), NICRF (Nepal Intensive Care Research Foundation), CCSK (Critical Care Society of Kenya) and South Africa.”

  3. Rapid evaluation: The process mapping will focus on locally-selected care processes from the list in Table 1. We have clarified this now in the manuscript, as below.

    “Stakeholders will be asked to map out the discrete activities (tasks) associated with locally-selected care processes from the list in Table 1, identify the actors and equipment involved, the interactions, decision making and who are the decision makers, for care as intended, and for care as it is actually delivered.”

  4. Organisational readiness: The MUSIQ 2 calculator will be used to enable stakeholders to reflect upon how their organisational culture may influence future quality improvement efforts. We have amended the discussion to make this clearer, as below.

    “The findings of the project will directly inform subsequent local projects aimed at improving patient care and the development of the CCAA registry to enable effective utilisation of data to drive quality improvement for stakeholders and their ICUs. By understanding organisational readiness for improvement, we will enable stakeholders to plan interventions best suited to their local cultures, needs and resources.  Further to this, given this project’s participatory nature, and having been co-designed by clinical stakeholders, novice researchers and clinicians will be exposed to new methods. We anticipate this approach will facilitate the building research and quality improvement capacity in the CCAA which will extend beyond this project.”

Discussion:

Comment 1: Why was the "learning by doing model" introduced in the Discussion? I’m not sure it added anything to the manuscript to bring it up at the end. If it helped in developing the protocol, it would be helpful to have it at the beginning of the manuscript and more fully described.

Response 1: We have removed the text referring to the learning by doing model from the discussion. It now reads as follows. “In this study, we aim to simultaneously lay the foundations for a culture of healthcare improvement and establish capacity for future institution-led research through which healthcare providers, families and patients reflect and appraise current practice, to identify problems and seek possible solutions.”

Comment 2: The discussion focuses a lot on participatory research, patient-centeredness, and stakeholder-led evaluation. While this is part of the work described in the protocol, it was not a key theme in until the Discussion. It may not be needed in the Discussion or it should be highlighted more in the Introduction and Abstract.

Response 2: We mention participatory research, patient-centeredness and stakeholder-led evaluation in the discussion because these are core parts of the methodology described for this protocol; as mentioned in the introduction, part of the challenge in measuring and improving quality of care in ICUs is involving key stakeholders (i.e. healthcare professionals, patients and carers). The project we describe will overcome these challenges through participatory research and stakeholder-led evaluations. We allude to patient centeredness in the introduction, but the term was not mentioned; we have now added that into the text. We have also amended the abstract and discussion slightly to make this clearer (see below) Abstract: “Methods: Using rapid evaluation methods, we will use four data collection methods: 1) registry embedded indicators to assess quality of care processes and their associated outcomes; 2) process mapping to provide a preliminary framework to understand gaps between current and desired care practices; 3) structured observations of processes of interest identified from the process mapping and; 4) focus group discussions with stakeholders to identify barriers and enablers influencing the gap between current and desired care practices. We will also collect self-assessments of readiness for quality improvement. Data collection and analysis will be led by local stakeholders, performed in parallel and through an iterative process across eight countries: Kenya, India, Malaysia, Nepal, Pakistan, South Africa, Uganda and Vietnam.” Intro: “In addition, patient-centred care and engagement with families and patients to share experiences of critical care is often overlooked despite being of itself, associated with improved outcomes 8.” Discussion: “Measurement of key quality indicators, which include patient-centred outcomes, to drive forward improvement is increasingly being promoted as part of benchmarking healthcare quality.”

Comment 3: the Discussion should have more on the impact on global ICU quality improvement that is expected from this protocol and the future impacts. This is how the manuscript was framed in the beginning and it did not return enough to this theme and impact at the end.

Response 3: Our focus in the introduction and discussion is around quality improvement and the ways in which quality improvement is measured. In the introduction, we discuss the importance of quality improvement in ICUs but the challenges of achieving this in LMICs and the global issue of not including healthcare professionals and patients in such initiatives. In the discussion, we summarise how and why the methods we outline can overcome these challenges, include healthcare professionals and patients, and inform quality improvement initiatives in ICUs in LMICs. We have however added some text to the discussion on the impact we expect this research to have on ICU quality improvement (see below). “In this study, we aim to simultaneously lay the foundations for a culture of healthcare improvement and establish capacity for future institution-led research through which healthcare providers, families and patients reflect and appraise current practice, to identify problems and seek possible solutions. As a result, this research has the potential to inform sustainable and effective ICU quality improvement in LMICs for both clinical outcomes and patient-family experiences.”

Comment 4: What was meant by “anthropological methods…may disenfranchise stakeholders from accepting findings, or using them to positive influence service improvement.” This is counter to a lot of anthropological research currently being conducted.

​​​​​​​ Response 4: The mention of anthropological methods is simply referring to the length of time it takes to make conclusions from anthropological studies and subsequent improvements to services. We have amended the text to make this clearer (see below). “Stakeholder-led rapid research can provide timely insight into specific, complex processes and systems from locally-defined perspectives, which if attempted using more traditional ethnographic methods may take many months before conclusions are made, risking disengagement of stakeholders and delaying service improvement 57.”

Wellcome Open Res. 2023 Feb 6. doi: 10.21956/wellcomeopenres.20748.r54370

Reviewer response for version 1

Maryati Mohd Yusof 1

The authors have outlined a comprehensive protocol for evaluating ICU quality. The methods are clearly described and well-planned. However, more details can be added:

  1. The interview agenda is missing;

  2. qualitative data analysis needs further clarification in terms of deductive/inductive approach, thematic analysis, coding, and data representation;

  3. process mapping specification: general or specific (e.g. Lean or BPM);

  4. study quality measures - e.g. member checking, construct validity.

I wish the authors all the best for a very promising study.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

Health information systems evaluation, qualitative methods, quality improvement, Lean healthcare

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Wellcome Open Res. 2023 Feb 15.
Tiffany Gooden 1

Dear Dr Maryati Yusof, Many thanks for reviewing our manuscript and proving such a positive assessment. Please see below our response to your individual comments.

  1. Within our supplemental materials ( https://doi.org/10.6084/m9.figshare.21763325), we have included the topic guide for the focus group discussions (file 3). Here we note that the topics, or agenda will comprise of the following, though these may be iteratively modified based on the findings of the process mapping and structured observations:
    1. For healthcare providers:
      1. Experiences with managing critically ill patients generally
      2. Challenges with specific processes of care for critically ill patients
    2. For patients and family members:
      1. Experiences of ICU generally
      2. Experiences specific to care processes

  2. The qualitative data will be analysed using an interpretive analysis approach, a form of thematic analysis, and coded both inductively and deductively using the Consolidated Framework for Implementation Research framework.

  3. We have added the following text to the 2nd paragraph of the ‘Data Collection’ section: 'We will use a general process mapping technique rather than one allied to a specific improvement methodology' followed by this reference which describes the methods we plan to use: Vindrola-Padros C, Johnson GA. Rapid techniques in qualitative research: A critical review of the literature. Qualitative Health Research. 2020;30(10):1596-1604. doi:10.1177/1049732320921835. 

  4. Please see existing descriptions of triangulation of findings within and between sites in the 3rd paragraph of ‘Data analysis and management’ section. Furthermore, we have added the following text: 'The validity of findings will be discussed and checked with process mapping, observation and focus group participants at every site' followed by this reference: Strauss ME, Smith GT. Construct validity: Advances in theory and methodology. Annual Review of Clinical Psychology. 2009 Apr 27;5:1-25. 

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Citations

    1. Rashan A, Beane A, Ghose A, et al. : Supplementary materials for 'Mixed methods study protocol for combining stakeholder-led rapid evaluation with near real-time continuous registry data to facilitate evaluations of quality of care in intensive care units'.[Dataset] Figshare. 2022. 10.6084/m9.figshare.21763325 [DOI] [PMC free article] [PubMed]

    Data Availability Statement

    Underlying data

    No underlying data are associated with this article.

    Extended data

    Figshare: Supplementary materials for 'Mixed methods study protocol for combining stakeholder-led rapid evaluation with near real-time continuous registry data to facilitate evaluations of quality of care in intensive care units'. https://doi.org/10.6084/m9.figshare.21763325 17

    This project contains the following extended data:

    • -

      File 1: Data Completion Guide

    • -

      File 2: Mapping Session Guide

    • -

      File 3: Focus Group Discussion Topic Guides

    • -

      File 4: Invitation Email to CCAA Country Site Leads

    • -

      File 5: Participant Information Sheet: Observations

    • -

      File 6: Participant Information Sheet: Interviews

    • -

      File 7: Observation Template Sheet

    • -

      File 8: Rapid Assessment Process (RAP) Sheet

    Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

    Articles from Wellcome Open Research are provided here courtesy of The Wellcome Trust

    RESOURCES