Abstract
Background
It is imperative to maintain accurate documentation of clinical interventions aimed at enhancing the quality of care for newborns and sick children. The National Newborn Health and IMCI programme of Bangladesh led the development of a standardised register for managing newborns and sick children under five years of age during inpatient care through stakeholder engagement. We aimed to assess the implementation outcomes of the standardised register in the inpatient department.
Methods
We conducted implementation research in two district hospitals and two sub-district hospitals of Kushtia and Dinajpur districts from November 2022 to January 2023 to assess the implementation outcomes of the standardised register. We assessed the following World Health Organization implementation outcome variables: usability, acceptability, adoption (actual use), fidelity (completeness and accuracy), and utility (quality of care) of the register against preset benchmarks. We collected data through structured interviews with health care providers; participant enrolment; and data extraction from inpatient registers and case record forms.
Results
The average usability and acceptability scores among health care providers were 73 (standard deviation (SD) = 14) and 82 (SD = 14) out of 100, respectively. The inpatient register recorded 96% (95% confidence interval (CI) = 95–97) of under-five children who were admitted to the inpatient department (adoption – actual use). The proportions of completed data elements in the inpatient register were above the preset benchmark of 70% for all the assessed data elements except ‘investigation done’ (24%; 95% CI = 23–26) (fidelity – completeness). The percentage agreements between government-appointed nurses posted and study-appointed nurses were above the preset benchmark of 70% for all the reported variables (fidelity – accuracy). The kappa coefficient for the overall level of agreement between these two groups regarding reported variables indicated moderate to substantial agreement. The proportion of newborns with sepsis receiving injectable antibiotics was 62% (95% CI = 47–75) (utility – quality of care). We observed some variability in the completeness and accuracy of the inpatient register by district and facility type.
Conclusions
The inpatient register was positively received by health care providers, with evaluations of implementation outcome variables showing encouraging results. Our findings could inform evidence-based decision-making on the implementation and scale-up of the inpatient register in Bangladesh, as well as other low- and middle-income countries.
Bangladesh has significantly reduced its under-five mortality rate from 48 deaths per thousand live births to 31 deaths per thousand live births between 2011 and 2022 [1–4]. Despite this progress, around 80 000 children under five years of age still die each year in Bangladesh [5]. Most of these deaths are avoidable and occur due to causes such as pneumonia; serious infections; prematurity and low birth weight; and intrapartum-related complications, with nearly two-thirds occurring within the first 28 days of life [4]. To prevent these deaths, it is important to improve the quality of care provided to newborns and sick children in hospital settings.
Improving documentation practices by strengthening the patient record system and documentation flow can significantly improve quality of care in health facilities, as it can enable effective disease monitoring, the exploration of health care delivery patterns, and the generation of practical evidence of the impact of medical care and service delivery models for children [6–8]. A comprehensive health systems approach which would include the timely provision of quality inpatient care, the identification of individuals at high risk, and the recording and reporting clinical data can save many lives and prevent morbidity [9–11].
Bangladesh has scaled up highly effective evidence-based interventions targeting major causes of under-five deaths. These include immediate newborn care and helping babies breathe for birth asphyxia; the administration of antenatal corticosteroid for suspected preterm labour; kangaroo mother care for preterm and low birth-weight babies; the administration of injectable antibiotics for serious infections, and the establishment of special care newborn units (SCANUs) for critically ill newborns [12–15]. The country has also introduced standardised service registers in corners related to newborn and child care targeting these interventions, such as postnatal care corners, integrated management of childhood illness (IMCI) corners; labour wards and operation theatres; SCANUs; and kangaroo mother care corners. Monthly reports are generated from these service registers and are submitted to routine health information systems. However, there is no dedicated service register or monthly reporting form in the paediatric inpatient department, where a substantial number newborns and sick children are managed every year.
Acknowledging this, the National Newborn Health and IMCI programme (NNHP & IMCI programme) of the Directorate General of Health Services took the initiative to develop and introduce a standardised register for managing newborns and sick children during inpatient care. It also decided to implement this standardised register in select districts and to conduct research to assess implementation outcomes before national scale up. In response, we wanted to assess the implementation outcomes of the standardised register for inpatient management of newborns and sick children in select districts to inform evidence-based scale up in Bangladesh.
METHODS
Development and implementation of inpatient register
The NNHP & IMCI programme led the design and development of this register, along with an implementation plan through stakeholder engagement. This register was rolled out in multiple districts across Bangladesh with technical assistance from development partners through four steps – the development process; training; district implementation; and monitoring and supervision.
Development process
The NNHP & IMCI programme established a technical committee to develop an inpatient register for newborns and sick children with members from government departments under the Directorate General of Health Services and in coloration with development partners. This committee investigated documentation practices in district and sub-district hospitals, particularly within the paediatric inpatient department, where they collected data from case record forms and registers to gain insights on how paediatric patients were being managed. They also reviewed various national and global guidelines and strategies, as well as registers and case record forms available worldwide. The committee examined similar inpatient registers used within Bangladesh, such as the kangaroo mother care register and the SCANU register. Afterwards, four consultative workshops were organised to develop the inpatient register, which was structured in such a way that the details of one patient can be documented on a single page. Each page is divided into two rows, with the first row to be completed upon admission and the second to be filled out at the time of discharge (Table S1 of the Online Supplementary Document). Nurses working in the inpatient departments are responsible for filling up this register based on the information recorded in the individual patient case record forms. The technical committee also developed a monthly reporting form based on the register which participating facilities are meant to complete every month and send to the NNHP & IMCI programme.
Training
In October 2022, the NNHP & IMCI programme organised four training sessions for master trainers in Dhaka. Each facility had at least two trained master trainers who subsequently conducted training for the above-mentioned nurses.
District implementation
The inpatient register was implemented in five districts between November 2022 and January 2023, with technical assistance from icddr,b, Save the Children, the United Nations Children’s Fund (UNICEF), and the World Health Organization (WHO). Forty health facilities were involved, including one medical college hospital, four district hospitals, and 35 sub-district hospitals.
Monitoring and supervision
The NNHP & IMCI programme and health managers at district and sub-district levels, along with the master trainers, were jointly responsible for monitoring and supervising the register’s implementation. The challenges of implementing the register and insights gained from monthly reporting forms were discussed during monthly meetings at participating facilities. Monthly reporting forms were sent to the NNHP & IMCI programme, based on which online meetings were organised to assess and review progress.
Study design
The icddr,b carried out an implementation research to assess the implementation outcomes of the standardised register in two districts in Bangladesh. The research itself was designed as a cross-sectional study. Based on the decision of the technical committee, the implementation outcomes were adapted from the framework for implementation research developed by the WHO [16]. The primary implementation outcomes were the usability, acceptability, adoption (actual use), fidelity (completeness and accuracy), and utility (quality of care) of the register. The technical committee set a benchmark to successfully demonstrate each outcome variable. The primary research questions are detailed in Table 1.
Table 1.
Primary research questions along with benchmark set by technical committee to successfully demonstrate each outcome variable based on framework for implementation research developed by WHO
| WHO’s framework | Research questions | Proposed indicator | Benchmark |
|---|---|---|---|
| Usability |
What is the usability of the inpatient register among health care provider? |
Average score using the SUS |
>70 |
| Acceptability |
What is the acceptability of the inpatient register among health care provider? |
Average score using the TAM |
>80 |
| Adoption |
Actual use: do health care providers use the inpatient register? |
Proportion of admitted newborn and children documented in the inpatient register |
>80% |
| Fidelity | Completeness: do health care providers fill up ‘investigation done’ column of the inpatient register? |
Proportion of completed ‘investigation done’ column among newborn and children recorded in the inpatient register |
>70% |
| Completeness: do health care providers fill up ‘care received during admission’ column in the inpatient register? |
Proportion of completed ‘care received during admission’ column among newborn and children recorded in the inpatient register |
>70% |
|
| Completeness: do health care providers fill up ‘drugs received during admission’ column in the inpatient register? |
Proportion of completed ‘drugs received during admission’ column among newborn and children recorded in the inpatient register |
>70% |
|
| Completeness: do health care providers fill up ‘final diagnosis’ column in the inpatient register? |
Proportion of completed ‘final diagnosis’ column among newborn and children recorded in the inpatient register |
>70% |
|
| Completeness: do health care providers fill up ‘outcome of treatment’ column in the inpatient register? |
Proportion of completed ‘outcome of treatment’ column among newborn and children recorded in the inpatient register |
>70% |
|
| Accuracy: what is the level of observed agreement between government-appointed nurses and study-appointed nurses for ‘oxygen given’ variable in the inpatient register? |
Level of observed agreement between government-appointed nurses and study-appointed nurses regarding recording ‘oxygen given’ variable in the inpatient register |
>70% |
|
| Accuracy: what is the level of observed agreement between government-appointed nurses and study-appointed nurses for ‘injectable gentamicin’ variable in the inpatient register? |
Level of observed agreement between government-appointed nurses and study-appointed nurses regarding recording ‘injectable gentamicin’ variable in the inpatient register |
>70% |
|
| Accuracy: what is the level of observed agreement between government-appointed nurses and study-appointed nurses for ‘diagnosis – severe pneumonia’ variable in the inpatient register? |
Level of observed agreement between government-appointed nurses and study-appointed nurses regarding recording ‘diagnosis – severe pneumonia variable in the inpatient register |
>70% |
|
|
|
Accuracy: what is the level of observed agreement between government-appointed nurses and study-appointed nurses for ‘refer’ variable in the inpatient register? |
Level of observed agreement between government-appointed nurses and study-appointed nurses regarding recording ‘refer’ variable in the inpatient register |
>70% |
| Utility | Quality of care: do the admitted children receive injectable antibiotic for newborn sepsis? | Proportion of admitted children receiving injectable antibiotic for newborn sepsis | >70% |
TAM – technology acceptance model, SUS – system usability scale
Study settings
We carried out the study in the Kushtia and Dinajpur districts of Bangladesh due to them having similar newborn and under-five mortality rates to the national estimates (Figure S1 of the Online Supplementary Document). The Kushtia district, situated in the Khulna division of western Bangladesh, has a population of more than two million. The Dinajpur district, located in the Rangpur division of northern Bangladesh, has around three and a half million inhabitants. Both districts primarily consist of rural areas [17]. These districts have well-functioning inpatient and outpatient care for newborns and sick children, owing to several maternal, newborn, and child health initiatives undertaken by the government. One district hospital and one sub-district hospital (also known as a upazila health complex) were selected from each district in consultation with the NNHP & IMCI programme and district health managers (civil surgeons). These were the Kumarkhali sub-district hospital from the Kushtia district and Hakimpur sub-district hospital from the Dinajpur district, which serve as the first referral facility for inpatient management of newborns and sick children. In turn, the Kushtia and Dinajpur district hospitals serve as the secondary referral facility covering the whole district.
Study participants
We included children aged between 0 and 59 months who had been admitted to the paediatric indoor department of the participating health facilities, as well as government-employed nurses who worked in the paediatric indoor department and had received training on the inpatient register.
Sample size and sampling
We calculated the sample size based on the preset benchmark so as to successfully answer each primary research question. Assuming a 5% margin of error, 95% confidence interval (CI), and 5% non-response rate, the highest adjusted sample size for the adoption, fidelity, and utility indicators was 340 children aged between 0 and 59 months who were admitted to the paediatric indoor department per facility per month. Therefore, the total sample size was 4080 children. We approached all the children admitted to the inpatient department of the participating health facilities for enrolment.
Assuming a 10% margin of error, 95% confidence interval, and 10% non-response rate, the highest adjusted sample size for the usability and acceptability indicators was 90 government-employed nurses who had worked in the paediatric indoor department and had received training on the inpatient register. We approached all such nurses in the Kushtia and Dinajpur districts for enrolment.
Data collection
We carried out our study during a three-month period from 1 November 2022 to 31 January 2023. We collected the data using structured interview of the health care providers; participant enrolment; and data extraction from inpatient register and case record forms. Prior to finalisation, all data collection tools, including a custom-designed Android application, underwent separate field testing with a minimum of five participants. We obtained administrative approvals from the NNHP & IMCI programme, including for the patient enrolment and data extraction from the inpatient registers and the case record forms.
Structured interviews
Using adapted versions of the System Usability Scale (SUS) [18] and the Technology Acceptance Model (TAM) tool [19], we conducted structured interviews with nurses who had received training on the paediatric inpatient register in the Kushtia and Dinajpur districts to assess the usability and acceptability of the inpatient register among health care providers. The SUS, a dependable tool for evaluating the usability of systems, products, or services, comprises a 10-item questionnaire featuring five response options, ranging from ‘strongly agree’ to ‘strongly disagree.’ SUS scores range from 0 to 100 and provide insights into overall user satisfaction and usability [18]. The TAM, meanwhile, is a widely recognised psychological model used to understand how health care providers perceive and embrace new information technology or systems [19]. We obtained written informed consents from the health care providers prior to the structured interview.
Participant enrolment
Using a structured participant enrolment tool, non-medical study staffs enrolled patients by collecting basic demographic information (name, age, gender, admission date and time, and registration number) for all admitted children in the paediatric inpatient department at the time of admission throughout the three-month study. We also obtained written informed consents from the mother/caregivers of the children at the time of enrolment.
Data extraction from inpatient register
Trained data collectors employed a structured data extraction tool to retrieve information from the inpatient register for children participating in the study and had subsequently been discharged from the hospital. A clinical supervisor oversaw the data extraction process, including the re-extraction of 5% of the observations.
Data extraction from case record forms
Trained study nurses extracted data from the case record forms for children whose information was obtained from the inpatient register using a specially designed data extraction tool similar to the inpatient register. We ensured that individuals involved in patient enrolment and data extraction from the case record forms had no access to the inpatient register's information. The same clinical supervisor monitored the case record form data extraction and performed 5% re-extraction, and provided feedback as required.
Data analysis
We assessed the inpatient register’s primary outcomes, including usability, acceptability, adoption, fidelity, and utility. We measured usability through the health care providers’ perception of the register’s usability and acceptability through their acceptance of the register. Moreover, we measured adoption by comparing the number of admitted children from patient enrolment with the children documented in the inpatient register. We measured fidelity – completeness by the completion of data elements such as investigation done, care received during admission, drugs received during admission, final diagnosis, and treatment outcome among children recorded in the inpatient register, and fidelity – accuracy through the percentage agreement on reporting variables (oxygen given; injectable gentamicin; diagnosis – severe pneumonia; and outcome of treatment – refer) between government-appointed nurses in the inpatient department and study-appointed nurses. Lastly, we measured utility through the reporting proportion of newborns with sepsis receiving injectable antibiotics.
We presented usability and acceptability as means with standard deviations (SDs) disaggregated by health care provider age, facility type, and district. We used independent t-tests to determine whether the difference of means between categories was significant. We presented adoption, fidelity, and utility as percentages with 95% CIs for patient’s age, sex, facility type, district, and month of assessment. We used proportion tests to determine whether the difference of percentage between sex, facility type, and district was significant. We also used analysis of variance (ANOVA) to check the difference for the categories of month of assessment and patient’s age. The statistical significance was reported at a 5% level of significance.
We also presented the agreement between government-appointed nurses and study-appointed nurses as a kappa (κ) coefficient to report accuracy, where agreement is interpreted per the following categories: poor (κ≤0.00), slight (κ = 0.01–0.19), fair (κ = 0.20–0.39), moderate (κ = 0.40–0.59), substantial (κ = 0.60–0.79), and perfect (κ≥0.80) [20]. Data analysis was conducted using Stata version 15.0 [21].
Ethical considerations
We received ethical approval (PR-21112) from the Institutional Review Board of the icddr,b. We also obtained informed written consent from the mother/caregiver of the children and the health care providers, and administrative approval from the NNHP & IMCI programme.
RESULTS
In total, 5052 under-five children were admitted in the paediatric inpatient department between 1 November 2022 and 31 January 2023. We extracted data from the inpatient register for 4849 (96%) children and from case record forms for 4180 (83%) children (Figure 1).
Figure 1.
Sample size for assessments.
For the inpatient register data, 2102 (43%) children were aged between 29 days and 11 months, 2770 (61%) were male, 4011 (83%) were from the Kushtia district, and 3909 (81%) were from district hospitals. Moreover, 2069 (52%) were discharged with advice, while 36 (1%) children died while hospital stay (Table 2).
Table 2.
Background characteristics of the participants according to data extracted from inpatient register (N = 4849)
| Characteristics | n (%) |
|---|---|
|
Age
|
|
| 0–28 days |
1210 (25.0) |
| 29 days to 11 months |
2102 (43.4) |
| 12 to 59 months |
1532 (31.6) |
| Missing |
5 |
|
Sex
|
|
| Male |
2770 (60.6) |
| Female |
1795 (39.3) |
| Others |
4 (0.1) |
| Missing |
280 |
|
District
|
|
| Kushtia |
4011 (82.7) |
| Dinajpur |
838 (17.3) |
| Missing |
0 |
|
Facility type
|
|
| District hospital |
3909 (80.6) |
| Sub-district hospital |
940 (19.4) |
| Missing |
0 |
|
Month of assessment
|
|
| Month 1 |
1517 (31.3) |
| Month 2 |
1705 (35.2) |
| Month 3 |
1627 (33.6) |
| Missing |
0 |
|
Outcome of treatment
|
|
| Discharge with advice |
2069 (52.1) |
| Discharge on request |
829 (20.9) |
| DORB |
78 (2.0) |
| Refer |
442 (11.1) |
| Absconded |
517 (13.0) |
| Death |
36 (0.9) |
| Missing |
878 |
| Total | 4849 |
DORB – discharge on risk bond
We enrolled 176 health care providers from the Kushtia and Dinajpur districts who had received training on inpatient register. All of them were nurses; 94 (53%) less than 35 years old, 111 (63%) were from the Dinajpur district, and 149 (85%) were from sub-district hospitals (Table S2 of the Online Supplementary Document).
For the assessments of the WHO implementation research outcomes for inpatient registers (Figure 2), the average usability score among health care providers according to the SUS was 73 (SD = 14) out of 100, while average acceptability score per the TAM was 82 (SD = 14) out of 100. The inpatient register recorded 96% (95% CI = 95–97) of under-five children who were admitted in the paediatric inpatient department (adoption- actual use). The proportions of completed data elements in the inpatient register were above the preset benchmark of 70% for all the assessed data elements, except for ‘investigation done’ (24%; 95% CI = 23–26) (fidelity – completeness). The percentage agreement between government-appointed nurses and study-appointed nurses were above the preset benchmark of 70% for all the reported variables (fidelity – accuracy), with the κ coefficient for the overall level of agreement between these two groups indicating moderate to substantial agreement (Table S3 of the Online Supplementary Document). However, the proportion of newborns with sepsis receiving injectable antibiotics according to data extracted from the inpatient register was 62% (95% CI = 47–75), which was below the preset benchmark of 70% (utility – quality of care).
Figure 2.
Assessments of WHO implementation research outcomes for inpatient register, presented as means with SDs for usability and acceptability indicators and percentages with 95% CIs for adoption, fidelity, and utility indicators (utility and acceptability: n = 176; adoption: n = 5052; fidelity-completeness and utility-outcome: n = 4849; fidelity-accuracy: n = 4180; utility-quality of care: n = 50).
In view of the influence of various factors on the implementation research outcomes for inpatient register, the mean SUS and TAM scores were significantly higher in the Dinajpur district, while the completeness and accuracy of the majority of the reported data elements were higher in the sub-district hospitals and the Dinajpur district (Table 3).
Table 3.
Influence of different factors on implementation research outcomes for inpatient register*
| Patient-related factors |
Provider-related factors |
Facility-related factors |
|||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Age
|
Sex
|
Age
|
Facility type
|
District
|
Month of assessments
|
||||||||||
|
|
0–28 days
|
29 d to 11 months
|
12–59 months
|
Male
|
Female
|
<35 years
|
≥35 years
|
District hospital
|
Sub-district hospital
|
Kushtia
|
Dinajpur
|
Month 1
|
Month 2
|
Month 3
|
|
| Usability |
x̄ (SD)
|
|
|
|
|
|
74(15) |
72(12) |
75 (19) |
73 (13) |
64 (16) |
78 (9) |
|
||
|
|
P-value
|
|
|
0.088 |
0.221 |
<0.001 |
|
||||||||
| Acceptability |
x̄ (SD)
|
|
|
|
|
|
83(14) |
81(13) |
72 (21) |
84 (11) |
69 (11) |
90 (7) |
|
||
|
|
P-value
|
|
|
0.169 |
0.005 |
<0.001 |
|
||||||||
| Adoption (actual use) |
% (95% CI)
|
96 (95–97) |
96 (95–97) |
96 (95–97) |
99 (98–99) |
99 (98–99) |
|
96 (95–97) |
96 (94–97) |
96 (95–97) |
96 (95–97) |
97 (95–97) |
96 (95–97) |
95 (94–96) |
|
|
|
P-value
|
0.256 |
0.675 |
|
0.669 |
0.838 |
0.287 |
||||||||
| Fidelity (completeness) investigation done |
% (95% CI)
|
24 (22–27) |
28 (26–30) |
20 (18–22) |
25 (23–26) |
24 (22–26) |
|
30 (28–31) |
2 (1–3) |
28 (27–29) |
8 (6–10) |
23 (21–26) |
27 (25–29) |
23 (21–25) |
|
|
|
P-value
|
<0.001 |
0.894 |
|
<0.001 |
<0.001 |
<0.001 |
||||||||
| Fidelity (completeness) – care received during admission |
% (95% CI)
|
80 (78–82) |
79 (77–81) |
73 (71–76) |
77 (75–78) |
79 (77–80) |
|
78 (77–79) |
75 (72–78) |
75 (73–77) |
80 (78–82) |
75 (73–77) |
80 (78–82) |
77 (75–79) |
|
|
|
P-value
|
<0.001 |
0.203 |
|
0.041 |
<0.001 |
0.006 |
||||||||
| Fidelity (completeness) – drugs received during admission |
% (95% CI)
|
86 (84–88) |
88 (87–89) |
80 (78–82) |
85 (84–87) |
85 (83–87) |
|
84 (83–85) |
89 (87–91) |
85 (84–86) |
85 (82–87) |
84 (82–86) |
85 (83–86) |
87 (85–88) |
|
|
|
P-value
|
<0.001 |
0.697 |
|
<0.001 |
0.838 |
0.113 |
||||||||
| Fidelity (completeness) – final diagnosis |
% (95% CI)
|
73 (71–77) |
79 (77–81) |
77 (75–79) |
77 (75–78) |
79 (77–80) |
|
76 (74–77) |
83 (80–85) |
75 (73–76) |
88 (86–90) |
73 (71–76) |
81 (79–83) |
76 (74–79) |
|
|
|
P-value
|
<0.001 |
0.125 |
|
<0.001 |
<0.001 |
0.915 |
||||||||
| Fidelity (completeness) – outcome of treatment |
% (95% CI)
|
78 (75–80) |
82 (84–86) |
83 (81–84) |
81 (80–83) |
83 (81–85) |
|
79 (78–81) |
93 (91–94) |
81 (79–82) |
87 (85–90) |
80 (78–82) |
83 (81–85) |
82 (80–84) |
|
|
|
P-value
|
<0.001 |
0.189 |
|
<0.001 |
<0.001 |
0.117 |
||||||||
| Fidelity (accuracy) – treatment oxygen given |
% (95% CI)
|
72 (69–74) |
69 (67–71) |
79 (77–81) |
72 (70–74) |
73 (71–76) |
|
68 (66–70) |
92 (90–94) |
73 (72–75) |
72 (69–75) |
68 (66–71) |
77 (75–79) |
73 (71–76) |
|
|
|
P-value
|
<0.001 |
0.405 |
|
<0.001 |
0.618 |
<0.001 |
||||||||
| Fidelity (accuracy) – treatment injectable gentamicin |
% (95% CI)
|
89 (87–91) |
90 (89–92) |
96 (94–97) |
92 (90–93) |
92 (90–93) |
|
91 (90–92) |
92 (90–94) |
90 (89–91) |
99 (99–100) |
90 (88–92) |
92 (90–93) |
93 (92–94) |
|
|
|
P-value
|
<0.001 |
0.869 |
|
0.638 |
<0.001 |
0.019 |
||||||||
| Fidelity (accuracy) – diagnosis – severe pneumonia |
% (95% CI)
|
97 (95–98) |
84 (82–85) |
91 (89–93) |
89 (88–90) |
90 (88–91) |
|
88 (87–89) |
93 (91–95) |
88 (89–90) |
90 (87–92) |
88 (87–90) |
89 (87–91) |
90 (88–92) |
|
|
|
P-value
|
<0.001 |
0.631 |
|
<0.001 |
0.549 |
0.436 |
||||||||
| Fidelity (accuracy) – refer |
% (95% CI)
|
90 (88–92) |
98 (97–98) |
98 (97–99) |
89 (88–90) |
90 (88–91) |
|
95 (94–96) |
99 (98–100) |
95 (94–96) |
99 (98–99) |
96 (95–97) |
95 (94–96) |
96 (95–97) |
|
|
|
P-value
|
<0.001 |
0.096 |
|
<0.001 |
<0.001 |
.0.552 |
||||||||
| Utility (quality of care) – injectable antibiotic for newborn sepsis |
% (95% CI)
|
50 (34–66) |
29 (4–71) |
33 (1–91) |
44 (24–65) |
43 (23–66) |
|
57 (39–73) |
15 (2–45) |
59 (41–75) |
19 (4–46) |
38 (20–59) |
50 (26–74) |
67 (22–96) |
|
| P-value | 0.537 |
0.972 |
|
0.009 |
0.007 |
0.435 |
|||||||||
CI – confidence interval, SD – standard deviation, x̄ – mean
*Utility and acceptability: n = 176; adoption: n = 5052; fidelity-completeness and utility-outcome: n = 4849; fidelity-accuracy: n = 4180; and utility-quality of care: n = 50. P-values <0.05 indicate a statistically significant difference in mean or proportion.
DISCUSSION
The introduction of standardised clinical registers has the potential to improve the quality of care and outcomes of patients admitted in the hospitals [22-24]. Here we reported on our assessment of WHO implementation outcome variables (usability, acceptability, adoption, fidelity, and utility [16]) for the implementation of a standardised register for the inpatient management of newborns and sick children. We found that health care providers accepted the inpatient register and found it usable; that it universally captured all eligible newborns and sick children admitted to the inpatient department; and that its data elements were mostly completed. The recording of variables in the inpatient register by health care providers showed moderate to substantial agreement with the recording done by study-recruited nurses. We set a benchmark prior to conducting this implementation research, this might have created a sense of accountability among the health care providers, which may have contributed to our positive findings.
The WHO has recommended the judicious and timely administration of injectable antibiotics for newborns admitted to the hospital with serious infections [25], as well as the administration of the first dose of injectable antibiotics and urgent referral for newborns visiting outpatient departments with suspected severe bacterial infections [26]. The Every Newborn Action Plan identified comprehensive health indicators used for monitoring newborn health progress effectively [27]. Population-based surveys such as Demographic and Health Surveys and Multiple Indicator Cluster Surveys are not suitable for collecting information on the coverage of treatment of newborn infections with injectable antibiotics, mostly due to issues with maternal recall error and bias [28–30]. This makes routine health information systems the most appropriate way to measure coverage of treatment of newborn infections with injectable antibiotics. In Bangladesh, IMCI was introduced nationally by the government in the early 1990s for the outdoor management of childhood illnesses, as recommended by the WHO for low- and middle-income countries [31]. Data on the management of newborn infections with injectable antibiotics has also been available in routine health information systems for outdoor management in Bangladesh. However, it is not possible to track this coverage indicator for the inpatient department, except for those newborns who are admitted to SCANUs, where structured registers and monthly reporting to routine health information systems are available. The inpatient register for newborns and sick children presents a unique opportunity to track the treatment of newborn infections with injectable antibiotics. However, we observed that only 62% of newborns admitted to the inpatient department with sepsis received injectable antibiotics. Three possible explanations exist for this low coverage: either the intervention was not performed; or it was performed, but not documented in the case record forms; or it was performed and documented in the case record forms, but not in the inpatient register. Exploring whether injectable antibiotics were administered or not was beyond the scope of this study. However, we identified issues with the design of the case record forms used to fill up the inpatient register, which might have contributed to the low coverage of injectable antibiotic treatment for newborn infections. In Bangladesh, unstructured case record forms are used in public hospital inpatient departments, resulting in inconsistency in documentation practices [30,32]. This also likely contributed to the low proportion of completion for the ‘investigations done’ data element in the inpatient register in our study. This is a consequence of the prevalent practice in Bangladesh, where laboratory investigation reports are typically handed over to patients at the time of discharge, along with the discharge certificate, leading to the documentation of investigation data in case record forms being inadequately maintained. Therefore, we recommend introducing structured case record forms in hospitals to address this issue. Regular monitoring and supervision from central and district-level health managers are essential in identifying reasons for the low coverage of injectable antibiotic treatment for newborn infections and the low proportion of completion of the ‘investigations done’ data element in the inpatient register.
We also assessed the completeness and accuracy of the inpatient register to determine the extent to which the intervention was executed in accordance with its intended design and objectives [16] and found some variability in both variables. This is consistent with several studies that reported similar issues with health registers [33–40]. Moreover, sub-district hospitals in our study had higher rates of completeness and accuracy than district hospitals, which may be due to differences in caseload between these two types of health facilities. Specifically, district hospitals serve as referral facilities for a district and provide services to patients from all over the district, while sub-district hospitals serve as primary-level health facilities with lower caseloads than district hospitals. These differences can also explain the higher rate of completeness and accuracy in the Dinajpur district than the Kushtia district. Notably, the Dinajpur district has a separate medical college hospital that serves as the highest-level referral facility, resulting in comparatively lower caseloads in other public health facilities in the Dinajpur district. Conversely, caseloads are higher in the district hospital of the Kushtia district as there was no functioning medical college hospital during our study period. Several studies have reported that poor documentation practice by health care providers is associated with higher caseloads [41–44]. To improve the completeness and accuracy of the register, dedicated staff can be appointed in facilities with higher caseloads for documentation in the inpatient register. Different data quality improvement initiatives (including routine audits by facility-, district-, and national-level health managers) and on-site mentoring have also proved effective [45,46]. Shifting towards e-registers can be a viable solution, although managing them alongside large databases can be challenging in low-resource settings. This challenge must be taken into account when considering the feasibility of implementing e-registers in such settings [47,48].
Strengths and limitations
Our study had several strengths. First, we designed this study based on the framework for implementation research developed by the WHO and reported most of the implementation outcomes. Second, we selected two high-burden facilities (the Kushtia district hospital and the Kumarkhali sub-district hospital) and two low-burden facilities (the Dinajpur district hospital and the Hakimpur sub-district hospital) to assess the effect of caseload on outcome variables. Lastly, we collected data from trained nurses using a custom-designed Android application to optimise data management.
However, our study also has some limitations. First, we did not choose the implementation districts and health facilities randomly, meaning our results cannot be generalised to the whole country due to selection bias. The Kushtia and Dinajpur districts predominantly have plain land, while other parts of the country with exposed coastal areas, hills, floods, and extreme drought zones might face different challenges in providing health care in the facilities, which we could not explore in this study. We also acknowledge that the implementing facilities are relatively better functioning because of other health initiatives taken in the Kushtia and Dinajpur districts over the past decade. Despite accounting for case mix and hospital differences, we observed limited variability in the background and clinical profiles of patients admitted to both district and sub-district hospitals in Bangladesh, making our study findings applicable to most parts of Bangladesh. Second, the presence of study staff in the inpatient department may have introduced the Hawthorne effect [49] in the performance of the health care providers in filling up the inpatient register. To mitigate this, we established rapport between health care providers and the data collection team through sustained interactions by conducting preparatory workshops at each facility. Third, we only extracted data from the inpatient register and case record forms, but did not explore whether care was actually provided to the patients by the health care providers. This prevented us from investigating whether an intervention was not given or not documented in the case record forms by looking at the data. Fourth, we did not explore other WHO implementation outcome variables such as implementation cost and sustainability due to time and resource limitations. We recommend conducting future studies to explore the cost-effectiveness of introducing an inpatient register in health facilities and post-implementation follow-ups to determine the sustainability of such interventions Lastly, we reported the accuracy of health care providers in filling up the inpatient register by comparing it with the performance of study-recruited nurses. We acknowledge that study-recruited nurses may not be considered the true gold standard for measuring the validity and accuracy of the inpatient register filled up by health care providers. We therefore reported percentage agreement and κ coefficient instead of diagnostic accuracy tests such as sensitivity, specificity, positive and negative predictive values, and area under the curve, which require such gold standards.
CONCLUSIONS
Healthcare providers have shown a positive attitude towards the register, while our assessment of implementation outcomes has shown promising results. This may be attributed to the extensive stakeholder engagement process led by the Government of Bangladesh from the development of the inpatient register to its implementation and assessments. However, we also found low proportions in some of the indicators and variability in the completeness and accuracy of the inpatient register by district and type of facility, mostly due to the differences in caseload. The experience acquired from this demonstration, coupled with the insights derived from this implementation research, can contribute to the formulation of data-driven decisions relating to the implementation and scale-up of a standardised register for inpatient management of newborns and sick children in Bangladesh and other low- and middle-income countries.
Additional material
Acknowledgements
We would like to thank all the technical committee members who participated in the development of the register. We would also like to thank district and facility health managers of Kushtia and Dinajpur districts for their kind support in implementation of the study.
Footnotes
Funding: This research was funded by the United States Agency for International Development (USAID) under the terms of USAID’s Research for Decision Makers (RDM) Activity cooperative agreement No. AID-388-A-17-00006. Views expressed herein do not necessarily reflect the views of the US Government or USAID. icddr,b is also grateful to the Governments of Bangladesh and Canada for providing long-term financial support.
Authorship contributions: AER was responsible for the conception and design of this study and secured funding with support from SA. MJI led the register development and implementation of the register with assistance from SA, AS, PKS, HMSA, SN, SS, and SDA from DGHS and GB, AEK, BA, SA, and MNK from development partners. SA developed the original draft of the manuscript as first author with support from SSS. AER provided guidance and feedback to SA at every stage as senior author. SA, KMT and SSS conducted the data analysis and data visualisation. All authors reviewed the first draft, provided inputs and approved the final manuscript.
Disclosure of interest: The authors completed the ICMJE Disclosure of Interest Form (available upon request from the corresponding author) and disclose no relevant interests.
REFERENCES
- 1.National Institute of Population Research and Training (NIPORT), Mitra and Associates, ICF International. Bangladesh Demographic and Health Survey 2011. Dhaka, Bangladesh and Calverton, Maryland, USA: NIPORT, Mitra and Associates, and ICF International; 2013. Available: https://dhsprogram.com/pubs/pdf/FR265/FR265.pdf. Accessed: 8 May 2024. [Google Scholar]
- 2.National Institute of Population Research and Training (NIPORT) , Mitra and Associates, ICF International. Bangladesh Demographic and Health Survey 2014. Dhaka, Bangladesh, and Rockville, Maryland, USA: NIPORT, Mitra and Associates, and ICF International; 2016. Available: https://dhsprogram.com/pubs/pdf/FR311/FR311.pdf. Accessed: 8 May 2024. [Google Scholar]
- 3.National Institute of Population Research and Training (NIPORT), ICF. Bangladesh Demographic and Health Survey 2017-18. Dhaka, Bangladesh, and Rockville, Maryland, USA: NIPORT and ICF; 2020. Available: https://dhsprogram.com/pubs/pdf/FR344/FR344.pdf. Accessed: 8 May 2024. [Google Scholar]
- 4.National Institute of Population Research and Training (NIPORT), ICF. Bangladesh Demographic and Health Survey 2022: Key Indicators Report. Dhaka, Bangladesh, and Rockville, Maryland, USA: NIPORT and ICF; 2023. Available: https://dhsprogram.com/pubs/pdf/PR148/PR148.pdf. Accessed: 8 May 2024. [Google Scholar]
- 5.United Nations Children’s Fund. Bangladesh. 2021. Available: https://data.unicef.org/country/bgd/. Accessed: 8 May 2024.
- 6.Gliklich RE, Dreyer NA, Leavy MB, editors. Registries for Evaluating Patient Outcomes: A User's Guide. 3rd ed. Rockville, Maryland: Agency for Healthcare Research and Quality; 2014. Available: https://www.ncbi.nlm.nih.gov/books/NBK208616/. Accessed: 24 April 2024. [PubMed] [Google Scholar]
- 7.Dokholyan RS, Muhlbaier LH, Falletta JM, Jacobs JP, Shahian D, Haan CK, et al. Regulatory and ethical considerations for linking clinical and administrative databases. Am Heart J. 2009;157:971–82. 10.1016/j.ahj.2009.03.023 [DOI] [PubMed] [Google Scholar]
- 8.Madhok R.Crossing the quality chasm: lessons from health care quality improvement efforts in England. Proc Bayl Univ Med Cent. 2002;15:77–83. 10.1080/08998280.2002.11927816 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.World Health Organization, United Nations Children’s Fund. Every newborn: an action plan to end preventable deaths. Geneva, Switzerland: World Health Organization; 2014. Available: https://www.who.int/publications-detail-redirect/9789241507448. Accessed: 24 April 2024.
- 10.Duggan AK, Starfield B, DeAngelis C.Structured encounter form: the impact on provider performance and recording of well-child care. Pediatrics. 1990;85:104–13. 10.1542/peds.85.1.104 [DOI] [PubMed] [Google Scholar]
- 11.Mwakyusa S, Wamae A, Wasunna A, Were F, Esamai F, Ogutu B, et al. Implementation of a structured paediatric admission record for district hospitals in Kenya–results of a pilot study. BMC Int Health Hum Rights. 2006;6:9. 10.1186/1472-698X-6-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Nahar T, Azad K, Aumon BH, Younes L, Shaha S, Kuddus A, et al. Scaling up community mobilisation through women’s groups for maternal and neonatal health: experiences from rural Bangladesh. BMC Pregnancy Childbirth. 2012;12:5. 10.1186/1471-2393-12-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Rahman AE, Hossain AT, Siddique AB, Jabeen S, Chisti MJ, Dockrell DH, et al. Child mortality in Bangladesh – why, when, where and how? A national survey-based analysis. J Glob Health. 2021;11:04052. 10.7189/jogh.11.04052 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Rahman AE, Herrera S, Rubayet S, Banik G, Hasan R, Ahsan Z, et al. Managing possible serious bacterial infection of young infants where referral is not possible: Lessons from the early implementation experience in Kushtia District learning laboratory, Bangladesh. PLoS One. 2020;15:e0232675. 10.1371/journal.pone.0232675 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Rubayet S, Shahidullah M, Hossain A, Corbett E, Moran AC, Mannan I, et al. Newborn survival in Bangladesh: a decade of change and future implications. Health Policy Plan. 2012;27:iii40–56. 10.1093/heapol/czs044 [DOI] [PubMed] [Google Scholar]
- 16.Peters DH, Tran NT, Adam T. Implementation research in health: a practical guide. Geneva: World Health Organization; 2013. Available: https://ahpsr.who.int/publications/i/item/9789241506212. Accessed: 8 May 2024. [Google Scholar]
- 17.Bangladesh Bureau of Statistics. Population and Housing Census 2022. Dhaka: Ministry of Planning, Government of the People’s Republic of Bangladesh; 2023. Available: https://sid.portal.gov.bd/sites/default/files/files/sid.portal.gov.bd/publications/01ad1ffe_cfef_4811_af97_594b6c64d7c3/PHC_Preliminary_Report_(English)_August_2022.pdf. Accessed: 8 May 2024. [Google Scholar]
- 18.Ng AW, Lo HW, Chan AH. Measuring the Usability of Safety Signs: A use of system usability scale (SUS). In: Proceedings of the International MultiConference of Engineers and Computer Scientists; 2011 March 16-18; Hong Kong, China. Hong Kong: International Association of Engineers; 2011. [Google Scholar]
- 19.Holden RJ, Karsh BT.The technology acceptance model: its past and its future in health care. J Biomed Inform. 2010;43:159-72. 10.1016/j.jbi.2009.07.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Sim J, Wright CC.The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85:257–68. 10.1093/ptj/85.3.257 [DOI] [PubMed] [Google Scholar]
- 21.Stata. Main page. 2021. Available: https://www.stata.com/. Accessed.
- 22.Hoque DME, Kumari V, Hoque M, Ruseckaite R, Romero L, Evans SM.Impact of clinical registries on quality of patient care and clinical outcomes: a systematic review. PLoS One. 2017;12:e0183667. 10.1371/journal.pone.0183667 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Pirkle CM, Dumont A, Zunzunegui MV.Medical recordkeeping, essential but overlooked aspect of quality of care in resource-limited settings. Int J Qual Health Care. 2012;24:564–7. 10.1093/intqhc/mzs034 [DOI] [PubMed] [Google Scholar]
- 24.Zegers M, de Bruijne MC, Spreeuwenberg P, Wagner C, Groenewegen PP, van der Wal G.Quality of patient record keeping: an indicator of the quality of care? BMJ Qual Saf. 2011;20:314–8. 10.1136/bmjqs.2009.038976 [DOI] [PubMed] [Google Scholar]
- 25.World Health Organization. Pocket book of hospital care for children: guidelines for the management of common childhood illnesses. Geneva, Switzerland: World Health Organization; 2013. Available: https://www.ncbi.nlm.nih.gov/books/NBK154447/. Accessed: 8 May 2024. [PubMed] [Google Scholar]
- 26.World Health Organization, United Nations Children’s Fund. Integrated Management of Childhood Illness: management of the sick young infant aged up to 2 months: Geneva: World Health Organization; 2019. Available: https://www.who.int/publications/i/item/9789241516365. Accessed: 8 May 2024. [Google Scholar]
- 27.World Health Organization. The WHO technical consultation on newborn health indicators. Geneva: World Health Organization; 2014. Available: http://apps.who.int/iris/bitstream/10665/184225/1/9789241509381_eng.pdf?ua=1. Accessed: 8 May 2024. [Google Scholar]
- 28.Day LT, Rahman QS, Rahman AE, Salim N, Ashish K, Ruysen H, et al. Assessment of the validity of the measurement of newborn and maternal health-care coverage in hospitals (EN-BIRTH): an observational study. Lancet Glob Health. 2021;9:e267–79. 10.1016/S2214-109X(20)30504-0 [DOI] [PubMed] [Google Scholar]
- 29.Ameen S, Siddique AB, Peven K, Rahman QS, Day LT, Shabani J, et al. Survey of women’s report for 33 maternal and newborn indicators: EN-BIRTH multi-country validation study. BMC Pregnancy Childbirth. 2021;21:238. 10.1186/s12884-020-03425-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Rahman AE, Hossain AT, Zaman SB, Salim N, KC A, Day LT, et al. Antibiotic use for inpatient newborn care with suspected infection: EN-BIRTH multi-country validation study. BMC Pregnancy Childbirth. 2021;21:229. 10.1186/s12884-020-03424-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.El Arifeen S, Blum LS, Hoque DE, Chowdhury EK, Khan R, Black RE, et al. Integrated Management of Childhood Illness (IMCI) in Bangladesh: early findings from a cluster-randomised study. Lancet. 2004;364:1595–602. 10.1016/S0140-6736(04)17312-1 [DOI] [PubMed] [Google Scholar]
- 32.Hossain AT, Ameen S, Salim N, Ashish K, Ruysen H, Tahsina T, et al. Measuring coverage and quality of supportive care for inpatient neonatal infections: EN-BIRTH multi-country validation study. J Glob Health. 2022;12:04029. 10.7189/jogh.12.04029 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Day LT, Gore-Langton GR, Rahman AE, Basnet O, Shabani J, Tahsina T, et al. Labour and delivery ward register data availability, quality, and utility-Every Newborn-birth indicators research tracking in hospitals (EN-BIRTH) study baseline analysis in three countries. BMC Health Serv Res. 2020;20:737. 10.1186/s12913-020-5028-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Hollung SJ, Vik T, Wiik R, Bakken IJ, Andersen GL.Completeness and correctness of cerebral palsy diagnoses in two health registers: implications for estimating prevalence. Dev Med Child Neurol. 2017;59:402–6. 10.1111/dmcn.13341 [DOI] [PubMed] [Google Scholar]
- 35.Alhaug OK, Kaur S, Dolatowski F, Småstuen MC, Solberg TK, Lønne G.Accuracy and agreement of national spine register data for 474 patients compared to corresponding electronic patient records. Eur Spine J. 2022;31:801–11. 10.1007/s00586-021-07093-8 [DOI] [PubMed] [Google Scholar]
- 36.Taylor L, Casson D, Platt M.Issues and experience around the paediatric register of inflammatory bowel disease. Arch Dis Child. 2003;88:891–3. 10.1136/adc.88.10.891 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.McCarthy H, Dixon M, Crabtree I, Eaton-Evans M, McNulty H.The development and evaluation of the Screening Tool for the Assessment of Malnutrition in Paediatrics (STAMP©) for use by healthcare staff. J Hum Nutr Diet. 2012;25:311–8. 10.1111/j.1365-277X.2012.01234.x [DOI] [PubMed] [Google Scholar]
- 38.Chiba Y, Oguttu MA, Nakayama T.Quantitative and qualitative verification of data quality in the childbirth registers of two rural district hospitals in Western Kenya. Midwifery. 2012;28:329–39. 10.1016/j.midw.2011.05.005 [DOI] [PubMed] [Google Scholar]
- 39.Lodge W, Menon G, Kuchukhidze S, Jumbam DT, Maongezi S, Alidina S, et al. Assessing completeness of patient medical records of surgical and obstetric patients in Northern Tanzania. Glob Health Action. 2020;13:1765526. 10.1080/16549716.2020.1765526 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.O’Hagan R, Marx MA, Finnegan KE, Naphini P, Ng’ambi K, Laija K, et al. National assessment of data quality and associated systems-level factors in Malawi. Glob Health Sci Pract. 2017;5:367–81. 10.9745/GHSP-D-17-00177 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Kebede M, Endris Y, Zegeye DT.Nursing care documentation practice: The unfinished task of nursing care in the University of Gondar Hospital. Inform Health Soc Care. 2017;42:290–302. 10.1080/17538157.2016.1252766 [DOI] [PubMed] [Google Scholar]
- 42.Avoka Asmani J, Delasi Amenorpe F, Babanawo F, Maria Ansah Ofei A.Nursing documentation of inpatient care in eastern Ghana. Br J Nurs. 2014;23:48–54. 10.12968/bjon.2014.23.1.48 [DOI] [PubMed] [Google Scholar]
- 43.Blake-Mowatt C, Lindo J, Bennett J.Evaluation of registered nurses’ knowledge and practice of documentation at a J amaican hospital. Int Nurs Rev. 2013;60:328–34. 10.1111/inr.12040 [DOI] [PubMed] [Google Scholar]
- 44.Tamir T, Geda B, Mengistie B.Documentation practice and associated factors among nurses in Harari regional state and Dire Dawa administration governmental hospitals, eastern Ethiopia. Adv Med Educ Pract. 2021. 12453–62. 10.2147/AMEP.S298675 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Gimbel S, Mwanza M, Nisingizwe MP, Michel C, Hirschhorn L.Improving data quality across 3 sub-Saharan African countries using the consolidated framework for implementation research (CFIR): results from the African health initiative. BMC Health Serv Res. 2017;17:828. 10.1186/s12913-017-2660-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Merai R, Keating R, Mubiri P, Kajjo D, Otare C, Mugume D, et al. Assessing effects of a data quality strengthening campaign on completeness of key fields in facility-based maternity registers in Kenya and Uganda. East Afr J Appl Health Monitor Eval. 2019;3:1–7. 10.58498/eajahme.v1i1.20 [DOI] [Google Scholar]
- 47.Asogwa BE.The challenge of managing electronic records in developing countries: Implications for records managers in sub Saharan Africa. Rec Manage J. 2012;22:198–211. 10.1108/09565691211283156 [DOI] [Google Scholar]
- 48.Iwhiwhu BE. Electronic records management in Africa: problems and prospects. In: Adomi EE, editor. Handbook of research on information communication technology policy: trends, issues and advancements. Hershey, PA: IGI Global; 2011. p. 161-85. [Google Scholar]
- 49.McCambridge J, Witton J, Elbourne DR.Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. J Clin Epidemiol. 2014;67:267–77. 10.1016/j.jclinepi.2013.08.015 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.