Long-term quality of integrated community case management care for children in Bugoye Subcounty, Uganda: a retrospective observational study

Abstract

Objective

Integrated community case management (iCCM) of childhood illness in Uganda involves protocol-based care of malaria, pneumonia and diarrhoea for children under 5 years old. This study assessed volunteer village health workers’ (VHW) ability to provide correct iCCM care according to the national protocol and change in their performance over time since initial training.

Setting

VHWs affiliated with the Ugandan national programme provide community-based care in eight villages in Bugoye Subcounty, a rural area in Kasese District. The first cohort of VHWs began providing iCCM care in March 2013, the second cohort in July 2016.

Participants

All children receiving iCCM care in 18 430 clinical encounters occurring between April 2014 and December 2018.

Primary and secondary outcome measures

The descriptive primary outcome measure was the proportion of patients receiving overall correct care, defined as adherence to the iCCM protocol for the presenting condition (hereafter quality of care). The analytic primary outcome was change in the odds of receiving correct care over time, assessed using logistic regression models with generalised estimating equations. Secondary outcome measures included a set of binary measures of adherence to specific elements of the iCCM protocol. Preplanned and final measures were the same.

Results

Overall, VHWs provided correct care in 74% of clinical encounters. For the first cohort of VHWs, regression modelling demonstrated a modest increase in quality of care until approximately 3 years after their initial iCCM training (OR 1.022 per month elapsed, 95% CI 1.005 to 1.038), followed by a modest decrease thereafter (OR 0.978 per month, 95% CI 0.970 to 0.986). For the second cohort, quality of care was essentially constant over time (OR 1.007 per month, 95% CI 0.989 to 1.025).

Conclusion

Quality of care was relatively constant over time, though the trend towards decreasing quality of care after 3 years of providing iCCM care requires further monitoring.

Strengths and limitations of this study.

• We used a record review approach to examine quality of care for all integrated community case management clinical encounters over a nearly 5-year period.

• A retrospective record review approach cannot assess all aspects of appropriate care and may overestimate quality of care.

• While this study assessed a large number of patient encounters, it included a fairly small number of village health workers in a single geographical area, limiting its generalisability.

Introduction

In integrated community case management (iCCM) of childhood illness, village health workers (VHWs) or other lay workers provide care for paediatric malaria, pneumonia and (in some settings) diarrhoea, usually in a community setting rather than at a health facility.¹ VHWs follow a defined protocol that directs specific diagnostic and therapeutic steps based on the patient’s presenting complaint and clinical exam findings. In some parts of Uganda, volunteer VHWs provide iCCM care, either in the patient’s home or at the VHW’s home, as part of the national Village Health Teams system.^2–4 These VHWs also retain their existing roles in health promotion efforts. In Bugoye Subcounty, Uganda, a rural, mountainous area in Kasese District (on the western border of Uganda), VHWs from the national programme have provided iCCM care since 2013, with financial and operational support from a long-standing collaboration with Mbarara University of Science and Technology (Mbarara, Uganda) and the Massachusetts General Hospital (Boston, Massachusetts, USA).

Prior studies have employed several different approaches to assess iCCM quality of care, including direct observation of VHWs (with or without re-examination of the patient by a trained clinician), review of VHWs’ clinical records, case scenarios and surveys of patients’ caregivers. Each approach comes with certain risks of bias and with different abilities to measure the varied aspects of quality of care. Direct observation of VHWs with re-examination of patients allows for the most comprehensive assessment of quality of care, though the presence of an observer may alter VHWs’ clinical practice (Hawthorne effect). A record review approach cannot assess all elements of quality of care and measures quality of recorded care rather than quality of actual care. However, record review has the benefit of easier repeated or widespread implementation. Prior studies comparing different methods of assessing quality of iCCM care, with direct observation of VHWs and re-examination of patients as the gold standard, have found that record review, case scenarios and direct observation alone all tend to overestimate quality of care somewhat.^{5 6} While acknowledging that limitation, in this study we employ a record review approach because it feasibly allows for continuous assessment of quality of care over multiple years.

In Uganda, prior studies using direct observation,⁷ record review,⁸ surveys of caregivers⁹ and a combination of approaches¹⁰ have generally demonstrated high quality of iCCM care, within the limits of each approach. In other countries in sub-Saharan Africa, some studies have found similar quality of care,^{11 12} while others documented lower quality of care overall¹³ or for specific areas such as severe illness¹⁴ or antibiotic overuse.¹⁵ However, there are few evaluations examining changes in iCCM care quality over time; most studies have assessed quality of care at a point in time or cumulatively for a single time period, or examined the impact of a particular intervention.^7–15

One prior study in Kenya examined quality of care at multiple time points using direct observation of VHWs and re-examination of patients, with three evaluations conducted over the 4-year period after VHWs’ initial iCCM training. This study demonstrated improvement in some quality measures and worsening of other measures over time but did not assess for an overall trend in quality of care over time.¹⁶ We previously examined trends in quality of care over the first 2 years of the iCCM programme in Bugoye using a record review approach, showing improvement in the proportions of patients receiving correct care (quality of care) over the initial 6 months after iCCM care initiation and stable quality of care for the remaining 18 months.¹⁷ Several other studies have examined quality of care provided by VHWs with differing levels of experience but did not examine the relationship between experience level and quality of care provided.^{18 19} Prior studies in Bugoye and other settings have also examined the specific iCCM skills of performing and interpreting rapid diagnostic tests (RDTs) for malaria, finding that VHWs are able to maintain these skills over a period of 1–4 years.^20–22

Since iCCM programmes provide front-line care for common but potentially fatal childhood illnesses, programmes must be able to monitor and ensure quality of care over time. Here we describe quality of iCCM care over a nearly 5-year period, as VHWs gain clinical experience but are also further from their initial iCCM training.

Methods

Study design

In this retrospective, observational study, we examined overall iCCM quality of care as well as specific components of iCCM care for two cohorts of VHWs using a record review approach for all clinical encounters occurring between April 2014 and December 2018. For each cohort, we also examined trends in quality of care over time. For the first cohort of VHWs, we sought to assess long-term trends in quality of care, building on the prior evaluation of the first 2 years of iCCM care provided by this cohort.¹⁷ For the second cohort of VHWs, we sought to assess quality of care over the initial 2.5 years of iCCM care provided by this cohort using a larger dataset of all clinical encounters rather than sampled encounters to further evaluate our prior findings on quality of iCCM care after initial training.

The descriptive primary outcome measure was the proportion of patients receiving overall correct care, defined as adherence to all elements of the iCCM protocol for the presenting condition (further description of the iCCM protocol is provided below). We have termed this outcome measure quality of care, acknowledging that a retrospective record review approach cannot assess all domains of quality of care. The analytical primary outcome measure examined trends over time in quality of care, again using a binary classification of overall correct or incorrect care. Secondary outcome measures also used a binary correct or incorrect classification, and included VHWs’ adherence to correct diagnostic protocol, correct prescribing practices and correct referral of patients to a health facility. Preplanned and final measures were the same.

Study setting

VHWs in Bugoye are selected by community members in their villages as part of the national Village Health Teams programme and serve their communities as part-time volunteers. In general, VHWs are working-age adults and have a paid job or practise subsistence farming, sometimes with an additional cash crop such as coffee, in addition to their work as VHWs. All VHWs are required to have achieved basic literacy, and most have completed primary school. In addition to iCCM care, VHWs also provide health education and collect demographic and health data. All VHWs received 3 days of initial general training, 5 days of initial iCCM training and half-day refresher trainings on a quarterly basis throughout the time period examined here. Clinical staff members at the government-funded health centre in Bugoye are trained as iCCM instructors (as part of a Ministry of Health-supported training of trainers approach) and lead the quarterly training sessions; they also provide field-based individual supervision. VHWs provide iCCM care for malaria, pneumonia and diarrhoea in children between 2 months and 5 years old. In keeping with WHO/UNICEF standards, VHWs perform RDTs for malaria and provide artemisinin combination therapy (ACT) if indicated, measure respiratory rate and treat presumed pneumonia with amoxicillin, and evaluate children with diarrhoea and treat with oral rehydration solution (ORS) and zinc, as well as assess children for signs of severe illness (danger signs) and provide health education.¹ VHWs refer children with other conditions or with danger signs to a health facility. The first cohort (24 VHWs) began providing iCCM care in March 2013 in five villages; one VHW was added in 2016 to replace a VHW who had moved away. The second cohort (14 VHWs) began providing iCCM care in July 2016 in three additional villages. Both cohorts had the same initial training, refresher trainings and supervision. In this programme, Mbarara University of Science and Technology and Massachusetts General Hospital funded programme administration and the purchase medications and malaria RDTs; these institutions also provided staff support to enhance data collection and quality improvement efforts. During the time period examined here, iCCM care had not yet been implemented throughout the national Village Health Teams programme.

Data collection

During the time period assessed in this study, VHWs in Bugoye used a paper form to record information from each clinical encounter. The form used was based on a Ministry of Health template and was modified to address local context and translated into the local language of Lukonjo. At the end of each month, programme staff collected the paper records and entered information from each clinical encounter into a customised Epidata form²³ (implemented separately from national reporting systems); data were then stored in a Research Electronic Data Capture database.²⁴ The paper record forms contain basic information about the patient (age, sex and presenting complaints), clinical assessment data (presence of danger signs, respiratory rate and malaria RDT result) and actions taken (medications administered and/or referral to the health facility).

Deidentified data for all encounters from April 2014 to December 2018 (reflecting all available months in the electronic clinical database) were extracted to create the research dataset used for this study. Data were then cleaned, with comparison to paper records in case of a mismatch between similar variables (eg, report month and visit date) or other suspected inaccuracies. Infants under 2 months of age were excluded from analysis as they were likely seen for newborn wellness checks separate from iCCM care. Stata V.15 was used to check for presumed duplicate entries in the database. Decision rules in Stata were used to determine whether the care each patient received (based on the paper record form) matched with correct care according to the iCCM protocol. For patients presenting with fever, elements of correct care include performing an RDT for malaria and recording the result, treatment with ACT if the test is positive and referral to a health facility if the test is negative. For patients presenting with cough/fast breathing, elements of correct care include measuring and recording the respiratory rate and treatment with oral amoxicillin if the respiratory rate is above age-based cut-offs (based on patients’ recorded age and respiratory rate we could assess whether the VHW made this determination appropriately). For patients presenting with diarrhoea, correct care constituted treatment with ORS and zinc. For patients recorded as having danger signs, we assessed whether the patient was referred to a health facility and received appropriate prereferral treatment (if indicated). Certain elements of correct care (eg, medication dosage and specific elements of health education provided) could not be assessed because they are not included on the paper record form.

Data analysis

The analytical methods for this study are quite similar to our prior study of iCCM care quality over the initial 2 years of this programme.¹⁷ In addition to the measures above, we also calculated the proportion of VHWs providing high-quality care for each of the main iCCM conditions, using thresholds of 70% and 84% of encounters with correct care. These thresholds were chosen to accord with our prior findings, which employed a lot quality assurance sampling approach.¹⁷ Since failure to refer a patient with subjective fever and a negative RDT was a common and likely low-consequence error,¹¹ we also calculated the overall proportion of patients receiving correct care if this error is excluded.

We assessed population average trends in quality of care over time using logistic regression models with generalised estimating equations (GEEs) with an exchangeable correlation structure and with robust SEs to account for clustering by VHW, but otherwise without adjustment for demographic variables. For this analysis, we used a binary outcome variable representing correct or incorrect care according to the iCCM protocol, with time in months since the VHW began providing iCCM care as the exposure variable, to examine changes in the odds of receiving correct care over time. For each cohort, we initially fit a base model without a spline. Based on graphical depiction of the data as well as our prior findings suggesting that quality of care improved over the initial 6 months of providing iCCM care,¹⁷ we then fit models that included a spline, as the relationship between experience (time elapsed since initial iCCM training) and provision of correct care might not be constant (eg, provision of correct care might initially improve with experience but eventually may reach a point at which further experience does not correlate with further improvement). We used the quasi-likelihood under the independence criterion (QIC) to compare model fit between the models with and without splines. The VHW who joined the programme later in a village from the first cohort was excluded from this portion of analysis.

Patient and public involvement

While patients’ families were not directly involved in developing the research question or outcome measures for this study, community members play a key role in selecting the VHWs who provide care in their communities. In a separate study, we have also sought to understand families’ experience of and satisfaction with iCCM care to identify programme strengths and areas for improvement.²⁵ Evaluation of iCCM quality of care has also helped tailor the content of VHW refresher trainings and other quality improvement efforts.

Results

After exclusion of 48 encounters for children under 2 months old, 339 presumed duplicate entries and 76 encounters with no clinical information recorded, VHWs completed a total of 18 430 clinical encounters between April 2014 and December 2018. VHWs in the first cohort accounted for 74% of these encounters (table 1). Because the data are deidentified and do not contain a unique identifier variable like a medical record number, it was not possible to determine how many unique children received care from a VHW during this period. For brevity, hereafter we refer to ‘patients’ rather than clinical encounters. The proportion of female patients and the mean age of patients were similar between the first cohort and second cohort of VHWs. Subjective fever was the most common presenting complaint, followed by cough/fast breathing, and then diarrhoea (59%, 45% and 28%, respectively; percentages add to >100% because some patients presented with multiple complaints). In both cohorts, 1% of patients were noted to have danger signs.

Table 1.

Patient demographic information and summary of all iCCM clinical encounters (April 2014–December 2018 for first cohort VHWs, July 2016–December 2018 for second cohort VHWs)

Measure	n (%) or mean (range)
	First cohort VHWs	Second cohort VHWs	Overall
Total encounters*	13 650 (74)	4780 (26)	18 430
Female	6780 (50)	2325 (49)	9105 (49)
Age (months)†	28.5 (2–60)	27.0 (2–60)	28.1 (2–60)
Presenting complaints‡
Fever	8418 (62)	2394 (50)	10 812 (59)
Cough/fast breathing	5890 (43)	2449 (51	8339 (45)
Diarrhoea	3818 (28)	1352 (28)	5170 (28%)
Other/not recorded	600 (4)	149 (3)	749 (4)
Patients with danger signs	171 (1)	58 (1)	229 (1)
Actions/outcomes
Respiratory rate measured	6377 (47)	2472 (52)	8849 (48)
Respiratory rate elevated	5756 (90)	2293 (93)	8049 (91)
RDT performed	9316 (68%)	2715 (57)	12 031 (65)
RDT positive	7060 (76)	1719 (63)	8779 (73)
Patients treated with ACT	7022 (51)	1706 (36)	8728 (47)
Patients treated with amoxicillin	5972 (44)	2457 (51)	8429 (46)
Patients treated with ORS and zinc	3841 (28)	1331 (28)	5172 (28)
Patients treated with rectal artesunate	95 (1)	38 (1)	133 (1)
Patients referred to health facility	783 (6)	303 (6)	1086 (6)
Medication reactions	2 (0.01)	0 (0)	2 (0.01)
Deaths	1 (0.01)	1 (0.02)	2 (0.01)

*339 presumed duplicate encounters as well as 76 encounters with no clinical information recorded are excluded from the analysis.

†48 infants under 2 months of age were excluded from the analysis as they were likely seen for newborn assessments rather than iCCM care.

‡Percentages add to >100%, as some patients presented with multiple complaints.

ACT, artemisinin combination therapy; iCCM, integrated community case management; ORS, oral rehydration solution; RDT, rapid diagnostic test; VHW, village health worker.

For patients with measured respiratory rate, 91% had a respiratory rate above age-based cut-offs. VHWs performed a total of 12 031 malaria RDTs, of which 8879 (73%) were positive. There were 8728 patients treated with ACT, 8429 patients treated with amoxicillin and 5172 patients treated with ORS and zinc. Higher proportions of patients treated by VHWs in the first cohort presented with fever (62% vs 50%), had an RDT performed (68% vs 57%) and had positive RDTs (76% vs 63%). Correspondingly, a higher proportion of patients treated by VHWs in the first cohort received ACT (51% vs 36%). There were 133 patients (1%) treated with rectal artesunate, and 1086 (6%) were referred to a health facility. There were two patients recorded as having adverse reactions to medications and two recorded deaths (table 1).

Regarding quality measures, 97% of patients presenting with subjective fever correctly had an RDT performed (98% of those treated by VHWs in the first cohort vs 94% of those treated by VHWs in the second cohort). Of patients diagnosed with malaria by RDT, 93% received correct management (94% vs 88%). However, of patients with a negative RDT, only 23% were appropriately referred to a health facility (21% vs 26%). Of patients presenting with cough or subjective fast breathing, 96% had their respiratory rate recorded (97% vs 94%). Of those with an elevated respiratory rate and thus presumed pneumonia based on the iCCM algorithm, 90% received correct treatment (92% vs 86%). Of patients presenting with diarrhoea, 92% received correct treatment (92% vs 90%). Of patients with danger signs, 77% were appropriately referred to a health facility (80% vs 69%), and 60% received appropriate prereferral treatment (56% vs 72%; see table 2).

Table 2.

Integrated community case management quality of care measures

Measure	First cohort VHWs n (%)	Second cohort VHWs n (%)	Overall n (%)
RDT performed for patient presenting with fever	8278 (98)	2258 (94)	10 536 (97)
Malaria patients receiving correct management	6864 (94)	1617 (88)	8841 (93)
Patients with negative RDT receiving correct management	323 (21)	172 (26)	495 (23)
Respiratory rate recorded for patient presenting with cough or subjective fast breathing	5693 (97)	2305 (94)	7998 (96)
Patients with elevated respiratory rate receiving correct treatment	5625 (92)	2200 (86)	7825 (90)
Patients with diarrhoea receiving ORS and zinc	3699 (92)	1281 (90)	4980 (92)
Patients inappropriately treated with ACT (out of total patients)	182 (1)	100 (2)	282 (2)
Inappropriate ACT prescriptions (out of total ACT prescriptions)	182 (3)	100 (6)	282 (3)
Patients inappropriately treated with amoxicillin (out of total patients)	343 (3)	256 (5)	599 (3)
Inappropriate amoxicillin prescriptions (out of total amoxicillin prescriptions)	343 (6)	256 (10)	599 (7)
Patients inappropriately treated with ORS, zinc or both (out of total patients)	188 (1)	73 (2)	261 (1)
Inappropriate ORS or zinc prescriptions (out of total ORS and zinc prescriptions)	188 (5)	73 (5)	261 (5)
Patients with danger signs appropriately referred to health facility	137 (80)	40 (69)	177 (77)
Patients with danger signs receiving appropriate prereferral treatment	96 (56)	42 (72)	138 (60)
Patients receiving overall correct management (all months)*	10 455 (77)	3244 (68)	13 699 (74)
Patients receiving overall correct management (July 2016–December 2018)	5560 (75)	3244 (68)	8804 (72)
Patients receiving overall correct management, excluding referral of patients with negative RDT (all months)*	11 486 (84)	3654 (76)	15 140 (82)

*April 2014–December 2018 for original VHWs, July 2016–December 2018 for expansion VHWs.

ACT, artemisinin combination therapy; ORS, oral rehydration solution; RDT, rapid diagnostic test; VHW, village health worker.

Overall, 74% of patients received correct management (77% vs. 68%). When comparing only the time period in which the second cohort of VHWs was providing iCCM care, 72% of patients received correct management (75% vs 68%). Excluding the common error of failing to refer patients with a negative malaria RDT, 82% of patients received correct management (84% vs 76%).

Inappropriate use of medications was fairly low, with 2% of patients receiving ACT inappropriately, 3% of patients receiving amoxicillin inappropriately and 1% of patients receiving ORS, zinc, or both inappropriately (table 2). This constituted 3%, 7% and 5% of prescriptions for those medications, respectively. These proportions were similar between the two cohorts of VHWs.

Regarding individual VHWs’ performance, of the 38 VHWs, all 38 VHWs provided correct care for at least 70% of patients with malaria, and 34 VHWs provided correct care for least 84% of patients with malaria, compared with 35 and 30 VHWs (respectively) for patients with presumed pneumonia, 36 and 29 VHWs (respectively) for patients with diarrhoea, and 24 and 7 VHWs (respectively) for all patients. If failure to refer patients with a negative RDT is excluded, then 35 VHWs met the 70% threshold for overall correct care, and 16 VHWs met the 84% threshold for overall correct care (table 3).

Table 3.

VHW-level quality of care measures

Measure	In >70% of encounters n (%)*	In >84% of encounters n (%)*
VHWs providing correct care for malaria	38 (100)	34 (89)
VHWs providing correct care for presumed pneumonia	35 (92)	30 (79)
VHWs providing correct care for diarrhoea	36 (95)	29 (76)
VHWs providing overall correct care	24 (63)	7 (18)
VHWs providing overall correct care, excluding referral of patients with negative RDT	35 (92)	16 (42)

*The 70% and 84% thresholds were chosen to accord with our prior lot quality assurance sampling approach, allowing for comparison between the two studies

RDT, rapid diagnostic test; VHW, village health worker.

For the first cohort of VHWs, modelling time as a continuous variable (months since the cohort began providing iCCM care) estimated a 0.6% decreased odds of correct care for each month elapsed (OR 0.994, 95% CI 0.988 to 0.999, p=0.032). Graphical depiction of overall correct care over time for the first cohort appeared to show a gradual increase until approximately March 2016 (3 years after iCCM care initiation), followed by gradual decrease (figure 1). Based on the graphical depiction, the addition of a spline knot at the 3-year mark demonstrated a slight trend towards increasing correctness of care up until the 3-year mark (OR 1.022, 95% CI 1.005 to 1.038, p=0.009), followed by decreasing correctness of care after that point (OR 0.978, 95% CI 0.970 to 0.986, p<0.001), with an improvement in model fit based on the QIC values (table 4).

Figure 1.

LOWESS smoothing plots for correct management over time. (A–C) Plots correspond to the first cohort of VHWs. (A) Overall correct care over time. (B) Correct referral of patients with a negative malaria RDT over time. (C) Overall correct care over time, excluding the error of failing to refer patients with a negative malaria RDT. (D–F) Second cohort of VHWs. (D) Overall correct care over time. (E) Correct referral of patients with a negative malaria RDT over time. (F) Overall correct care over time, excluding the error of failing to refer patients with a negative malaria RDT. iCCM, integrated community case management; LOWESS, locally weighted scatterplot smoothing; RDT, rapid diagnostic test; VHW, village health worker.

Table 4.

GEE logistic regression models for quality of care over time

Measure	OR	95% CI	P value	Model QIC*
First cohort VHWs, overall correct care†				14 587
Months since iCCM services initiation	0.994	0.988 to 0.999	0.032
First cohort VHWs, overall correct care, with spline knot at 3 years after iCCM care initiation†				14 524
Months since iCCM services initiation, months 14–36	1.022	1.005 to 1.038	0.009
Months since iCCM services initiation, months 36–70	0.978	0.970 to 0.986	<0.001
First cohort VHWs, overall correct care excluding referral of patients with negative RDT†				11 747
Months since iCCM services initiation	0.999	0.993 to 1.004	0.632
First cohort VHWs, overall correct care excluding referral of patients with negative RDT, with spline knot at 3 years after iCCM care initiation†				11 718
Months since iCCM services initiation - Months 14–36	1.023	1.005 to 1.043	0.015
Months since iCCM services initiation - Months 36–70	0.984	0.976 to 0.993	<0.001
Second cohort VHWs, overall correct care				6057
Months since iCCM services initiation	1.007	0.989 to 1.025	0.475
Second cohort VHWs, overall correct care, with spline knot at 6 months after iCCM care initiation				6061
Months since iCCM services initiation, months 1–6	1.068	0.971 to 1.175	0.175
Months since iCCM services initiation, months 7–30	0.998	0.973 to 1.022	0.845
Second cohort VHWs, overall correct care, with spline knot at 18 months after iCCM care initiation				6047
Months since iCCM services initiation, months 1–18	1.027	1.004 to 1.050	0.023
Months since iCCM services initiation, months 19–30	0.963	0.926 to 1.000	0.053
Second cohort VHWs, overall correct care excluding referral of patients with negative RDT				5272
Months since iCCM services initiation	1.026	1.007 to 1.045	0.006

*Quasi-likelihood under the independence model criterion. This is a modification of the Akaike information criterion so that it can be applied to GEE regression models to assess goodness of fit of different models. A lower QIC term reflects a better-fitting regression model.

†One VHW joined the first cohort of VHWs later and is thus excluded from this analysis.

GEE, generalised estimating equation; iCCM, integrated community case management; QIC, quasi-likelihood under the independence criterion; RDT, rapid diagnostic test; VHW, village health worker.

Failure to refer patients with a negative RDT to a health facility was a common error. We conducted a secondary analysis to assess whether the trends observed in the main models were driven solely by this error. Graphical depiction of correct referral of patients with a negative RDT for the first cohort of VHWs appeared to show worse performance over time; exclusion of this error appeared to moderate the trend towards decreasing quality of care in later months (figure 1). Repeating the model with the outcome variable as overall correct care excluding referral of patients with negative RDT estimated essentially constant correctness of care over time (OR 0.999, 95% CI 0.993 to 1.004, p=0.632). The addition of a spline knot at the 3-year mark resulted in similar findings to using the main outcome variable of overall correct care (slightly increasing correctness of care up until the 3-year mark, followed by slightly decreasing correctness of care; see table 4 and figure 2).

Figure 2.

Postestimation margins plots for generalised estimating equation logistic regression models with spline knots, which display the predicted probability of receiving correct care at specified time points (6-month intervals). (A, B) Plots depict the first cohort of VHWs. (A) Predicted probability of overall correct care, with a spline knot 3 years after the first cohort began providing iCCM care. (B) Predicted probability of overall correct care excluding the error of failing to refer patients with a negative malaria RDT, again with a spline knot at 3 years. (C, D) Second cohort of VHWs. (C) Predicted probability of overall correct care, with a spline knot 6 months after the second cohort began providing iCCM care. (D) The predicted probability of overall correct care is again shown, but with a spline knot at 18 months. iCCM, integrated community case management; RDT, rapid diagnostic test; VHW, village health worker.

For the second cohort of VHWs, modelling time as a continuous variable (months since the cohort began providing iCCM care) estimated essentially constant correctness of care over time (OR 1.007, 95% CI 0.989 to 1.025, p=0.475; see table 4). Graphical depiction of overall correct care over time for the second cohort appeared to show a slight increase in correct care over approximately the first 6 months after iCCM programme initiation, followed by a plateau and then a further increase until approximately 18 months, followed by a gradual decline (figure 1). Based on the graphical appearance as well as our prior findings,¹⁷ we created two additional models, one with a spline knot at 6 months and one with a spline knot at 18 months. The model with the spline knot at 6 months did not meaningfully change the result (months 1–6: OR 1.068, 95% CI 0.971 to 1.175, p=0.175; months 7–30: OR 0.998, 95% CI 0.973 to 1.022, p=0.845). The model with the spline knot at 18 months showed some evidence of a trend towards increasing correctness of care over the first 18 months and decreasing correctness of care over the remaining 12 months (months 1–18: OR 1.027, 95% CI 1.004 to 1.050, p=0.023; months 18–30: OR 0.963, 95% CI 0.926 to 1.000, p=0.053), with an improvement in model fit based on the QIC values (table 4).

We again conducted a secondary analysis using the outcome variable of overall correct care excluding referral of patients with negative RDT. Similar to the first cohort, graphical depiction of correct referral of patients with a negative RDT by VHWs in the second cohort appeared to show worse performance over time; exclusion of this error resulted in a relatively linear-appearing trend towards increasing correctness of care over time, followed by a slight decline over the final 6 months (figure 1). Repeating the model with the outcome variable as correct care excluding referral of patients with negative RDT demonstrated a trend towards increasing correctness of care over time (OR 1.026, 95% CI 1.007 to 1.045, p=0.006; see table 4).

Discussion

Patient demographic data and presenting complaints were broadly similar between patients treated by the two cohorts of VHWs. Overall quality of care was slightly higher for the first cohort of VHWs compared with the second cohort (77% vs 68%). Both cohorts demonstrated high quality of care for uncomplicated malaria, pneumonia and diarrhoea (>90% on all measures for the first cohort and >85% for all measures for the second cohort). Encouragingly, overuse of medications was fairly low in both cohorts as well. To the best of our knowledge, the programme never experienced stockouts of medications, so VHWs’ ability to provide correct care would not have been affected by unavailability of the necessary medication.

These findings are broadly similar to our prior evaluation of quality of care in Bugoye as well as other studies in Uganda.^{7–9 17} Studies on iCCM care in other countries in sub-Saharan Africa have demonstrated greater heterogeneity. Some have documented correct care for 70%–90% of encounters.^{18 26–28} Others have documented lower quality of care overall^{19 29} or in the control group or preassessment evaluation in several intervention studies.^{30 31} Local context might explain some of the variation in quality of care. In this setting, all scheduled trainings took place (to the best of our knowledge); external funding ensured a consistent supply of diagnostic tests and medications; and monthly collection of paper record forms might have helped provide accountability. These factors might result in higher quality of care compared with settings in which iCCM is implemented on a larger scale.

In both cohorts, the proportion of patients with fever and negative malaria RDT appropriately referred to a health facility was quite low. Our prior evaluation also identified this as a common error,¹⁷ which prompted refresher training on this topic. The persistence of this error despite refresher training might indicate that this represents an intentional decision by VHWs rather than an unintentional error; perhaps VHWs have observed that such patients often have self-limiting viral illnesses and recover without intervention. For this error (as well as other errors), caregiver preferences or requests might also have influenced VHWs. For instance, caregivers might not wish to be referred to the health facility because of the time spent travelling to the facility and waiting to be seen. Some iCCM programmes advise symptomatic management and at-home follow-up rather than referral for patients with subjective fever and a negative malaria RDT.¹¹ When this error was excluded, overall quality of care for all other patients was 84% and 76% in the first and second cohorts, respectively. While inappropriate use of medications was fairly low, pressure from caregivers to provide a medication could also have influenced VHWs’ treatment decisions.

VHWs did not perform as well for appropriate referral and prereferral treatment of patients with danger signs. This finding is somewhat concerning as these patients are likely at highest risk of poor outcomes or death. Very few deaths were reported, though deaths might be under-reported as VHWs might not update their clinical encounter record if a child later dies during that illness episode. Additionally, deaths might have occurred in children who were never evaluated by a VHW; caregivers who recognise that a child has severe illness might proceed directly to a health facility rather than seeking care from a VHW.

We employed GEE regression models to examine trends in quality of care over time (GEE models estimate a population average effect, so these trends should be interpreted as such). For the first cohort of VHWs, quality of care appeared to increase modestly until approximately 3 years after their initial iCCM training, followed by a gradual decrease in quality of care. For the second cohort of VHWs, quality of care was essentially constant over time. This finding is in contrast to our prior study demonstrating increasing quality of care over the first 6 months in the first cohort of VHWs.¹⁷ There are several potential explanations for the trend towards decreasing quality of care for the first cohort in the later years. Uganda relies on an all-volunteer VHWs workforce, which could result in decreased motivation over time or a need to focus on income-generating activities instead of volunteer work. Alternatively, as VHWs gain experience and are further from their initial training, they may begin to rely more on clinical judgement and follow the iCCM protocol less closely. However, based on the secondary analysis, the decline in quality of care persists even when the error of failing to refer patients with a negative RDT is excluded. Another potential explanation for this trend would be higher engagement and interest from supervisors in the earlier years of the programme, though we are not aware of any concrete changes in frequency of trainings, supervision schedule, incentives or other such factors in this programme. Alternatively, there could have been changes in care-seeking behaviours or expectations over time (eg, caregivers bringing children with a broader range of illnesses to VHWs as the programme became more established); we could not meaningfully examine this using the available records.

More broadly, several years represents a substantial follow-up period for a study but not such a substantial period for a health system. From this standpoint, the modest decline in quality of care over time might point to the challenge of relying on a volunteer workforce to fill a critical role in the health system. Some prior studies examining integrated management of childhood illness and long-term trends in quality of facility-based care for malaria or other life-threatening paediatric illnesses have documented consistent improvements in quality of care,^{32 33} though other factors apart from the paid status of these healthcare workers could explain this discrepancy. While iCCM has the potential to increase access to potentially lifesaving care, broader implementation and longer-term reliance on this approach should be accompanied by monitoring and evaluation of quality of care.

This study has at least five limitations. First, while we conducted some data cleaning and assessed for duplicate entries as detailed earlier, these data were not double-entered. Given the large volume of clinical encounters, some data entry errors likely persist. However, such errors are less likely to cause systematic bias. Second, some elements of correct care are not captured on the paper registers and thus cannot be assessed here (eg, correct drug dosages), which may overestimate quality of care. More broadly, a record review approach cannot capture whether or how much health education is provided, or VHWs’ ability to assess children’s overall health and to identify more chronic issues such as malnutrition. Third, this study assesses quality of recorded care rather than quality of actual care. In some instances, this may overestimate quality of care; for example, we cannot assess whether a VHW correctly performed and interpreted a malaria RDT or measured respiratory rate correctly. In other cases, a VHW could provide correct care but have the encounter classified as incorrect due to incomplete record-keeping. For instance, because the words ‘fever’ and ‘malaria’ are often used interchangeably in the local language, some VHWs seem to have recorded a patient as having subjective fever only if the malaria RDT was positive. For a patient with a negative RDT, this misunderstanding would make it appear that the VHW had incorrectly performed an RDT for a patient without subjective fever (when in fact the RDT was appropriately performed). Overall however, prior research suggests that record review somewhat overestimates quality of care when compared with direct observation of VHWs with reexamination by a clinician.^{5 6} Fourth, because the data are deidentified and lack a reliable unique identifier variable, two or more illness episodes for the same child may be included. However, this issue is unlikely to affect the results significantly, as quality of care for different episodes of illness for a given individual are not necessarily correlated (beyond the correlation of receiving care from the same VHW). Fifth, while this study assesses a large number of patient encounters, it includes a fairly small number of VHWs in a single geographical area, limiting its generalisability.

Conclusion

Overall, we found that VHWs continue to provide quality care for uncomplicated malaria, pneumonia and diarrhoea nearly 5 years after initial training, though with a trend towards decreasing quality of care during the later period and with lower quality of care for patients with danger signs. The trend towards decreasing quality of care starting several years after iCCM care initiation points to the importance of long-term monitoring of quality of care by iCCM programmes.

Data availability statement

Data are available upon reasonable request. Data are available for non-commercial use from the corresponding author (jsmiller@post.harvard.edu) upon reasonable request.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

This study involves human participants and was approved by the Partners Healthcare IRB (2016P001423) and the research ethics committee at the Mbarara University of Science and Technology (14/06-16). Because the study involved only analysis of deidentified clinical records, consent was not obtained from individual patients’ caregivers for the study.