Diagnosing Post-Cesarean Surgical Site Infections in Rural Rwanda: Development, Validation, and Field Testing of a Screening Algorithm for Use by Community Health Workers

Abstract

Background: We aimed to develop and validate a screening algorithm to assist community health workers (CHWs) in identifying surgical site infections (SSIs) after cesarean section (c-section) in rural Africa.

Methods: Patients were adult women who underwent c-section at a Rwandan rural district hospital between March and October 2017. A CHW administered a nine-item clinical questionnaire 10 ± 3 days post-operatively. Independently, a general practitioner (GP) administered the same questionnaire and assessed SSI presence by physical examination. The GP's SSI diagnosis was used as the gold standard. Using a simplified Classification and Regression Tree analysis, we identified a subset of screening questions with maximum sensitivity for the GP and CHW and evaluated the subset's sensitivity and specificity in a validation dataset. Then, we compared the subset's results when implemented in the community by CHWs with health center-reported SSI.

Results: Of the 596 women enrolled, 525 (88.1%) completed the clinical questionnaire. The combination of questions concerning fever, pain, and discolored drainage maximized sensitivity for both the GPs (sensitivity = 96.8%; specificity = 85.6%) and CHWs (sensitivity = 87.1%; specificity = 73.8%). In the validation dataset, this subset had sensitivity of 95.2% and specificity of 83.3% for the GP-administered questions and sensitivity of 76.2% and specificity of 81.4% for the CHW-administered questions. In the community screening, the overall percent agreement between CHW and health center diagnoses was 81.1% (95% confidence interval: 77.2%–84.6%).

Conclusions: We identified a subset of questions that had good predictive features for SSI, but its sensitivity was lower when administered by CHWs in a clinical setting, and it performed poorly in the community. Methods to improve diagnostic ability, including training or telemedicine, must be explored.

Surgical site infections (SSIs) create a significant burden for patients and health systems [1–3]. The burden of SSIs is especially high in low- and middle-income countries (LMICs) [4], where incidence rates as high as 30.9% have been reported [5]. Most SSIs develop 5–10 days after surgery [6], and SSI rates are underestimated severely when the observation period for infections is limited to a patient's hospital stay [7–9]. However, after discharge, there is no reliable method to identify infections among surgical patients in the community [10].

Globally, one in every 14 operations is a cesarean section (c-section), and in low-resource settings, c-sections constitute about 30% of the total volume of surgery [11]. In rural Rwandan district hospitals, approximately 60% of all operations are c-sections [12], and women receive minimal to no clinical follow-up after discharge. Notably, our group has reported approximately 11% SSIs among women who have had a c-section in eastern Rwanda [13]. Given the high frequency of c-sections and the prevalence of SSIs, monitoring SSIs after discharge from the hospital is critical. Studies on methods of tracking post-c-section infections among patients who have been discharged to home have been performed predominantly in high-income countries [14–16], with few carried out in sub-Saharan Africa (SSA) [17,18]. The SSA papers did not specify the criteria used to diagnose an incision as infected. A valid and reliable method for detecting post-discharge SSI in LMICs could result in timely provision of care to women after c-section, thus avoiding serious complications and improving options for SSI surveillance.

In many countries, community health workers (CHWs) provide care for mothers and their children in their own homes or villages [19]. In Rwanda, a special cadre of maternal CHWs provides pre-natal care to all mothers and post-partum care for women who deliver vaginally [20]. However, these CHWs do not provide care for women after c-section delivery, requiring that these mothers travel to their health center for post-discharge care. Leveraging this existing cadre of specialized CHWs to provide post-discharge SSI monitoring using a mobile telephone could be feasible, cost effective, and scalable.

We aimed to develop and validate a screening algorithm for use by CHWs via mobile health technology to identify SSIs and for prompt referral of women to receive health care after c-section if a SSI is suspected. We report data from two distinct phases. In the first phase, we developed and validated a simple screening algorithm to identify post-c-section SSIs. In the second phase, we compared the results of the screening tool when implemented in the field setting with health center-reported SSI. Combined, these results provide a basis for other teams considering community screening for SSIs after discharge.

Patients and Methods

Study setting

This study included women who underwent c-section at Kirehe District Hospital (KDH) in rural eastern Rwanda. KDH serves a population of approximately 370,000 people and is operated by the Rwandan Ministry of Health with technical and logistic support from the non-governmental organization Partners In Health. Typically, in Kirehe District, a laboring mother first presents at one of the district's 16 health centers for delivery under the supervision of a nurse. If the nurse identifies an emergency condition, the mother is referred to KDH for delivery. At KDH, a midwife examines the mother, and a general practitioner (GP) determines whether a c-section is indicated. In rare cases, a mother can present directly to KDH, bypassing the health center, either because of an emergency or if she has private insurance. After cesarean delivery, the mother is admitted to the maternity ward for recovery and post-operative follow-up. Typically, the mother is discharged on post-operative day (POD) 3 unless there is a clinical indication necessitating a longer stay. At discharge, nurses recommend the mother to return to her nearest health center every two or three days for the subsequent three weeks for post-partum follow-up and care, including dressing changes and monitoring of the incision.

Study population, enrollment, and follow-up

Figure 1 displays the study timeline.

FIG. 1.

Study timeline for development and validation of surgical site infection detection algorithm in rural Rwanda.

Phase 1

All women 18 years of age or older who delivered via c-section at KDH between March 23 and October 18, 2017 were eligible for inclusion. We excluded women who were not residents of Kirehe District; women who were residents of Mahama Refugee Camp, as they did not have autonomy to complete study follow-up; and women who were not discharged by POD 10. All eligible women were identified after c-section and before discharge, informed of the study objectives and procedures, and invited to participate. We obtained signed informed consent forms from all consenting participants. At discharge, participants were asked to return to KDH at POD 10 ± 3 days to have their incisions evaluated and undergo the screening questionnaire for SSI. Participants received a voucher to reimburse transportation costs and compensate them for survey participation, payable at the time of return for their screening visit. Study staff attempted to contact participants a day before their screening visit to remind them of their appointment. For any participants who missed their first screening visit, the study staff attempted to contact them by telephone and reschedule their visit for the next screening day. Participants who missed both screening days were considered lost to follow-up. Further details on study methods for Phase 1 are described elsewhere [13].

Phase 2

All women 18 years of age or older who underwent cesarean delivery between November 1, 2017 and September 4, 2018 were eligible for inclusion; the same eligibility criteria and enrollment procedures used in Phase 1 were applied. For Phase 2, at discharge, the enrolled participants were randomized to one of three arms: Arm 1 = a CHW visits the participant at home post-operatively to administer an SSI screening questionnaire; Arm 2 = a CHW calls the participant on the telephone post-operatively to administer the questionnaire, with as many as three call attempts; Arm 3 = the standard of care, with no specialized follow-up. For both intervention arms, participants were screened by the CHW (at home or on the telephone) on POD 10 ± 3. On POD 30, a study staff member called the participants to inquire about any further visits to healthcare facilities after discharge from the hospital. Additional details on study methods, including randomization process, for Phase 2 are described elsewhere [21].

Data collection and analysis

For both phases, data collectors compiled sociodemographic information through interviews with patients and health information through chart extraction.

Phase 1

At the screening visit, each participant underwent an assessment by a CHW, who administered 10 questions pertaining to the patient's incision. The CHW was hired exclusively for the study using the same community nomination and selection process used for CHW selection in Rwanda [20].

Only female CHWs were hired to suit the Rwandan cultural norms. The CHWs underwent intensive training on administering the protocol questions and physical examination. The candidate questions were adapted from those used in previous research in Haiti [22] with input from Rwandan clinicians and included:

• 1.Reported increase in pain since discharge;
• 2.Any reported fever since discharge;
• 3.Erythema (redness);
• 4.Edema (swelling);
• 5.Induration (firmness);
• 6.Dehiscence (gaping);
• 7.Drainage from the incision;
• 8.Thick drainage;
• 9.Drainage with discoloration; and
• 10.Drainage with a foul odor.

Next, the participant was screened by a GP with experience in cesarean deliveries and post-operative care. The GP administered the same 10 questions independently and made a diagnosis of the presence or absence of an SSI. All data were entered directly into a REDCap [23] database on a password-protected study computer.

Data from March 23 to July 22, 2017 were used to develop the screening algorithm (“development dataset”). Data from July 23 to October 18, 2017 were used to validate the algorithm (“validation dataset”). For both analyses, we considered the GP SSI diagnosis to be the gold standard. To develop the screening algorithm, we compared the responses to each of the screening questions collected by the CHW with those collected by the GP. Next, we assessed each question's sensitivity and specificity, looking separately at the recorded GP and CHW responses.

We considered two approaches for building the SSI screening algorithm. First, we considered a participant SSI positive if she answered yes to a certain number of original screening questions and mapped every possible cut-off on a receiver operating characteristic (ROC) curve for both GP and CHW responses. Second, we applied a simplified Classification and Regression Tree (CART) analysis. For this approach, we started with the most sensitive question. We then removed all participants with a positive record for this question and considered the most sensitive question among the remaining participants. We continued to remove participants with a positive recorded value iteratively and considered the next most sensitive question until none of the remaining questions detected any more of the SSIs. If there were questions with equally high sensitivity at any stage, we explored paths with each possible combination of questions. At each stage of question selection, we reported the sensitivity and specificity of the algorithm. From these two approaches, we determined a recommended SSI screening algorithm, a subset of the original candidate questions that we thought were the most appropriate to detect SSIs in this context. Using the validation dataset, we assessed the sensitivity and specificity and 95% confidence interval (CI) of the SSI screening algorithm independently, looking separately at the recorded GP and CHW responses.

Phase 2

We collected responses to the SSI screening algorithm for participants in Arms 1 and 2 using a hand-held Android-based tablet. The screening questions used in this phase were those identified in Phase 1 (detailed below) and included: (1) Reported fever since discharge; (2) reported increased pain since discharge; and (3) drainage with discoloration. If a participant answered “yes” to any of the three screening questions, the CHW instructed her to seek further evaluation at the participant's health center. Study staff reviewed healthcare center's logs to record any patients having c-section who were evaluated post-operatively and whether a patient was found to have an SSI and treated at the health center. At the hospital, the same clinical information was captured from the medical charts. On POD 30, participants were telephone-called to ask whether they returned for care at a health center or hospital. Health center records were largely incomplete; however, 228 participants from Arms 1 and 2 (41.5% of the total from these arms) had both a POD 30 call and health center documentation, and the reports of the two sources were identical for all of these participants.

For participants in Arms 1 and 2, we compared the SSI diagnosis by the screening algorithm administered by the CHW to the health center SSI diagnosis. We report the percent positive agreement and percent negative agreement and 95% CI of these two diagnoses, considering the health center diagnosis as the bronze standard.

We received scientific and ethical approval from the Rwanda National Ethics Committee (Kigali, Rwanda, No. 848/RNEC/2016) as well as from Partners Human Research Committee (Boston, MA USA, No. 2016P001943/MGH).

Results

In Phase 1, a total of 596 women met the inclusion criteria. Of these, 525 (88.1%) returned on POD 10 ± 3 days for evaluation of their incision site. This group included 294 (56.0%) in the development dataset and 231 (44.0%) in the validation dataset.

In the development dataset, 31 women (10.5% of 294) had a GP-diagnosed SSI. Among these, the questions that received the most affirmative responses when administered by a GP were having a fever since coming home from the hospital (n = 13; 41.9%), increasing pain (n = 15; 48.4%), and discolored drainage (n = 24; 77.4%) (Table 1). The questions that yielded the the largest number of “yes” responses when administered by a CHW for those with GP-diagnosed SSI were presence of thick drainage (n = 21; 67.7%), discolored drainage (n = 25; 80.7%), and drainage with a foul odor (n = 20; 64.5%). A gaping incision was indicated by the GP for 13 of the women (41.9%) found to have SSI but gaping was not identified by the CHW for these patients. There was disagreement between the GP-administered and CHW-administered questions (Appendix I). Responses were nearly always consistent for redness (96.9%) and least consistent for the presence of discolored drainage (83.3%). Drainage alone was not sensitive for GP-diagnosed SSI and so was considered only in the presence of discoloration, foul odor, or purulence, leaving nine possible screening questions for use when developing the final algorithm.

Table 1.

General Practitioner (GP) and Community Health Worker (CHW) Responses for Participants with and without Surgical Site Infection (SSI) (Algorithm Development Dataset, March 23-July 22, 2017)

Protocol Questions	GP-screened positive for SSI (n = 31)		GP-screened negative for SSI (n = 263)
	Answered “yes” to screening question (%)	Answered “no” to screening question (%)	Answered “yes” to screening question (%)	Answered “no” to screening question (%)
GP responses for participants with and without SSI
According to the GP screening, did the participant have:
Fever since coming home from the hospital?	13 (41.9)	18 (58.1)	20 (7.6)	243 (92.4)
Increasing pain?	15 (48.4)	16 (51.6)	8 (3.0)	255 (97.0)
Redness?	7 (22.6)	24 (77.4)	3 (1.1)	260 (98.9)
Swelling?	8 (25.8)	23 (74.2)	3 (1.1)	260 (98.9)
Firmness?	6 (19.4)	25 (80.7)	12 (4.6)	251 (95.4)
Any drainage? If yes,	28 (90.3)	3 (9.7)	46 (17.5)	217 (82.5)
Is the fluid thick?	15 (48.4)	16 (51.6)	2 (0.8)	261 (99.2)
Is the fluid brown, yellow, green, or white?	24 (77.4)	7 (22.6)	15 (5.7)	248 (94.3)
Does the fluid smell bad?	19 (61.3)	12 (38.7)	0	263 (100)
An incision that gaped open suddenly?	13 (41.9)	18 (58.1)	5 (1.9)	258 (98.1)
CHW responses for participants with and without SSI (according to GP)
According to the CHW screening, did the participant have:
Fever since coming home from the hospital?	8 (25.8)	23 (74.2)	19 (7.2)	244 (92.8)
Increasing pain?	10 (32.3)	21 (67.7)	11 (4.2)	252 (95.8)
Redness?	1 (3.2)	30 (96.8)	0	263 (100)
Swelling?	0	31 (100)	1 (0.4)	262 (99.6)
Firmness?	1 (3.2)	30 (96.8)	2 (0.8)	261 (99.2)
Any drainage? If yes,	27 (87.1)	4 (12.9)	86 (32.7)	177 (67.3)
Is the fluid thick?	21 (67.7)	10 (32.3)	37 (14.1)	226 (85.9)
Is the fluid brown, yellow, green, or white?	25 (80.7)	6 (19.4)	47 (17.9)	216 (82.1)
Does the fluid smell bad?	20 (64.5)	11 (35.5)	34 (12.9)	229 (87.1)
An incision that gaped open suddenly?	0	31 (100)	0	263 (100)

Figure 2 shows the ROC curve comparing cut-offs for the number of questions with positive responses by GP and CHW between those with GP-diagnosed SSI and those without. Diagnosing SSI by having an affirmative indication on three or more questions yielded moderate sensitivity and high specificity for both the GP (sensitivity = 74.2%; specificity = 98.9%) and CHW (sensitivity = 71.0%; specificity = 89.0%). The results of the CART analysis are shown in Figure 3. Color of drainage was the most sensitive question for GPs (77.4%) and CHWs (80.7%). Two combinations of the GP-administered questions maximized sensitivity: fever–pain–discolored drainage (sensitivity = 96.8%; specificity = 85.6%) and fever–gaping incision–discolored drainage (sensitivity = 96.8%; specificity = 86.7%). For the CHW-administered questions, fever–pain–discolored drainage maximized sensitivity (sensitivity = 87.1%; specificity = 73.8%). Given the stability of this algorithm for both GP and CHW screenings, we moved forward with this combination of questions for validation.

FIG. 2.

Comparing number of questions (Qs) with positive responses by general practitioners (GPs) and community health workers (CHWs) between those with GP-diagnosed surgical site infection and those without.

FIG. 3.

Algorithm based on sensitivity and specificity of general practitioner (A) and algorithm based on sensitivity and specificity of community health workers (B). SSI = surgical site infection.

In the validation dataset, 21 women (9.1% of 231) were considered SSI positive by the GP. In this dataset, the subset of questions fever–pain–discolored drainage had a sensitivity of 95.2% (95% CI: 76.2–99.9%) and specificity 83.3% (95% CI: 77.6–88.1%) for the GP-administered questions and a sensitivity of 76.2% (95% CI: 52.8–91.8%) and specificity of 81.4% (95% CI: 75.5–86.4%) for the CHW-administered questions.

In the community screening, the overall percent agreement between CHW and health center diagnoses for those in Arm 1 was 78.6% (95% CI: 73.0%–83.5%), the percent positive agreement was 32.0% (95% CI: 14.9%–53.5%), and the percent negative agreement was 83.7% (95% CI: 78.2%–88.3%) (Table 2). For those in Arm 2, the overall percent agreement between the CHWs and health center diagnoses was 84.3% (95% CI: 78.6%–89.0%), the percent positive agreement was 37.5% (95% CI: 18.8%–59.4%), and the percent negative agreement was 90.6% (95% CI: 85.3%–94.4%). When the two arms were combined, the overall percent agreement between the CHWs and health center diagnoses was 81.1% (95% CI: 77.2%–84.6%), the percent positive agreement was 34.7% (95% CI: 21.7%–49.6%), and the percent negative agreement was 86.7% (95% CI: 83.0%–89.9%).

Table 2.

Percent Agreement between Community Health Worker and Health Center Diagnosis of Surgical Site Infection

Agreement Type	n (%)	95% CI
Arm 1
Percent positive (N = 25)	8 (32.0)	14.9-53.5
Percent negative (N = 227)	190 (83.7)	78.2-88.3
Overall percent (N = 252)	198 (78.6)	73.0-83.5
Arm 2
Percent positive (N = 24)	9 (37.5)	18.8-59.4
Percent negative (N = 180)	163 (90.6)	85.3-94.4
Overall percent (N = 204)	172 (84.3)	78.6-89.0
Both arms
Percent positive (N = 49)	17 (34.7)	21.7-49.6
Percent negative (N = 407)	353 (86.7)	83.0-89.9
Overall percent (N = 456)	370 (81.1)	77.2-84.6

CI = confidence interval.

Discussion

In this study, we aimed to develop and validate a simple screening algorithm for CHWs to assist in detecting SSIs after discharge to the patients' homes. We found that the combination of questions fever–pain–discolored drainage had moderate sensitivity (76.2%) and specificity (81.4%) when used in a hospital setting; however, the same screening algorithm had low positive agreement (34.7%) for the CHWs compared with the health center diagnosis when administered in the community.

We identified only two other studies in East Africa, one in Kenya [17] and another in Tanzania [18], that aimed to validate telephone-based assessments for SSI. These studies, where post-operative patients were called and asked a series of questions about the surgical site and then were evaluated by a clinician in person, showed sensitivity similar to what we observed for the CHW-administered questions in the hospital setting, but it was considerably higher than the positive agreement we observed when CHWs administered the questions in the community. One plausible explanation for the difference is that in the other two regional studies, questions were administered by clinically trained staff or nurses, whereas in our study, questions were administered by CHWs with only primary education and minimal clinical training.

We found differences in screening accuracy between GP- and CHW-administered questions in the clinical setting, with the screening protocol being less sensitive and specific when administered by a CHW. Participant responses by GPs and CHWs were conflicting, including for patient-reported assessments such as “have you had pain since discharge?” Because CHWs are recruited from within the community where they live, they are, presumably, more easily accessible to and accepted by community members and understand the context of the community better than health facility-based clinicians. However, in some SSA countries, the perceived inability of CHWs to maintain confidentiality [24,25] and lack of formal training [26,27] were reported as barriers to the community's acceptance of CHWs. Further research is needed to understand why women in Rwanda who have undergone a c-section communicate their symptoms differently to a GP than to a CHW.

Although we are reluctant to endorse this SSI screening algorithm because of its poor performance in the non-clinical setting, its high accuracy in the clinical setting and the consistency of the questions selected during the development stage, whether used by GPs or CHWs, suggests that this algorithm has promise. Future studies need to examine ways to improve CHW-administered screening consistency and accuracy. The CHWs might need more intensive training to be able to detect SSIs [28]. CHWs with more education generally have better performance [29], and therefore raising the educational requirements for CHWs may benefit SSI diagnostic accuracy. Refresher training has improved performance among CHWs in Uganda [30]. On the other hand, perhaps SSIs are too complex to be detected by a questionnaire. Health systems could consider supplementary methods such as telemedicine to ensure women who have undergone a c-section receive the care they need in a timely manner. Studies in the United States have demonstrated that adding incision photos to patient-reported clinical information improved diagnostic accuracy from 67% to 76% (p < 0.001) [31] and that simple abdominal operations with short post-operative stays (< 4 days) were the procedures most amenable to telemedicine follow-up [32]. Our team is exploring the feasibility of photograph-based diagnosis of post-cesarean SSIs in rural Rwanda.

Our study has a few limitations. Our sample was small compared with national SSI surveillance datasets in high-income countries. However, the development and validation of an algorithm to detect post-cesarean SSI in SSA has had little to no previous exploration, and the proposed algorithm had stable properties in the development and validation dataset. Another limitation is that in Phase 2, we did not have a GP-confirmed SSI; rather, we relied on the SSI documented in the health center chart or reported by the participant on follow-up. In addition, we did not have the timing of the health center SSI diagnoses, making it plausible that the screening occurred at a different time than the diagnosis. However, the prevalence of confirmed SSIs in Phase 1 was approximately the same as what we observed in Phase 2 (10.9% versus 10%), and, given that most SSIs develop by POD 10, we have strong confidence in these data. A final limitation is that we used CHWs whose sole task was to administer the questionnaire; thus, their workload was not as heavy as it would be for a standard CHW in Rwanda. Additional effectiveness research on the feasibility and accuracy of application of SSI screening algorithms by CHWs who have other demands would guide the scalability of this approach.

Conclusion

We believe that the combination of questions—fever–pain–discolored drainage—has sufficient sensitivity and specificity when administered in a clinical setting and is simple enough for CHWs to use for a community-based screening for SSIs after cesarean delivery. The CHW-administered questions had lower sensitivity in the clinical setting and performed poorly in the field setting. Additional research is needed to stabilize the algorithm so that it excels both in and outside of clinical settings. Telemedicine intervention, along with screening questions, might improve the diagnostic accuracy of SSIs that develop after c-section.

Appendix

Appendix A1.

Distribution of Answers to Screening Protocol by General Practitioners (GPs) and Community Health Workers (CHWs) (Algorithm Development Dataset, March 23–July 22, 2017)

Did the Participant Have	GP and CHW answered “Yes” (%)	GP answered “Yes”, CHW answered “No” (%)	GP and CHW answered “No” (%)	GP answered “No,” CHW answered “Yes” (%)
Fever since coming home from the hospital?	16 (5.4)	17 (5.8)	250 (85.0)	11 (3.7)
Increasing pain?	13 (4.4)	10 (3.4)	263 (89.5)	8 (2.7)
Redness?	1 (0.3)	9 (3.1)	284 (96.6)	0
Swelling?	0	11 (3.7)	282 (95.9)	1 (0.3)
Firmness?	0	18 (6.1)	273 (92.9)	3 (1.0)
Any drainage? If yes,	64 (21.8)	10 (3.4)	171 (58.2)	49 (16.7)
Is the fluid thick?	16 (5.4)	1 (0.3)	235 (79.9)	42 (14.3)
Is the fluid brown, yellow, green, or white?	31 (10.5)	8 (2.7)	214 (72.8)	41 (14.0)
Does the fluid bad?	15 (5.1)	4 (1.4)	236 (80.3)	39 (13.3)
An incision that gaped open suddenly?	0	18 (6.1)	276 (93.9)	0

Funding Information

This study was funded by the National Institutes of Health, grant number R21EB022369.

Author Disclosure Statement

The authors do not have any potential conflicts of interest.