Initiating a Standardized Regional Referral and Counter-Referral System in Guatemala: A Mixed-Methods Study

Abstract

Background:Few data exist for referral processes in resource-limited settings. We utilized mixed-methods to evaluate the impact of a standardized algorithm and training module developed for locally identified needs in referral/counter-referral procedures between primary health centers (PHCs) and a Guatemalan referral hospital. Methods: PHC personnel and hospital physicians participated in surveys and focus groups pre-implementation and 3, 6, and 12 months post-implementation to evaluate providers’ experience with the system. Referred patient records were reviewed to evaluate system effectiveness. Results: A total of 111 initial focus group participants included 96 (86.5%) from PHCs and 15 from the hospital. Of these participants, 53 PHC physicians and nurses and 15 hospital physicians initially completed written surveys. Convenience samples participated in follow-up. Eighteen focus groups achieved thematic saturation. Four themes emerged: effective communication; provision of timely, quality patient care with adequate resources; educational opportunities; and development of empowerment and relationships. Pre- and post-implementation surveys demonstrated significant improvement at the PHCs (P < .001) and the hospital (P = .02). Chart review included 435 referrals, 98 (22.5%) pre-implementation and 337 (77.5%) post-implementation. There was a trend toward an increased percentage of appropriately referred patients requiring medical intervention (30% vs 40%, P = .08) and of patients requiring intervention who received it prior to transport (55% vs 73%, P = .06). Conclusions: Standardizing a referral/counter-referral system improved communication, education, and trust across different levels of pediatric health care delivery. This model may be used for extension throughout Guatemala or be modified for use in other countries. Mixed-methods research design can evaluate complex systems in resource-limited settings.

Introduction

Most developing countries have formal or informal referral processes in place within their tiered health care system, where an acutely ill patient evaluated at a primary level of care is referred to a higher level of care. The system may or may not include a process for counter-referral, where patients are then sent back to their primary care provider for follow-up and continued care. Several observational studies suggest that the functionality of these systems is often limited.^1-5 Barriers include inappropriate or inadequate use of the process by personnel, misunderstanding of a tiered system by patients, and lack of available resources needed for a functional referral system.^1,3-13 Many global projects and processes, including Integrated Management of Childhood Illness,^5,7,10,14,15 rely heavily on these systems in providing definitive care for their sickest patients. There are few data on referral system quality improvement or interventions. The majority of intervention and outcome data for emergent referral is from high-risk obstetrics in resource-limited settings.^6,16-19 The literature suggests that development of standards, communication of those standards to providers, quality monitoring and feedback, as well as team-based quality improvement problem solving are useful strategies to employ in referral process design^{1,3-8,16,17,20}

This project developed from an ongoing multi-institutional collaborative effort among Hospital Nacional Pedro Bethancourt (HNPB; a public referral hospital in Antigua, Guatemala), the Guatemalan Ministry of Health (MoH), the Pan American Health Organization (PAHO), and Baylor College of Medicine/Texas Children’s Hospital (BCM/TCH) aimed at building the capacity of hospitals in Latin America by improving the recognition and initial stabilization of acutely ill children through triage training programs and process implementation.^21-23 Previously, we developed and implemented a locally led, self-sustaining, high-quality, and effective Spanish version of the WHO Emergency Triage Assessment and Treatment (ETAT) program (Clasificación, Evaluación y Tratamiento de Emergencias Pediátricas [CETEP]).²¹ Since then HNPB has implemented a formal CETEP training program for pediatric personnel and triage process in the pediatric emergency department (PED) that has reduced admission rates, length of stay, and mortality.²² Leaders at HNPB self-identified the next step in the quality improvement initiative as standardization and improvement of the referral/counter-referral system with primary health centers (PHCs) alongside extension of CETEP trainings to pediatric personnel in the PHCs within their region in Guatemala.²³

The objective of this study was to standardize and enhance the existing referral and counter-referral system for acutely ill pediatric patients between PHCs and the regional referral hospital. We evaluated the impact of system standardization on the referral/counter-referral process, communication, resources, education, patient care, and relationship building between healthcare providers.

Methods

Design and Setting

This was a prospective mixed-methods cohort study conducted in 1 of 2 districts in the Sacatepéquez region of Guatemala (population: 329 547 inhabitants over 466 km²; 39% <15 years of age) from April 2012 to October 2013. The overall mortality rate is 5.3/1000, and the mortality rate for children <5 years of age is 32/1000 live births.²⁴ The region has a tiered health care system with numerous health posts staffed primarily by nurses, 4 PHCs served by general medical practitioners, and a secondary-level public referral teaching hospital. The 4 PHCs located within the region of Sacatepéquez vary greatly in personnel and resource availability, distance from the referral hospital, and patient volume. A site assessment of the PHCs was completed in February 2012 to understand resources and barriers including physical space, personnel, equipment, supplies, and logistical processes (Table 1). The hospital treats approximately 20 000 children 0 to 12 years of age per year for acute illnesses through the PED and has 11 000 encounters per year in the outpatient pediatric clinic.

Table 1.

Composition of Health Centers.

	Health Center
	1	2	3	4
Demographics
Driving distance from nearest referral facility (in minutes)	<5 minutes	~30 minutes	~30 minutes	~40 minutes
Time open	M-F (8 am to 4 pm)	M-F (8 am to 4:30 pm)	M-F (8 am to 4:30 pm)	7 days/week, 24 hours/day
Pediatric population in catchment area	81 000	40 000	49 000	46 000
# of pediatric sick visits (per day)	15-25	10-25	15	50-55
# of pediatric referrals (per month)	12-16	4	20	30-60
Daily staffing
# of physicians	3	2	3	6
# physician participants	3	2	3	4
# of professional nursesa	2	3	1	4
# of professional nurse participants	1	3	1	2
# of auxiliary nursesb	9	6	4	14
# of auxiliary nurse participants	9	6	4	10
# of dentists	2	1	0	1
# of dentist participants	2	0	0	1
Available resources
BVM	No	No	Yes	Yes
Pediatric mask for BVM	No	No	No	Yes
Supplemental oxygen	No	No	No	Infrequently
Nebulized breathing treatments	Yes	Yes	Yes	Yes
Pediatric nebulization masks	No	No	No	No
Oral rehydration solution	Yes	Yes	Yes	Yes
Nasogastric/orogastric tube	No	No	No	Yes
Pediatric IV catheters	No	No	No	Yes
Intravenous fluids	Yes	Yes	Yes	Yes
Antibiotics	Yes	Yes	Yes	Yes
Antipyretics	Yes	Yes	Yes	Yes
Anticonvulsant	No	Yes (limited)	No	Yes
Ambulance available at center	No	No	No	Yes
Able to call ambulance	Yes	Yes	Yes	Yes

Abbreviations: BVM, bag-valve-mask; F, Friday; IV, intravenous; M, Monday.

^aProfessional nurses: several years of education after high school.

^bAuxiliary nurses: minimal education after high school.

Study Population

All staff members caring for children at the 4 PHCs as well as hospital PED and inpatient pediatric physicians were invited to participate. PHC personnel included general medical practitioners, dentists, professional nurses, auxiliary nurses, ambulance drivers, and administrative personnel in clinical areas. Professional nurses have several years of formal education following high school, while auxiliary nurses may have minimal training after high school. Hospital personnel included attending pediatricians and pediatric residents. Chart review included all referred pediatric patients (0-13 years of age) referred from the PHCs to the referral hospital PED within the region during the study period. All personnel at the PHCs were included since the referral of an emergent or priority patient required a team approach, with the physician stabilizing the patient while nursing, administration, and transport assist with referral paperwork and phone calls. Only the physicians were included at the hospital since they were the only ones responsible for the verbal and written counter-referral process that would occur in a nonemergent manner at patient discharge.

Intervention

Although a referral/counter-referral form designed by the MoH existed, we performed a feasibility assessment in February 2012 with a large sample of the PHC and referral hospital stakeholders, which revealed multiple systems issues that thwarted a successful process. These included lack of provider understanding of the system, no routine use of verbal communication, and unclear expectations of PHC and hospital providers (Figure 1). Based on these locally identified needs, our collaborative team designed a standardized algorithm (Supplement 1; all supplemental materials are available in the online version of the journal) and training module for referrals (from PHCs to the hospital) and counter-referrals (from the hospital back to PHCs for continued care; Figure 2). In October 2012, the BCM/TCH team performed the initial referral/counter-referral training to a convenience sample of available personnel responsible for transfers from both the PHCs and HNPB PED and inpatient pediatric personnel. Selected HNPB providers were trained to perform subsequent trainings. This module was completed alongside CETEP trainings for the PHC participants. All participants received copies of the algorithms related to their role in the referral/counter-referral system. Algorithms were posted throughout clinical areas at the hospital and the PHCs. Following the training, all personnel were directed to use the enhanced referral/counter-referral procedures, which incorporated the MoH form already in existence and were available at all sites. MoH and institutional review board approvals were obtained. The MoH was involved throughout all stages of this project, supporting regional trainings and data collection.

Figure 1.

Fishbone diagram illustrating challenges in the referral/counter-referral system.

Figure 2.

Diagram of Patient Flow through the Referral/Counter-referral System.

CETEP, Clasificación, Evaluación y Tratamiento de Emergencias Pediátricas (Spanish version of the WHO Emergency Triage Assessment and Treatment [ETAT] program); CR, counter-referral; dc, discharge; ED, emergency department; PHC, primary health center.

Outcome Measurement

We evaluated the referral/counter-referral process with mixed methodology utilizing quality improvement, qualitative and quantitative methods, including focus groups, individual surveys, and retrospective and prospective hospital and PHC chart review of referred patients over an 18-month period (Figure 3). We used Revised Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) as detailed in Figure 4.²⁵

Figure 3.

Project Timeline.

Figure 4.

Project Outline in SQUIRE guidelines format.

We conducted focus groups with all available personnel from the 4 PHCs and hospital physicians immediately prior to training and implementation, and at 3, 6, and 12 month post-implementation to identify needs and barriers and to rapidly incorporate necessary quality improvement changes/modifications identified by the focus groups after each periodic evaluation (Supplement 2). Experienced, trained moderators (MAS, HLC) and transcribers fluent in written and spoken Spanish conducted focus groups following a standardized script to make sure all components of the referral/counter-referral system were explored throughout the study.

We also designed an individual survey (Likert scale 1-5) for PHC and hospital physicians and PHC nurses that were given at the same time intervals. The survey constructs were determined a priori based on identified needs and barriers. The survey was piloted with 16 volunteers in the participant group and modified in response to this feedback. The participants used in the pilot were included in the survey when it was administered at each time point throughout the study.

Both focus groups and surveys evaluated the provider experience with the referral/counter-referral system. This included bidirectional written and verbal communication between the referral hospital and PHCs, the ease of using the referral process, and provider confidence that the patient is actually arriving at the hospital and being followed-up at the PHC. We also used these tools to explore the relationship and trust between PHC and hospital staff.

We reviewed hospital and PHC medical records of all children from the 4 PHCs referred to HNPB PED for care 6 months prior to implementation (April 2012 to October 2012) and then monthly for 1 year after implementation (October 2012 to October 2013) to evaluate the effectiveness of the standardized referral system. Patients referred from the PHCs were already recorded on weekly registers kept at the PHC. Two referral hospital pediatric residents used a standardized data collection form that was created by BCM/TCH to record the following data from medical charts, when available: age, gender, referral PHC, referral category based on CETEP guidelines (emergent, priority, or nonemergent), need for intervention prior to transport, whether the patient received the appropriate intervention prior to transport (defined a priori). BCM/TCH worked with the HNPB residents on site to pilot the data forms, create the electronic database, and complete a sample of 15 medical record reviews side-by-side to assure quality of extraction. These 2 pediatric residents reviewed all records and cross-checked their results with each other before data entry; if there was variation between categories assigned, they discussed the case together until reaching consensus. We only considered interventions that the PHCs would be able to perform based on our site assessment. We considered emergent and priority referrals appropriate and nonurgent referrals as inappropriate.

Statistical Analysis

Focus Groups

The day following each focus group, all trained transcribers met to compare and consolidate their notes and memos that contained the thoughts and ideas of the PHC and hospital personnel as they evolved throughout the study. After compilation of all transcriptions, one researcher (RK) independently coded the transcriptions from each time point using an iterative coding process to identify patterns of responses, ensure reliability, and examine discrepancies through thematic analysis. Rigorous memos of coding decisions were kept to provide consistency in coding as the coding progressed. The codes were then categorized and emerging themes were identified. Each time point was compared to all others to identify changes over time. A second researcher (EMS) was then asked to review transcripts, codes, and themes and peer debrief the findings to establish triangulation, achieve consensus, and trustworthiness.

Surveys

Percentages were calculated for demographic characteristics. Dentists were excluded from subgroup analyses due to small sample size and limited contact with pediatric patients at the PHCs. Statistical analysis was conducted by using the Statistical Package for the Social Sciences (SPSS), version 23, software (IBM Corp, Armonk, NY). Surveys were Likert-based scales analyzed at each time point by nonparametric analyses. A priori comparisons were made to evaluate changes in written communication, verbal communication, ease of referral process, provider feelings of certainty of patient arriving at hospital, provider certainty that patient would be followed-up, relationship between PHC and hospital staff, and trust between PHC and hospital staff. We used the Mann-Whitney test to analyze the data. Only surveys without any missing values were included in the analysis. After initial analysis, auxiliary and professional nurse data were combined, given the low number of professional nurses.

Chart Review

Percentages were calculated for demographic characteristics. Age was significantly skewed to the right (P < .001), so Kruskal-Wallis testing was utilized when comparing all 4 PHCs, otherwise χ² or Mann-Whitney analysis was used to analyze the remainder of the variables.

Results

Focus Groups

There were a total of 18 focus groups consisting of a total of 111 initial participants: 96 from the PHCs (11 general medical practitioners, 9 professional nurses, 29 auxiliary nurses, 3 dentists, 3 ambulance drivers, 41 administrative personnel) and 15 physicians from the hospital. The 3-, 6-, and 12-month focus groups included a convenience sample of available personnel from the aforementioned initial participants. Four principle themes emerged regarding the components of a successful referral/counter-referral system—the need for effective communication; provision of timely, quality patient care with adequate resources; educational opportunities; and development of empowerment and relationship building (Supplement 3).

The first theme that emerged is that the referral/counter-referral system should foster clear, legible, and comprehensive written and verbal communication that is essential between health care providers:

There is better communication between the health center and hospital. Communication is better with the transport personnel, now we involve them in care. We can always find the number to call the hospital because it is posted in the office and I can even call it for advice. When the ambulance gets to the hospital, they are getting better care. Treatment is more cordial. (PHC physician)

Now we are speaking the same language, because we have training in classification like they do at the hospital. (PHC nurse)

Furthermore, participants stressed patients requiring referral should experience safe, timely, and quality medical care. Resource limitations include basic emergency medications/equipment at the health center and during patient transport, a reliable phone system, a dedicated ambulance for each health center, the cost of patient transport, and the time necessary to adequately communicate (fill out forms and make phone calls):

The care that we give to the patients here has changed. Now we know who to refer, we treat the sicker kids faster, and we know when they get there they are being cared for sooner. Now we know the danger signs and triage and treat according to that. (PHC nurse)

A child came in choking and blue. We tried to do the maneuvers we learned. But he died. We didn’t have an ambu bag. (PHC physician)

Just because we know what to do, doesn’t mean we can always do it. Like sometimes for dehydrated patients, we don’t have the right angiocaths. (PHC nurse)

Another important component was the availability of educational opportunities instructing all health care providers and patients of their responsibility in the referral/counter-referral system and focusing on training health center and transport personnel in identification and stabilization of acutely ill children:

Now we stabilize more before transfer. People have learned CETEP and know how to classify [triage] the patients. We are applying the skills that we learned to stabilize and care for the patients. Even the new people that have not been trained are learning what we do from us. We can start treating an emergency right away based on their classification [triage category]. (PHC nurse)

Now the transport personnel is more helpful and can do basic things for the patient. It’s much better. We [PHC staff] have learned a lot about what to do in relation to CETEP emergencies and priorities. (PHC nurse)

It would improve patient care if we had an educational plan for the patient prior to referring to the hospital and in the hospital prior to coming back. It would be better for the patient. (PHC nurse)

Finally, the relationships established from the development of a referral/counter-referral system aids in development of self-confidence, mutual respect, and trust essential to the emergence of a team dynamic between caregivers and patients:

Before, patients told us that the hospital threw the referral sheet in the trash, but that has changed. There is more respect for the system and from the doctors for the referral forms and for us. Everything that has changed about the system has been positive. (PHC physician)

Things are better because the project involved everybody, even the transport personnel ask about the patient and know more, because more people feel part of the project. (Hospital physician)

We have noted change. There is more communication. The doctors’ relationships are better. More referrals are being sent and more follow-up forms are being received. We are also seeing more patients in follow-up. The personal relationship between the hospital and health centers is better. (PHC nurse)

Surveys

Surveys were initially completed by a total of 53 PHC physicians and nurses and 21 hospital physicians. Frequency of completion at each time is noted in Table 2. Using a 5-point Likert-type scale, the overall range of possible points for the PHC survey is 35 to 175 and for the hospital survey it is 38 to 190. There is a statistically significant improvement between pre-intervention and 3 months (P = .001), 6 months (P < .001), and 12 months (P < .001) post-intervention PHC total survey scores (Table 3). Hospital survey data were also statistically significant when comparing pre-intervention to 3 months (P = .004), 6 months (P = .02), and 12 months (P = <.001) post-intervention total survey scores (Table 3).

Table 2.

Frequency of Completed Surveys over Time.

Time	PHC (N = 53), n (%)	Hospital (N = 21), n (%)
Pre-intervention	23 (43.4%)	17 (81.0%)
3 Month	24 (45.3%)	13 (61.9%)
6 Month	23 (43.4%)	7 (33.3%)
12 Month	20 (37.7%)	15 (71.4%)
Completion of all 4 times	2 (3.8%)	3 (14.3%)

Abbreviation: PHC, primary health centers.

Table 3.

PHC and Hospital Survey Scores Over Time.

PHC	Pre Total Score, Median (IQR) (N = 23)	3-Month Total Score, Median (IQR) (N = 24)	6-Month Total Score, Median (IQR) (N = 23)	12-Month Total Score, Median (IQR) (N = 20)	P a
	100.0 (86.0, 118.0)	125.0 (112.0, 132.8)			.001
	100.0 (86.0, 118.0)		135.0 (120.0, 139.0)		<.001
	100.0 (86.0, 118.0)			133.0 (117.75, 147.8)	<.001
Hospital	Pre Total Score, Median (IQR) (N = 17)	3-Month Total Score, Median (IQR) (N = 13)	6-Month Total Score, Median (IQR) (N = 7)	12-Month Total Score, Median (IQR) (N = 15)	P a
	113.0 (100.0, 126.5)	137.0 (129.0, 148.5)			.004
	113.0 (100.0, 126.5)		149.0 (116.0, 159.0)		.02
	113.0 (100.0, 126.5)			141.0 (133.0, 158.0)	<.001

Abbreviations: IQR, interquartile range; PHC, primary health center.

^aP value was calculated using the Mann-Whitney test for unpaired data.

Differences between time periods were further stratified by constructs defined a priori. From the PHC perspective, survey scores showed a significant improvement by the conclusion of the study period (12 months) compared to pre-intervention in the following measures: PHC written communication (P = .001), Hospital written communication (P < .001), PHC verbal communication (P = .004), Hospital verbal communication (P = .01), Ease of using the referral/counter-referral system (P = .01), and Trust and relationship with the hospital (P < .001; Table 4). No significant difference was seen in Patient follow-up (P = .22) or Positive effect of the system on patient care (P = .15; Table 4).

Table 4.

PHC Survey Data over Time Stratified by Construct.

Pre-Intervention, Median (IQR)	3-Month, Median (IQR)	6-Month, Median (IQR)	12-Month, Median (IQR)	P
Written Communication—PHC Responsibility
N = 47	N = 32	N = 29	N = 23
13.0 (10.0, 16.0)	15.0 (13.0,17.0)	16.0 (11.5, 19.0)	17.0 (15.0, 19.0)	.004a
13.0 (10.0, 16.0)	15.0 (13.0,17.0)			.06b
13.0 (10.0, 16.0)		16.0 (11.5, 19.0)		.03b
13.0 (10.0, 16.0)			17.0 (15.0, 19.0)	.001b
Written Communication—Hospital Responsibility
N = 34	N = 29	N = 27	N = 22
27.50 (17.0, 33.25)	33.0 (24.0, 38.50)	35.0 (33.0, 38.0)	36.0 (31.25, 39.0)	<.001a
27.50 (17.0, 33.25)	33.0 (24.0, 38.50)			.02b
27.50 (17.0, 33.25)		35.0 (33.0, 38.0)		<.001b
27.50 (17.0, 33.25)			36.0 (31.25, 39.0)	<.001b
Verbal Communication—PHC Responsibility
N = 41	N = 32	N = 30	N = 23
7.0 (5.50, 7.0)	7.0 (6.0, 7.0)	7.0 (6.0, 8.25)	9.0 (6.0, 10.0)	.02a
7.0 (5.50, 7.0)	7.0 (6.0, 7.0)			.83b
7.0 (5.50, 7.0)		7.0 (6.0, 8.25)		.19b
7.0 (5.50, 7.0)			9.0 (6.0, 10.0)	.004b
Verbal Communication—Hospital Responsibility
N = 41	N = 33	N = 29	N = 24
1.0 (1.0, 1.0)	1.0 (1.0, 2.0)	1.0 (1.0, 2.0)	1.0 (1.0, 2.0)	.051a
1.0 (1.0, 1.0)	1.0 (1.0, 2.0)			.10b
1.0 (1.0, 1.0)		1.0 (1.0, 2.0)		.03b
1.0 (1.0, 1.0)			1.0 (1.0, 2.0)	.01b
Ease of Using the Referral/Counter-Referral System
N = 39	N = 29	N = 28	N = 22
27.0 (23.0, 30.0)	28.0 (24.0, 30.0)	28.50 (24.25, 32.0)	31.0 (26.0, 34.25)	.04a
27.0 (23.0, 30.0)	28.0 (24.0, 30.0)			.93b
27.0 (23.0, 30.0)		28.50 (24.25, 32.0)		.20b
27.0 (23.0, 30.0)			31.0 (26.0, 34.25)	.01b
Patient Follow-up at the PHC
N = 40	N = 32	N = 30	N = 24
5.0 (4.0, 6.75)	5.0 (4.0, 6.75)	5.0 (4.0, 6.25)	6.0 (4.0, 7.0)	.50a
5.0 (4.0, 6.75)	5.0 (4.0, 6.75)			.98b
5.0 (4.0, 6.75)		5.0 (4.0, 6.25)		.88b
5.0 (4.0, 6.75)			6.0 (4.0, 7.0)	.22b
Positive Effect of Referral System on Patient Care
N = 29	N = 29	N = 26	N = 23
16.0 (13.0, 25.0)	25.0 (15.0, 32.50)	32.50 (17.75, 35.0)	17.0 (14.0, 33.0)	.01a
16.0 (13.0, 25.0)	25.0 (15.0, 32.50)			.02b
16.0 (13.0, 25.0)		32.50 (17.75, 35.0)		.001b
16.0 (13.0, 25.0)			17.0 (14.0, 33.0)	.15b
Trust and Relationship Between PHC and Hospital
N = 39	N = 31	N = 29	N = 24
8.0 (6.0, 10.0)	11.0 (8.0, 12.0)	11.0 (9.50, 12.0)	12.0 (10.25, 13.0)	<.001a
8.0 (6.0, 10.0)	11.0 (8.0, 12.0)			.002b
8.0 (6.0, 10.0)		11.0 (9.50, 12.0)		<.001b
8.0 (6.0, 10.0)			12.0 (10.25, 13.0)	<.001b

Abbreviations: IQR, interquartile range; PHC, primary health center.

^aP value calculated using the Kruskal-Wallis test for unmatched data.

^bP value was calculated using the Mann-Whitney test for unpaired data.

From the Hospital perspective significant improvements were seen when comparing pre-interventions survey scores with 12-month survey scores for the following measures: PHC written communication (P = .004), Hospital written communication (P < .001), PHC verbal communication (P = .01), Hospital verbal communication (P = .01), Ease of using the referral/counter-referral system (P = .002), Patient follow-up (P = .01), and Trust and relationship with the PHCs (P < .001; Table 5). No significant difference was seen for positive effect of the system on patient care (P = .21; Table 5).

Table 5.

Hospital Survey Data Over Time Stratified by Construct.

Pre-Intervention, Median (IQR)	3-Month, Median (IQR)	6-Month, Median (IQR)	12-Month, Median (IQR)	P
Written Communication—PHC Responsibility
N = 19	N = 15	N = 10	N = 17
32.0 (27.0, 35.0)	36.0 (32.0, 40.0)	41.0 (27.0, 41.25)	40.0 (34.50, 42.0)	.02a
32.0 (27.0, 35.0)	36.0 (32.0, 40.0)			.10b
32.0 (27.0, 35.0)		41.0 (27.0, 41.25)		.07b
32.0 (27.0, 35.0)			40.0 (34.50, 42.0)	.004b
Written Communication—Hospital Responsibility
N = 19	N = 14	N = 10	N = 17
18.0 (14.0, 23.0)	26.0 (22.75, 29.0)	27.50 (24.0, 29.0)	28.0 (24.50, 29.50)	<.001a
18.0 (14.0, 23.0)	26.0 (22.75, 29.0)			.001b
18.0 (14.0, 23.0)		27.50 (24.0, 29.0)		<.001b
18.0 (14.0, 23.0)			28.0 (24.50, 29.50)	<.001b
Verbal Communication—PHC Responsibility
N = 21	N = 16	N = 10	N = 15
4.0 (2.0, 5.50)	5.0 (4.0, 6.0)	5.0 (4.75, 6.25)	6.0 (4.0, 7.0)	.03a
4.0 (2.0, 5.50)	5.0 (4.0, 6.0)			.10b
4.0 (2.0, 5.50)		5.0 (4.75, 6.25)		.03b
4.0 (2.0, 5.50)			6.0 (4.0, 7.0)	.01b
Verbal Communication—Hospital Responsibility
N = 19	N = 16	N = 10	N = 17
1.0 (1.0, 2.0)	2.0 (1.0, 2.0)	1.50 (1.0, 3.0)	2.0 (1.50, 3.0)	.08a
1.0 (1.0, 2.0)	2.0 (1.0, 2.0)			.06b
1.0 (1.0, 2.0)		1.50 (1.0, 3.0)		.32
1.0 (1.0, 2.0)			2.0 (1.50, 3.0)	.01
Ease of Using the Referral/Counter-Referral System
N = 21	N = 15	N = 8	N = 16
32.0 (28.0, 33.50)	33.0 (31.0, 38.0)	36.0 (28.75, 39.75)	38.0 (35.0, 40.0)	.01a
32.0 (28.0, 33.50)	33.0 (31.0, 38.0)			.11b
32.0 (28.0, 33.50)		36.0 (28.75, 39.75)		.12b
32.0 (28.0, 33.50)			38.0 (35.0, 40.0)	.002b
Patient Follow-up at the PHC
N = 20	N = 16	N = 9	N = 17
4.0 (2.0, 4.0)	5.50 (4.0, 7.0)	5.0 (4.0, 8.0)	6.0 (3.50, 8.0)	.02a
4.0 (2.0, 4.0)	5.50 (4.0, 7.0)			.02b
4.0 (2.0, 4.0)		5.0 (4.0, 8.0)		.03b
4.0 (2.0, 4.0)			6.0 (3.50, 8.0)	.01b
Positive Effect of Referral System on Patient Care
N = 19	N = 15	N = 10	N = 16
19.0 (9.0, 25.0)	24.0 (15.0, 26.0)	25.50 (16.0, 29.0)	23.0 (10.25, 29.50)	.34a
19.0 (9.0, 25.0)	24.0 (15.0, 26.0)			.22b
19.0 (9.0, 25.0)		25.50 (16.0, 29.0)		.11b
19.0 (9.0, 25.0)			23.0 (10.25, 29.50)	.21b
Trust and Relationship Between PHC and Hospital
N = 18	N = 16	N = 8	N = 17
6.0 (4.0, 9.0)	7.50 (6.25, 10.0)	9.50 (7.0, 11.50)	10.0 (8.50, 12.0)	.001a
6.0 (4.0, 9.0)	7.50 (6.25, 10.0)			.06b
6.0 (4.0, 9.0)		9.50 (7.0, 11.50)		.02b
6.0 (4.0, 9.0)			10.0 (8.50, 12.0)	<.001b

Abbreviations: IQR, interquartile range; PHC, primary health center.

^aP value calculated using the Kruskal-Wallis test for unmatched data.

^bP value was calculated using the Mann-Whitney test for unpaired data.

We compared all 4 time intervals across all 4 health centers as well as both medical professional degrees. There was no significant difference between the health centers in total survey scores at any time interval, pre-intervention (P = .4), 3 months (P = .45), 6 months (P = .42), or 12 months (P = .83). There was no significant difference between MD and RN survey scores over time except at the 12-month interval, where physicians scored the referral/counter-referral system significantly higher than RNs (P = .01).

Chart Review

There were 435 referrals, 98 (22.5%) pre-intervention and 337 (77.5%) post-intervention (Table 6). There was a significant difference between patient age and health center (P = .001; Table 6) but not gender (P = .85). There was no statistical difference in the number of appropriate referrals (emergent and priority) versus inappropriate referrals (nonurgent) between pre- and post-intervention periods (Table 6). The number of patients that were appropriately referred that required medical intervention increased from 30% pre-intervention to 40% post-intervention, approaching significance (P = .08). Of those that were appropriately referred and required medical intervention, the number of patients that received an intervention prior to transport increased from 55% pre-intervention to 73% post-intervention, also approaching significance (P = .06).

Table 6.

Referral Status of Patients Pre- and Post-Intervention From Chart Review.

Center	Total # Referrals		Median Age in Years (IQR)			Referral Status
						Appropriate Referrals				Inappropriate Referrals		P a
						Emergent (% of Total)		Priority (% of Total)		Nonurgent (% of Total)
	Pre	Post	Pre	Post	P =	Pre	Post	Pre	Post	Pre	Post	P =
1	12	24	1.0 (0.6, 6.5)	1.0 (0.5, 3.2)	.70	0 (0%)	1 (4%)	7 (58%)	14 (58%)	5 (42%)	9 (38%)	.81
2	25	55	1.1 (0.2, 5.5)	1.0 (0.3, 5.0)	.87	1 (4%)	2 (4%)	15 (60%)	34 (62%)	9 (36%)	19 (35%)	.9
3	24	78	1.0 (0.9, 7.5)	1.0 (0.4, 3.0)	.18	1 (4%)	2 (3%)	14 (58%)	45 (58%)	9 (37.5%)	31 (40%)	.84
4	37	180	3.5 (0.1, 8.8)	0.5 (0.1, 1.7)	.004	5 (14%)	5 (3%)	22 (60%)	139 (77%)	10 (27%)	36 (20%)	.34
Total	98	337	2.0 (0.3, 7.2)	0.92 (0.2, 2.2)	.001	7 (7%)	10 (3%)	58 (59%)	232 (69%)	33 (34%)	95 (28%)	.29

Abbreviations: IQR, interquartile range; Pre, before the intervention; Post, after the intervention.

^aPearson χ² comparing Appropriate and Inappropriate referrals in the pre/post time period.

Discussion

An effective referral/counter-referral system is essential to resource-limited health care systems to maximize efficiency, decrease unnecessary resource utilization, and provide safe and timely patient care within a tiered system. The challenges of successfully developing such a system are multifactorial since limited resources, multiple layers of communication, education, and relationships among providers and patients are all involved. Many programs in a tiered system depend on a functioning referral/counter-referral system.

The successful standardization of the referral/counter-referral system was due to several factors. First, the project was identified and proposed by the local care providers who determined a need for improved care for referred patients. This buy-in allowed for stakeholders at all levels to communicate their vision for an ideal system and identify barriers, and then participate in the design and evaluation of the program. Building capacity through education and participation at all levels, including working as a team, produced empowered local providers and a network that can be used for future projects within a tiered system. Additionally, rapid incorporation of changes to the system in order to address identified barriers throughout the project built a strong base for continuous communication and action. This supports literature suggesting the utility of quality improvement methodology in referral system design. There is often difficulty in assessing the outcomes for complex health systems–based processes in resource-limited settings. The use of mixed-methods evaluation allowed us to successfully capture meaningful change for the referral/counter-referral system through triangulation of data, including focus groups, surveys, and chart review.

Limitations

There were several limitations to our study. Over time, the number of study participants decreased, which may have led to selection bias. In the survey results, there was no significant improvement in positive effect on patient care. Review of the survey with stakeholders after completion of data collection suggests that there may have been an issue with the translation of the word “effect,” where it inherently holds a negative connotation in Spanish, which was not recognized in initial review, and could be a reason that this construct is an outlier. The chart review did not show a statistical difference in appropriate versus inappropriate referrals when comparing pre- and post-intervention time periods. Despite employing labor-intensive methods to follow the referral and counter-referrals as well as admissions and discharge summaries through the patient cycle, we were unable to detect a significant difference at the .05 α level due to a lack of statistical power. A larger number of patients would have to be tracked in order to definitively show a quantitative improvement in the referral system that would support the qualitative improvements we were able to successfully glean from the focus groups and survey data. We used a mixed-methods design specifically because we anticipated this possibility. Complex paper record keeping systems made it nearly impossible to track patient follow-up so we are unable to comment on long-term patient outcomes. Prior to this study, it was embedded in the PHC culture to refer every pediatric patient to the hospital due to a lack of comfort in evaluating children, which is another reason there may be no significant change in referral patterns of emergent and priority patients. We anticipate it may require further time and training to overcome this practice. Another design limitation is that patients were not included in the focus groups or surveys due to time constraints, but would undoubtedly add perspective and depth in evaluation of the referral/counter-referral system.

Future Directions

The next steps in achieving positive change would include establishing a continuous supply chain for provision of adequate resources including transportation, emergency medications, regular education for both staff and patients, and working phone lines to improve verbal communications. Additionally, the development of data and record management systems would undoubtedly enhance the ability to track outcomes, identify barriers, give feedback, and take action based on real-time data.

Conclusion

Our program design, which involved assessing, standardizing, and rapid cycle testing the referral/counter-referral system, successfully improved the communication, education, relationships, and trust needed to build a sustainable process based on multiple methods of evaluation. We anticipate that this design will serve as a model for extension of referral/counter-referral process improvement throughout Guatemala, and it can be modified for use in other countries.

Author Contributions

RK: Contributed to conception and design; contributed to acquisition, analysis, and interpretation; drafted manuscript; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.

LA: Contributed to design; contributed to acquisition and analysis; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.

MAS: Contributed to conception and design; contributed to acquisition and analysis; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.

ATC: Contributed to design; contributed to analysis and interpretation; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.

EMS: Contributed to design; contributed to analysis and interpretation; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.

MAS: Contributed to conception and design; contributed to acquisition and analysis; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.

EAC: Contributed to analysis and interpretation; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.

HLC: Contributed to conception and design; contributed to acquisition, analysis, and interpretation; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.