Skip to main content
Public Health Action logoLink to Public Health Action
. 2012 Jun 5;2(2):32–37. doi: 10.5588/pha.11.0028

Simplified methods of determining treatment retention in Malawi: ART cohort reports vs. pharmacy stock cards

A K Chan 1,2,3,4,, E Singogo 1,5, R Changamire 1, Y E C Ratsma 1, J-M Tassie 6, A D Harries 7,8
PMCID: PMC4536654  PMID: 26392945

Abstract

Setting:

Rapid scale-up of antiretroviral therapy (ART) has challenged the health system in Malawi to monitor large numbers of patients effectively.

Objective:

To compare two methods of determining retention on treatment: quarterly ART clinic data aggregation vs. pharmacy stock cards.

Design:

Between October 2010 and March 2011, data on ART outcomes were extracted from monitoring tools at five facilities. Pharmacy data on ART consumption were extracted. Workload for each method was observed and timed. We used intraclass correlation and Bland-Altman plots to compare the agreeability of both methods to determine treatment retention.

Results:

There is wide variability between ART clinic cohort data and pharmacy data to determine treatment retention due to divergence in data at sites with large numbers of patients. However, there is a non-significant trend towards agreeability between the two methods (intraclass correlation coefficient > 0.9; P > 0.05). Pharmacy stock card monitoring is more time-efficient than quarterly ART data aggregation (81 min vs. 573 min).

Conclusion:

In low-resource settings, pharmacy records could be used to improve drug forecasting and estimate ART retention in a more time-efficient manner than quarterly data aggregation; however, a necessary precondition would be capacity building around pharmacy data management, particularly for large-sized cohorts.

Keywords: ART retention, drug forecasting, monitoring and evaluation of HIV programme, Malawi


The scale-up of antiretroviral therapy (ART) in low- and middle-income countries is one of the most remarkable public health achievements of the last decade. Routine monitoring of patients who have started on ART and those who are retained on ART is crucial for measuring programme success.1 Consistent access to data on the number of new patients starting ART and the number retained on treatment over a specific period of time is necessary for forecasting for commodities,2 provides a yardstick by which to assess progress, and helps to determine whether funds are used effectively.

In Malawi, like other countries in southern Africa, rapid scale-up to over 350 000 patients ever enrolled has challenged the capacity of the public health infrastructure to effectively monitor and retain large numbers of patients.3,4 Currently, the Department of HIV/AIDS (DHA) at the Malawi Ministry of Health (MOH) compiles monitoring and evaluation data on the ART programme through quarterly supervisions, during which 10 teams of ART providers (31 individuals from the MOH and from non-governmental organisation [NGO] partners) are deployed to the field to collect reports aggregated from a standardised paper-based monitoring system of registers and master cards from all 433 ART facilities in the country.5 Routine reporting of accurate numbers of patients retained on ART and the numbers of patients on each ART regimen is complicated, and for medium sized cohorts (2000–5000 patients) quarterly aggregation of data is labour intensive. For the largest sized cohorts (>10 000 patients), paper records are virtually impossible to aggregate in a timely manner, and many have adopted electronic data management systems out of necessity.

Other ways of estimating the number of patients retained on therapy are needed. Development of electronic data system capacity may be a solution, but this is fraught with challenges in low-technology settings. In addition to the hardware costs and the need for a sustained electricity supply, the electronic system needs well-trained information technology personnel to routinely monitor performance and resolve problems. Another possible option is using pharmacy records to streamline the monitoring process.6,7 Pharmacy stock card information on drug dispensation, if well kept, could be used as an indicator of treatment retention over a defined time period.

The objectives of this pilot study were 1) to determine whether pharmacy records can provide comparable estimates of patients retained on ART, including numbers retained on different regimens, at different levels of the health system, and 2) to examine the workload and time taken to determine retention on therapy comparing use of pharmacy records with clinic cohort analysis. Findings from this pilot study can inform the feasibility of using pharmacy stock cards to simplify the process of monitoring the number of patients alive and on ART in low-resource settings.

METHODS

Study setting

This pilot study used a purposeful sampling technique to select five ART facilities 1) that represent various levels of medium- to large-sized ART facilities in the South-East Zone, 2) that followed >1000 patients on ART (i.e., not a small-sized cohort, where aggregating cohort data is not difficult and does not require a lot of time), and 3) that were selected based on the explicit agreement of the DHA supervision teams and the District Health Management Teams in the South-East Zone (population 3.2 million).

Data were collected from one peri-urban health centre, three rural secondary referral hospitals and one urban tertiary referral centre. The central hospital clinic is an MOH clinic that uses an electronic patient data management system supported by Dignitas International, a Canadian medical NGO. During the period of data collection, the central hospital transitioned from data entry of paper data into a database to a direct entry electronic patient management system (Baobab Malawi). All other sites used paper-based management, and service delivery was provided solely by the Malawi MOH. All sites provided ART services as per standardised national guidelines.8

Data collection

In all ART facilities, quarterly cohort reports were collected from clinic-based paper or electronic records for both Quarter 4 (1 October to 31 December 2010) and Quarter 1 (1 January to 31 March 2011). Data on ‘the number of new patients registered, including new ART initiations and transfer-in patients’, and ‘the number of patients retained on therapy at the end of a quarter’ were extracted and calculated from the ART register. The number retained on therapy was also categorised by type of regimen (first-line, alternative first-line and second-line). The number of full-time equivalent (FTE) staff needed to conduct the analysis was determined by timing all staff responsible for compiling cohort reports during the quarterly supervision.

The number of ART continuation bottles in the pharmacy at all sites was documented at the start and end of a specific quarter from pharmacy stock cards for the following drugs: stavudine-lamivudine-nevirapine (d4T-3TC-NVP), zidovudine (AZT)-3TC-NVP, AZT-3TC, D4T-3TC, efavirenz (EFV), tenofovir, 3TC, NVP, didanosine, abacavir and ritonavir-lopinavir. Data for reallocation or expiry were also recorded from stocks and delivery notes and were used to calculate the total bottles dispensed to patients in the evaluation period. Although dispensing intervals varied, the average rate of dispensing per patient during this period was assumed to be stable over time. In the regimen-specific analysis for number of people retained on various alternative first-line regimens, the amount of first-line AZT-3TC was used as a proxy for patients on AZT-3TC + EFV as alternative first-line treatment (for prevention of mother-to-child transmission of HIV, first- and second-line stocks of AZT-3TC are recorded on separate stock cards), and the amount of d4T-3TC was used as a proxy for patients on d4T-3TC + EFV. For alternative first-line retention calculations, an adjustment for number of new patients (i.e., [total number of ART tins used over 3 months − (1.5 × number of new patients enrolled during the quarter)/3]) was not used, as most patients remain on first-line treatment during the first 3 months. This adjustment was made to calculate the retention of patients on first-line treatment, i.e., d4T-3TC-NVP.

Data analysis

Data were excluded in measurement accuracy from Mulanje DH Quarter 1, 2011, due to missing pharmacy stock cards. The timing of workload of the supervision team for the Mulanje DH data collection was not observed due to scheduling issues between the study team and the supervision team. The ART cohort data collection for Zomba Central Hospital was terminated early due to discrepancies in data in Quarter 4, 2010, as a result of data system migration.

Measurement accuracy, comparing both methods of determining retention on treatment, was assessed by calculating the intraclass correlation coefficient (ICC) and visually exploring Bland-Altman plots. All statistical analyses were conducted using Intercooled Stata version 9.2 (Stata Corp, LP, College Station, TX, USA); the significance level was fixed at P < 0.05.

Ethics approval

This pilot study was approved by the Ethics Advisory Group of the International Union Against Tuberculosis and Lung Disease, Paris, France, and sent to the Malawi National Health Sciences Research Council for oversight.

RESULTS

Comparison of quarterly ART clinic cohort report and pharmacy stock cards

Table 1 demonstrates the number of patients retained on ART based on data collected from five ART facilities over two quarters. The range in ‘precision’ (determined by the number of people retained on treatment using pharmacy data/clinic cohort data) varied from 78% to 150% in Quarter 4, 2010, and from 28% to 103% in Quarter 1, 2011. The number of ART bottles dispensed was stratified according to ART regimen, with results in Table 2 again demonstrating wide variability in ‘precision’. For second-line regimens, small numbers prevented any meaningful comparison.

TABLE 1.

Comparison of total number of patients retained on ART between clinic cohort report and those estimated from pharmacy stock records

ART facility Quarter 4 (October–December 2010)
Quarter 1 (January–March 2011)
Clinic cohort report n Pharmacy stock cards n Precision (pharmacy/clinic) % Clinic cohort report n Pharmacy stock cards n Precision (pharmacy/clinic) %
Health centres
Matawale HC 1054 836 79 1151 1187 103
 District hospitals
 Holy Family MH 2660 4 015 150 4233 1183 28
 Mulanje DH 4392 4 003 91 2508 NA NA
 St. Luke’s MH 1858 1 761 78 1971 1805 92
  Subtotal (district hospitals) 8910 9 779 110 8712 2988 48*
Central hospital
 Zomba CH 8685 10 874 125 9144 7052 77
*

Calculated without the Mulanje DH data, as pharmacy data were NA.

ART = antiretroviral treatment; HC = health centre; MH =mission hospital; DH =district hospital; NA = data not available, as pharmacy stock cards missing; CH =central hospital.

TABLE 2.

Comparison of number of patients alive and on ART between clinic cohort report and those estimated from pharmacy stock records by regimen type

ART facility First-line regimen
Alternative first-line regimen
Second-line regimen
ART cohort n Pharmacy stock cards n Precision % ART cohort n Pharmacy stock cards n Precision % ART cohort n Pharmacy stock cards n Precision %
Matawale HC, Q4 2010 886 691 78 168 145 87 0 0
Matawale HC, Q1 2011 943 939 100 208 247 119 0 0
St. Luke’s MH, Q4 2010 1706 1499 88 147 262 178 4 0
St. Luke’s MH, Q1 2011 1780 1635 92 188 169 90 3 0
Holy Family MH, Q4 2010 2622 3901 149 38 113 298 0 0
Holy Family MH, Q1 2011 2434 1096 45 74 86 117 0 0
Mulanje DH, Q4 2010 4298 3889 90 94 113 121 0 0
Mulanje DH, Q1 2011 4115 NA NA 118 NA NA 0 0
Zomba CH, Q4 2010* 7233 8824 122 1414 2050 145 38 (7 children) 51 134
Zomba CH, Q1 2011 7572 5615 74 1528 1436 94 44 (13 children) NA NA

ART = antiretroviral treatment; HC =health centre; Q =quarter; MH = hospital; DH = district hospital; NA = data not available due to missing stock cards; CH = central hospital.

Table 3 and the Figure demonstrate statistical measures of agreement between the two methods of determining retention on treatment, as there is no gold standard for the actual number of patients retained. Results are disaggregated into specific regimen type. For Table 3, an ICC > 0.7 denotes good agreeability between methods. The overall ICC was 0.93 (95% confidence interval [CI] 0.74–0.98) and was not significant. For specific regimen types, all except AZT-3TC + EFV had ICCs > 0.7; however, none were statistically significant. Figure A is a collection of Bland-Altman plots that graphically represent the agreement between the two different methods of measurement. Figure B again disaggregates the data by specific regimen type. In an ideal Bland-Altman plot there is little variation around the zero-difference line ( y-axis), indicating that the measures produced very similar results from one measurement to the next. Measures with greater agreement display points closer to zero across the entire range of mean scores. Bias is demonstrated when the points are grouped above or below the zero-difference line ( y-axis). Bias indicates that one administration is consistently higher or lower than the other. There appears to be greater agreement (clustering around the y-axis) when the numbers retained on treatment are lower, with significant divergence (outliers) as cohort sites get larger. The apparent association of agreement with size of cohort holds true even with results disaggregated by specific type of regimen.

TABLE 3.

Comparison of number of patients alive and on ART between clinic cohort report and those estimated from pharmacy stock records by regimen type using ICC*

Regimen type ICC 95%CI P value
First-line
 d4T-3TC-NVP 0.92 0.71–0.98 0.95
Alternative first-lines
 AZT-3TC-NVP 0.91 0.69–0.98 0.96
 AZT-3TC-EFV 0.59 −0.03–0.89 0.75
 D4T-3TC-EFV 0.89 0.64–0.97 0.95
  Total alternative first-line regimens 0.94 0.79–0.99 0.97
   Total retained on treatment (first-line, alternative first-line and second-line regimens) 0.93 0.74–0.98 0.96
*

ICC > 0.7 denotes good agreeability between methods; however, statistically none of the ICCs were shown to be significant due to wide CIs.

ICC = intraclass correlation coefficient; CI =confidence interval; d4T-3TC-NVP = stavudine-lamivudine-nevirapine; AZT = zidovudine; EFV = efavirenz.

FIGURE.

FIGURE

FIGURE

FIGURE

FIGURE

FIGURE

Cohort vs. pharmacy data: number retained on treatment. Bland-Altman plots displaying the agreement between measuring number retained on treatment comparing ART cohort data and pharmacy stock card data; the x-axis is ‘the number of patients as a mean of both methods’ and the y-axis is ‘the difference in number of patients between methods’. Correlation R = 0.21 (P = 0.58). Slope = −0.067 (P = 0.58). Intercept = 370.4 (P = 0.59). A. There appears to be greater agreement when the numbers retained on treatment are lower, with higher divergence at the sites with larger cohorts. Correlation R = 0.21 (P = 0.58). Slope = −0.067 (P = 0.58). Intercept = 370.4 (P = 0.59). B. The association of agreement with size of cohort holds true even with results disaggregated by specific type of regimen. a. Cohort data vs. pharmacy data: number retained on d4T-3TC-NVP. Correlation R = 0.16 (P = 0.67), slope = −0.069 (P = 0.68), intercept = 375 (P = 0.58). b. Cohort data vs. pharmacy data: number retained on AZT-3TC-NVP. Correlation R = 0.84 (P < 0.05), slope = −0.347 (P < 0.05), intercept = 8 (P = 0.83). c. Cohort data vs. pharmacy data: number retained on AZT-3TC-EFV. Correlation R = 0.55 (P = 0.12), Slope = 0.547 (P = 0.12), intercept = 2 (P = 0.86). d. Cohort data vs. pharmacy data: number retained on d4T-3TC-EFV. Correlation R = 0.37 (P = 0.29), slope = −0.162 (P = 0.29), intercept = −12 (P = 0.65).

Workload and time taken in compiling cohort report

Table 4 shows the time and workload required to compile clinic cohort reports and pharmacy stock data. The range of patients followed in the clinics varied from 1323 to 15 269. It took on average 573 min (range 40–1473 min) to compile the clinic cohort report. In contrast, estimation of cohort data from pharmacy stock cards took only 81 min on average (range 33–185 min). More staff were also required to complete the ART cohort report (range 2–8 persons) compared with the pharmacy stock cards (1–3 persons).

TABLE 4.

Comparison of time taken to compile cohort reports between ART clinic register and pharmacy data stock cards

ART facility Type of facility ART patients registered n ART cohort report
Pharmacy stock cards
Staff n Min Staff n Min
Matawale HC, Q4 2010 MOH Health Center (Level 1) 1 323 3 534 3 57
Matawale HC, Q1 2011 Health Center (Level 1) 1 325 2 290 1 37
St. Luke’s MH, Q4 2010 CHAM Mission Hospital (Level 2) 2 888 4 532 3 66
St. Luke’s MH, Q1 2011 CHAM Mission Hospital (Level 2) 3 067 4 563 2 33
Holy Family MH, Q4 2010 CHAM Mission Hospital (Level 2); acts as District Hospital 4 384 4 1473 1 80
Holy Family MH, Q1 2011 CHAM Mission Hospital (Level 2); acts as District Hospital 4 561 2 444 2 141
Mulanje DH, Q4 2010 MOH District Hospital (Level 2) 7 912 8 1176 2 99
Mulanje DH, Q1 2011 MOH District Hospital (Level 2) 8 328 NA NA 1 59
Zomba CH, Q4 2010 MOH Central Hospital (Level 3); NGO supported ART Clinic 15 269 2 106* 3 185
Zomba CH, Q1 2011 MOH Central Hospital (Level 3); NGO supported ART Clinic 14 840 2 40 1 55
Average 6 392 3 573 2 81
*

The report was terminated early by the supervision team because of inconsistencies between the paper and electronic database, due to merging of the old electronic database (Dignitas International) with the new electronic database (MOH Baobab).

ART = antiretroviral treatment; HC =health centre; MOH = Ministry of Health; Q =quarter; MH = mission hospital; CHAM =Christian Health Association of Malawi; DH = district hospital; NA = data not available, supervision team was not timed; CH = central hospital; NGO = non-governmental organisation.

There was a significant reduction in time to complete cohort reporting when electronic data reporting was used, and that was only at the largest site, Zomba Central Hospital. Using an NGO-supported paper-to-electronic data transfer method and an Access database in Quarter 4, 2010, the time to generate a report was 106 min. Using the Baobab Electronic Data System in Quarter 1, 2011, which has automated macros for quarterly supervision reporting and is direct entry by provider, the time to generate a report was 40 min. These do not account for the time and human resources required to clean the electronic data, which is substantial using either method of electronic data management.

DISCUSSION

This is one of the first studies to explore the feasibility of using pharmacy stock cards to simplify the process of monitoring the number of patients alive and on ART in low-resource settings. First, the study demonstrates the limitations of using operational data in research with respect to the reliability and accuracy of the data. For example, data had to be excluded from one site in one quarter due to missing stock cards. In addition, there were concerns about the reliability of the cohort data generated at the largest site due to problems during migration of data from an old electronic data system to a new system. One would have expected the data to be more accurate at the smaller sites; however, there was clearly wide variation in precision between sites and also within sites at two different time points, independent of the size of the cohort. The inconsistencies in paper data reflect findings from a recent paper published from Malawi, which looked at the extent of inaccuracies between electronic records and paper registers in five health facilities and found up to 24% of total case registrations missing in one hospital and major discrepancies in the number of deaths recorded at two sites.9

This study strongly suggests that using pharmacy stock data to determine the number of patients retained on treatment may be cost-effective due to the reduction in the time health care workers spend estimating the numbers of patients on ART. Capacity building in improving the accuracy of pharmacy stock data is warranted, as it may be a far more cost-effective way to manage commodities, monitor the number of patients on treatment and flag defaulters than resource-intensive quarterly clinic cohort monitoring. Data from a study looking at the use of a computerised pharmacy tracking system (iDART) in South Africa to identify loss to follow-up at a community-based ART clinic suggest that a well-managed pharmacy system can also be used to detect potential defaulters and flag them for recall.10

Although the pharmacy stock cards appear, overall, to be a viable method of assessing ART patient retention, the data demonstrate that significant differences between the pharmacy and ART registers remain to be addressed. Although the ICC for almost all regimens was upwards of 0.9, which suggests good agreeability between the two methods, none of the results were significant (P < 0.05), due to a wide CI. The Bland-Altman plots suggest that this wide variability is primarily due to divergence in data at the sites with large cohorts. Our data do not allow us to determine whether the inaccuracies are due to errors in the ART cohort report or the pharmacy stock records as the cohort size gets larger. However, the Malawi DHA uses the cohort analysis performed by the quarterly supervision teams as the ‘gold standard’ in this setting, and most sites were awarded ‘Certificates of Excellence’ from the Malawi MOH based on their assessment of the quality of the cohort data. This implies an urgent need for pharmacy management strengthening.

Pharmacy operations for ART drug dispensation vary according to facility type, and this may contribute to discrepancies. In district hospitals, all ART drugs are stored in the main district pharmacy, and drug disbursement and transfers are documented on stock cards. When the ART clinics request drugs, the drugs are transferred to a separate location within the main district pharmacy, e.g., a locked cabinet, to signify their assignment for a specific location. Drugs are then collected from this location and brought to the ART clinic for dispensing in clinic rooms. Each dispensing room has an ART register that is updated every time the nurse dispenses ART drugs to a patient. The data for this model were collected from the main pharmacy, as the lockable cabinets did not contain stock cards. Lack of documentation of the drugs collected from cabinets made it hard to confirm whether all drugs dispensed from the main pharmacy were actually given to patients in the ART facility, and thus may have led to an overestimate of ART retention. It is of note that since the study results were collected, the Pharmacy Coordinator for Dignitas International has had multiple mentoring sessions with pharmacy staff at the rural hospital sites on both record keeping and other drug management issues. In addition, a nurse has been allocated at a site identified as having stock management issues to assist the pharmacy in updating their stock cards on ART and other HIV-related drugs.

Finally, although the implementation and running costs of electronic patient management systems are high, the promotion of electronic systems in high-burden ART facilities is advisable purely for data management efficiency. Our data confirm that it is also time-saving for generating data cohort reports. The findings from the accuracy of pharmacy cohort data reporting suggest that a wise investment in capacity building in pharmacy data management should also include concurrent development of electronic pharmacy data monitoring for sites with high numbers of patients, given the divergence of paper-based pharmacy records at those sites.

Acknowledgments

The authors would like to thank the Malawi Ministry of Health Department of HIV/AIDS Quarterly Supervision Teams and the staff at the various study sites; A Datta and A E Olayinka for assistance with statistical analysis; A L C Martiniuk and M van Lettow for reviewing the paper and statistics; F Cataldo, Director of Research at Dignitas International; and Y Souteyrand at the World Health Organization. This study was funded by the World Health Organization.

References

  • 1.World Health Organization, Joint United Nations Program on HIV/AIDS, UNICEF. Towards universal access: scaling up priority HIV/AIDS interventions in the health sector. Progress report 2009. Geneva, Switzerland: World Health Organization; 2009. http://www.who.int/hiv/pub/tuapr_2009_en.pdf Accessed January 2010. [Google Scholar]
  • 2.Harries A D, Schouten E J, Makombe S D, et al. Ensuring uninterrupted supplies of antiretroviral drugs in resource-poor settings: an example from Malawi. Bull World Health Organ. 2007;85:152–155. doi: 10.2471/BLT.06.032060. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Department of HIV/AIDS, Ministry of Health, Malawi. Quarterly report on the Antiretroviral Treatment Programme in Malawi with results up to 31st March 2011. Lilongwe, Malawi: Ministry of Health; 2011. [Google Scholar]
  • 4.Makombe S D, Hochgesang M, Jahn A, et al. Assessing the quality of data aggregated by antiretroviral treatment clinics in Malawi. Bull World Health Organ. 2008;86:310–314. doi: 10.2471/BLT.07.044685. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Libamba E, Makombe S, Mhango E, et al. Supervision, monitoring and evaluation of nationwide scale-up of antiretroviral therapy in Malawi. Bull World Health Organ. 2006;84:320–326. doi: 10.2471/blt.05.023952. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Nachega J B, Hislop M, Dowdy D W, et al. Adherence to highly active antiretroviral therapy assessed by pharmacy claims predicts survival in HIV-infected South African adults. J Acquir Immune Defic Syndr. 2006;43:78–84. doi: 10.1097/01.qai.0000225015.43266.46. [DOI] [PubMed] [Google Scholar]
  • 7.Wood R, Kaplan R, Bekker L G, Brown S, Rivett U. The utility of pharmacy dispensing data for ART programme evaluation and early identification of patients lost to follow-up. Afr J HIV Med. 2008;9:44–48. [Google Scholar]
  • 8.Ministry of Health, Malawi. Treatment of AIDS: guidelines for the use of antiretroviral therapy in Malawi. 3rd ed. Lilongwe, Malawi: Ministry of Health; 2008. [Google Scholar]
  • 9.Gadabu O J, Munthali C V, Zachariah R, et al. Is transcription of data on antiretroviral treatment from electronic to paper-based registers reliable in Malawi? Public Health Action. 2011;1:10–12. doi: 10.5588/pha.11.0002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Nglazi M D, Kaplan R, Wood R, Bekker L G, Lawn S D. Identification of losses to follow-up in a community-based ART clinic in South Africa using a computerized pharmacy tracking system. BMC Infect Dis. 2010;10:329. doi: 10.1186/1471-2334-10-329. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Public Health Action are provided here courtesy of The International Union Against Tuberculosis and Lung Disease

RESOURCES