Skip to main content
HHS Author Manuscripts logoLink to HHS Author Manuscripts
. Author manuscript; available in PMC: 2022 Sep 12.
Published in final edited form as: Contemp Clin Trials. 2021 Mar 29;105:106377. doi: 10.1016/j.cct.2021.106377

Optimizing Drug Inventory Management with a Web-based Information System: The TBTC Study 31 / ACTG A5349 Experience

Nigel A Scott a,*, Kara K Lee a, Claire Sadowski a,b, Ekaterina V Kurbatova a, Stefan V Goldberg a,1, Pheona Nsubuga c, Rene Kitshoff d, Colleen Whitelaw e, Hanh Nguyen Thuy f, Kumar Batra g, Cynthia Allen-Blige g, Howard Davis g, Jay Kim g, Mimi Phan h, Pamela Fedrick g, Kuo Wei Chiu h, Charles M Heilig a, Erin Sizemore a; The AIDS Clinical Trials Group and The Tuberculosis Trials Consortium
PMCID: PMC9466280  NIHMSID: NIHMS1692771  PMID: 33794353

Abstract

Introduction:

Efficient management of study drug inventory shipments is critical to keep research sites enrolling into multisite clinical treatment trials. A standard manual drug-management process used by the Tuberculosis Trials Consortium (TBTC), did not accommodate import permit approval timelines, shipment transit-times and time-zone differences. We compared a new web-based solution with the manual process, during an international 34-site clinical trial conducted by the TBTC and the AIDS Clinical Trials Group (ACTG); TBTC Study 31/ACTG A5349.

Material and Methods:

We developed and implemented a technological solution by integrating logistical and regulatory requirements for drug importation with statistical simulations that estimated stock-out times in an online Drug Management Module (DMM). We measured the average shipment-related drug stock-outs and time to drug availability, to assess the efficiency of the DMM compared to the manual approach.

Results:

An Interrupted Time-Series (ITS) analysis showed a 15% [p-value=0.03; 95% C.I. (–28.8%, –2.0%)] reduction in average shipment-related study drug stock-out after DMM implementation. The DMM streamlined the restocking process at study sites, reducing median transit-time for sites associated with a depot by 2 days [95% C.I. (–3.0, –1.0)]. Under the DMM, study drugs were available for treatment assignment on the day received, compared to one day after receipt under the manual process.

Discussion:

The DMM provided TBTC’s Data and Coordinating Center and site staff with more efficient procedures to manage and consistently maintain study drug inventory at enrolling sites. This DMM framework can improve efficiency in future multicenter clinical trials.

Keywords: Study drug inventory management, study drug stock-out, Interrupted Time-Series Analysis, statistical simulations, trial pharmaceuticals, web-based system

INTRODUCTION

Drug inventory management for public health programs and clinical trials is an important issue in disease control. Previous studies of pharmaceutical management have focused primarily on drug inventory systems implemented for national programs to manage inventory at local health clinics1,2. One study among HIV clinics showed that stock-out of combination antiretroviral therapy doubled the risk of treatment interruption3, highlighting the importance of preventing drug stock-outs. In 2011, the World Health Organization (WHO) published a drug management manual4 for national disease control programs; it recommended 12 core indicators for use in monitoring and evaluating drug procurement and supply management (PSM) in programs for infectious disease control. Six of these indicators identify early-warning signs of stock-outs and overstocking of medications and were used to assess a drug PSM system in Nigeria in 20125. The assessment demonstrated that the indicators were useful in ensuring no drug stock-outs affected patient management. These reports demonstrate high-level interest in drug management for national programs.

The ability of clinical trial networks to manage drug inventory effectively and to keep trial sites sufficiently stocked with study drugs is critical for timely enrollment and completion of clinical trials. This ability helps to avoid treatment interruptions for enrolled participants and thus reduces failure to complete treatment on time. However, only a few reports, from the United Kingdom6 and North America 7,8 describe investigational drug management in clinical trials recruited at primary, secondary or tertiary care facilities. These reports describe successful web-based systems which reduced loss to follow-up rates, decreased time spent dispensing investigational-agents and improved other daily operations. However, these studies were not in international multisite settings, and so do not address complications based on import permit approvals, transit timelines, and differences across multiple time-zones.

In 2016, the Tuberculosis Trials Consortium (TBTC) collaborated with the AIDS Clinical Trials Group (ACTG) to enroll participants in TBTC Study 31/ACTG A5349 (S31/A5349)9 (ClinicalTrials.gov Identifier: NCT02410772) at 34 study sites across 13 different countries (Brazil, China (Hong Kong), Haiti, India, Kenya, Malawi, Peru, the Republic of South Africa (R.S.A.), Thailand, Uganda, the United States (U.S.), Vietnam, and Zimbabwe). The objective of this phase 3 study was to evaluate efficacy and safety of two treatment-shortening regimens (Supplementary Table S1) for drug-susceptible tuberculosis (TB). TBTC’s Data and Coordinating Center (DCC) at the U.S. Centers for Disease Control and Prevention (CDC) implemented and directed most aspects of S31/A5349, including study drug management. This trial provided a chance to determine how to manage complexities in drug management related to international multisite settings. This manuscript describes the study drug management experience in S31/A5349, including the development of a Drug Management Module (DMM), to streamline shipping and maintain drug supply at sites. Our primary objective was to compare a standard manual approach with the DMM approach, to evaluate which method minimized pauses in enrollment due to shipment-related study drug stock-outs. Secondary objectives were to determine which approach was more efficient throughout the shipment processing timeline in assuring drug availability at sites and reducing person-hours spent in shipment processing at pharmacies.

MATERIALS AND METHODS

Drug Management Module (DMM) Motivation and Design

At the start of enrollment into S31/A5349, DCC personnel managed the supply and distribution of study drug kits using manual procedures (Supplementary Figure S4). These procedures included multiple email correspondences and several versions of different documents to track and verify kit shipments. However, as more sites opened for enrollment, manually managed drug shipments became increasingly burdensome and complicated. The manual process did not account for the varying times required by study sites to obtain import permit approvals, nor for transit-time from S31/A5349 pharmaceutical collaborator, Sanofi’s warehouse to the sites’ pharmacies. In this manual approach, restocking was initiated only when site inventory in any study arm fell below pre-specified thresholds of the number of unallocated kits. Additionally, time-zone differences between the DCC and non-U.S. sites often caused delays in processing requests or documents received by email.

The cyclical procedure for restocking sites offered an opportunity to automate recurring steps to save time and increase quality-control. We developed a technological solution to improve the efficiency of the supply and distribution process as more sites opened, and to minimize the negative effects of study drug management issues on enrollment - thus the Drug Management Module (DMM) was conceived.

The DMM was comprised of three core components:

  1. A shipment-related study drug stock-out algorithm. We defined shipment-related study drug stock-out as a complete consumption of kits in any one arm, due to the failure of a shipment to resupply a site with study drugs on time. The algorithm estimated the time to drug stock-out at each site. These estimates were used to determine when sites should start the restocking process to reduce the possibility of an enrollment pause imposed by study drug depletion;

  2. A tracking list of country-specific regulatory requirements for import permit approvals. This allowed accounting for shipping timelines between central supply at Sanofi’s warehouse and study site pharmacies;

  3. Real-time and automated functionalities, which allowed sites to confirm receipt of drug shipments in real-time and provided automated notifications of when to submit restock requests.

These core components were seamlessly integrated, as the DMM, into TBTC’s previously developed web-based trial management system, TBTC2. TBTC2 supports both electronic data capture and regulatory management for TBTC’s clinical trials. It is accessible to local site personnel, contract research organization associates, and DCC staff.

Shipment-Related Study Drug Stock-out Algorithm

To simulate time to stock-out, average enrollment rates and quantities of kits in an upcoming shipment for each site were used as inputs. The simulations generated the 5th and 95th percentiles of the mean number of participants recruited and the mean time to stock-out based on 200 iterations. This process projected stock-out date based on mean time to stock-out. The Supplementary Appendix includes a detailed description of the shipment-related study drug stock-out algorithm, referring to the Strengthening the Reporting of Empirical Simulation Studies (STRESS) guidelines10.

Country-Specific Regulatory Requirements

Regulatory approval requirements for drug importation imposed critical timing issues during the processes of clearance and delivery to sites. These requirements vary across countries and can change over time. Each country had its own set of required documents, such as certificates of analysis, conformity, or release; Good Manufacturing Practice (GMP) Certificates; commercial invoice; sponsor’s donation letters; and other requirements including the specific lot/batch numbers and pictures of study drugs being shipped. In addition to the various documents, countries have different scheduled lengths of time for obtaining approvals and different durations for the validity of the approvals. The scheduled lengths of time for obtaining approvals may vary for shipments. The number of importations allowed per approval can also vary. These are summarized in Table 1. Before shipping study drugs to sites, the DCC obtained key country-specific information on the importation approval process. Sites completed surveys to provide the most current information at the beginning of the study; these were updated when the DCC became aware of changes. Details of these documents and timelines, coupled with the projected study drug stock-out date, allowed the DCC to estimate when to start the approval process to have study drugs at sites prior to stock-out. The schema in Supplementary Figure S3 outlines the timing process.

Table 1:

Summary of Selected Characteristics of Import Permit Restrictions

Characteristic of Import Permit Requirement (Unit of measurement) Median
(Minimum, Maximum)a
Number of Shipments per permit 1 (1, Unlimited)
Length of Approval Process (days) 30 (7, 120)
Duration of permit validity (days) 365 (30, 365)
Number of Documents Required for overall submission 8 (3, 10)
a

Based on requirements from 12 countries: Brazil, China (Hong Kong), Haiti, India, Kenya, Malawi, Peru, Thailand, Uganda, the United States, Vietnam, and Zimbabwe. Study sites within these countries had the same import permit restrictions. Restrictions from the Republic of South Africa are not included since this country’s drug authority, Medicines Control Council (MCC), now referred to as South African Health Products Regulatory Authority (SAHPRA), required a one-time approval process that was valid for entire study.

Real-time and Automated Functionalities

To capitalize on the cyclical nature of the restocking process, real-time and automated functionalities were incorporated in the DMM. A drug receipt function provided sites the ability to immediately update kit statuses in the system from a shipment delivered on-site. This eliminated update delays caused by time-zone differences between sites and the DCC, and it made kits immediately available for new enrollments. Additionally, automated functions incorporated data from the shipment-related drug stock-out algorithm and details of approval requirements, to send notifications informing site personnel when restock requests should be submitted to avoid inventory stock-outs. Additional functionalities provided data entry screens that allowed sites to submit timely requests for restocking study drugs, as well as functionalities that displayed available inventory on-site. Supplementary Figures S5a-d display the process flow scheme and show how these functions operated. Selected screenshots of the DMM are also displayed in Supplementary Pictures S2 and S3.

Further Applications of Stock-out Algorithm

Application of the stock-out algorithm went beyond solely estimating drug stock-out at sites. Sanofi procured study drugs for all regimens (Supplementary Table S1). At several points throughout the trial, particularly as the trial neared the enrollment target, overall supply needs were projected to allow Sanofi to begin the procurement process for additional drug supplies and to schedule new packaging runs. This process was initiated approximately 6 months in advance of when study drugs would need to be available for shipping to sites. The purpose of these stock projections was to estimate the total amount of each study drug needed to complete enrollment while minimizing wastage.

These projections estimated enrollment from each study site. The average enrollment over the previous 3 months was the main factor in these estimates, however, anticipated slowdowns due to holidays, and wastage caused by accidents or temperature excursions during shipping, and storage, also were considered. The actual projection calculations were conducted by applying the stock-out algorithm to estimate when the current drug inventory at the site would be depleted, and how many participants would be enrolled with this supply. The DCC then considered the number of kit pairs (Supplementary Table S1) for each regimen available at Sanofi’s warehouse and the depots in R.S.A. and U.S. (Supplementary Appendix Section 2.3), the reported storage space at the site pharmacy and the site enrollment rate, to decide how many kit pairs should be resupplied to the sites. This process of applying the stock-out algorithm and resupplying the sites was repeated across all sites until the cumulative enrollment reached 2,500 participants, the target enrollment number for the trial. This provided the estimated month when all kits at Sanofi’s warehouse, and the R.S.A. and U.S. depots would be depleted. The difference between the stock on-hand and the quantity needed to reach the targeted enrollment would be the quantity needed for the new supply.

Statistical Methods

To measure the impact of the DMM implementation on shipment-related study drug stock-outs, an Interrupted Time Series (ITS) analysis was performed using a Segmented Regression model with autoregressive errors11. This pre-post analysis evaluated how effective DMM-processed shipments were compared with manual-processed shipments, in minimizing the average monthly shipment-related study drug stock-outs. The time-series model was tested and adjusted for stationarity (using the Dickey-Fuller test12), autocorrelation (using Partial Autocorrelation Functions) and seasonality. To improve the validity of the ITS model and meet the requirement that the DMM implementation was the only change being implemented, exogenous variables, observed to have changed over time, were identified graphically using cross-correlation plots. These variables were investigated in the segmented regression model, to test for significance. There is limited guidance on estimating the statistical power for ITS analyses. However, it has been determined that statistical power for ITS analyses depends on the number of data points, the autocorrelation, and the expected effect size13. For this analysis we focused on the power to detect a level change and defined the effect size as the expected/measured level change over its standard deviation13,14. We estimated statistical power based on simulation tables described in Zhang et al (2011)13 and report the number of data points before and after the DMM intervention. A comparable control group was not available for this ITS analysis and is discussed as a limitation. The time-series data is presented graphically, along with pre and post trendlines and level changes. The counterfactual to the DMM implementation is included on this graph to help with interpretation of a hypothetical control group.

To further compare the performance of manual-versus DMM-processed shipments, secondary comparisons were conducted using a DMM-restricted time-period in which an equal number of DMM shipments were sequentially selected to match the number of manually processed shipments. This allowed equal chances of observing and evaluating events of interest in the two processes. A DMM-unrestricted analysis using all DMM shipments was conducted to determine whether these additional data would change findings.

One secondary analysis used the number of shipment-related study drug stock-out events during the manual and the DMM-restricted time-periods. Corresponding proportions were calculated. The DMM-unrestricted comparison included all DMM shipment-related stock-out events. 95% Wald Asymptotic Confidence Intervals (C.I.) for the difference between proportions of stock-out events, were provided. A graph comparing the timing of stock-out events by processing type was generated to detect possible patterns in shipment-related stock-out events. Temperature excursion-related events were excluded from these analyses since they were independent of the manual and DMM processes.

Other events of interest used in the secondary restricted and unrestricted analyses include three time-intervals, which were calculated to compare the efficiency of both processing methods:

  1. Before shipment sent: Time from restock request by site to submission of shipment order.

  2. During-shipment: Time from shipment order submission to receipt at site pharmacy. This interval was further stratified by sites inside and outside the R.S.A. and the U.S. (i.e., sites with and without depots).

  3. After shipment arrived: Time from study drugs receipt at site pharmacy to being available for enrollment.

The median time with 25th and 75th quantiles were provided for these time-intervals. 95% C.I. for the differences in median times between processing methods was calculated using the Inversion of Rank test15. In scenarios where the Inversion of Rank test was unable to calculate differences in median times, the Hodges-Lehmann estimator16 for location shift was used to calculate 95% C.I. for the median of differences between processing times.

To determine the impact of the DMM implementation on site pharmacies, pharmacists from specific sites were surveyed to collect qualitative data comparing the manual- and DMM-processed shipments. The selected sites were limited to high enrolling sites, with either a large number of shipments, but small number of kits per shipment, or small number of shipments, but large number of kits per shipment. Pharmacists from these sites were asked to describe how the DMM affected their pharmacy procedures and estimated changes in person-hours during shipment receipt between the two processes. They were also asked to comment on how the DMM could be improved to better meet pharmacy requirements.

An assessment of time-savings at the DCC, between both shipment processes, was conducted. Numbers of steps needed to approve shipments and time to complete were estimated.

RESULTS

Human Resource Commitment

Planning for developing the DMM began in January 2016 with discussions between drug management personnel at the DCC and application development personnel from Northrop Grumman, who maintained TBTC2. These teams met over several months, to better understand the challenges in the manual process, and how the DMM might be designed to address these challenges. Sanofi and site pharmacies were contacted for feedback and input as needed. User Acceptance Testing was conducted with early draft-versions of the DMM, in late 2016. The first production version of the DMM was deployed on April 27th, 2017 (when 34% of the trial’s 2,516 participants had been enrolled).

With this deployment, the DCC saw an estimated person-hour reduction of 2 hours per shipment from manual- to DMM-processed shipments. Supplementary Figure S4 summarizes the manual drug shipment process. Automated, real-time functions in the DMM shortened step 1 and eliminated steps 2 and 3 in this process.

Comparing Efficiency between DMM and Manual Processes

A total of 337 study drug shipments were completed in S31/A5349. We processed 115 manually (an average of 7 shipments/month), from July 21st, 2015 to April 13th, 2017, prior to DMM deployment. The next 222 shipments with the DMM (an average of 12 shipments/month) were processed, from May 2nd, 2017 to October 19th, 2018. The ITS model was implemented with first-order autocorrelation and annual seasonality. The original time-series was determined to be stationary. The number of sites enrolling increased throughout the study as new sites completed start-up requirements. This increase in enrolling sites increased with the time-series.

Nineteen sites opened for enrollment throughout the manual period, and 15 additional sites opened throughout the DMM period. The number of sites enrolling was assessed to determine its influence on the model and was found to be non-significant. The number of data points before and after DMM implementation were 22 and 18 respectively. Both numbers exceed the suggested minimum of 8 observations each for pre- and post-intervention periods11 to have enough power in an ITS analysis. We estimated 96% power to detect the level change in this ITS analysis, by using simulation-based power calculation tables13, a calculated level effect size of 2.3 and autocorrelation of 0.3.

DMM implementation reduced the average monthly shipment-related stock-out (the level change) by 15% [p-value=0.03; 95% C.I. (–28.8%, –2.0%)] (Table 2). During the manual period (pre-DMM implementation), the average monthly shipment-related stock-out trend increased significantly at a rate of 0.8% [p-value=0.028; 95% C.I. (0.1%, 1.5%)]. There was a non-significant reduction of 0.6% [p-value=0.279; 95% C.I. (–1.8%, 0.5%)] in the trend post-DMM implementation (Table 2). Figure 1 shows the time-series data with linear trendlines for both the pre-DMM and post-DMM implementation periods. The trendline for the pre-DMM implementation period is extended to provide the counterfactual during the post-DMM implementation period.

Table 2:

Interrupted Time Series Segmented Regression Model

ITS Model Parameters Estimates 95% Wald
Confidence Interval
p-Value
Intercepta − 0.011 (− 0.103, 0.081) 0.810
Pre-DMM intervention slopeb 0.008 (0.001, 0.015) 0.028
DMM Implementation level changec − 0.154 (− 0.288, − 0.020) 0.030
Pre- to post-DMM intervention slope changed − 0.006 (− 0.018, 0.005) 0.279

Notes:

DMM – Drug Management Module.

a

Intercept, β0 : estimated starting level of the average monthly shipment-related stock-out.

b

Pre-DMM intervention slope, β1 : trend of the average monthly shipment-related stock-out during Manual period, that is before DMM intervention.

c

DMM implementation level change, β2 : change in the level of the average monthly shipment-related stock-out immediately after DMM implementation.

d

Pre- to post-DMM intervention slope change, β3 : change in the trend of the average monthly shipment-related stock-out from pre-DMM intervention to post-DMM intervention.

Figure 1:

Figure 1:

Time-Series of Shipment-related Study Drug Stockouts, with Linear Trendlines for Manual and DMM Period

Notes:

DMM – Drug Management Module

Slope of linear trendline in Manual Period (pre-DMM implementation) is 0.008.

Counterfactual is a hypothetical representation of trendline with no DMM intervention.

Slope of linear trendline in DMM Period (post-DMM implementation) is 0.002. A non-significant reduction of 0.006 in the slope from pre-DMM to post-DMM intervention (Table 2).

For the secondary comparisons, we defined the DMM-restricted time-period from May 2nd, 2017 to March 13th, 2018, in order to select the first 115 DMM-processed shipments for comparison with the 115 manual-processed shipments. Seventeen (17) shipment-related stock-out events occurred during the manual period, while 7 events occurred after DMM deployment. Four (4) of those 7 shipment-related events occurred during the DMM-restricted period (Table 3).

Table 3:

Measures of Efficiency

DMM-Restricted Period compared with Manual DMM-Unrestricted Period compared with Manual
Selected Measures of Efficiency Manually Processed Shipments (N = 115) a DMM Processed Shipments (N=115) b Difference between Manually and DMM Processed Shipments (95% C.I. of difference) Manually Processed Shipments (N = 115) a DMM Processed Shipments (N=222) c Difference between Manually and DMM Processed Shipments (95% C.I. of difference)
Number of shipment-related study drug stock-outs: n (%) 17 (14.8) 4 (3.5) −11.3d
(−18.6, −4.0)e
17 (14.8) 7 (3.2) −11.6d
(−18.5, −4.8)e
Median time in days. Before shipment sent: Time from restock request by site to submission of shipment order: (25th, 75th quantiles) 2.0
(1.0, 6.0)
1.0
(0.0, 3.0)
−1.0
(−2.0, 0.05) f
2.0
(1.0, 6.0)
0.0
(0.0, 3.0)
−2.0
( −2.7, −1.3)f
Median time in days. During shipment: Time from shipment order submission to receipt on site: (25th, 75th quantiles)
Sites Outside South Africa and U.S.:DMMnf = 41; Manualnf = 38 52.0
(27.0, 65.0)
51.0
(16.0, 70.0)
−1.0
(−23.1, 21.1)f
52.0
(27.0, 65.0)
54.0
(31.0, 77.0)
2.0
(−16.2, 20.2)f
Sites Inside South Africa and U.S.: DMMnf = 74; Manualnf = 77 5.0
(3.0, 7.0)
3.0
(2.0, 6.0)
−2.0
(−3.3, −0.7) f
5.0
(3.0, 7.0)
3.0
(2.0, 6.0)
−2.0
(−3.0, −1.0)f
Median time in days. After shipment received: Time from study drugs receipt on site to available for enrollment: (25th, 75th quantiles) 1.0
(0.0, 3.0)
0.0
(0.0, 0.0)
−1.0
(−1, 0)h
1.0
(0.0, 3.0)
0.0
(0.0, 0.0)
−1.0
(−1, 0) h
a

Manual shipments processed from July 21st, 2015 to April 13th, 2017.

b

DMM shipments processed from May 2nd, 2017 to March 13th, 2018. (DMM-restricted data)

c

All DMM shipments processed on or after May 2nd, 2017. (DMM-unrestricted data)

d

Difference is between the percentage of shipment-related study drug stock-outs for Manual-processed and DMM-processed shipments.

e

Wald Asymptotic Confidence Interval

f

Confidence interval (C.I.) of difference in median based on Inversion of Rank Test.

g

DMMn = number of DMM processed shipments; Manualn = number of manually processed shipments.

h

Unable to calculate confidence interval using Inversion of Rank Test due to small variance between two processes (DMM and Manual) at their median values. Instead, the 95% confidence interval based on Hodges-Lehmann Estimation for location shift (i.e. 95% C.I. for median of differences between processes).

The comparisons of manual- and DMM-processed shipments, using the selected measurements under the DMM-restricted time-period, are displayed in Table 3. Among manual-processed shipments, 14.8% were associated with a shipment-related drug stock-out, compared to 3.5% of DMM-processed shipments: a statistically significant 11.3% reduction [95% C.I. (–18.6%, –4.0%)]. The median time before shipment sent, had a non-significant reduction of 1 day [95% C.I. (–2.0, 0.05)] with DMM implementation. During-shipment time was stratified by sites inside and outside the R.S.A. and the U.S., that is, stratified by sites associated with and without a depot. The DMM reduced median time by 1 day [95% C.I. (–23.1, 21.1)], a non-significant difference, during shipment for sites outside the R.S.A. and the U.S. For sites within the R.S.A. and the U.S., median transit-time had a statistically significant reduction of 2 days [95% C.I. (–3.3, –0.7)].

In the DMM-unrestricted comparison, the measurements of efficiency were recalculated using all 222 DMM-processed shipments (Table 3) and so included 3 additional shipment-related stock-out events. These adjustments showed similar results for the stock-out events with a significant reduction of 11.6% [95% C.I. (– 18.5%, –4.8%)]. The median time before shipment sent, had a statistically significant decline of 2 days [95% C.I. (–2.7, –1.3)]. The median time during shipment had a non-significant increase under DMM implementation of 2 days [95% C.I. (–16.2, 20.2)] for sites outside the R.S.A. and the U.S., but like the DMM-restricted analysis, declined statistically significantly by 2 days [95% C.I. (–3.0, –1.0)] for sites within the R.S.A. and the U.S.

Figure 2 compares the timing of all shipment-related stock-out events by manual- and DMM-processed shipments. Events associated with manual-processed shipments, were primarily multiple events clustered at different time points. Events associated with DMM-processed shipments, occurred as single events over time.

Figure 2:

Figure 2:

Timing of Stock-out Events by Processing Type (Manual vs. DMM)

Effect on Study Site Pharmacies

The introduction of the DMM allowed pharmacists to finalize restock requests and confirm shipment receipts at a centralized location, through the automated notifications. Details of the kit numbers, regimens, phases, batches and expiration dates were loaded into the DMM. Site personnel physically verified the status of the kits received as either available or damaged (due to temperature excursions, water or structural damage), and updated the DMM, making kits available for use in real-time and reducing negative effects of time-zone differences. Based on median values, study drugs were available on the same day after shipment received, when processed with the DMM, compared to 1 day later [95% C.I. (–1.0, 0.0)] when processed manually (Table 3).

Additionally, pharmacists indicated that the number of person-hours during shipment receipt, depended on the quantity of kits in a shipment. Sites with larger numbers of shipments (minimum=21; maximum=33), but smaller numbers of kits per shipment (minimum=40; maximum=48), reported an average person-hour reduction of 1 hour/shipment when moving from manual- to DMM-processed shipments. Similarly, sites with smaller numbers of shipments (minimum=10; maximum=11), but larger numbers of kits per shipment (minimum=90; maximum=240), reported an average person-hour reduction of 2 hours/shipment.

Additional benefits reported by pharmacists included improvements in ease of monthly stock-inventory verification. Information for drug accountability logs were directly retrievable from the DMM, allowing pharmacists to confirm physical inventory. The DMM also provided a reference and comparative source for reports such as inventory on-hand, number of kits quarantined due to damage or temperature excursions, and number of used or expired kits. These reports were frequently required by local drug regulatory bodies or site monitors. The DMM also presented an efficient and accurate system for tracking expiration dates. The Last Allowable Enrollment Date (LAED), which provided enough time for participants to complete treatment prior to kit expiration, was difficult to identify by site pharmacists using the manual system.

The pharmacists surveyed suggested several improvements be included in future versions: more site-specific downloadable Excel or PDF outputs showing expiry reports, onsite stock, and disposition of kits for Good Pharmacy Practice (GPP). Notifications of expiration dates and deactivation of expired kits would also be helpful to pharmacists. There were a few instances when restock notifications were sent too early because of unanticipated reductions in enrollment rates. These notifications caused sites to ask whether they needed to submit a restock request, since they had enough kits to continue enrolling. In response, the DCC would inform sites to ignore these unexpected notifications, and the DMM timing was adjusted.

DISCUSSION

Trial networks conducting international multicenter studies often face drug management challenges. Among the challenges experienced by the DCC during S31/A5349 were complexities around the design and size of kits, the study drug procurement process, and the process for shipment to countries of participating sites (Supplementary Appendix Section 2). We developed the DMM, a web-based drug management system, to provide a technological solution to these problems. The DMM enabled us to estimate, through simulations, the timing of site study drug stock-out. It streamlined shipment procedures through knowledge of local importation processes, real-time updates and automated notifications. In order to allow the DMM to be implemented on other web-based platforms, the simulations for the shipment-related study drug stock-out algorithm are detailed in Supplementary Appendix Section 1, using STRESS guidelines10, and the DMM Process Flow Scheme is provided in Supplementary Figures S5a–d.

The DMM allowed timely resupply of study drugs to clinical sites. The ITS analysis showed the DMM intervention was significantly associated with a 15% [95% C.I. (–28.8%, –2.0%)] reduction (level change), of the average number of shipment-related stock-outs compared to the manual approach. Similar reductions were seen in the proportion of these stock-out events: 11.3% with DMM-restricted data, and 11.6% with DMM-unrestricted data. These improvements in efficiency were attributed to knowing when site inventory would be depleted and how long it would take to restock sites. A reduction in shipment-related stock-outs, allowed sites to maintain enrollment rates and minimize interruptions from lack of study drugs. Shipment-related stock-outs associated with manual shipments were clustered, largely because kits at multiple sites reached their LAED, rending those kits unsuitable for use. The manual approach did not efficiently predict when multiple sites, with varying importation requirements, should start the restocking process to avoid these situations. DMM implementation prevented occurrence of simultaneous stock-out events by sending notifications to initiate the restocking process with sufficient time accounting for shipment timelines and the LAED of kits. Consequently, shipment-related stock-out events after DMM implementation occurred as individual events spread over time. Unanticipated increases in enrollment at individual sites, resulting in depletion of inventory sooner than the stock-out algorithm projected, caused most of these events. Unexpected delays in shipments or import permit approvals also caused these events.

Determining improvements in efficiency of timing was a secondary objective. Comparison of median time-intervals before shipments were sent showed a non-statistically significant reduction of 1 day using DMM-restricted data, but a statistically significant reduction of 2 days with DMM-unrestricted data. Direct submission of restock requests by site staff into the DMM, automatically notifying DCC staff for review, approval, and submission to Sanofi or one of the depots for fulfillment, accounted for this time-saving. Despite the increase in productivity with DMM-unrestricted data, additional evaluation would be needed to confirm this difference because of the initial non-significant finding under the DMM-restricted data.

The median number of days during-shipment to sites outside the R.S.A. and the U.S., that is, sites not associated with a depot, revealed no time-savings with the DMM. Both DMM-restricted and unrestricted data showed non-statistically significant differences. The failure to detect differences can be attributed to two issues. Sites with more lengthy and complicated importation approval processes had more shipments included during the DMM-unrestricted period. Additionally, any unexpected changes in the approval process increased median time during-shipment. Transit-time increased because of changes in approval processes, for at least one shipment.

Sites within the R.S.A. and the U.S. were provided with inventory through depots. During-shipment time showed a consistent statistically significant improvement of 2 days with DMM implementation, using DMM-restricted or DMM-unrestricted data. This consistency in time-savings resulted from the importation approvals being managed centrally by the R.S.A. and the U.S. depots, and from having a large supply of kits readily available at a facility close to sites within those countries.

Pharmacy Benefits of DMM Implementation

Study sites experienced improved productivity in pharmacies with the DMM. Prior to the DMM, the standard pharmacy practice of drug receipt and storage, generally involved site pharmacists taking physical inventory of drugs received. This physical inventory additionally involved making entries in the study-specific drug accountability logs for each kit received. Site staff completed kit lists and notified DCC about kit status. DCC staff manually updated verified kits in TBTC2, with delays due to time-zone differences between DCC and sites. Following DMM implementation, one of the highest enrolling sites in S31/A5349 reported improved productivity for their site personnel through shortening physical stock intake times. This was reflected in the reduced median time after shipment received on-site, from 1 day to same day availability. These time-savings were attributed to real-time updates that site pharmacists could make to kit statuses, allowing kits to be ready for enrollment immediately after receipt at site. Though small, these time-savings proved valuable especially at high enrollment sites. Reducing delays in enrollment 1) accelerated completion of the study and 2) improved study quality and patient care by enabling research participants to start study regimens on the shortest possible end of their eligibility period after diagnosis and enrollment – minimizing time on non-study treatment.

Under the manual system, sites depended primarily on pharmacists to keep careful and accurate drug inventory in all arms, to decide on reorder levels and available storage space, and to account for import permit approval timelines, when ordering new shipments. This process was subject to human error, especially when staff changed or when there were fluctuations in factors affecting timing of reordering, such as enrollment rates. Even though the DCC utilized a pre-specified threshold as an indicator to start the restocking process, the manual approach was less responsive to enrollment rate changes and import permit timelines and was thus neither very effective nor informative for pharmacists. The DMM introduced an automated alert system that readily incorporated the timelines for import permit requirements at individual sites. The DMM reduced many time-consuming tasks, from manual drug inventory counts during stock receipt at sites through monthly inventory reviews, allowing site pharmacists to focus on other drug management tasks especially at high enrolling sites. Additionally, drug shipments under the DMM were more timely and more consistent, compared to the sporadic or sudden restock requests under unreasonable timelines that occurred with the manual approach. Finally, the DMM’s automatic tracking of expiration dates across different drug lots, kit batches, regimens, and shipments greatly facilitated maintenance of sufficient kit inventories.

Limitations

Despite the success of the DMM, there were limitations to both its evaluation and design. The ITS analysis did not include a comparable control group, potentially reducing the strength of the inference. However, the inclusion of the counterfactual acted as a proxy for a control group and provided an indication of the likely performance of the manual process, had there not been the DMM intervention. Throughout any trial, several external factors can influence enrollment rates, making them difficult to predict. In S31/A5349 unanticipated enrollment spikes or slowdowns occurred at some sites, influencing the previous 3-month enrollment average. These changes in enrollment rates would not have been accounted for during the estimate of a site’s stock-out. In order to reduce the consequences of this limitation, inventory checks were utilized throughout the trial. Initially, the stock-out algorithm was applied to perform these checks weekly. This provided regular, overall, updated views of study drug inventory at all study sites. When inventory became limited and the frequency of enrollments increased, these inventory checks were performed daily. The checks captured the need for unplanned shipments, particularly at high enrolling sites supplied from depots. Additionally, there were numerous uncontrolled timing factors that could not be accounted for by the DMM, such as transit from central warehouse facilities to Custom clearance and delivery to sites. These factors increased the shipment time in the DMM, since many more shipments occurred after the DMM was implemented. Neither the manual nor DMM approaches were designed to account for shortages at the study drug supplier level, so events from such situations were considered unevaluable. Finally, for comparisons of shipments outside the R.S.A. and the U.S., the number of shipments, during both manual and DMM periods, were relatively small resulting in wide confidence intervals. Therefore, care should be taken when interpreting these results.

Overall, the DMM approach of estimating study drug stock-out dates, accounting for shipment timelines through programmed recording of local requirements and enabling real-time automated functions on a web-based platform, proved more efficient than a manual approach. The primary benefit of DMM implementation was the significant reduction in average shipment-related stock-out at sites. Additional transit time-saving benefits occurred for sites associated with a depot, such as within the R.S.A. and the U.S. Marginal improvements also were observed before shipments were sent, and after shipments were received on-site, although additional data would be needed to verify these savings. The DMM provided a framework in which study drug inventory was managed successfully by DCC and study site staff. It demonstrated potential for implementation at other networks and health institutions conducting international multicenter clinical trials or requiring drug management.

Supplementary Material

1

Highlights.

  • Novel solutions to drug management challenges for international clinical trials.

  • Drug system with improved communications, simulations and importation approvals.

  • Reduces drug stock-out via efficient maintenance and deployment of drug inventory.

Acknowledgments

Special Thanks

We especially thank the Centers for Disease Control and Prevention Drug Service and Lekoko PMC for storage and shipments of study drugs to sites within R.S.A. and U.S.; all site Pharmacists and pharmacy personnel for their work on importation approvals and assistance with local study drug management.

Funding and Role of the Funder/Sponsor

Funding support for this trial was provided by the Centers for Disease Control and Prevention, National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention, Division of Tuberculosis Elimination; and by the National Institute of Allergy and Infectious Diseases of the National Institutes of Health under Award Numbers UM1 AI068634, UM1 AI068636 and UM1 AI106701. Sanofi (Paris, France, and Bridgewater, New Jersey) donated rifapentine and all other study drugs, supported shipping of study drugs to all sites, and provided funding support for pharmacokinetic testing.

Disclaimer

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention, the National Institutes of Health, or the authors’ affiliated institutions. References in this manuscript to any specific commercial products, process, service, manufacturer, or company do not constitute endorsement or recommendation by the U.S. Government.

Footnotes

Disclaimer

The contents of this report are solely the responsibility of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention (CDC), the National Institute of Allergy and Infectious Diseases, or the U.S. Department of Health and Human Services. References in this manuscript to any specific commercial products, process, service, manufacturer, or company do not constitute endorsement or recommendation by the U.S. Government.

Declaration of conflicting interests

The authorship team members have declared (below or attached) any potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Sanofi commercial interests did not influence the study design; the collection, analysis, or interpretation of data; the preparation of this manuscript; or the decision to submit this manuscript for publication. A Sanofi technical expert served on the protocol team.

Ethical Approval

This study was approved by the institutional review board(s)/research ethics committees at Centers for Disease Control and Prevention (CDC). Additionally, each site followed local review policies and procedures.

Clinical Trials Registration

This trial was registered with ClinicalTrials.gov (Study 31: NCT02410772)

IND Number / IND Sponsor

46,954 / U.S. Centers for Disease Control and Prevention

Declaration of interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Trial registration: This trial was registered with ClinicalTrials.gov (Identifier: NCT02410772) on April 8, 2015.

Submission Categories & Reviewers: Clinical Trial Management & Optimization

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  • 1.Mabirizi D, Phulu B, Churfo W, et al. Implementing an Integrated Pharmaceutical Management Information System for Antiretrovirals and Other Medicines: Lessons From Namibia. Global Health: Science and Practice (2018) Vol 6; Number 4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Leung N, Chen A, Yadav P, et al. The Impact of Inventory Management on Stock-Outs of Essential Drugs in Sub-Saharan Africa: Secondary Analysis of a Field Experiment in Zambia. PLoS ONE (2016) DOI: 10.1371/journal.pone.0156026 [DOI] [PMC free article] [PubMed]
  • 3.Pasquet A, Messou E, Gabillard D, et al. Impact of Drug Stock-Outs on Death and retention to Care among HIV-Infected Patients on Combination Antiretroviral Therapy in Abidjan, Cote d’Ivoire. PLoS ONE (2010) Vol. 5; Issue 10; e13414. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.World Health Organization. Harmonized monitoring and evaluation indicators for procurement and supply management systems. Early-warning indicators to prevent stock-out and overstocking of antiretroviral, anti-tuberculosis and antimalarial medicines. 2011. Geneva: World Health Organization. [Google Scholar]
  • 5.Jatau B, Avong Y, et al. Procurement and Supply Management System for MDR-TB in Nigeria: Are the Early Warning Targets for Drug Stock Outs and Over Stock of Drugs Being Achieved? PLoS ONE (2015) DOI: 10.1371/journal.pone.0128500 [DOI] [PMC free article] [PubMed]
  • 6.Rogers A, Flynn RWV, McDonnell P, et al. A novel drug management system in the Febuxostat versus Allopurinol Streamlined Trial: A description of a pharmacy system designed to supply medications directly to patients within a prospective multicenter randomised clinical trial. Clinical Trials 2016. vol. 13, 6: pp. 665–670. [DOI] [PubMed] [Google Scholar]
  • 7.Iteen SA and Ceppaglia J. Investigational drug information through a hospital-wide computer system. Am J Hosp Pharm 1992; 49: 2746–2748. [PubMed] [Google Scholar]
  • 8.Sweet BV, Tamer HR, Siden R, et al. Improving investigational drug service operations through development of an innovative computer system. Am J Health Syst Pharm 2008; 65: 969–973. [DOI] [PubMed] [Google Scholar]
  • 9.Dorman SE et al. High-dose rifapentine with or without moxifloxacin for shortening treatment of pulmonary tuberculosis: Study protocol for TBTC Study 31/ACTG A5349 phase 3 clinical trial. Contemp Clin Trials. 2020. Mar;90:105938. doi: 10.1016/j.cct.2020.105938. Epub 2020 Jan 22. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Monks Thomas, Currie Christine S. M., Bhakti Stephan Onggo Stewart Robinson, Kunc Martin, Taylor Simon J. E. (2019) Strengthening the reporting of empirical simulation studies: Introducing the STRESS guidelines, Journal of Simulation, 12:1, 55–67, DOI: 10.1080/17477778.2018.1442155 [DOI] [Google Scholar]
  • 11.Penfold RB, Zhang F. Use of interrupted time series analysis in evaluating health care quality improvements. Acad Pediatr. 2013. Nov-Dec;13(6 Suppl):S38–44. doi: 10.1016/j.acap.2013.08.002. PMID: 24268083. [DOI] [PubMed] [Google Scholar]
  • 12.Dickey DA; Fuller WA (1979). Distribution of the Estimators for Autoregressive Time Series with a Unit Root. Journal of the American Statistical Association. 74 (366): 427–431. [Google Scholar]
  • 13.Zhang F, Wagner AK, Ross-Degnan D. Simulation-based power calculation for designing interrupted time series analyses of health policy interventions. J Clin Epidemiol 2011;64:1252–61. [DOI] [PubMed] [Google Scholar]
  • 14.Zhang B, Liu W, Lemon SC, et al. Design, analysis power, and sample size calculation for three phase interrupted time series analysis in evaluation of health policy interventions. J Eval Clin Pract. 2020;26:826–841. 10.1111/jep.13266 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Keonker R. (1994). Confidence Intervals for Regression Quantiles. In Asymptotic Statistics, edited by Mandl P, and Huskova M, 349–359. New York: Springer-Verlag. [Google Scholar]
  • 16.Hodges JL, Lehmann EL (1963). Estimation of location based on ranks. Ann. Math. Statist. 34 (2): 598–611. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1

RESOURCES