Validation of a metered dose inhaler electronic monitoring device: implications for asthma clinical trial use

Abstract

Background

The SmartTouch Ventolin monitor (Adherium, Auckland, New Zealand) is an electronic monitor for use with a Ventolin metered dose inhaler, which records the date and time of inhaler actuations. This technology has the potential to allow in-depth analysis of patterns of inhaler use in clinical trial settings. The aim of this study was to determine the accuracy of the SmartTouch Ventolin monitor in recording Ventolin actuations.

Methods

20 SmartTouch Ventolin monitors were attached to Ventolin metered dose inhalers. Bench testing was performed over a 10-week period, to reflect the potential time frame between visits in a clinical trial. Inhaler actuations were recorded in a paper diary, which was compared with data uploaded from the monitors.

Results

2560 actuations were performed during the 10-week study period. Monitor sensitivity for diary-recorded actuations was 99.9% with a lower 97.5% confidence bound of 99.7%. The positive predictive value for diary-recorded actuations was 100% with a 97.5% lower confidence bound of 99.9%.

Conclusions

The SmartTouch Ventolin monitor is highly accurate in recording and retaining electronic data. It can be recommended for use in clinical trial settings in which training and quality control systems are incorporated into study protocols to ensure accurate data acquisition.

Key messages.

• Electronic monitoring of inhaled asthma therapy has the potential to allow in-depth analysis of patterns of inhaler use in clinical trial settings.

• The SmartTouch Ventolin monitor is highly accurate in recording and retaining electronic data.

• The SmartTouch Ventolin monitor can be recommended for use in clinical trial settings in which training and quality control systems are incorporated into study protocols to ensure accurate data acquisition.

Introduction

Electronic monitoring of inhaled asthma therapy allows for the collection of data on the date and time of inhaler actuation.¹ The application of this technology has the potential to greatly enhance data collection in the clinical trial setting by allowing assessment of total medication exposure and patterns of medication use.

Prior to the use of an electronic monitor in a clinical trial, it is essential to validate that monitor's accuracy under standardised conditions (ie, to perform bench studies).

The SmartTouch Ventolin monitor (Adherium, Auckland, New Zealand) has been developed specifically for use with a Ventolin pressurised metered dose inhaler (pMDI), and does not affect its use through a spacer device. The SmartTouch Ventolin is an updated version of the Smartinhaler Tracker, which was found to be over 99% accurate on bench testing,² and subsequently successfully used in a 6-month multicentre randomised controlled trial (RCT) in over 300 patients with high-risk asthma.^3–5 The Smartinhaler Tracker detected inhaler actuations using a small switch inside its plastic casing which was activated every time the pMDI was used. The new SmartTouch Ventolin incorporates a small sensor situated under the base of a pMDI, which detects inhaler actuation (figure 1). The SmartTouch Ventolin can also record the date and time an inhaler is inserted or removed from its case. Data are stored in the monitor for upload, via USB cable to Adherium's SmartinhalerLive website.

Figure 1.

SmartTouch Ventolin monitor.

The aim of the study was to assess the proportion of actuations correctly recorded by SmartTouch Ventolin monitors during a 10-week study period. The results will be used to guide the use of the monitors in the clinical trial setting.

Materials and methods

Study protocol

Overview

Twenty non-rechargeable, USB upload compatible SmartTouch Ventolin monitors were used with Ventolin pMDIs. A study period of 10 weeks was selected, to mimic the time between visits for a clinical trial. During the 10-week testing period, inhaler actuations, inhaler insertions and removals, battery tests and data uploads were performed to mimic use of the monitors in the clinical trial setting (table 1).

Table 1.

Monitor tests and subsequent outcome measures during the 10-week study period

Timing	Action	Outcomes
Day 0	Initial monitor screening check	Proportion of inhalers that passed/failed initial screen for 10-week period ▸Proportion of inhalers that failed due to the battery test ▸Proportion of inhalers that failed due to failed accuracy* in detection of actuations Usability of Smartinhaler Connection Centre and SmartinhalerLive website
On two occasions each week	Low use actuations†	Accuracy* in detection of 20 episodes of low use per monitor (1600 actuations in total)
On three occasions during study period	High use actuations‡	Accuracy* in detection of three episodes of high use per monitor (960 actuations in total)
Day 21	Within study monitor screening check§	Proportion of inhalers that passed/failed within study monitor check for 10-week period ▸Proportion of inhalers that failed due to the battery test ▸Proportion of inhalers that failed due to failed accuracy* in detection of actuations Usability of Smartinhaler Connection Centre and SmartinhalerLive website
Day 70	Final data upload and battery test	▸Proportion of inhalers with green battery check ▸Usability of Smartinhaler Connection Centre and SmartinhalerLive website

*Accuracy is determined by actuations correctly detected by the monitor and a lack of spurious actuations.

†Two sets of two actuations performed in the same day.

‡Two sets of eight actuations performed in the same day.

§Performed after actuations for that day (if any), and in half of the monitors only (monitors 1–10).

Based on initial testing of the monitors prior to the 10-week period and feedback from Adherium, it was identified that it was essential for the inhaler and monitor to be held correctly to ensure that monitor actuations were accurately recorded. This requires the inhaler to be held upright with the forefinger on top and thumb beneath the monitor (not on the product mouthpiece), as per the Medsafe Ventolin Inhaler (CFC-Free) Data Sheet.⁶ The monitor does not record actuations performed by pressing the top without the thumb pressing on the base, as might occur when the patient is using a spacer. Inhaler actuations in this study were performed only by the investigators named above, all of whom had training in correct inhaler technique.

Paper diaries were used to record when an inhaler was actuated, and when it was inserted or removed from a monitor. Monitors were connected via a USB cable to the computer, and data were uploaded via the Smartinhaler Connection Centre software to the SmartinhalerLive website. The website displayed the date and time of each actuation performed, as well as when the monitor detected an inhaler had been inserted or removed. The SmartTouch Ventolin monitor has the facility for Bluetooth communication with mobile devices, but we did not test that function here.

For detail on inhaler actuations, inhaler insertion and removal, inhaler screening checks, battery testing and data upload, see table 1.

Initial study and within-study monitor screening checks

These checks mimic what could be undertaken in the clinical trial setting to ensure a monitor is functioning correctly, and are similar to those performed in a previous RCT using the Smartinhaler Tracker.^{3 4} An ‘initial screen’ is what would take place prior to the first time a monitor is dispensed, while the ‘within-study screen’ would take place during the course of a clinical trial when the participant brought their inhalers to a study visit for replacement.

Both types of screening started with data upload. Data upload was performed by connecting the monitor via USB to a computer with Smartinhaler Connection Centre software. The software transferred data to the SmartinhalerLive website and automatically set the monitor's clock to the computer's clock. The monitor was then disconnected and a battery check was performed (see below for detail).

For the initial screen, a Ventolin pMDI inhaler (which could be new or partially used) was inserted into the monitor. Two inhaler actuations were performed, separated by approximately 10–20 s. At least 15 min later, 2 further actuations, separated by approximately 10–20 s, were performed.

For the within-study screen, the Ventolin pMDI inhaler attached to the monitor was replaced with another one (which could be new or partially used). Two inhaler actuations were then performed, separated by approximately 10–20 s.

The actuation times were recorded in a paper diary and a data upload was subsequently performed for both the initial study and within-study checks. Monitors were deemed to have failed their check if either the battery test did not display green, or there were missing or spurious actions when the uploaded data were compared with the paper diary.

Inhaler actuations

Actuations were performed in dedicated office areas under standardised conditions by at least two investigators. The pMDI was actuated into a plastic bag rather than inhaled. One investigator was responsible for inhaler actuation while the other investigator maintained a paper diary. This method was used to reduce investigator error affecting the interpretation of electronic actuation data.

Actuations were performed in either a low or high use pattern, to reflect possible real-life use. Low use actuations were performed by removal of the inhaler cap and two actuations separated by 5–30 s, followed by replacement of inhaler cap. At least 1 h later, the inhaler cap was removed and a further two actuations separated by 5–30 s were performed, followed by replacement of inhaler cap.

The high use actuation pattern involved removal of the inhaler cap and eight actuations separated by 5–20 s, followed by replacement of inhaler cap. At least 15 min later, the inhaler cap was removed and eight actuations separated by 5–20 s were performed, followed by replacement of the inhaler cap.

Inhaler removal/insertion and battery tests

To test whether the monitors could detect whether a participant had removed and reinserted an inhaler in one of their allocated monitors, every time an inhaler was removed or inserted into a monitor the date and time were also recorded in the paper diary.

The monitor batteries were tested by pressing a button on the side of the monitor case. The light is intended to glow green if the battery is functioning, orange if the battery is trending low but the device is still operating in an acceptable range, and red if the battery has entered an inoperable range, in which case the device will have ceased to log usage.

Statistical analysis

The primary outcome for this study was the accuracy, expressed as sensitivity and positive predictive values for diary-recorded actuations, of the SmartTouch Ventolin monitors over a 10-week period of use. Other outcomes included the accuracy of the monitors in detecting inhaler insertion or removal, the proportion of monitors that passed initial study and within-study checks, the proportion of monitors which passed battery testing, and the usability of the Smartinhaler Connection Centre and SmartinhalerLive.

Based on our previous studies,^{2 7} we predicted that a sample size of 20 monitors would allow us to make precise estimates for our primary outcomes. Sensitivity and positive predictive value for diary-recorded actuation were estimated by the relevant proportions, and a lower 97.5% confidence bound was calculated by the Clopper-Pearson method in R V.3.02.

Results

A total of 2560 actuations were performed over the 10-week study period. The proportion of actuations correctly recorded by the monitors, when compared with the paper diaries, was 2558/2560 (99.9%), as shown in table 2. As a result, the sensitivity for diary actuation was 99.9% with a lower 97.5% confidence bound of 99.7%. The positive predictive value for a diary actuation was 100.0% with a 97.5% lower confidence bound of 99.9%. The sensitivities were similar for low and high use actuations (table 2) and there were no spurious actuations during the study period.

Table 2.

Monitor function results from the 10-week study period

Monitor function checked	Outcome
	N/N (%) (lower 97.5% confidence bound)
	Sensitivity	Positive predictive value
Inhaler actuations
Total	2558/2560 (99.9%) (99.7)	2558/2558 (100.0%) (99.9)
Low-use actuations	1600/1600 (100.0) (99.8)
High-use actuations	958/960 (99.8) (99.2)*
Inhaler insertion and removal
Insertion	29/30 (96.7) (82.8)†	29/29 (100.0) (88.1)
Removal	9/10 (90.0) (55.5)	9/9 (100.0) (66.4)

Sensitivity is the proportion of diary actuations electronically recorded by the monitor expressed as a percentage. Positive predictive value is the proportion of monitor recorded actuations that were recorded in the diary expressed as a percentage (lower 95.7% confidence bound).

All data from week 6 was 1 h discrepant (to within 7 min), in keeping with the time prior to the start of the New Zealand daylight savings period.

*Monitor 8 and monitor 17 each missed one actuation during high use.

†In monitor 9, there was a failure to record one insertion and one removal.

The discrepancy between the monitor clock and diary times increased over time, so that by the end of 10 weeks without data upload (monitors 11–20), the times were between 4 and 7 min faster on monitor data compared with diary data.

Over 96% of inhaler insertions and removals were detected (table 2). All monitors passed their initial study check, within-study check (monitors 1–10 only) and battery tests. There were no problems using the Smartinhaler Connection Centre or SmartinhalerLive website.

Discussion

This study demonstrated that the SmartTouch Ventolin electronic monitor is an accurate device for measuring pMDI actuations over a 10-week period, over a range of usage patterns and lengths of data storage. The monitor can therefore be recommended for use in the clinical trial setting, provided initial study and within-study checks are performed and there is participant and investigator education on correct inhaler use.

The 99.9% accuracy of the SmartTouch Ventolin monitor is similar to or better than other electronic monitors available,^{8 9} including its predecessor, the Smartinhaler Tracker monitor.² The Smartinhaler Tracker was found to be 99.7% accurate in a bench study² prior to its use in our previous RCT, which ran for 6 months in over 300 patients with at-risk asthma.^{3 4} In the RCT, complete data were available from 98% of the returned monitors, which were essential for the assessment of patterns of inhaler use (including participant adherence and overuse), overall medication exposure (including corticosteroid exposure), and relationships between inhaler use and poor asthma outcomes.^{3 10–13}

Feedback from Adherium was that it is important to correctly handle the inhaler and monitor, with a thumb on the base under the pMDI canister. Failure to do so (as may occur when the patient is using a spacer, or if the investigator put his or her thumb on the inhaler mouthpiece rather than the base) may result in missed actuation recordings by the monitor. In the clinical trial setting, this would require education of participants on the correct use of the inhaler (as per manufacturers’ instructions) especially when using a spacer, and education of investigators to ensure accurate recording during initial study and within-study monitor checks.

The time discrepancy between the diary and monitor data was up to 7 min over 10 weeks; this is within the specified internal clock accuracy of ±1 h over 12 months. It is therefore important to consider the timing of study visits to allow the return of monitors for data upload and automatic synchronisation of the monitor clock with the local time on the computer clock. In addition, as the monitors do not automatically correct for a country's daylight savings period, this needs to be considered when interpreting monitor results.

This study was of bench test design to ensure accurate documentation of the date and time that inhaler actuations occurred, and accurately identify when there were discrepancies between diary and monitor data. While it is limited in not being ‘real-world’ design, variations in patterns of inhaler use, insertion and removal of inhalers from monitors, and variable data upload timing, were selected to reflect the real-life setting.

In conclusion, SmartTouch Ventolin monitoring system was found to be accurate in recording and retaining electronic data. It can be recommended for use in the clinical trial setting, providing there is adequate education of investigators and participants regarding inhaler technique, and monitor checks are incorporated into the study protocol.