Three-month validation of a turbuhaler electronic monitoring device: implications for asthma clinical trial use

Abstract

Background

Electronic monitoring of inhaled asthma therapy is suggested as the ‘gold standard’ for measuring patterns of medication use in clinical trials. The SmartTurbo (Adherium (NZ) Ltd, Auckland, New Zealand) is an electronic monitor for use with a turbuhaler device (AstraZeneca, UK). The aim of this study was to determine the accuracy of the SmartTurbo in recording Symbicort actuations over a 12-week period of use.

Methods

Twenty SmartTurbo monitors were attached to the base of 20 Symbicort turbuhalers. Bench testing in a research facility was undertaken on days 0, 5, 6, 7, 8, 9, 14, 21, 28, 56 and 84. Patterns of ‘low-use’ (2 sets of 2 actuations on the same day) and ‘high-use’ (2 sets of 8 actuations on the same day) were performed. The date and time of actuations were recorded in a paper diary and compared with data uploaded from the SmartTurbo monitors.

Results

2800 actuations were performed. Monitor sensitivity was 99.9% with a lower 97.5% confidence bound of 99.6%. The positive predictive value was 99.9% with a 97.5% lower confidence bound of 99.7%. Accuracy was not affected by whether the pattern of inhaler use was low or high, or whether there was a delay in uploading the actuation data.

Conclusions

The SmartTurbo monitor is highly accurate in recording and retaining electronic data in this 12-week bench study. It can be recommended for use in clinical trial settings, in which quality control systems are incorporated into study protocols to ensure accurate data acquisition.

Introduction

Electronic monitoring of inhaled asthma therapy is suggested as the ‘gold standard’ for measuring patterns of medication use in clinical trials.¹ A number of monitors capable of recording the date, time and number of actuations have been developed and used in clinical research.¹ It is important to evaluate the internal validity of monitor devices prior to their use, to ensure accuracy, ease of use and reliability. This can be undertaken using ‘bench’ (laboratory) studies under standardised conditions. We have previously bench validated the Smartinhaler Tracker (Adherium (NZ) Limited, Auckland, New Zealand), an electronic monitor for use with a pressurised metered-dose inhaler (pMDI), which was found to be over 99% accurate.² This device detected an actuation through the downward actuation of the MDI canister triggering a small switch inside the plastic casing. The accurate recoding, retention and upload of pMDI actuation data in the bench study were subsequently shown to have external validity to a ‘real world’ multicentre randomised controlled trial (RCT) of two management approaches in the treatment of high-risk asthma.^{3 4} In this RCT, the data on patterns of inhaler use enabled a more detailed and informative assessment of the comparative risks and benefits of the two inhaled asthma treatment regimens than previously, and set a new benchmark for asthma clinical trial design.⁵

The SmartTurbo is a new device in production that is designed, manufactured and supplied by Adherium (NZ) Limited (http://www.smartinhaler.com/).

It is an electronic monitor for use with a turbuhaler dry powder inhaler dispensing a range of medications manufactured by AstraZeneca, such as budesonide/formoterol from the Symbicort turbuhaler. As with the pMDI Smartinhaler Tracker, data from the monitor are uploaded to the SmartinhalerLive website (Adherium (NZ) Limited) database using a USB computer connection and dedicated software (Connection Centre, Adherium (NZ) Limited).

The aim of the present bench study was to determine the accuracy of the SmartTurbo in recording Symbicort actuations over a 12-week period of use. This included assessment of the reliability of the SmartTurbo in recording different patterns of use, not recording spurious actuations during periods without use, storing and uploading data, and testing the performance of the computer software and website database. The findings were considered in relation to the potential use of the SmartTurbo in a clinical trial setting.

Materials and methods

Study protocol

Twenty SmartTurbo monitors (allocated reference A to T) were attached to the base of 20 Symbicort turbuhalers in this 12-week study (figures 1 and 2).

Figure 1.

SmartTurbo monitor.

Figure 2.

SmartTurbo monitor attached to Symbicort turbuhaler.

When installed onto the Turbuhaler, the SmartTurbo monitor moves with the Turbuhaler grip. To deliver a dose of medication, the user turns the monitor in the same way as they would turn the grip of the Turbuhaler (fully in one direction and then the other, listening for a click). The SmartTurbo monitor uses electromechanical sensors (switches) to detect the absence or presence of the cap on the mouthpiece of the inhaler. It uses a proprietary torque switch mechanism to detect the completion of each full turn of the Turbuhaler grip required for correct administration of medication. The SmartTurbo software is designed to log a medication delivery event when both turns required to generate a dose occur within a preset time frame and when both turns occur in a ‘cap off’ state. Data were uploaded using the Smartinhaler Connection Centre programme and the SmartinhalerLive website. Once uploaded, the website software applies an algorithm to the data, identifying pairs of anticlockwise and clockwise turns as inhaler actuations. Other key features of the SmartTurbo monitoring system are summarised in the online supplementary table S1.

Testing was undertaken on days 0, 5, 6, 7, 8, 9, 14, 21, 28, 56 and 84 (table 1). This pattern was undertaken to include various combinations of use, including consecutive days of use and study windows in a 12-week clinical trial.

Table 1.

Monitor function testing over 84 days

Action	Day 0	Days 5, 6, 8, 9	Days 7, 14 and 21	Day 28	Day 56	Day 84
Attachment of SmartTurbo to a Symbicort turbuhaler prior to any inhaler actuations	X			X*	X*
Initial screen of monitor	X
Low-use actuations	X	X	X	X	X	X
High-use actuations			X	X	X	X
Storage of electronic data for 28-day period					X
Upload of data to website following and on the same day as inhaler actuations	X	X	X	X		X
Upload of data obtained on day 56						X
Comparison of uploaded website data to diary data†	X	X	X	X		X
Comparison of website data to diary data time	X	X	X	X		X
Monitor battery charge check						X

Testing may occur ±1 day.

*Previous inhaler removed on same day.

†Recording of any missed actuations or spurious actuations.

Turbuhaler testing occurred in a dedicated office area under standardised conditions by two investigators (RB and JP). One investigator was responsible for turbuhaler actuation while the other investigator maintained a paper diary. Monitors were attached to Symbicort turbuhalers (budesonide/formoterol 100/6 μg/actuation (n=3), 200/6 μg per actuation (n=12) and 400/12 μg/actuation (n=5)). Half of the turbuhalers had previous use, with counter displays showing that over half the number of available doses had been administered. The date and number of each actuation were recorded in the diary. The time of each of the low-use actuations was recorded. The first and last of the eight high-use actuations had their times recorded, as well as the number of actuations that occurred between them (at a maximum of 20 s apart). The time was taken from the clock on the computer used for data upload from the monitors. This method was utilised to minimise the chance of investigator error affecting the interpretation of electronic actuation data. When a turbuhaler was inserted into or removed from a monitor, this was also recorded.

Monitors were actuated on the basis of the following Symbicort turbuhaler instructions: Unscrew and lift off the cover and then hold the inhaler upright with the grip downwards. Turn the grip as far as it will go in both directions, listening for a click. Do not hold the mouthpiece when you load the inhaler. If a click was not heard, this was documented on the case report form.

The patterns of use were—Low-use: On the same day, two actuations were performed up to 2 min apart, followed by no use for at least 1.5 h, then two further actuations were performed up to 15 min apart (total 4 actuations). This pattern was chosen to reflect maintenance or ‘low’ reliever medication use. High-use: On the same day, eight actuations were performed over a period of up to 5 min, followed by no use for at least 1.5 h, and then eight actuations were performed over a period of up to 5 min (total 16 actuations). This pattern was chosen to reflect ‘high’ reliever medication use, for instance, during a severe exacerbation.

On day 0, an initial screen took place (table 1). Monitors were attached to the turbuhalers and the low-use actuation pattern was performed. Data were uploaded from the monitors on to the SmartinhalerLive website on the same day. Monitors were deemed to have passed screening if they recorded the time, date and number of the four low-use actuations correctly and did not produce spurious data.

Subsequently, combinations of low-use and high-use patterns were performed, as summarised in table 1. After each day of use, all data from the monitors were uploaded to the website, except on day 56. To check monitor data storage, data from day 56 were uploaded on day 84, prior to initiating any actuations on that day.

The turbuhalers were removed and reattached to the monitors prior to testing on days 28 and 56 to replicate replacement of used turbuhalers in the setting of a clinical trial.

On day 84, the battery charge was checked using a button on the side of the monitor, which glowed green if the battery was functioning, orange if the battery was functioning but low (<8 weeks of use left) and red if the battery was very low/flat (monitor no longer recording). Batteries were new at the start of the study.

The accuracy measures were: (1) the proportion of actuations accurately recorded (sensitivity); (2) the proportion of non-spurious actuations (positive predictive value); (3) these proportions for specified groups: (i) low-use and high-use inhaler actuations, (ii) four time periods: days 0, 1–28, 29–56 and 57–84; (iii) days when monitors were and were not attached or reattached to turbuhalers (days 0, 28 and 56); (4) days when data retention and upload were on the same day and delayed (data from day 56, uploaded on day 84); (5) the proportion of monitors which recorded inaccurate reading(s); (6) monitor clock accuracy; (7) website upload usability and (8) battery charge on day 84.

Statistical analysis

SmartTurbo data were compared with written diary data. A series of 20 monitors were attached to the turbuhalers, which performed a total of 2800 actuations. This number was used to address variability in the functioning of different monitors, as well as accuracy. Sensitivity for diary actuation and a positive predictive value for diary actuation were estimated by the relevant proportions and a lower 97.5% confidence bound by the Clopper-Pearson method R V.3.02 was used.

Results

Accuracy in recording inhaler actuations

The proportion of actuations correctly recorded by the monitors, when compared with the paper diaries, was 2796/2800 (99.9%), as shown in table 2. There were four actuations that were not recorded by the monitors. Monitors S and T each failed to electronically record one single actuation on day 7 during the high-use regimen. Monitor A failed to record one actuation during the low-use regimen on day 56 and another actuation during the low-use regimen on day 84. Monitor I recorded one and monitor P recorded two spurious actuations during the period of low-use testing on day 0 (screening).

Table 2.

Overall results from the testing process

Monitor electronically recorded actuation	Written diary actuation log
	Yes	No
Yes	2796	3
No	4
Sensitivity* 99.9% (99.6%) Positive predictive value* 99.9% (99.7%)

1. Data from day 0 uploaded to V.7.7 Smartinhaler Connection Centre programme. Subsequently, all data were uploaded to V.7.8 Smartinhaler Connection Centre programme.

2. Investigators were unable to access the Adherium website to upload data on day 14, so data were uploaded on day 15 when access was gained.

3. All data from day 14 (320/320 actuations) were 1 h discrepant (to within 3 min), in keeping with the time prior to the cessation of the New Zealand daylight savings period on day 8. This discrepancy applied to data on day 14 only.

*Sensitivity is the proportion of diary actuations electronically recorded by the monitor expressed as a percentage (lower 97.5% confidence bound). Positive predictive value is the proportion of monitor recorded actuations that were recorded in the diary expressed as a percentage (lower 97.5% confidence bound).

As a result, the sensitivity for diary actuation was 99.9% with a lower 97.5% confidence bound of 99.6%. The positive predictive value for diary actuation was 99.9% with a 97.5% lower confidence bound of 99.7%, as shown in table 2.

The spurious results from day 0 were identified as being caused by an error in the online software algorithm for the Smartinhaler Connection Centre programme V.7.7, which identified individual turns of the inhaler as actuations. As a result of the software error, the online software programme was subsequently updated to the Smartinhaler Connection Centre programme V.7.8 prior to testing on day 5. Since the spurious actuations were the result of the algorithm rather than the recording function of the monitors themselves, all monitors were deemed to have passed screening and continued to subsequent testing. There were no further spurious actuations recorded.

Accuracy in recording inhaler insertion and removal

One inhaler removal and one inhaler installation were not recorded by monitor G on day 28. On day 56, one inhaler removal and one inhaler installation were not recorded by monitor Q. All other inhaler installations, 58/60 (97%), and removals, 38/40 (95%), were correctly recorded.

Factors potentially affecting accuracy

The sensitivity and positive predictive values for inhaler actuation of monitors were not affected by inhaler use regimen (high compared to low), period of use (days 0, 1–28, 29–56 and 57–84), attachment of the turbuhaler (days 0, 28 and 56) or delay in data upload (data from day 56), as shown in table 3. For a detailed summary of actuation data, see online supplementary table S2.

Table 3.

Detailed results from the testing process

Monitor function check	Outcome
Monitors passed initial screening test	20/20 (100%)
Sensitivity and positive predictive value by factors potentially affecting accuracy
	Sensitivity*	Positive predictive value*
Regimen
Low-use actuations	878/880 (99.8) (99.2)	878/881 (99.7) (99.0)
High-use actuations	1918/1920 (99.9) (99.6)	1918/1918 (100) (99.8)
Period, days
0 (screening)	80/80 (100) (95.5)	80/83 (96.4) (89.8)
1–28	1918/1920 (99.9) (99.6)	1918/1918 (100) (99.8)
29–56	399/400 (99.8) (98.6)	399/399 (100) (99.1)
57–84	399/400 (99.8) (98.6)	399/399 (100) (99.1)
Upload pattern
Same day upload	2397/2400 (99.9) (99.6)	2397/2400 (99.9) (99.6)
Delayed upload (days 56–84)	399/400 (99.8) (98.6)	399/399 (100) (99.1)
Turbuhaler attached to monitor prior to testing
No	1917/1920 (99.8) (99.5)	1917/1917 (100) (99.8)
Yes (days 0, 28 and 56)	879/880 (99.9) (99.4)	879/882 (99.7) (99.0)
Accuracy by monitor**
Number of monitors with at least one missed actuation recording	3/20 (15)
Number of monitors with at least one spurious actuation recording	2/20 (10)
Monitor battery check**
Pass on day 84	20/20 (100)

1. Data from day 0 uploaded to V.7.7 Smartinhaler Connection Centre programme. Subsequently, all data were uploaded to V.7.8 Smartinhaler Connection Centre programme.

2. Investigators were unable to access the Adherium website to upload data on day 14, so data were uploaded on day 15 when access was gained.

3. All data from day 14 (320/320 actuations) were 1 h discrepant (to within 3 min), in keeping with the time prior to the cessation of the New Zealand daylight savings period on day 8. This discrepancy applied to data on day 14 only.

*Sensitivity is the proportion of diary actuations recorded by the monitor expressed as N/N (%) (lower 97.5% confidence bound). Positive predictive value is the proportion of monitor recorded actuations that were recorded in the diary expressed as N/N (%) (lower 97.5% confidence bound).

**N/N (%) (95% CI).

Monitor clock accuracy

Actuation times recorded by the monitors were accurate up to 1 min of the recorded times in the paper diaries, with the exception of data from testing on 2 days. On day 14, times were exactly 1 h (to 1 min) different due to New Zealand daylight saving ending. This was subsequently corrected in the Smartinhaler Connection Centre programme. The day 56 actuation times, uploaded on day 84, were accurate up to 3 min from the times recorded in the paper diaries.

Website upload and battery check

On day 14, the investigators were not able to log into the Smartinhaler Connection Centre website to upload data. Adherium was contacted and data were subsequently uploaded on day 15. There were no other problems with uploading or checking the data on the Adherium website. All monitors demonstrated a green light on their battery check on day 84.

Discussion

This study demonstrated that the SmartTurbo electronic monitor is an accurate device for measuring turbuhaler actuations over a 12-week period. Our study has provided information on the reliability and accuracy of this monitor over a range of inhaler usage patterns and demonstrated retention of stored data between uploads with no evidence of spurious activations during periods of no use. As a result, the SmartTurbo monitor system should allow for accurate monitoring of turbuhaler actuations in the context of a clinical trial, in conjunction with quality control processes and participant education on correct inhaler technique.

The accuracy of the SmartTurbo monitor (99.9%) was at least comparable to that demonstrated by the pMDI Smartinhaler Tracker monitor (99.7%).² Results of the pMDI Smartinhaler Tracker bench testing provided the basis of the detailed monitor quality control programme⁴ successfully used in a multicentre RCT studying the use of multiple pMDIs over 6 months in over 300 patients.³ This programme identified that complete data were available from 98% of returned monitors. Electronic monitoring allowed in-depth analyses of the medication regimens assessed in the RCT which would otherwise not have been possible. These included safety assessments, such as systemic corticosteroid exposure, identification of poor adherence and inhaler overuse, evaluation of asthma control and analysis of the relationships between inhaler use and poor outcomes.^{3 6–9} However, one limitation to the use of the Smartinhaler Tracker was that trial medication (inhaled budesonide/formoterol) had to be delivered via an pMDI, which made its use ‘off-label’ according to the single maintenance and reliever therapy (SMART) regimen.

The accuracy of the SmartTurbo monitoring system used in this study now provides the required validation to warrant its use in clinical trials of Symbicort turbuhaler, including the combination inhaled corticosteroid/long-acting β-agonist inhaler reliever therapy regimen in mild asthma, as recently proposed.¹⁰

As well as demonstrating the accuracy of the SmartTurbo monitor in measuring turbuhaler use, this bench study informs protocol development for the clinical trial setting. First, screening prior to dispensing monitors will allow the identification of inaccurate monitors and/or programme inconsistencies, to reduce the risk of future inaccuracies and data loss.^{3 4} As seen with the screening test during this study, the detection of spurious actuations and close communication with the manufacturer allowed resolution of the issue prior to the next day of testing. Close cooperation with the manufacturer was essential for the success of the use of the pMDI Smartinhaler Tracker in our previous RCT, where data loss was limited through the identification of monitors which failed screening and their return for analysis and replacement.^{3 4} Second, it was noted that while the time discrepancy between the computer clock and monitor data was small, it did increase to 3 min in the data from day 56, when upload was delayed until day 84. This is in keeping with the devices’ specified internal clock accuracy (±20 min over 12 months). When designing a trial incorporating the Smartinhaler Tracker, this can be considered when planning the frequency of data uploads, which allow automatic synchronisation of the monitor clock with the computer clock.

A limitation of this study's design is that it is a bench, rather than real world, validation. Bench testing was selected to allow for testing in an environment under investigator control to ensure true and accurate documentation of the date and time that inhaler actuations occurred, and to ensure that discrepancies between diary and monitor data were accurately identified. In addition, aspects such as varied patterns of inhaler use, delayed data upload and monitor to canister reattachments were incorporated into the protocol to reflect variables that occur in the real-life setting.

In contrast to monitors for pMDIs,^{2 11 12} SmartTurbo monitors require two turns of the turbuhaler (one in each direction) to register the inhaler as having been actuated. As a result, trial patients will need to be clearly educated about the correct use of their inhaler, as per the manufacturer’s instructions. Patients could also be asked to demonstrate inhaler technique at study visits to allow for correction of any poor technique. Incorrect use of the inhaler (eg, 1 turn or 3 turns prior to inhalation from the turbuhaler) could potentially result in errors in both medication delivery and the monitor recording whether an actuation occurred. In addition, participants should be told not to remove monitoring devices from their inhaler, or use non study-specific inhalers.

Other electronic monitor devices have been developed for use with dry powder inhalers, which include technology to measure actuations by flow generation and sound measurement.^13–15 These may have advantages over the detection of a clockwise and anticlockwise turn, as flow and sound measurements would give further indication that the user actually inhaled their medication. However, the accuracy and algorithms to interpret data are still being developed.^13–15

In conclusion, we have performed a 12-week validation of the SmartTurbo monitor and found it to be highly accurate in recording and retaining electronic data, allowing recommendation for its use in the clinical trial setting, with quality control systems and inhaler technique education incorporated into study protocols to ensure accurate data acquisition.