The insignificant structural break in the relationship between improved observed hand hygiene on methicillin-resistant Staphylococcus aureus bloodstream infection rates in Ireland: a data-driven re-analysis

To the Editor

Segmented regression models are commonly used for ‘before-and-after’ comparison in interventional program evaluation [1]. Smiddy et al. [2] analysed the bloodstream infection (BSI) rate time series in exploring the change in their temporal trends before and after the implementation of observational hand hygiene auditing in Ireland. They reported that the decreasing trends of methicillin-resistant Staphylococcus aureus (MRSA) BSI rate significantly slowed, with P-value = 0.007, from 5% to 2% decrease per quarter pre- and post-intervention, respectively, see their Table 2. In other words, they found that the structural break, i.e. the change in slope in [2], in the decreasing trends of MRSA BSI rate is statistically significant. However, by observing the temporal trends presented in their Figure 2 (the panel at row 1 and column 2), I considered this structural break is not obvious, and it may not reach statistical significance. In this study, I re-analysed the data in [2] to examine the likelihood of the structural break in the decreasing trends of MRSA BSI rate.

The raw data in [2] were unavailable, and thus I digitised the panel at row 1 and column 2 in Figure 2 of their work using WebPlotDigitizer (version 4.2, https://automeris.io/WebPlotDigitizer). I considered two regression models as follows. They were

• the baseline model: E[ln(BSI_t)] = α₀ + α₁t + α₂Gap, and

• the full model: E[ln(BSI_t)] = β₀ + β₁t₁ + β₂t₂ + β₃Gap, which was the segmented regression as the same as in [2].

The E[⋅] denoted the expectation and the BSI_t denoted the MRSA BSI rate at the tth time interval. The settings, meaning and interpretation of other notations were exactly the same as in [2], but the fitting was conducted with full likelihood frameworks. In this re-analysis, I examined the following three aspects relevant to the estimate of a structural break in BSI_t. They included

• the consistency, i.e. the effect size and statistical significance, between the β₁ and β₂ estimates in this study and those in [2],

• the consistency between the statistical significance of the change in slope, i.e. (β₁ − β₂), in this study and that in [2], and

• the statistical significance of the likelihood-ratio (LR) test of the full model against the baseline model.

All analyses were carried out in R software (version 3.6.0) [3].

I reported that the β₁ and β₂ estimates were consistent with those in Table 2 of [2]. Different from the significant level estimated in [2], I found the P-value = 0.088 appears not statistically significant at the 5% level, against P-value = 0.007 < 0.05 in [2], for the change in slope, which indicated the difference in slopes was statistically unclear. Moreover, I noticed that the 95% confidence intervals (95%CI) of exp(β₁) (from 0.93 to 0.97) and exp(β₂) (from 0.97 to 0.99) estimates in [2] were extremely close. This might also be a sign of no difference in β₁ and β₂, and thus a structural break may not have occurred. For the LR test, the P-value = 0.066, which implied the structural break unlikely occurred. As shown in Figure 1 (of this study), the fitting results of the baseline and full models almost overlapped.

Fig. 1.

The fitting results of the association between MRSA BSI incidence per 1000 bed days used (BDU) in Ireland from 2009 to 2016. The black dots are the observed data in Figure 2 (the panel at row 1 and column 2) of [2]. The blue curves are the fitting results from the full model and the green curves are the fitting results from the baseline model. Bold curves are the mean fitting results and the dashed curves are the 95%CI.

I note that different fitting framework, as well as, the testing methods were adopted in this study and in [2]. The likelihood inference with LR test was used in this study, whereas the robust estimation with Wald test was carried out to obtain an estimate of the variance (Huber-White) for the data as a whole in [2]. However, as pointed out in [4],

‘white robust standard errors are universally used in econometrics, (and) their finite sample properties lead to over-rejection under the null hypothesis, sometimes by a large amount.’

This difference in thecourse of analysis probably leads to the inconsistency in the estimates or testing outcomes.

Although this inconsistency between Smiddy et al.'s findings and the findings in this study was unlikely to affect the main conclusions in [2], the inconsistency in analysis outcomes from different model setting or frameworks should be considered with caution.

Author contributions

SZ conceived the study, carried out the analysis, discussed the results, drafted the first manuscript, critically read and revised the manuscript, and gave final approval for publication.

Financial support

This work was not funded.

Consent for publication

Not applicable.

Disclaimer

The funding agencies had no role in the design and conduct of the study; collection, management, analysis and interpretation of the data; preparation, review or approval of the manuscript or decision to submit the manuscript for publication.

Ethical standards

Ethical approval or individual consent was not applicable.

Availability of data and materials

The raw data digitised from the panel at row 1 and column 2 in Figure 2 of [2], and R codes were both available via the online supplementary files.