Reporting and Analysis of Process-of-Care Time Measures for Hyperacute Stroke Interventions

Abstract

BACKGROUND:

Process-of-care time measures provide important information about hyperacute stroke interventions and performance of systems of care. These measures are often highly skewed, and mean-based summary statistics may be misleading. Percentile-based statistics (eg, median) conveniently and unbiasedly summarize the proportion of patients treated within a given timeframe. Despite this, mean-based synthesis methods are primarily recommended by the Cochrane Handbook for Systematic Reviews of Interventions for meta-analyses. We aim to investigate the reporting and statistical comparison of process-of-care time measures in published hyperacute stroke trials and systematic reviews of trials and provide methodological foundation for meta-analysis in future studies.

METHODS:

A systematic scoping review of studies reporting or comparing time measures between events in the delivery of hyperacute stroke care was undertaken (Protocol DOI: 10.11124/JBIES-23-00136). Of 2321 studies identified through the database search, 146 studies were included. We analyzed the statistical agreement of transformation-based methods using process-of-care time data from hyperacute stroke clinical trials. We graphically and statistically compared mean- and percentile-based meta-analysis techniques for process-of-care time measures.

RESULTS:

Overall, 49 of 146 (34%) studies reported a process-of-care time measure using only the mean/SD. Under the normal distribution assumption, impossible (negative) time values would have been observed in 40 of 49 (82%) of those studies. Transformation-based imputation methods showed moderate agreement with the true SD (Lin Concordance Correlation Coefficient=0.88). Mean-based and median-based meta-analysis provided comparable results when analyzing summary measures of between-arm treatment effects and differed substantially when analyzing individual-arm summary measures.

CONCLUSIONS:

We recommend that stroke researchers: (1) summarize a sufficient set of process-of-care time measures using the median/interquartile range and use consistent rank-based analysis methods; (2) exercise caution in utilizing transformation-based methods to impute the mean/SD of process-of-care time measures; and (3) use median-based meta-analytical techniques for the synthesis of process-of-care time measures.

CLINICAL PERSPECTIVE.

What Is New?

• Process-of-care time measures are frequently reported in a way that does not encapsulate the nature of their non-normal distribution, which negatively affects the interpretability of published results.

What Are the Clinical Implications?

• Inconsistent or inaccurate reporting restricts the validity of comparison and analysis of the workflow efficiency and potential time-saving impact of hyperacute stroke interventions by the wider clinical audience.

• The use of percentile-based measures and median-based meta-analysis provides clinically interpretable and meaningful targets for process-of-care interventions, for example, reducing the upper quartile of delay.

In the hyperacute phase of stroke,¹ providing rapid intervention is crucial for improving patient outcomes.² For patients with ischemic stroke,² each minute of time between stroke onset and intravenous thrombolysis (IVT) saved is estimated to result in an average of 1.8 days of extra healthy life.^3,4 Treatment effects of IVT^5,6 and endovascular thrombectomy^7–10 in ischemic stroke and of reducing blood pressure in hemorrhagic stroke¹¹ are substantively modified by the time delay between stroke onset and intervention.¹² In particular, clear recognition of this fact led to the introduction of mobile stroke units^13–17 (MSUs, specialized ambulances equipped with a computed tomography scanner), which facilitate faster access to appropriate treatment in hyperacute stroke. Process-of-care time measures describe time periods between key events in the hyperacute stroke care pathway.¹⁸ The accuracy and consistency of the reporting, analysis, and quantitative synthesis (ie, meta-analysis) of such process-of-care time measures are of paramount importance¹⁹ for monitoring the performance of systems of care, proposing and evaluating potential improvements and time-saving strategies in hyperacute stroke care, and have an immediate impact on patient care guidelines.

Process-of-care time measures have a well-defined lower bound of zero as various events of interest follow each other and negative values for time periods between such events are not possible. Such measures are rarely normally distributed, often following skewed and nonsymmetrical distributions.²⁰ In the methodological literature, various process-of-care time measures have been described and modeled using a variety of non-normal distributions that capture their bounded and often positively skewed nature, including, for example, the exponential, Weibull, beta, and phase-type distributions.^21–24

The skewed nature of process-of-care time measures presents a significant challenge for their appropriate reporting, analysis, and meta-analysis due to the following reasons:

• Summarizing process-of-care time measures using the mean and the SD may not appropriately encapsulate the nature of a skewed time distribution.^19,25 Means and SDs are strongly affected by the presence of outliers and may substantially distort the understanding of the underlying process-of-care time distributions. In addition, without knowing or assuming the true underlying distribution of the data being summarized, the mean and SD do not provide clinical researchers with clinically interpretable information. Instead, describing these process-of-care time measures with the median and interquartile range (IQR) is recommended as more appropriate.^19,25 Unlike the mean and SD, the median and IQR are percentile-based measures and, as such, are clinically interpretable regardless of the true underlying distribution of the data. In addition, well-developed methods exist for the comparison of medians in a given sample of data.²⁶

• Despite existing median-based meta-analysis methods that estimate the pooled difference of medians,^27,28 the Cochrane Handbook for Systematic Reviews²⁹ of Interventions primarily recommends mean-based data synthesis approaches, estimating difference of means³⁰ for meta-analyses. According to these recommendations,²⁹ outcomes of individual studies reporting alternative summary measures (eg, medians and IQRs) can either be excluded from meta-analysis or only be included after the use of various transformation-based methods^31–33 to impute the mean and SD from the set of the reported alternative measures.

The combination of these reasons is particularly challenging for the reporting, analysis, and meta-analysis of process-of-care time measures as clinical researchers may be influenced by the Cochrane Handbook guidelines²⁹ to choose less appropriate methods (eg, only reporting mean and SD even in the case of a skewed distribution) for describing and analyzing such measures to ensure the inclusion of their studies in future systematic reviews and meta-analyses. Although the implementation of transformation-based methods ensures the inclusion of studies that report the median/IQR in mean-based meta-analysis, it still encourages the synthesis, analysis and interpretation of time metrics and results through a mean-based lens.

Despite recent encouragement for more standardization in reporting hyperacute stroke process-of-care time measures,¹⁹ there is varying practice within the research community as to how process-of-care time measures are to be reported and analyzed. This leads to uncertainty surrounding best practice and a lack of consistency among published studies.

To address this challenge, the aim of this study is 2-fold:

1. To understand how process-of-care time measures are being reported, compared, and analyzed in hyperacute stroke trials and systematic reviews of trials.

2. To provide methodological foundation for the appropriate reporting of process-of-care time measures in hyperacute stroke trials and the incorporation of these measures into future meta-analyses.

Methods

The data that support the findings of this study can be requested from the principal investigators of the original trials that were utilized for this analysis. The code executed within R (Version 4.3.2) used to conduct this analysis can be shared on reasonable request to the corresponding author.

To achieve aim 1, we conducted a systematic scoping review, which aimed to investigate the reporting and comparison of process-of-care time measures in hyperacute stroke trials and systematic reviews of trials (referred to as studies in the remainder of the article). To achieve aim 2, we first assessed the performance of existing transformation-based methods for imputing mean/SD based on median (IQR) using real-world process-of-care time data from hyperacute stroke clinical trials. We then investigated the performance of median-based data synthesis techniques for the meta-analysis of process-of-care time measures in hyperacute stroke.

This study did not require ethical approval as it does not report any individual-level patient data, and only summarizes previously published research findings or utilizes previously collected data for the purposes covered by existing ethical approvals.

Aim 1: Systematic Scoping Review

We followed the methodological framework for systematic scoping reviews described by the Joanna Briggs Institute³⁴ and reported according to the extension of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Statement for Scoping Reviews.³⁵ The protocol was published in Joanna Briggs Institute Evidence Synthesis.³⁶

As per the published protocol,³⁶ we aimed to estimate the following outcome measures:

1. The proportion of included studies that reported each different process-of-care time measure.

2. The proportion of included studies that reported process-of-care time measures using either mean and SD, median and IQR, or other methods.

3. The proportion of included studies where reported summary measures of mean and SDs resulted in impossible values of time (as time is a bounded and often positively skewed variable that cannot be negative) under the assumption of time being normally distributed (eg, if the SD is larger than the mean value, under the assumption of the normal distribution, this would result in at least 16% of the data being negative).

4. The proportion of included studies that compared process-of-care time measures between groups where this comparison is congruent with the method used to report the process-of-care time measures.

The comprehensive account of scoping review methods, including the inclusion criteria, search strategy and data extraction, can be found in the published protocol³⁶ and are provided in Systematic Scoping Review Methods in the Supplemental Material.

Aim 2: Investigating Agreement Between the True Mean and the Imputed Mean Process-of-Care Times Using Transformation-Based Methods

We statistically investigated the performance of transformation-based methods in imputing the mean and SD of process-of-care times from a combination of the median, IQR, range, and sample size utilizing process-of-care time data from hyperacute stroke clinical trials (as detailed in Summary of Data Sources in the Supplemental Material). Specifically, we implemented the quantile estimation (QE) method recommended by McGrath et al,³⁷ and as opposed to alternative transformation-based methods, it does not involve an assumption of normality in the underlying distribution of the data. As per existing literature, we utilized the transformation-based methods to impute the mean and SD from the following sets of reported summary statistics:

• Scenario 1.

  Minimum, median, maximum, sample size.

• Scenario 2.

  25th percentile, median, 75th percentile, sample size.

• Scenario 3.

  Minimum, 25th percentile, median, 75th percentile, maximum, and sample size.

We summarized the process-of-care time measures into 3 distinct epochs: (1) MSU-related times, referring to measures that are specifically recorded for MSU processes; (2) prehospital times, including all process-of-care times from stroke onset-to-hospital arrival; and (3) in-hospital times, including all process-of-care times from hospital arrival to revascularization.

To investigate the performance of transformation-based methods, we graphically and statistically analyzed the following outcome comparisons in R (version 4.3.2)³⁸:

1. True mean versus imputed mean process-of-care times: The agreement between the true mean derived from the original data set and the imputed mean that results from utilizing the transformation-based methods on different sets of reported summary statistics.

2. True SD versus imputed SD process-of-care times: The agreement between the true SD derived from the original data set and the imputed SD that results from utilizing the transformation-based methods on different sets of reported summary statistics.

3. Coefficient of variation versus relative error: The relative error is calculated as the difference between the true mean and the imputed mean divided by the true mean. The coefficient of variation³⁹ represents a standardized measure of dispersion around the mean and is calculated as the true SD divided by the true mean, with larger values indicating greater variability in the data. This outcome comparison is utilized to understand the impact of variability in the data on the successful imputation of the true mean through transformation-based methods.

For statistical analysis, we estimated Lin Concordance Correlation Coefficient (LCCC),⁴⁰ mean absolute error, and the slope and intercept of a standard linear regression model utilizing the true mean/SD as the independent variable for outcome comparisons (1) and (2). LCCC was utilized as the primary measure of agreement for this analysis, as it consolidates the assessment of both the precision and accuracy of the continuous measurements. The slope of the fitted regression line that is different to the slope of the perfect concordance line (slope=1) is indicative of the presence of proportional bias; for example, the error in the imputed mean/SD that changes depending on the value of the true mean/SD. In the absence of proportional bias, the intercept of the fitted regression line different to 0 represents the fixed bias, for example, the error in the imputed mean/SD that remains constant regardless of the value of the true mean/SD. For graphical analysis, statistical agreement is represented via the proximity of the individual points to the line of perfect concordance.

Aim 2: Investigating Agreement Between Median-Based Meta-Analysis Techniques Using Transformation-Based Methods

We conducted an illustrative statistical analysis to evaluate and compare the performance of median-based meta-analysis with mean-based meta-analysis using real-world hyperacute stroke process-of-care time data.

For this analysis, we utilized summary data from 6 published studies comparing the effect of MSU use on functional outcomes for patients with ischemic stroke undergoing thrombolytic therapy.^13,41–45 These studies included 2 randomized controlled trials, 2 nonrandomized controlled studies and 2 observational studies that had varying sample sizes and were previously included in a published meta-analysis.¹⁸ For illustrative purposes, we analyzed the onset-to-IVT process time results from these studies as they reported both the true mean/SD as well as the true median/IQR, enabling the implementation of transformation-based methods utilizing the scenario 2 set of summary statistics (25th percentile, median, 75th percentile, sample size). For this analysis, we utilized the Hozo,³¹ Luo-Wan³² (LW), and QE³⁷ transformation-based methods. We utilized the metamedian⁴⁶ package in R to implement these analyses.

For each individual study, as well as the pooled effect size across all 6 studies, we used true medians,²⁶ true means, and imputed means^31,32,37 (utilizing the 25th percentile, median, 75th percentile, sample size), to estimate the following effect measures:

1. Between-arm difference in mean/median onset-to-IVT time between MSU and control groups.

2. Individual-arm mean/median onset-to-IVT time for the MSU group.

3. Individual-arm mean/median onset-to-IVT time for the control group.

To illustrate the performance of different meta-analytical approaches, we graphically summarized the between-arm differences and individual-arm summary measures for the true means, true medians, and imputed means through a multipanel forest plot. To statistically analyze the performance of the meta-analytical approaches based on transformation-based methods, we estimated the bias and relative efficiency of the pooled imputed means compared with the pooled true mean for the between-arm difference in mean onset-to-IVT time and individual-arm mean onset-to-IVT time (for MSU and control arms). Bias is calculated as the absolute difference between the imputed mean and the true mean and represents a systematic difference between the expected value of an estimator and the actual population average. Relative efficiency is calculated as the SE of the imputed mean divided by the SE of the true mean and compares the precision of an estimator and the true population value. Relative efficiency larger than 1 represents the factor increase in the 95% CI width of the imputed mean compared with the true mean. For example, if the relative efficiency has a value of 1.5, the 95% CI for the imputed mean will be 50% wider than the CI for the true mean.

Results

Aim 1: Scoping Review

Summary of Included Studies

We identified 2321 studies through the systematic database search of PubMed, EMBASE, 3 clinical trial registries (ANZCTR [Australia & New Zealand Clinical Trials Registry], ISRCTN [International Standard Randomised Controlled Trial Number Registry], and ClinicalTrials.gov), as well as additional reference list searches. From 1744 unique identified studies, 328 (18.8%) full-text articles were assessed for eligibility after title and abstract screening and 146 (8.4%) studies were extracted and included in this review. This included 121 (82.9%) individual trials and 25 (17.1%) systematic reviews or review articles. A PRISMA 2020 flow diagram is provided to describe the results and movement of sources through the study selection process and is provided in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow chart in the Supplemental Material.

Frequency of Reporting of Different Process-of-Care Time Measures

The most frequently reported time epochs among all included studies started at either the last known well (LKW) or the onset. Due to their conceptual similarity, for the purposes of aim 1, we jointly referred to these time points as onset/LKW. The most commonly reported time was onset/LKW-to-needle (ie, intravenous thrombolytic administration) time, which was reported in 64 of 146 (43.8%) studies, and onset/LKW-to-puncture (ie, endovascular therapy) time, which was reported in 40 of 146 (27.4%) studies (Figure 1). Each individual study reported a median of 3 distinct process-of-care time measures.

Figure 1.

Number and proportion of included studies that reported each individual process-of-care time measure as defined by the conceptual model in Figure S1. Rows represent the activity start time and columns indicate the activity end time. Cells represent the process-of-care time measures according to the conceptual model in Figure S1 starting at the time indicated by the row and finishing at the time indicated by the column. Cell shading indicates the frequency of the reported time measure. EMS indicates emergency medical services; ER, emergency room; EVT, endovascular thrombectomy; and LKW, last known well.

Among the 121 included individual trials, the most frequently reported time periods were onset/LKW-to-needle time, which was reported in 50 of 121 (41.3%) studies, and onset/LKW-to-door time, which was reported in 32 of 121 (26.4%) studies. Among the 25 included systematic reviews, the most frequently reported time epochs were onset/LKW-to-needle time, which was reported in 14 of 25 (56%) studies, and onset/LKW-to-puncture time, which was reported in 13 of 25 (52%) studies.

Methods Used to Report Process-of-Care Time Measures

The most frequently observed combination of methods (eg, mean, SD, median, IQR, range) used by included studies to report process-of-care time measures was the median and IQR, which was utilized by 83 of 146 (57%) studies overall (Table 1). The combination of the mean and SD was utilized by 34 of 146 (23%) studies overall. In addition, the combination of the mean, SD, median and IQR was utilized by 9 of 146 (6.2%) studies. There were 15 (10%) studies that utilized different methods to report different process-of-care time measures within the 1 study.

Table 1.

Proportion of Included Studies That Utilized Different Combinations of the Mean, SD, Range, Median, and IQR to Report Process-of-Care Times, Stratified by Type of Study (Individual Trial or Systematic Review)

Among the 121 of 146 individual trials included in this review, the most used method of reporting process-of-care time measures was the median and IQR, which was utilized by 76 of 121 (62.8%) studies. Among the 25 of 146 systematic reviews included in this review, the most used method of reporting process-of-care time measures was the mean and SD, which was utilized by 9 of 25 (36%) studies.

Of the 49 of 146 (34%) studies that utilized the mean and SD (along with any other combination of methods) to report process-of-care time measures, 40 of 49 (82%) included at least 1 measure that was reported with values that resulted in an impossible value of time under the assumption of the normal distribution. This includes 23 of 32 (72%) individual trials and 15 of 17 (88%) systematic reviews.

Methods Used to Compare and Analyze Process-of-Care Time Measures

Overall, 78 of 146 (53.4%) included studies either included a statistical group comparison or meta-analysis based on process-of-care time measures. The most utilized statistical method used by these studies to analyze or compare process-of-care time measures across groups or to synthesize measures across studies was the Mann-Whitney U test, utilized by 37 of 78 (47%) studies, followed by a difference of means approach utilized by 13 of 78 (17%) studies (Table 2). Of the 78 of 146 included studies that performed a statistical analysis, 14 of 78 (18%) utilized a method that was not congruent with the method that was used to report the process-of-care time measures that were being analyzed (eg, reporting process-of-care time measures with the median and IQR but statistically comparing them utilizing the difference of means).

Table 2.

Proportion of Included Studies That Utilized Different Methods for Statistical Group Comparison of Process-of-Care Times, Stratified by Type of Study (Individual Trial or Systematic Review)

Aim 2: Performance of Transformation-Based Methods

The imputed means (Figure 2A) from each of the transformation-based methods showed strong agreement with the true means (LCCC, 0.999 [95% CI, 0.998–0.999]), mean absolute error, 3.22; slope, 1.02; intercept, −0.54). The imputed SDs (Figure 2B) from each of the transformation-based methods showed moderate agreement with the true SDs (LCCC, 0.88 [95% CI, 0.80–0.92]), mean absolute error, 8.53; slope, 1.13; intercept, −1.4). As the coefficient of variation (the ratio of the true SD to the true mean) increased (Figure 2C), the absolute relative error (the difference between the true mean and the imputed mean divided by the true mean) also increased. Figure 2 demonstrates the results for the QE transformation method when imputing the mean and SD using the scenario of the median, IQR, and sample size (scenario 2).

Figure 2.

Graphs for agreement for the estimates produced from the quantile estimation transformation-based method using the median, interquartile range and sample size (Scenario 2) compared with the true means and SDs. A, Statistical agreement between the true mean (calculated from the real data) and the imputed mean from the transformation-based method. Statistical agreement is indicated via the proximity of the individual points to the line of perfect concordance. B, Statistical agreement between the true SD (calculated from the real data) and the imputed SD from the transformation-based method. C, The relationship between the coefficient of variation, calculated as the ratio of the true SD to the true mean, and the relative error, calculated as the difference between the true mean and the imputed mean divided by the true mean. The coefficient of variation represents a standardized measure of dispersion around the mean, with larger values indicating greater variability in the data. DIRECT-SAFE indicates A Randomized Controlled Trial of DIRECT Endovascular Clot Retrieval Versus Standard Bridging Thrombolysis With Endovascular Clot Retrieval Within 4.5 Hours of Stroke Onset; EXTEND, Extending the Time for Thrombolysis in Emergency Neurological Deficits; MSU, mobile stroke unit; STOP-MSU, Stopping Haemorrhage With Tranexamic Acid for Hyperacute Onset Presentation Including Mobile Stroke Units; and TASTE-A, Tenecteplase Versus Alteplase for Stroke Thrombolysis Evaluation Trial in the Ambulance.

Aim 2: Performance of Median-Based Meta-Analysis Methods

Figure 3 illustrates the performance of median-based and mean-based meta-analytical techniques in estimating effect sizes for the meta-analysis of both the individual mean/median onset-to-needle time as well as the difference in mean/median for onset-to-needle time across the included studies.

Figure 3.

Multipanel forest plots. For 6 individual studies, as well as the pooled effect size across all 6 studies, each panel represents the effect measure being synthesized as follows: (A) between-arm difference in mean/median onset-to-intravenous thrombolysis (IVT) time between mobile stroke unit (MSU) and control groups, (B) individual-arm mean/median onset-to-IVT time for the MSU group, (C) individual-arm mean/median onset-to-IVT time for the control group. Within each panel, studies are arranged by sample size in ascending order. For each study, results are reported for the following data synthesis methods (indicated by color): (1) difference of true medians, (2) difference of true means, (3) difference of imputed means (Hozo Method³¹), (4) difference of imputed means (Luo-Wan [LW] Method³²), and (5) difference of imputed means (quantile estimation [QE] Method³⁷).

Figure 3A represents the difference between the mean/median onset-to-needle time across the MSU and control treatment groups and shows strong agreement between the different data synthesis methods across all 6 individual studies, as well as for the pooled estimate across all studies. Figure 3B and 3C represent the individual mean/median onset-to-needle time for the MSU and control treatment groups, respectively, and demonstrate disagreement in the effect size estimation between the true medians and true means. There was strong agreement in the effect size estimation between the imputed means utilizing the Hozo transformation method, and the imputed means using the LW transformation method; however, neither of these showed strong agreement with the true mean. In addition, there was strong agreement in the effect size estimation between the imputed means utilizing the QE transformation method and the true means. The level of agreement in the effect size estimates between different data synthesis methods was consistently larger for studies with a larger sample size. These results were consistent across all 6 studies and the pooled estimates.

Table 3 summarizes the bias and relative efficiency of the transformation-based methods in estimating the pooled true mean across all 6 individual studies for the between-arm difference and the individual-arm onset-to-IVT time for the MSU and control arms. The 3 transformation-based methods demonstrated similar unbiased estimates of the mean (Bias: Hozo method, 0.37; LW method, 0.35; QE method, −0.48). These estimates were more efficient (narrower 95% CI) compared with the true mean (relative efficiency: Hozo method, 0.76; LW method, 0.76; QE method, 0.98) for estimating the between-arm difference in onset-to-IVT time. For the individual-arm onset-to-IVT time, the QE transformation method provided an unbiased estimate of the true mean (MSU, 0.47; control, 1.62), whereas both the LW (MSU=−9.05, control=−5.89) and Hozo (MSU, −9.39; control, −6.29) methods showed more bias. All 3 transformation-based methods provided less efficient estimators than the true mean for both the MSU (Hozo, 1.09; LW, 1.09; QE, 1.41) and control (Hozo, 1.21; LW, 1.22; QE, 1.16) arms.

Table 3.

Bias and Relative Efficiency of Imputed Means Compared With the True Mean for the Between-Arm Difference in Mean Onset-to-IVT Time and Individual-Arm Mean Onset-to-IVT Time (for MSU and Control Arms) for Each Transformation-Based Method for the Pooled Estimate Across All 6 Individual Studies

Discussion

In the systematic scoping review part of this study, we found that published hyperacute stroke studies consistently captured process-of-care times that begin with either stroke onset or LKW time or hospital arrival (door) time, but other process-of-care measures were rarely reported. In addition, there was a clear lack of diversity among the process-of-care time measures that were reported, with each study only reporting a median of 3 distinct measures (IQR, 1–4). Only reporting a select few process-of-care time measures and omitting several key decision-making time points restricts the ability of stroke researchers to analyze the efficiency of hyperacute stroke interventions reliably and consistently and to conduct informative meta-analyses.¹⁹

Are the Most Appropriate Methods Being Used for Reporting and Statistically Comparing Process-of-Care Time Measures?

When reporting process-of-care time measures, published hyperacute stroke studies consistently utilized one of either the median and IQR (57%) or the mean and SD (23%), with a clear lack of studies utilizing both types of measures simultaneously (6.2%), which has been previously recommended by minimal reporting guidelines.¹⁹ Furthermore, most individual trials appropriately reported process-of-care times using the median and IQR (62.8%), whereas the majority of systematic reviews utilized the mean and SD (36%). Of the 49 of 146 (34%) studies that reported process-of-care times using the mean and SD (including those that additionally utilized any combination of other methods), the majority (82%) reported at least 1 process-of-care time measure with values that would result in impossible (negative) values of time under the assumption of the normal distribution. In total, 78 (53.4%) included studies (57 trials and 21 systematic reviews) performed a statistical group comparison of process-of-care time measures, with most of these studies utilizing the Mann-Whitney U test (47%) or a difference of means approach (17%) for this comparison. We found that 14 of 78 (18%, 10 trials and 4 systematic reviews) of these studies utilized an analysis method that was not consistent with the method that was used to report the process-of-care time measures being analyzed (eg, reporting with the mean and SD but utilizing the Mann-Whitney U test for statistical comparison). Individual trials that performed a statistical comparison of process-of-care time measures more often utilized a rank-based comparison method, such as the Mann-Whitney U test (63%). In contrast, systematic reviews that performed a statistical comparison through data synthesis of process-of-care time measures more often utilized a mean-based method, such as the difference of means (52%).

The observed difference in reporting process-of-care time measures between individual trials and systematic reviews indicates that while in the majority of individual trials, researchers use medians and IQRs for reporting, they are predominantly utilizing mean-based meta-analytical techniques for synthesis of these measures in systematic reviews. This is further reflected in the highlighted disparity between the rank-based methods that are utilized for the statistical comparison of process-of-care time measures in individual trials versus mean-based methods in systematic reviews.

The findings of this study illustrate that process-of-care time measures are frequently reported in a way that does not encapsulate the nature of their non-normal distribution. This negatively affects the interpretability of the published results,^19,25 as inconsistent or inaccurate reporting restricts the validity of comparison and analysis of the workflow efficiency and potential time-saving impact of these interventions by the wider clinical audience.

Is It Safe to Rely on Mean/SD Imputation Techniques for Process-of-Care Time Measures When Means/SDs Are Not Reported?

When means and SDs are not reported, transformation-based methods can successfully and accurately impute the mean (LCCC, 0.999), and to a lesser extent, the SD (LCCC, 0.88), from the 3 common sets of rank-based summary statistics that were considered. Transformation-based methods are most accurate at imputing the mean and SD for symmetrical distributions with a low mean-to-SD ratio, and they may become unreliable for skewed distributions such as those that are expected for process-of-care time measures. For the analysis of process-of-care times, which we expect to have a higher mean-to-SD ratio due to the skewed nature of time distributions, these methods are unlikely to be as accurate. This has implications for the applicability of transformation-based methods to the analysis of process-of-care time measures in hyperacute stroke studies, as despite their recommendation by the Cochrane Handbook,²⁹ they may not provide valid inputs for incorporation in meta-analysis, especially in cases of a higher mean-to-SD ratio.

What Are the Implications for Meta-Analysis of Process-of-Care Time Measures?

Overall, median-based and mean-based (including all transformation-based methods) data synthesis methods provided similar results when implemented for the illustrative meta-analysis of the difference in onset-to-IVT time between treatment arms, as illustrated in Figure 3A. However, as demonstrated in Figure 3B and 3C, we found substantial differences in results when utilizing mean-based versus median-based methods for the synthesis of individual-arm results. The emergent differences between the 2 methods that appear when analyzing individual-arm results underline the importance of implementing median-based synthesis methods where possible for process-of-care time measures.

Out of the 3 transformation-based methods that were investigated in this study, the QE method³⁷ provided the most accurate estimate of the true mean when synthesizing individual-arm results, as shown in Table 3. This is to be expected given the ability of this method to account for lack of normality in the underlying data when compared with the Hozo³¹ and LW³² methods, which rely on an assumption of normality for the underlying distribution of the data.

Based on the above, the following recommendations to stroke researchers can be provided. Although this study utilized the specific context of process-of-care time measures in hyperacute stroke to address this problem, our recommendations are generalizable to other data that are known to be bounded and skewed, such as time-related measures in other healthcare domains. Furthermore, the implications of these recommendations are extended to all types of studies, including registry-based and quality-based studies, that report or analyze process-of-care time measures.

Recommendation 1: Summarize a Sufficient Set of Process-of-Care Time Measures Using the Median/IQR and Use Consistent Rank-Based Analysis Methods

Future studies should follow the published minimal reporting standards for workflow process-of-care time measures¹⁹ to ensure that this detail is accurately captured. The reporting of a sufficient quantity of distinct process-of-care time measures has implications for the identification of workflow improvements in the hyperacute stroke care process,¹⁹ specifically ensuring that improvements in efficiency can be tracked longitudinally and compared across studies.

Process-of-care time measures should be reported and statistically compared utilizing the median and IQR to avoid any potential inaccuracies and inconsistencies in the interpretation of their distribution. Utilizing solely the mean and SD for reporting and statistically comparing measures that naturally follow a non-normal distribution may lead to lack of validity in the interpretation of results, as it does not encapsulate the nature of their underlying distributions. Although it is known that these measures commonly follow skewed non-normal distributions, stroke researchers may simply not be aware of the statistical reasoning behind utilizing mean-based summary statistics versus rank-based summary statistics when reporting this type of data. There should be further education for stroke researchers about the use of mean-based versus rank-based summary statistics for skewed data. This includes the fact that, despite the Central Limit Theorem permitting reasoning based on means and SDs for any distribution, this does not make them informative, as opposed to quantiles such as the median and IQR that are clinically interpretable regardless of the true underlying distribution of the data.

Recommendation 2: Exercise Caution in Utilizing Transformation-Based Methods to Impute the Mean and SD of Process-of-Care Time Measures

Future studies should avoid where possible the use of transformation-based methods to impute the mean and SD of process-of-care time measures for the purpose of analysis or inclusion in mean-based meta-analysis. Stroke researchers may be driven by the Cochrane guidelines for meta-analysis,²⁹ which recommend mean-based approaches to data synthesis, which may lead researchers to utilizing the recommended transformation-based methods to impute missing mean and SD values for process-of-care time measures. These methods are most accurate at imputing the mean and SD for data that follows a symmetrical distribution, while process-of-care time measures are generally skewed, and as such mean-based analysis is less reliable or informative.

The use of transformation-based methods may be acceptable under limited conditions, such as for data with a highly symmetrical and unbounded (ie, no maximum or minimum value) underlying distribution. It is important to emphasize that although a large sample size guarantees the normality of means themselves, it does not guarantee that means are interpretable in the presence of skewness in the underlying distribution. If researchers choose to use transformation-based methods regardless, it is recommended that methods, such as the QE method,³⁷ which do not rely on the assumption of underlying normality, are utilized for data where there is expected to be a lack of normality in the underlying distribution.

Recommendation 3: Use Median-Based Meta-Analytical Techniques for the Synthesis of Process-of-Care Time Measures

Researchers should implement median-based meta-analytical techniques^27,28,46 for the synthesis and analysis of process-of-care time data in future hyperacute stroke reviews. Overall, an increase in the utilization of median-based meta-analysis approaches will enable studies to report rank-based summary statistics, to validly represent data with skewed distributions, such as process-of-care time measures in hyperacute stroke, and still be included in future meta-analyses. Further education and awareness are required for stroke researchers on the utilization of median-based data synthesis as an alternative to imputing the mean and SD from the median and IQR using transformation-based methods. This will address the existing preference of researchers to report mean-based summary statistics in individual trials in the hopes of enabling their trial to be included in future systematic reviews and meta-analyses. This is exacerbated by the fact that stroke researchers may be driven by the Cochrane guidelines for meta-analysis,²⁹ which recommend mean-based data synthesis approaches, and therefore may report mean-based summary statistics in an attempt to ensure that their studies are included in future meta-analyses. Use of median-based analysis avoids the unrealistic CI estimates produced by mean-based analysis, which often extend into negative territory, thereby improving clinical interpretation. The use of percentile measures also provides a more useful assessment of variation when undertaking quality improvement activities, as, for instance, reducing the upper quartile of delay may be a relevant target for intervention.

In conclusion, we recommend that stroke researchers report a sufficient set of process-of-care time measures in studies of hyperacute stroke interventions, and that they are summarized using the median/IQR and consistent rank-based methods are used for analysis. Second, we recommend that stroke researchers avoid transformation-based methods where possible for imputing mean and SD values for process-of-care time measures; however, if necessary, use methods that are appropriate for skewed data (eg, QE method³⁷). Lastly, we recommend that stroke researchers use available methods for median-based meta-analysis to ensure that such studies are not excluded from systematic reviews.

ARTICLE INFORMATION

Sources of Funding

None.

Disclosures

None.

Supplemental Material

Scoping Review Methods

Summary of Data Sources for Transformation-Based Methods Analysis

PRISMA Flow Chart