Conducting sensitivity analyses to identify and buffer power vulnerabilities in studies examining substance use over time

Abstract

Introduction:

A priori power analysis is increasingly being recognized as a useful tool for designing efficient research studies that improve the probability of robust and publishable results. However, power analyses for many empirical designs in the addiction sciences require consideration of numerous parameters. Identifying appropriate parameter estimates is challenging due to multiple sources of uncertainty, which can limit power analyses’ utility.

Method:

We demonstrate a sensitivity analysis approach for systematically investigating the impact of various model parameters on power. We illustrate this approach using three design aspects of importance for substance use researchers conducting longitudinal studies – base rates, individual differences (i.e., random slopes), and correlated predictors (e.g., co-use) – and examine how sensitivity analyses can illuminate strategies for controlling power vulnerabilities in such parameters.

Results:

Even large numbers of participants and/or repeated assessments can be insufficient to observe associations when substance use base rates are too low or too high. Large individual differences can adversely affect power, even with increased assessments. Collinear predictors are rarely detrimental unless the correlation is high.

Conclusions:

Increasing participants is usually more effective at buffering power than increasing assessments. Research designs can often enhance power by assessing participants twice as frequently as substance use occurs. Heterogeneity should be carefully estimated or empirically controlled, whereas collinearity infrequently impacts power significantly. Sensitivity analyses can identify regions of model parameter spaces that are vulnerable to bad guesses or sampling variability. These insights can be used to design robust studies that make optimal use of limited resources.

1. Introduction

Advances in longitudinal study design and analysis have greatly increased understanding of causes and effects of substance use with respect to a number of cognitive, affective, and behavioral processes and associated mental and physical health characteristics (e.g., Trull & Ebner-Priemer, 2013; Wilhelm, Perrez, & Pawlik, 2012). At the same time, these discoveries generally require a great deal of time, human, and monetary investment. It is critical that studies of substance use behaviors and disorders be robustly designed to maximize informational value (Witkiewitz, Finney, Harris, Kivlahan, & Kranzler, 2015).

A priori power analysis (Cohen, 1962, 1988, 1992) is one method for enhancing empirical reliability that has recently gained renewed attention in the addiction sciences (e.g., Addiction, 2018; Hallgren & Witkiewitz, 2013; National Institutes of Health, 2015; Tackett et al., 2017). In brief, power analysis usually consists of estimating anticipated effect(s) of a planned study and calculating the sample size necessary to have some predetermined probability (e.g., 80%) of correctly rejecting the null hypothesis (see Cohen, 1988, 1992; Maxwell, 2004; for detailed discussions of statistical power). Appropriately powering a planned study has a number of benefits, including reducing Type II error, improving the ability to distinguish true nulls from underpowered effects, increasing precision, and ensuring optimal resource utilization.

Several tools exist for estimating power for some basic and more complex designs, such as G*Power (Faul, Erdfelder, Lang, & Buchner, 2007) and Optimal Design (Raudenbush, Spybrook, Liu, & Congdon, 2011). In addition, we have recently elaborated a flexible simulation-based power analysis method (see also, Bolger & Laurenceau, 2013; Gelman & Hill, 2006; Muthén & Muthén, 1998–2017). This approach is flexible to any model specification and is particularly useful when closed-form power calculation tools are unavailable. A detailed discussion of this approach (and corresponding syntax) is available in Lane and Hennes (2018).

Despite the availability of simulation- and equation-based tools, power analyses often entail considerable uncertainty on a number of dimensions. Many analytic models employed by substance use researchers, such as multilevel (e.g., Fitzmaurice, Laird, & Ware, 2012; Raudenbush & Bryk, 2002) or robust designs (e.g., Liang & Zeger, 1986), entail observations at multiple levels of analysis, individual differences in associations across clusters/individuals, and a large number of additional parameters that may significantly impact power but are infrequently reported. Despite known mathematical associations between certain statistical constructs and power (e.g., base rates and variability; Preacher, Rucker, MacCallum, & Nicewander, 2005), the magnitude of these effects as a function of model type and other parameters is essentially unexplored. Researchers often have poor intuitions about different parameters’ impact on power (Bakker, Hartgerink, Wicherts, & van der Maas, 2016), and there are few available and accessible tools that elucidate such relationships. Where available, they are generally limited to variations in between-subjects sample size and a single effect size (e.g., G*Power; Faul et al., 2007). Additionally, researchers must consider the phenomenology of processes of interest, such as how often use occurs, whether it is con-founded/collinear with other behaviors (e.g., co-use), and how often use will be observed given the assessment schedule.

As a result, even with available power estimation tools, challenges in identifying appropriate point estimates across a long vector of parameters can lead to highly misleading sample size determinations. Conducting a single power analysis using a “best guess” does not reveal vulnerabilities in the estimated model where misjudgments may lead to significant under- or overestimates of required sample size (nor areas where misjudgments have limited impact). Without this knowledge, a researcher will not have the requisite information to know whether additional pilot testing or literature review is necessary to obtain more precise parameter estimates, nor whether to consider alternative study designs that buffer power. Individual power analyses, while incredibly useful, may give researchers a sense of false confidence, leading them to make design decisions that do not maximize statistical power and methodological efficiency.

To address these challenges, the current study demonstrates the utility of sensitivity analyses, in which power analyses are conducted across a multivariate range of parameter estimates (see also Bolger, Stadler, & Laurenceau, 2012; Gelman & Hill, 2006). Rather than relying on heuristic recommendations that may not generalize to their model or in which the magnitude of impact on their own data is unknown, sensitivity analyses enable researchers to explore their own empirical models to independently make efficient design decisions.

1.1. Current study

We draw on results reported in two articles that adopt different longitudinal designs to demonstrate the added benefit of a sensitivity analysis approach over individual power analysis. The first employed a multi-year longitudinal design to characterize differences in hangover trajectories as a function of sex and family history of alcoholism (Piasecki, Sher, Slutske, & Jackson, 2005). The second used an ambulatory assessment design (Trull & Ebner-Priemer, 2013) and examined temporal associations between alcohol/cannabis use and affect (Trull, Wycoff, Lane, Carpenter, & Brown, 2016). We demonstrate a series of sensitivity analyses that a researcher might initiate if they aimed to replicate findings from either article in subsequent research. We adopt the simulation approach described elsewhere to conduct all power and sensitivity analyses (e.g., Bolger & Laurenceau, 2013; Gelman & Hill, 2006; Lane & Hennes, 2018; Muthén & Muthén, 1998–2017).

We use the authors’ statistical models and parameter estimates as representative starting points, and systematically vary:

• Number of participants and assessments per participant (Examples 1 and 2)

• Variability (i.e., base rates, frequency, distributions) of model predictors (Examples 1 and 2)

• Collinearity (i.e., co-use, correlation) of predictors (Example 2)

• Individual differences (i.e., random slopes) in the association between model predictors and outcomes (Examples 1 and 2)

In light of classic power formulae (Cohen, 1988), we hypothesize that, 1) base rates that maximize variability, 2) lower correlations between predictors, and 3) smaller individual differences will be associated with increased power. Based on recent research using similar designs (Lane & Hennes, 2018; see also Rast & Hofer, 2014; Rouder & Haaf, 2018), we hypothesize that 4) increasing participants would benefit power more than increasing the number of assessments per participant. However, given the nonlinear effects of varying such parameters as a function of effect size (c.f., Lane & Hennes, 2018), it was unclear the extent to which these two parameters would buffer power in the context of base rates, correlated predictors, or random slopes in the current examples of substance use over time and in daily life. Thus, the findings from the current simulations can inform studies characterized by similar design characteristics, possibly across substantive domains. At the same time, the aim of the current manuscript is not merely to report specific findings regarding these limited study characteristics, but to illustrate how researchers can adopt such practices to independently optimize their designs.

2. Method

2.1. Example 1 – Effects of family history on hangover frequency in early adulthood

Piasecki et al. (2005) examined hangover frequency (0 = no past-year hangover, 8 = 40 or more hangovers in the past year) across 6 assessments spanning 11 years (Years 1, 2, 3, 4, 7, and 11) among 486 college freshmen (at study entry). They modeled linear trajectories over time as moderated by participants’ family history of alcoholism (51% family history positive [FHP; family history negative = FHN]; 1 = FHP, 0 = FHN) and sex (53% female; 0 = female, 1 = male). They also included a time-varying covariate of the number of heavy-drinking days in the past month and its interactions with family history and sex. Four hundred ten participants were assessed at Year 11, indicating 16% missing data at the final wave. We assumed a constant (1.7% per year) rate of dropout to generate an approximate pattern of missing data.

We focus on the interaction between family history and year, which the authors report to be negative and significant (b = −0.05, p < .05; Table 1). This intriguing finding warrants replication. While the authors replicate previous findings indicating that FHP individuals are at higher risk for hangover in college (e.g., Newlin & Pretorious, 1990), their data suggests that this risk diminishes as individuals enter adulthood, consistent with an interpretation of FHP as a “developmentally limited” risk factor of hangover (Sher & Gotham, 1999).

Table 1.

Model estimates, observed power, and N required for 80% power to replicate each effect using the same assessment schedule for the linear growth model of hangover frequency reported by Piasecki et al. (2005).

Main effects	Estimate	Observed power	N for 80% power
Intercept	1.42***	100%
Family History	0.53**	89%	368
Sex	0.42*	72%	588
Linear Slope	−0.06***	90%	363
Linear Slope * Family History	−0.05 *	67%	655
Linear Slope * Sex	0.05*	71%	609
Heavy-drinking Covariate	1.62***	100%	40
Heavy-drinking * Family History	−0.30*	57%	827
Heavy-drinking * Sex	−0.51***	95%	294
Random effects			Variance
Person intercept			2.79***
Person linear slope			0.02***
Person heavy-drinking covariate			0.70***

Note. Bold-type indicates the effect of interest for the simulation analyses.

^*p < .05.

^**p < .01.

^***p < .001.

Researchers interested in conducting replications or extensions of this study would need to make a number of potentially costly design decisions, such as number of participants, number and timing of assessments, distributions of gender, family history, and heavy-drinking within the sample, and inclusion of additional covariates or moderators. However, the magnitude of impact of such decisions on power may not be obvious. Here we illustrate sensitivity analyses exploring the impact of a few of such factors, (a) sample size, (b) number of assessments (c) FHP proportion, and (d) individual differences in the linear slopes over time (that are unexplained by FHP or sex), on power to replicate the linear slope by family history interaction effect. Before examining these factors, we first simulated datasets that reproduced the reported effect sizes of Piasecki et al. (2005) as closely as possible. Syntax and details about this approach are provided in Appendix A.

2.2. Example 2 – Effects of alcohol and cannabis use on positive affect among individuals with borderline personality or depressive disorder

Trull et al. (2016) report results from a sample of 93 psychiatric outpatients diagnosed with either borderline personality or depressive disorders who reported using alcohol and/or cannabis at least once (on average) over the course of 27 days. Individuals were randomly prompted six times each day and asked to report, in part, on their felt positive affect (1 = very slightly or not at all, to 5 = extremely) and their alcohol and cannabis use (1 = yes, 0 = no) since the last prompt. On average, individuals completed 144.5 prompts each. The authors report parameter estimates for a three-level multilevel model in which current and previous occasion substance use (level-1), current and previous day substance use (level-2), and person average substance use (level-3), along with covariates, were used to predict positive affect.

The authors find statistically significant positive associations between positive affect and alcohol use on the current day, but significant negative associations between positive affect and alcohol use on the previous day (b = −0.17, 95% confidence interval [−0.33, −0.02]; Table 2). In light of the concurrent positive effect of alcohol use, the negative lagged effect could be interpreted in line with negative re-inforcement models and would be important to replicate given established inconsistencies in observing such a mechanism across studies (Baker, Piper, McCarthy, Majeskie, & Fiore, 2004).

Table 2.

Model estimates, observed power, and N required for 80% power to replicate each effect using the same assessment schedule from Trull et al. (2016) of the associations between concurrent and lagged alcohol and cannabis use and positive affect.

Main effects	Positive affect		Observed power	N for 80% power
	Estimate	95% CI
Intercept	2.33***	[2.16, 2.50]	100%
Occasion level
Current occasion alcohol use	0.12***	[0.06, 0.18]	98%	48
Previous occasion alcohol use	−0.07*	[−0.14, −0.01]	66%	131
Current occasion cannabis use	0.02	[−0.11, 0.15]	6%	8716
Previous occasion cannabis use	0.02	[−0.05, 0.09]	8%	3134
Day level
Current day alcohol use	0.33***	[0.16, 0.49]	97%	49
Previous day alcohol use	−0.17 *	[−0.33, −0.02]	61%	143
Current day cannabis use	0.11	[−0.21, 0.43]	10%	1596
Previous day cannabis use	0.11	[−0.16, 0.39]	13%	1221
Person level
Degree of alcohol use	0.41	[−0.90, 1.73]	10%	1596
Degree of cannabis use	0.58	[−0.15, 1.32]	36%	266
Random effects				Variance
Person intercept				0.11***
Person(day) intercept				0.31***
Current occasion alcohol use				0.02
Previous occasion alcohol use				0.02
Current occasion cannabis use				0.08**
Previous occasion cannabis use				0.01
Current day alcohol use				0.07
Previous day alcohol use				0.04
Current day cannabis use				0.34
Previous day cannabis use				0.21

Note. 95% CI = 95% confidence interval. Bold-type indicates the effect of interest for the simulation analyses.

^*p < .05.

^**p < .01.

^***p < .001.

As in Example 1, a number of parameters in such a replication are under the control of the researcher. We illustrate sensitivity analyses examining the impact of (a) sample size, (b) number of assessments, (c) alcohol use base rate, (d) cannabis co-use, and (e) individual differences in the association between alcohol use and affect. As in Example 1, we first simulated datasets that reproduced the effect sizes reported in Trull et al. (2016). Syntax and details are provided in Appendix A.

2.3. Simulation procedure

For each model parameter of interest, we simulated hypothetical data using a range of possible population parameter values. This process was repeated 10,000 times for each combination of model values and subsequently analyzed using the models depicted in Eqs. S1 and S2. In each case, we coded the dichotomous statistical significance of each effect (ɑ = 0.05), and aggregated the dichotomous codes across the 10,000 simulated samples to create estimates of power for each parameter (see Lane & Hennes, 2018, for more information about this approach). Analyses were conducted using SAS 9.4 (SAS Institute, 2014; see Appendix A for specific syntax and parallel scripts in R).

3. Results

3.1. Initial power analyses

We first report the results of two individual a priori power analyses for replicating the negative effect of 1) FHP status on yearly linear trajectories of hangover (Example 1), and 2) previous day’s alcohol use on positive affect (Example 2). Observed power for the respective effects was approximately 67% (Example 1) and 61% (Example 2). We would need approximately 655 individuals observed across 6 waves to achieve 80% power to replicate the negative FHP by linear slope interaction (Table 1). We would need approximately 143 individuals observed across 27 days to achieve 80% power to replicate the negative effect of previous day alcohol use on positive affect (Table 2).

3.2. Sensitivity analyses: Number of individuals vs. number of assessments

We now illustrate how conducting additional sensitivity analyses can offer insight into optimizing design efficiency and buffering against “bad guesses”. We first report analyses examining the impact of adding participants (N - solid gray line) versus assessments (n - solid black line) for both studies (Fig. 1), holding total number of person-assessments constant. In Example 1, we would achieve 80% power (holding number of individuals constant at 486) by increasing the number of assessments up to 8 in the 11-year span (Panel A). In contrast, holding the number of individuals in Example 2 constant at 93 but increasing the number of assessment days shows that power begins to asymptote and does not reach 80% until 95 days. This indicates that even hundreds of assessments per person may be insufficient to power an effect given an insufficient number of participants (Panel B; c.f., Lane & Hennes, 2018; Rast & Hofer, 2014; Rouder & Haaf, 2018). The difference between Examples 1 and 2 is primarily due to the fact that increasing the number of assessments in Example 1 also increases the variance of the linear predictor of year. Together, these findings demonstrate the utility of sensitivity analyses to assess the benefit of increasing assessments given particular design characteristics. While in some cases increasing assessments can be valuable, often such efforts will not lead to appreciable benefits.

Fig. 1.

Power for the Piasecki et al. (2005) FHP by year interaction effect (Panel A) and the Trull et al. (2016) lagged effect of day-level alcohol use on positive affect (Panel B), assuming 25% (triangle - short dash) 50% (square - solid), and 75% (circle - long dash) proportions of FHP/drinking as a function of increasing participants (N) or assessments (n).

3.3. Sensitivity analyses: Base rates

Piasecki et al.’s (2005) sample was specifically recruited to have equal proportions of FHP and FHN individuals, which maximizes power (Fig. 1A). Trull et al. (2016) indicated that their sample reported drinking on approximately 25% of days, which is comparable to epidemiological estimates among drinkers in the United States (Dawson, Goldstein, Saha, & Grant, 2015). Fig. 1B indicates that if a researcher instead recruited participants who drank on average 50% of days, 80% power is achieved with only 77 participants or only 23 assessment days. However, increasing average drinking further to 75% of days does not further enhance power, but rather achieves similar power as using 25%-of-days drinkers.¹ This demonstrates that base rates increase power as a function of variability, not frequency, in substance use, and suggests that researchers could reduce study time and expense by developing a targeted recruitment strategy or altering assessment frequency to achieve a 50% rate of use.

3.4. Sensitivity analyses: Correlation between predictors (e.g., co-use)

The results reported by Trull et al. (2016) indicate minimal alcohol and cannabis co-use (see Appendix A for details on extracting this correlation from their reported results). Nevertheless, the larger literature suggests that daily alcohol and cannabis co-use can be quite large (r ≈ 0.60) in certain subpopulations (e.g., Walsh et al., 2004). Therefore, it may be unwise to assume the same low correlation in subsequent power analyses. But what estimate should be used, and how much does it matter? Fig. 2 shows that although larger correlations between daily alcohol and cannabis use are associated with lower power, the effect is small. There is less than a 15% difference in power between correlations of 0.00 and 0.70. Increasing participants is associated with consistent gains (Fig. 2 – black lines), but increasing assessments is associated with smaller and smaller gains (Fig. 2 – gray lines). This indicates that researchers may need not worry about acquiring an accurate estimate of collinearity in their power analysis, nor about the impact on power of recruiting co-users unless co-use is very high and/or the fixed effect size of interest is small (although they should also consider the substantive implications of co-use inclusion criteria).²

Fig. 2.

Power of Trull et al. (2016) lagged alcohol effect on positive affect as a function of the correlation between daily alcohol and cannabis use when increasing participants (black lines) or assessment days (gray lines).

3.5. Sensitivity analyses: Individual differences in the association between predictors and outcomes (i.e., random slopes)

Individual differences (e.g., in linear trajectories [Example 1]; between lagged alcohol use and positive affect [Example 2]) are estimated by modeling random slopes, but these parameters are infrequently reported. Therefore, a researcher may not have a good estimate of the random slope (s), intuitions about its impact on power, or strategies for buffering it. Fig. 3 shows that as the random slope magnitude increases, power decreases nonlinearly. Increasing participants (N) buffers against large random effects, however, only to a point. Increasing assessments (n) provides much less buffer, and the benefit it does have dissipates quickly. This illustrates that random slopes can substantially impact power, so it is worthwhile to obtain good estimates (e.g., via contacting authors or conducting pilot studies). Faulty estimates can be somewhat buffered by large samples; however, the number of observations may be unfeasible. The researcher might also consider reducing unexplained variability with moderators.

Fig. 3.

Power of Piasecki et al. (2005) Year*Family History interaction (Panel A) and Trull et al. (2016) lagged alcohol effect (Panel B) as a function of the random slope magnitude when increasing participants (black lines) and assessment days (gray lines).

4. Discussion

Power analyses represent a valuable tool for optimizing limited resources to conduct informative research. However, the impact of inaccurate parameter estimates on power calculations can vary dramatically across model dimensions and is rarely intuitive. Sensitivity analysis can complement power analyses by buffering study designs to vulnerabilities due to uncertainty. To be sure, sensitivity analyses fall prey to the same uncertainty limitations as individual power analyses, as they are simply iterative power analyses. However, mapping a landscape of possible scenarios enables researchers to identify opportunities to optimize study methodology (e.g. assessment schedule, recruitment restrictions, moderators) and provides insight into parameters for which effort to identify precise estimates is more and less necessary. Table 3 provides a list of (incomplete) recommendations for conducting sensitivity analyses that can help researchers understand and mitigate factors that influence the power of their studies.

Table 3.

General guidelines for conducting sensitivity analyses.

Step	Description
1.	Determine the complete statistical model intended to test the study hypotheses.
2.	Generate best guesses for all model parameters, based on published data, pilot studies, etc.
3.	Conduct a traditional power analysis, ensuring that the chosen method (i.e., hand calculations, software, simulation) fully accommodates the predicted model.
4.	Consider the parameters of the model in terms of (a) their centrality to the research question, (b) your certainty about their magnitude, and (c) their controllability by the researcher.
5.	Construct a range of possible values for each parameter of central interest, with endpoints corresponding to the smallest effect of interest and the largest effect that could practically be expected. Use your “best guess”, minimum, and maximum as the iterations of the sensitivity analysis. Additional values can be included if more resolution is desired.
6.	Repeat Step 5 for secondary parameters (e.g., random effects), with endpoints informed by their controllability and certainty.
7.	Conduct iterative power analyses across the permutations of secondary parameter values for each primary parameter value.
8.	Identify parameters based on these sensitivity analyses that have the largest impact on power to detect the hypothesized effects.
9.	Use this information to optimize study design (e.g., increase participants, increase assessments, adjust sampling frame, create inclusion criteria, increase reliability, add moderators).

Although not the primary goal of the current research, the simulations illustrated here also provide specific insights along several dimensions for studies that assess binary indicators (e.g., group membership [Example 1] or frequency of substance use [Example 2]) across time. Consistent with previous research (Lane & Hennes, 2018; Rast & Hofer, 2014; Rouder & Haaf, 2018), we find that, holding person-assessments constant, increasing participants is more valuable for power than increasing assessments. This is likely to be true more generally when there are expected individual differences in (within-cluster) associations of interest. However, the degree to which this is true depends on other factors, some of which we illustrate, that are difficult to infer despite well-known statistical rules. Conducting sensitivity analyses in these cases can be extremely helpful for determining if investigators are better off recruiting a few more individuals and scaling back on repeated assessments or sampling fewer participants many times.

We also demonstrate that base rates of substance use, because they directly translate to variability, can be critical for observing hypothesized associations. A sample of 93 participants similar in their drinking frequency to the average American drinker (i.e., 25% of days) did not drink enough to achieve 80% power to observe the association of interest even after 27 days of assessment. Increasing assessments had little benefitwhile increasing participants was more powerful. Alternatively, recruiting the same number of participants who instead drank on 50% of days would achieve adequate power (but recruiting those who drank on 75% of days would not). Because power increases as predictor variance increases, power is often optimized when participants are assessed approximately twice as often as use occurs.³ Researchers might thus adjust their inclusion criteria and/or their assessment schedule (or implement event-contingent assessments) to maximize power.⁴

In contrast, the correlation between predictors can be quite large with relatively little decrement in power, assuming that the fixed effect of interest is not small. However, faulty random slope estimates can have a strong impact on power. Because such effects are infrequently reported in the literature, researchers should make efforts to estimate them as accurately as possible or control their impact with large sample sizes or other methods (e.g., moderators; restrictive inclusion criteria).

5. Conclusions

Collecting the rich data made available by advancements in intensive longitudinal designs comes with an array of practical and conceptual design considerations that, when ignored or left to intuition, can sabotage a study’s likelihood of identifying true associations. We advocate for the use of power and sensitivity analyses to more fully understand the nuances of such designs and take preemptive action to ensure the robustness and reliability of studies conducted in addictions science.

HIGHLIGHTS.

• Sensitivity analysis is a tool for identifying model vulnerabilities.

• Base rates, collinearity, and random slopes were considered.

• Low/high base rates, large random slopes impede power; collinearity less so.

• Increasing participants buffer power more than increasing assessments.

• Sensitivity analyses can empower researchers to optimize study design.