Influence of random measurement error on estimated rates of headache chronification and remission

Abstract

Objective:

To examine the potential influence of random measurement error on estimated rates of chronification and remission.

Background:

Studies of headache chronification and remission examine the proportion of headache sufferers that move across a boundary of 15 headache days per month between two points in time. At least part of that apparent movement may represent measurement error or random variation in headache activity over time.

Methods:

A mathematical simulation was conducted to examine the influence of varying degrees of measurement error on rates of CM onset and remission. Using data from the American Migraine Prevalence and Prevention Study (AMPP), we estimated a starting distribution of headache days from 0 to 30 in the migraine population. Assuming various levels of measurement error, we then simulated two sets of data for time 1 and time 2. The “individuals” in this study were assumed to have no real change in headache frequency from time 1 to time 2. The observed variations in headache frequency were those influenced by imputed random variance to resemble typical measurement error or natural variability. Using this simulation approach, we estimated the amount of chronification and remission rates that might be attributed simply to statistical artifacts such as unreliability or regression to the mean.

Results:

As the degree of measurement error increased, the amounts of illusory chronification and remission increased substantially. For example, if the headache frequency of sufferers randomly varies by only 2 headache days each month due to chance alone, a substantial degree of illusory chronification (0.6% to 1.3%) and illusory remission (10.3% to 23.5%) rates are expected simply due to random variation.

Conclusions:

Random variation, without real change, has the potential to influence estimated rates of progression and remission in longitudinal migraine studies. The magnitude of random variation needed to fully reproduce observed rates of progression and remission are implausibly large. Recommendations are offered to improve estimation of rates of progression and remission, reducing the influence of random variation.

Introduction

Migraine and tension-type headache are thought to increase in frequency over time among a minority of individuals.¹ In this series, we consider a number of methodological and statistical issues that arise in assessing headache progression and remission. In the first paper in the series,² we introduced the argument that observed rates of headache chronification are confounded by measurement error in headache reporting, regression to the mean, and study design elements including extreme score selection using two measurement occasions. In the second paper in the series, we demonstrated how one such source of measurement error, rounding (or “heaping”) in the reporting of headache frequency, contributes to unreliable estimates of headache frequency and thus to errors in the classification of headache sufferers as either episodic (< 15 headache days/month) or chronic (≥ 15 days/month).³ To what extent might these multiple methodological threats influence estimated rates of progression and remission?

To address this question, we performed a mathematical simulation that examines the influence of random variation in headache frequency on estimated rates of CM onset and remission reported in the previously published studies on headache chronification.^4–9 In this simulation, we “created” individuals who suffer from 0 to 30 headache days/month and who were assessed at two time periods (T1 and T2). However, the individuals in this study were assumed to have no real or systematic change in headache frequency from T1 to T2. Instead, the observed variations in headache frequency were those influenced by imputed random variance. Using this simulation approach, we endeavored to determine the extent to which chronification and remission rates might be attributable simply to measurement or statistical artifacts such as unreliability or regression to the mean. Finally, recommendations are offered to reduce the impact of these issues in future chronification studies.

Methods

Using a modification of the methods elegantly described by Zhang and Tomblin,¹⁰ this simulation study was performed by generating data using random, computer-sampled draws from statistical distributions thought to resemble headache frequency in the population. As such, this research is not subject to ethical review by a human subjects review board. The population distributions from which the simulations were based were derived from those reported in published, peer-reviewed studies on headache chronification and remission.

Simulation Methods

The simulations involved three sequential conceptual steps as illustrated in Figure 1.

Figure 1.

A summary of the simulation methods. Step 1: The simulations created hypothetical individual headache sufferers by assigning them a headache frequency ‘true’ score. Step 2: A unique random error is added once to create an observed Time 1 score and then again to create a Time 2 score (ie, two random draws from a normal distribution with a mean of 0 and SD = σ). Step 3: The proportion of individuals that exhibited an episodic headache frequency at Time 1 and a chronic headache frequency at Time 2 (illusory chronification), or a chronic headache frequency at Time 1 and an episodic frequency at Time 2 (illusory remission), was calculated.

Step 1: Create a ‘true’ score for each simulated headache sufferer.

To create a set of true scores, we drew for each hypothetical individual a random number from 0 to 1 from a uniform distribution, ~U (0, 1). The resulting numbers in this uniform numerical dataset were then partitioned such that their distribution reflected the proportion of discrete monthly headache frequencies observed in the general population. In this way, each individual’s monthly headache frequency was commensurate with that expected among an individual randomly drawn from the general population. The simulation was conducted iteratively using each of three different assumptions (A to C, below) about the ‘true’ score population distribution of headache frequency.

A. Actual monthly headache frequency data from the American Migraine Prevalence and Prevention (AMPP) study.

This distribution matches the actual headache frequencies reported in the AMPP study.⁶ As demonstrated in a companion manuscript, this distribution contained ‘heaping’ of scores around headache frequency multiples of ‘5’ as individuals (particularly those with frequent headache) commonly rounded their reports of their ‘true headache frequency’ to a multiple of 5 (eg, 17 days/month might be rounded to 15 days/month).³ The AMPP distribution was used to illustrate the expected rates of chronification and remission potentially due to unreliability/regression artifact in the seminal work of Bigal et al.⁶ when using the actual study data. In this analysis, we explore a range of values for this unreliability and regression artifact because we do not know the actual magnitude of this variation.

B. The smoothed Negative Binomial Model of Headache Frequency (NBS)

This distribution is based on the companion manuscript that modeled headache frequency after removing the heaping that occurred around multiples of ‘5’, such that the data were smoothed and then modeled using a negative binomial distribution (as described in Houle et al, ³). This distribution was used to illustrate the chronification/remission rates that might be expected due to unreliability/regression artifact within a headache study in which participants did not exhibit a tendency to round their reports with increasing headache frequencies.

C. The smoothed Negative Binomial Model of Headache Frequency with Imputed Rounding Behavior (NBR)

This distribution is identical to the smoothed negative binomial except that the rounding model developed in our second manuscript of this series examining headache chronification was used to impute rounding back into the distribution (ie, the estimated probability that someone will round a score was used to create a distribution that had heaping in multiples of ‘5’).³ This distribution was used to illustrate the chronification/remission rates due to unreliability/regression artifact that might be expected within a headache study in which participants exhibited a known tendency to round their reports with increasing headache frequencies.

Step 2: Create error variance in the ‘true scores’ to produce observed scores.

So as to isolate the potential effects of measurement error on rates of chronification and remission, the hypothetical headache sufferer was assigned a ‘true’ score reflecting an unchanging rate of headache frequency. Of course, this ‘true’ score may not be the score that would actually be reported by the “individual,” due to natural but subtle fluctuations in frequency/variability over time, measurement error (unreliability), or other random influences that affect how the score is reported by the participant.

To introduce measurement error, for each participant a randomly drawn score was selected from a normal distribution with a mean of zero and some variance, ~N(0, σ). A normal distribution was selected because in classical test theory, measurement errors are considered random and thus normally distributed.¹¹ Even if some measurement error is not random, the assumption of a normal distribution seems justified given the sheer number of factors that could cause an individual to either report or actually experience subtle differences in headache frequency. Notably, the choice of distribution is crucially important to the present analysis. The extent to which month-to-month variability in headache frequency estimates is a function of headache frequency itself has not been critically examined. It is intuitive to postulate that higher headache frequencies are associated with greater fluctuation (ie, more error variance) than lower frequencies, and indeed the second paper in this series shows that higher frequencies are associated with greater heaping and thus likely higher variability.³ However, for the purposes of the simulation, a constant error was imposed across all levels of headache frequency (ie, the size of the error did not increase with increasing headache frequency).

The random error was added to the hypothetical sufferers’ ‘true’ score to create an observed score. For example, for a true score of 15 headache days/month, a random component was added to this error to produce an observed score centered on 15 ± the random error. This process was conducted twice to reflect two different measurement occasions, Time 1 (T1) and Time 2 (T2), both of which are impacted by a different realization of random error. Importantly, in using this procedure no actual change in the ‘true’ score has occurred or is being represented to have occurred. Any observed differences in headache frequency within a hypothetical headache sufferer between T1 and T2 are solely due to random errors that are impacting the observed score. In other words, in this simulation no participants are actually exhibiting headache chronification or remission as a function of any bona fide deterioration or improvement; any observed differences between the two occasions potentially suggestive of chronification or remission are simply due to error variance and regression artifacts.

Step 3. Estimate the degree of chronification/remission between Times 1 and 2 that is potentially attributable to random error or artifact.

Each sufferer’s observed score at Time 1 (ie, ‘true’ score + error) was used to categorize her into a headache frequency group. Individuals were dichotomized into either an episodic (< 15 headache days/month) or chronic (≥ 15 days/month) frequency group. Frequency of hypothetical headache sufferers who exhibited episodic headaches at T1 but chronic headaches at T2 was used to estimate rates of potentially illusory chronification. Similarly, the frequency of hypothetical sufferers who exhibited chronic headaches at T1 but episodic headaches at T2 was used to estimate illusory remission rates.

Running all simulations

The simulations were then conducted iteratively using all combinations of the aforementioned distributions and conditions. The chronification and remission rates within each simulation were estimated as a function of a sample size similar to that used in previous studies (N = 2000 headache sufferers). The influence of differing error variances in observed frequency scores was examined using SDs ranging from 0.25 to 6.0 (in increments of 0.25 SDs). A separate simulation run was conducted for each of the three ‘true’ score distributions, and for each scenario 100 replications were conducted (ie, each combination of factors was examined 100 times). Thus, simulations were run as a permutation of the 3 distributions (AMPP, NBS, NBR) and 24 headache frequency SDs (0.25 to 6.0 in increments of 0.25), for a total of 72 simulations, each of which was replicated 100 times. Table 1 summarizes the conditions used in the simulations.

Table 1.

Summary of Simulation Conditions

		‘True’ Score Distributions
Sample Size	Measurement Error (SD)	AMPP	Negative Binomial- Smoothed (NBS)*	Negative Binomial- Rounded (NBR)*
N = 2000	0.25 to 6.0 by 0.25	100 replications	100 replications	100 replications

^* Houle et al, 2013

Statistical Analysis

All analyses and simulations were conducted in SAS 9.2 (SAS, Inc., Cary, NC). The code to run the simulations is available upon request from the first author. All models were conducted separately for each replication (1 to 100) and aggregated for reporting. Chronification proportions, remission proportions, and predicted probabilities from the simulations are reported as the median result from the 100 replications per scenario, with the 5^th and 95^th percentile scores from the replications representing a 95% confidence interval (CI). Sensitivity analyses were also conducted as described below.

Results

Verifying the simulation

The simulations performed as expected in regard to the imposed true score distributions, error distributions, and sample characteristics. Appendix A contains several plots illustrating the outputs under a variety of conditions. Of great importance, there was no change in the mean or variance levels of headache frequency between T1 and T2 for respective distributions. Across all replications for all conditions, no T1 and T2 differed by more than ± 0.003 headache days/month, with differences of variances of less than 0.002 SDs. Thus, although there could be considerable error in the observed T1 and T2 scores (ie, between 0.25 and 6.0 SDs in respect to the true scores), there was no actual mean chronification or remission in headache frequency, such that the same underlying distributions were observed at each occasion (as designed).

Expected Illusory Chronification & Remission Rates

Because the only actual differences in T1 and T2 headache frequencies were those imposed on the true score by a random error process, the resulting observed rates of headache chronification and remission were only those due to illusory artifacts that did not reflect durable change. Although some participants temporarily crossed the chronic versus episodic threshold during data collection, this occurred because of inherent unreliability in the observed scores, dichotomization of the frequency distribution into extreme scores (< 15, ≥ 15 days/month), and use of only two measurement occasions wherein T1 is used to predict T2 and predicated on extreme score selection (as articulated in the first two papers of this series ^2,3).

Figures 2A and 2B illustrate the expected amount of illusory chronification and remission rates under varying levels of measurement error. As the amount of measurement error increased, the amount of illusory chronification increased substantially. It is difficult to know what level of measurement error is encountered in reality. In powering clinical trials for preventive medications, a typical assumption might be that active drug and placebo would differ by 0.5 SD. Less is known about the typical within-person variation in headache frequency from month to month. It is unlikely that the magnitude of random variation is large, relative to the magnitude of effect introduced by a medication. Using that as an example, when the amount of measurement error was SD = 0.5, rates of illusory chronification will range from 0.05% to 0.5%, depending on the true score distributional assumptions. In contrast, when the measurement error was larger, SD = 2.0, the rates of illusory chronification will range from 0.6% to 1.3%. The expected rates of illusory chronification were impacted by the assumptions underlying the true score distribution, such that the illusory rates were always highest in the NBR true score distribution, followed by the NBS distribution, and the AMPP distribution.

Figure 2.

A. The increasing rates of illusory chronification (A) and remission (B) with increasing error. As the standard deviation of the random fluctuations in observed headache frequency increases, the amount of illusory chronification and remission increases substantially. Three different underlying headache frequency distributions were simulated (AMPP data, negative binomial [NBS] model without rounding, negative binomial [NBR] with rounding to ‘5’). Random fluctuations in observed headache frequencies could occur due to natural variability in headache activity or measurement error (eg, rounding).

A much higher rate of illusory remission is expected under similar conditions. For example, when the amount of assumed measurement error was SD = 0.5, the rates of illusory remission will range from 0.8% to 9.0%, depending on the true score distributional assumptions. In contrast, when the measurement error was SD = 2.0, the rates of illusory remission will range from 10.3% to 23.5%. The expected rates of illusory remission were similar for the NBR and NBS true score distributions, and least among the AMPP distribution.

How much of the published chronification/remission rates are illusory?

This simulation was designed to estimate the expected degree of illusory chronification/remission rates when measurement error in the predominant study designs is considered. Because “real” chronification and remission processes would also serve to increase the amount of variability between T1 and T2 scores, the effects of chronification confound this approach to some degree, and it is thus impossible to compare the observed rates to these expected rates with certainty. However, even if the headache frequency of sufferers randomly varies by only 1 headache days each month (SD = 1), a substantial proportion of the published chronification rates (2.5% to 3.0%) and remission rates (15% to 57%) may instead reflect illusory chronification (rates of 0.4% to 0.7%) and illusory remission (rates of 5.8% to 15.1%) attributable to measurement error.

Discussion

This simulation study demonstrates the perils of estimating rates of headache chronification and remission using an experimental design with two measurement occasions in the context of measurement error, regression to the mean, and extreme score selection. Because the process of chronification involves an individual’s headache frequency to exceed both the population mean as well as their own historical levels of headache activity, it seems likely that random error fluctuations rather than regression to the mean plays a major role. In contrast, remission from a chronic headache frequency will be prone to regression to the mean that comes with extreme score selection. The simulation demonstrated that illusory estimates of both chronification and remission can be obtained merely as a function of the designs that have been used to study this issue. While it is clear that random variation can contribute to estimates of chronification and remission, interpretation of these results depends critically on the assumptions one makes about realistic levels of random variation in headache frequency. The simulation leaves us with more questions than answers and we are left to ask: What are the true rates of chronification and remission in the population?

To ponder this question, one must first focus upon the weaknesses of the current simulation. Unlike prior headache chronification research, these are not actual data collected from actual headache sufferers. The data used to estimate these rates are based on computer simulations that themselves are based on a number of assumptions that may not be entirely accurate. We represented the headache frequencies that might be encountered in a group of headache sufferers using distribution results fitted from the seminal AMPP study. To the extent that these distributions are reflective of the headache frequencies of patients presenting for treatment at any particular clinic is unclear. We also had to assume how much measurement error, or the nature of that measurement error (eg, normal variation that is constant across headache frequency), occurs in the average person over time. Given the limitations in these simulations, we encourage our readers to view these results as illustrations of the significant threats to this research paradigm, rather than inscrutable findings. In this context, it is impossible to conclusively determine what proportion of the rates of chronification/remission uncovered in previous studies actually are illusory; our findings nevertheless indicate that these illusory proportions represent a substantive threat regarding the accuracy of accepted rates of chronification and remission.

Despite these limitations, several conclusions can be drawn from this effort. At a minimum, this analysis raises the specter that some degree of the presently accepted rates of chronification and remission could be replicated by a data-generating process that assumes only random variation. Extending this finding, it will be crucial for future studies on this topic to reduce the proportion of the chronification/remission rates that could be due to either measurement error or statistical artifact. Because headache is a progressive disease in some individuals, these individuals should exhibit stable and longstanding changes that can be distinguished from temporary fluctuations in observed scores.

A second conclusion relates to the importance of rounding behavior for these illusory rates. The model where rounding behavior was imposed (NBR) exhibited substantially more illusory chronification/remission than wherein such behavior did not exist (NBS). This finding demonstrates that the observed tendency to round headache frequencies to multiples of ‘5’ has substantial impacts on the study of headache chronification. Simulations based on the actual AMPP data exhibited the least amount of illusory rates, and this is almost certainly due to the fact that this underlying distribution did not exhibit a full range of headache frequencies (ie, few/no individuals in that large sample reported a headache frequency of 8, 17, 18, 19, 27, and 29). It is difficult to interpret what these lower illusory rates reflect beyond than the idiosyncrasies of the selective reporting in this particular sample.

To improve the study of headache chronification (and remission), several lessons can be gleaned from this series of manuscripts to improve the future study of headache chronification/remission. The following recommendations are offered for researchers considering this endeavor.

1. Minimize Measurement Error

Any attempts to quantify headache frequency using more reliable methods than retrospective recall at two time-points will substantially reduce threats to the interpretations of headache chronification research. Because of the difficulties in estimating frequency rates,² the recommendations in Houle et al³ and Lipton et al¹² for estimating headache frequencies should be implemented in future headache chronification research. Of these, the recommendation to use the shortest recall period as possible, with repeated assessments (ie, use of a daily headache diary), has the most potential as a powerful solution to reduce unreliability.

2. Interpret designs that use only two measurement occasions with great caution

Although many elegant designs use two measurement occasions (eg, cohort study, case-control, pretest-posttest), having only two measurement occasions does not provide sufficient information on meaningful fluctuations in headache activity over time and also forces the researcher to make simple change calculations that can be fraught with difficulty (as evidence by this simulation). Instead, adding additional measurement occasions allows for evaluation of durable change and indexes any such change in the context of naturally-occurring variability in headache frequency. Vickers¹³ presents a nice introduction to this issue, especially in regard to measurement occasions with low correlations such as might be expected in headache frequency studies.

3. Ignore the ICHD Frequency Criteria, if Only During Analysis

By adhering to the ICHD diagnostic criteria^14,15 (episodic headache = < 15days/month; chronic headache = ≥ 15 days/month) in a statistical model, groups of participants are artificially created around a relatively arbitrary cut-off. As demonstrated, this practice creates problematic interpretations given expected sample dynamics of extreme scores, as well as for several other well-known reasons not addressed in this series.¹⁶ Instead, researchers should focus on analyzing the actual scores as continuous variables rather than an arbitrarily-created dummy-code reflecting group status. Any model predictions can be later imposed on the diagnostic criteria for clinical communication.

4. Model Trajectories, not just Transitions

Related to the above, modeling the probability that an individual crosses an arbitrary cut-off can certainly be useful, but given the issues involved in headache chronification research it is far superior to instead model the ‘trajectory’ of headache over time (for introduction to trajectory analysis see: ¹⁷). This approach focuses either on the longitudinal course of an actually observed value, such as headache frequency, or on an unobserved (ie, latent) concept that is instead reflected in several observed variables (eg, headache intensity, disability, or associated symptoms). For example, if a group of investigators utilized daily headache diaries, they could model the probability that an individual experienced a headache for each day during a 6-month time period. In this way, changes in the daily probability that an individual might experience a headache could be examined over a longitudinal course.

5. Always be Aware of Regression Artifacts

Regression artifacts, such as regression to the mean, will always be a competing hypothesis in observational studies. These effects are very easily overlooked and can be readily interpreted as anything other than what they are: a statistical inevitability whenever the level of one imperfectly reliable variable is used to predict the level of another. Interested readers are referred to Campbell and Kenny’s accessible text.¹⁸ We encourage these readers to be gentle with us when they discover previously unrecognized regression artifacts in our own published research results.

Appendix A.

A1.

The error variances for SD = 1. The plot illustrates a distribution of error variances for each true score from one of the simulation runs. The proportion of individuals in the simulation who received each level of error variance for Time 1 is illustrated using a curve (though the actual errors were discrete headache frequency values). Because the errors were randomly distributed, the means of Time 1 and Time 2 were virtually identical across simulations.

A2.

The illusory rates of progression for individuals with low headache frequency at Time 1 (<= 8 headache days/month) to progress to an intermediate frequency (9 to 14) or for incident chronic headache at Time 2 using the NBS distribution.