Near real-time surveillance for consequences of health policies using sequential analysis

Abstract

Background

New health policies may have intended and unintended consequences. Active surveillance of population-level data may provide initial signals of policy effects for further rigorous evaluation soon after policy implementation.

Objective

This study evaluated the utility of sequential analysis for prospectively assessing signals of health policy impacts. As a policy example, we studied the consequences of the Food and Drug Administration's warnings cautioning that antidepressant use could increase suicidal risk in youth.

Method

This was a retrospective, longitudinal study, modeling prospective surveillance, using the maximized sequential probability ratio test (maxSPRT). We used historical data (2000-2010) from 11 health systems in the US Mental Health Research Network. The study cohort included adolescents (ages 10-17) and young adults (ages 18-29), who were targeted by the warnings, and adults (ages 30-64) as a comparison group. Outcome measures were observed and expected events of two possible unintended policy outcomes: psychotropic drug poisonings (as a proxy for suicide attempts) and completed suicides.

Results

We detected statistically significant (p<0.05) signals of excess risk for suicidal behavior in adolescents and young adults within 5-7 quarters of the warnings. The excess risk in psychotropic drug poisonings was consistent with results from a previous, more rigorous interrupted time series analysis but use of the maxSPRT method allows timely detection. While we also detected signals of increased risk of completed suicide in these younger age groups, on its own it should not be taken as conclusive evidence that the policy caused the signal. A statistical signal indicates the need for further scrutiny using rigorous quasi-experimental studies to investigate the possibility of a cause-and-effect relationship.

Conclusions

This was a proof-of-concept study. Prospective, periodic evaluation of administrative healthcare data using sequential analysis can provide timely population-based signals of effects of health policies. This method may be useful to employ as new policies are introduced.

Introduction

Governments often introduce health policies in order to increase patient access to services including prescription medicines, improve safety and/or quality of care, and lower costs.^1-3 Such policies include, but are not limited to, formulary restrictions, copayments, prior authorization requirements, and US Food and Drug Administration (FDA) regulatory actions.^3-5

Policies can have intended and unintended consequences. Monitoring the outcomes of policies is an important public health issue. For instance, well-controlled studies demonstrated that prior authorization requirements in state Medicaid programs reduced initiation of effective medications for bipolar illness and decreased initiation of antidepressants.^{6, 7} While rigorous quasi-experimental approaches to policy evaluation such as interrupted time series (ITS) analysis are useful and important,^{1, 8} they require many pre- and post-policy data points to assess the statistical significance of a change in the rate of outcomes.¹ Therefore, these approaches are not optimal for the timely detection of possible policy effects.

Methods for real-time surveillance of safety of medical products have been developed and used by the Centers for Disease Control sponsored Vaccine Safety Datalink,^9-11 the Health Care Systems Research Network's (formerly HMO Research Network) Center for Education and Research on Therapeutics (CERT),¹² and the FDA-sponsored Sentinel initiative.^{13, 14} These methods could potentially also enable prospective surveillance of policy changes and timely detection of potential intended and unintended consequences of policies for subsequent study. Routine surveillance would enable policymakers to investigate sooner the possible unintended impacts of policies as needed.

The FDA is responsible for formulating policies to improve patient safety related to use of marketed drugs.¹⁵ When safety information emerges after drug approval, the FDA systematically evaluates and responds to it. If the FDA identifies new safety concerns, it must decide how to effectively communicate the information about risks to the general public and providers.¹⁵ These communications range from minor revisions to the label of the drug to boxed warnings – FDA's strongest warning about a drug or drug class – when risks may be severe.^{5, 15} FDA safety communications influence prescribing and medication use behavior, especially when they are widely publicized by the media.^16-18 For example, longitudinal studies found that the 2004 boxed warnings for antidepressants^{19, 20} and media reports^21-23 led to decreases in antidepressant use,^24-27 and there is evidence suggesting increases in suicide attempts among youth.²⁷ Increased suicidality may have resulted from under-treated depression or failure to increase outpatient monitoring of young people with depression.²⁸ Importantly, these outcomes were not reported years after the warnings.

This article describes our application of sequential analysis and the maximized sequential probability ratio test (maxSPRT)²⁹ to detect signals of effects of health policies. We studied the consequences of the FDA's warnings cautioning that antidepressant use could increase suicidal risk in youth.^{19, 20} Specifically, the objective of the present study was to evaluate the utility of prospective near real-time sequential analysis for quickly assessing the need for more rigorous confirmatory analysis of policy impact.

Methods

Study design

This was a retrospective, longitudinal study using sequential analysis and the Poisson maxSPRT²⁹ to simulate prospective surveillance. This study was part of a larger project designed to examine the effects of FDA boxed warnings and media reports regarding use of antidepressants and suicidal behavior.

Data source

This study used 2000-2010 data from 11 US healthcare organizations involved in the Mental Health Research Network (www.mhresearchnetwork.org).^{27, 30-32} Together the 11 healthcare systems provided care to a diverse population of 10 million people in 12 states. Members were enrolled through employer-sponsored insurance, individual insurance plans, and capitated Medicare and Medicaid programs. Members served by these systems are generally representative of each system's geographic service area. See our previously published article²⁷ for a comparison of demographic characteristics between the study cohort and the US general population.

Data were obtained from the Health Care Systems Research Network Virtual Data Warehouse (VDW).^33-35 The VDW has been developed and is maintained using a series of data standards and automated processes that guide the generation of similarly constructed data tables at each organization. In this federated data structure, common data definitions and formats facilitate sharing of de-identified data for research across multiple sites. The VDW data include demographics, health plan enrollment, inpatient and outpatient care utilization, diagnoses, procedures, and outpatient pharmacy data. At each site, source data are extracted from the health system's administrative and claims databases as well as electronic medical records. The VDW also includes date (month, year) and information on causes of death, derived from state death registries and internal health plan data. This study was approved by Institutional Review Boards at each participating site as well as from state departments of public health when required for the use of information from death certificates.

Intended and unintended consequences of policies

Intended consequences of a policy are defined by its goals. Unintended consequences of a policy, however, are difficult to classify because they are frequently side effects of the process of the policy change. Unintended consequences have been characterized by the following attributes by Rogers and Bloomrosen et al.:^{36, 37} (i) Desirable or undesirable effects: is the consequence positive, negative, or mixed (good in some ways and bad in other ways)? (ii) Anticipated or unanticipated effects: can the consequences be predicted or anticipated, and if so, by whom? Such consequences might include events that are easily anticipated, events that are only anticipatable by experts, and events that are completely unanticipatable by anyone (i.e., a total surprise). How anticipatable the consequences might be relates to the presence or absence of an event as well as its magnitude; (iii) Direct or indirect effects: does the policy change cause the consequence directly or is there a chain of events leading to it? It is worth noting that unanticipated, unintended consequences are frequently the result of an indirect causal chain; and (iv) Obvious or latent effects: is the consequence easily visible or does it become obvious only in a certain environment or at a later time?

Policy exposure

The intervention of interest in this study was the policy exposure. The FDA released several advisories beginning in 2003 followed by a boxed warning in late 2004 stating that use of antidepressants could increase risk of suicidality in youth and explicitly recommended monitoring of patients in the initial phase of treatment. These warnings were widely publicized by media reports.²¹ Many news stories emphasizing the risk of pediatric antidepressant use;²¹ thus, distorting the message of the well-intended safety warnings. Given the widespread media coverage, we considered the combination of FDA warnings and media reports as the policy exposure. Previous studies have found reductions in antidepressant treatment in all age groups following the warnings and no increases in use of treatment alternatives (e.g., psychotherapy, atypical antipsychotics),^{24, 27, 28, 38-40} suggesting an overall reduction in treatment of mood disorders. Because depression is an independent risk factor for suicidality and appropriate treatment with antidepressants is effective in reducing depressive symptoms,^41-43 the falling rates of treatment of mood disorder following the policy and lack of increased outpatient monitoring of youth with mood disorder⁴⁴ had the potential to increase suicide attempts and completed suicides at the population level – the two measures of unintended (indirect, latent) impacts of the policy considered in this study. Based on the messages of the warnings and management of mood disorders, Figure 1 outlines the conceptual model of the likely intended and unintended consequences of this policy exposure.

Figure 1.

Conceptual model of likely consequences of FDA warnings and media reports regarding antidepressants

In order to evaluate the impacts of policies, one key consideration is the precise timing of the policy. Unexposed person-time was defined as time before the policy (the baseline period; the first quarter of 2000 to the third quarter of 2003). The last quarter of 2003 to the last quarter of 2004 was considered a “phase-in” period that spanned the entire period of the FDA advisories, the boxed warning, and intense media coverage to deal with the possibility of an anticipatory response to the warnings. We did not compare observed and expected event rates during the phase-in period in the main analysis. The phase-in period allowed for patients and clinicians to learn about the evidence and consider changing their patterns of antidepressant use. Thus excluding the phase-in period assessed the effects of the warnings at ‘full strength.’ Anticipatory effects of policies are common. Without accounting for anticipatory effects, the signal might have been detected sooner or later but the interpretation might be difficult. We conducted a sensitivity analysis to include the phase-in period in the post-policy period. Policy exposed time began in the first quarter of 2005 and continued through the last quarter of 2010 for the poisoning outcome and through the last quarter of 2008 for the suicide outcome (we only had death data up to 2008 for all participating sites at the time of study due to lag in availability of such data). Because of delays in obtaining the completed suicide data from one study site, which provided ∼3% of the data, and to be consistent with our previously reported study, we used completed suicide data through 2008 for the main analysis. We also conducted a sensitivity analysis using suicide data up to 2009 but including only 10 of 11 study sites.

Study population

Study cohorts included adolescents (ages 10-17), young adults (ages 18-29), and adults (ages 30-64). We used these age cut-offs because the prevalence of serious suicidal thoughts, planning, and attempts is higher among young adults aged 18–29 years than among adults aged 30 years and older.⁴⁵ Adults were not targeted by the warnings and were therefore included in this study as a control cohort where no policy effect should be seen. To avoid introducing selection bias, we did not limit our cohorts to individuals with a coded depression diagnosis.

This is because previous studies showed that rates of depression diagnosis changed after the warnings,^{24, 38, 40} including in populations that are part of our study sample.⁴⁶ Further, outpatient claims are frequently incomplete for mental health conditions like depression.^{47, 48}

Outcome measures

While encounters for suicide attempts can be identified in administrative databases using external cause of injury codes (E-codes), they are incompletely captured in commercial plan databases.⁴⁹ Our previous analysis found that E-code completeness varied across study sites, across treatment settings, and across years.³⁰ Therefore, instead of deliberate self-harm E-codes, we used poisoning by psychotropic agents (International Classification of Diseases, 9th revision, Clinical Modification [ICD-9-CM] code: 969), a more reliable proxy for population-level suicide attempts.^{30, 49} Poisoning by drugs or toxic substances is the most frequent method of suicide attempt leading to hospitalization⁴⁹ and emergency room treatments.^{50, 51} Non-fatal poisoning by psychotropic drugs (predominantly tranquilizers) has a positive predictive value of 79.7% for suicide attempts (sensitivity was 38.3% and specificity was 99.3%), outperforming other injury/poisoning types.⁴⁹ In addition, ICD-9-CM coding is more stable over time as compared to E-codes and therefore can detect sudden changes in incidence rates in large populations over time. We identified deaths with suicide as a cause of death (ICD-10 codes: X60-X84, Y87.0), consistent with the algorithm used by the US Centers for Disease Control and Prevention.²⁷

To examine changes in suicide attempts following the warnings, we identified a rolling cohort of continuous enrollees (individuals who had enrolled for the full 90 days in the quarter). We calculated the quarterly numbers of enrollees and encounters in hospitals or in emergency rooms by these enrollees for psychotropic drug poisoning.

To examine changes in completed suicides after the warnings, we identified a rolling group of individuals who had enrolled at any time in a given quarter or the month immediately prior to that quarter. We did not require continuous enrollment because health plan membership is terminated by death. We then calculated the quarterly numbers of enrollees as denominators and completed suicides as numerators.

Calculating observed and expected events

For each quarter the analysis requires information about the number of observed events during the quarter as well as the expected number, under the assumption that the null hypothesis of no excess risk following the warnings is correct.²⁹

To calculate expected event rates, we used a Poisson regression with a linear term to model the temporal trend in the quarterly rates during the baseline and extrapolated the baseline trend to the post-warning period to calculate the expected event rates for comparison purposes for all post-warning quarters. A key assumption was that the baseline trend would have continued if the policy had not occurred.

Sequential Analyses

We performed analyses that mimicked quarterly prospective surveillance. Sequential analysis is used when there are repeated queries of the data over time, on a periodic basis (e.g., monthly, quarterly). Adjustments are made for multiple tests inherent in the repeated looks at the gradually accumulating longitudinal data. We used the Poisson maxSPRT in this study because it does not require an ‘a priori’ specification of the magnitude of the risk level under the alternative hypothesis.²⁹ The null hypothesis is rejected if there are sufficiently more observed cases than expected in the accumulated data to date. An event signal is generated the first time the log likelihood ratio (LLR) exceeds a critical value, B (i.e., when LLR(t)>B) during the post-policy period, indicating a statistically significant association between the policy and the outcome. Relative risk (RR) is the increased relative risk associated with the policy; RR was calculated as observed events divided by expected counts.

To establish the critical value, it is necessary to specify the alpha level and a pre-specified upper limit on the length of surveillance defined in terms of the expected number of observations (events) under the null hypothesis. We chose alpha to be 0.05 so that the overall probability of rejecting the null at any time during the surveillance was 0.05. We chose the length of surveillance to be a maximum of five years, which meant specifying a maximum of 2000 expected events under the null for psychotropic drug poisonings. That is, surveillance ends if and when a cumulative number of 2000 expected events has been reached without rejecting the null. The critical value was calculated using the R Sequential package (https://www.r-project.org/), and a signal was generated when the LLR exceeded 4.42 for psychotropic drug poisoning for all age groups. For completed suicides among adolescents and young adults, we specified a maximum of 100 and 500 expected events respectively under the null for completed suicides to correspond to approximately five years of surveillance; corresponding critical values are 3.95 and 4.22. We specified a maximum of 2000 expected events under the null for completed suicides among adults; the corresponding critical value is again 4.42. The Poisson models adjusted for health plan, sex, and educational status (defined by whether the individual resided in neighborhoods with less than 25% of the population having a college degree, Y/N). We conducted separate analyses for the three age groups.

Results

We studied approximately 1.1 million adolescents, 1.4 million young adults, and 5 million adults in each quarter. Characteristics of the three age groups included in the study have been reported previously.²⁷

Psychotropic Drug Poisoning

Table 1 presents the results using sequential analysis to assess the effect of the FDA warnings on psychotropic drug poisoning for all three age groups. Figure 2 displaysLLR and RR plots for psychotropic drug poisoning for the three age groups.

Table 1.

Sequential analysis results detecting signals in changes in psychotropic drug poisoning and suicide outcomes by age group

	Psychotropic drug poisoning			Suicide
	When critical value* was reached	Log likelihood	Relative risk	When critical value* was reached	Log likelihood	Relative risk
Adolescents	2007 Q4	5.12	1.34	2006 Q1	11.59	4.77
Young adults	2007 Q2	7.89	1.30	2006 Q3	4.62	1.79
Adults	N/A	N/A	N/A	N/A	N/A	N/A

^*Critical value = 4.42 based on alpha=0.05 and 2000 expected events under the null hypothesis for psychotropic drug poisonings for all age groups. Critical value = 3.95, 4.22, and 4.42 for adolescents, young adults, and adults respectively based on alpha = 0.05 and 100, 500, and 200 expected events under the null for completed suicides.

Figure 2.

Sequential analysis results on psychotropic drug poisonings among (a) adolescents, (b) young adults, and (c) adults.

The sequential analysis detected a signal in psychotropic drug poisonings in the last quarter of 2007 among adolescents and in the second quarter of 2007 among young adults. We did not detect a signal among adults for this outcome. Results remained the same in a sensitivity analysis that included the phase-in period (last quarter of 2003 through last quarter of 2004) in the post-policy period.

Completed Suicides

The results for completed suicides are shown in Table 1. Figure 3 displays LLR and RR plots for this outcome for all three age groups. Among adolescents, sequential analysis detected a signal in the first quarter of 2006 and among young adults, a signal in the third quarter of 2006. We did not detect a signal among adults for the suicide outcome. A sensitivity analysis that included the phase-in period (last quarter of 2003 through last quarter of 2004) in the post-policy period found the same results as the main analysis. Another sensitivity analysis that used data up to 2009 among 10 of 11 participating sites did not change the timing of the detected signals.

Figure 3.

Sequential analysis results on completed suicide among (a) adolescents, (b) young adults, and (c) adults.

Discussion

To our knowledge this is the first study to apply near real-time sequential analysis methods to examine potential unintended consequences of health policies. With the sequential approach, only pre-policy data are needed to estimate the expected count. Prospective, near real-time surveillance for selected outcomes can be implemented immediately after a policy change using accumulating data to rapidly assess the outcomes of policies. Using historical data, we modeled prospective surveillance for suicidality risk among youth following FDA antidepressant warnings and media reports. We assessed the timing of such signals and found that, had these innovative methods been employed prospectively, signals of increased risk of psychotropic drug poisoning (e.g., suicide attempts) – an unintended (indirect, latent) policy impact – might have been identified in adolescents and young adults within 10 quarters of the policy. Indeed, our previous study²⁷ of the same policy using ITS analysis and data from the same organizations also found increases in psychotropic drug poisonings among adolescents and young adults following the widely publicized FDA warnings, but not among adults. Our prior study was conducted in 2012 after the 2010 data had been collected; hence, we only detected the effect eight years after the warnings were issued. An ITS analysis of data only up to the end of 2006 (two years after the warnings) did not detect increases in psychotropic drug poisonings among adolescents, demonstrating that longer follow-up data were essential using this method.

We also detected signals of increased risk of completed suicides in adolescents and young adults within 5-7 quarters of the policy. This, however, might be a false alert because our time series study²⁷ did not detect increases in completed suicides after the policy. Sequential analysis works equally well for rare and common outcomes except for the difference in sample sizes that influences power and time to signal. On its own a signal from the sequential analysis should not be taken as conclusive evidence that the policy led to excess risk. A statistical signal indicates the need for further scrutiny using rigorous quasi-experimental studies (ITS for example) to investigate the possibility of a cause-and-effect relationship.

Near real-time sequential analysis may also be useful for other policy changes with possible intended and unintended consequences, for example, cost-containment policies (e.g., copayments, prior authorization) that are commonly used for prescription drugs. Other key benefits of the method relate to its use of routinely collected health plan encounter and dispensing data that are commonly used in health services and policy research, minimal data requirements in terms of needed data elements, the ability to simultaneously apply the method within a number of data systems and the use of a highly summarized data structures for aggregation across study sites.^{33, 35} Most public and private health insurers in the US have data that could support sequential analyses. Prospective surveillance of policies that affect commercially insured and/or publicly insured populations is therefore possible.

Our conceptual model presents an approach for designing policy research by examining different types of consequences. Ideally the possible consequences should be examined using data from the same database that contains sufficient observations before and after the policy; this guides the selection of databases for policy research. In our more rigorous quasi-experimental study, we examined three distinct outcomes (antidepressant use, psychotropic drug poisonings as proxy for suicide attempts, and completed suicides) using the same database and in a study population representative of the US general population.²⁷ There are a number of databases that also contain information on rates of suicide attempts in the US.⁵² These data, however, would not allow examination of multiple likely consequences of this policy in the same study population and some provide insufficient data points before and after the policy to control for secular trends (history bias). They would not meet the simple inclusion criteria of international systematic reviews.⁵³ The appropriateness of these databases for examining this policy is discussed in detail in our Counterpoint article.⁵⁴

There are potential study limitations. Developing and applying robust measures that are consistent over time to examine policy consequences are important and could be challenging. We used psychotropic drug poisonings as a proxy for suicide attempts. These poisonings include both suicidal and non-suicidal overdoses but they underestimate suicide attempts.³⁰ However, psychotropic drug poisonings were the most appropriate measure in our data for examining this policy over time because use of other metrics (self-harm E-codes and an established algorithm for suicide attempts) would introduce ascertainment bias. The limitations of this proxy measure are discussed in detail in our Counterpoint article.^{30, 49, 54} Our other measure – completed suicides in this policy example – derived predominantly from state death registries, is not vulnerable to similar limitations. Another challenge for policy evaluation is that there might be unanticipated consequences; thus, measures for such outcomes might not be developed and implemented a priori. Sequential analysis using automated healthcare claims data will only be useful if it has reasonable sensitivity and does not generate an unacceptable number of false positives. False-positive findings might be possible early in the study when relatively few data have accumulated and test statistics are less stable. We used quarterly data in this study because we studied rare outcomes. Many intended and unintended outcomes of policies are more frequent (e.g., therapeutic substitution); thus, prospective surveillance of policy impacts using sequential analysis could take advantage of administrative data that might be updated as frequently as weekly or monthly to allow near real-time detection of signals. This is a proof-of-concept study. This method should be tested for other FDA regulatory actions and policy changes. Further applications and accumulated experience with implementation, analysis and reporting of results would help investigators establishing methodological criteria to address issues of policy exposure, events, and setting a minimum number of observations before accepting a signal.

In summary, signals of excess risk for psychotropic drug poisonings in adolescents and young adults were detected after the warnings. These results were confirmed by our more rigorous previous study using interrupted time series analysis that can establish causal relationship.⁸ Our results support the continued investigation of sequential analysis as a potentially important tool for other health policy changes. Prospective, periodic evaluation of observational healthcare data can complement strong quasi-experimental studies as it holds the potential to rapidly identify early signals of intended and unintended effects of large-scale interventions on population health.