Outcome Measures Utilized in Clinical Trials of Interventions for Post-Cardiac Arrest Syndrome: A Systematic Review

Abstract

BACKGROUND

The post-cardiac arrest syndrome (period of critical illness following return of spontaneous circulation [ROSC]) is a promising window of opportunity for clinical trials of therapeutic interventions to improve outcome from cardiac arrest. However, the methodological rigor of post-ROSC trials and the ability to compare or pool data on treatment effects across studies requires consistent and appropriate outcome measures. We aimed to determine the current degree of uniformity of outcome measures in clinical trials of post-ROSC interventions.

METHODS

We conducted a systematic review of Cochrane Library, MEDLINE, EMBASE, CINAHL, conference proceedings, and clinical trial registrations using a comprehensive strategy. We identified experimental or quasi-experimental trials testing post-ROSC interventions in adults. Four authors independently extracted data and assessed study quality using standardized instruments.

RESULTS

The search yielded 33 potential studies, of which 13 randomized controlled trials (n=1937) were included in the final analysis. Seven trials tested pharmacologic therapies and six tested non-pharmacologic therapies. Our main finding is that heterogeneity in the selection and reporting of outcomes limited comparability of results across studies. No two trials used exactly the same primary outcome, and timing of measurement varied widely. We found only two commonalities: (1) indices of functional survival were used rather than survival alone, and (2) ordinal scales of neurological function were collapsed into clinically meaningful groups (“good” versus “bad” outcome).

CONCLUSION

Currently there is a lack of uniformity in selection and reporting of outcome measures among trials of post-ROSC interventions. Achieving consensus would be an important advance for resuscitation science.

INTRODUCTION

Sudden cardiac arrest represents the most common lethal manifestation of cardiovascular disease.^{1, 2} Although the vast majority of clinical research on therapeutic interventions for cardiac arrest to date has been focused on techniques to achieve return of spontaneous circulation (ROSC) [e.g. cardiopulmonary resuscitation (CPR) or defibrillation methods], it is now recognized that therapeutic interventions initiated after ROSC has been achieved can have a profound effect on outcome.^{3, 4} Thus the post-cardiac arrest syndrome,^{5, 6} defined as the period of critical illness immediately following ROSC, has emerged as a crucial link in the classical "Chain of Survival" paradigm for treating cardiac arrest, as well as a promising window of opportunity for clinical trials of novel therapeutic interventions.⁷

As state-of-the-art therapy for post-cardiac arrest syndrome continues to evolve, selection of valid and well-accepted outcome measures will be essential for the design of clinical trials. The methodological quality of individual trials will be dependent upon the reliability and reproducibility of the outcome measures utilized, and uniformity in outcome measures is needed in order to permit comparison of treatment effects across studies. Recently, the resuscitation science community identified the potential lack of uniformity in outcome measures as an important problem for cardiac arrest trials in general and the American Heart Association convened a scientific conference on the topic.⁸ However, we know of no previous study that has examined the degree of heterogeneity in outcome measures among cardiac arrest clinical trials in a systematic fashion. Whether or not there is uniformity in the use of outcome measures for clinical trials of interventions for the post-cardiac arrest syndrome is currently unclear. Understanding the current degree of homogeneity among outcome measures in post-ROSC trials could have important implications for trial design in the future.

The aims of this study are (1) to perform a systematic review of the literature comprised of a comprehensive search strategy and standardized analysis techniques in order to determine the outcome measures utilized in clinical trials of post-ROSC interventions; (2) to compare the outcome measures across trials; and (3) to determine whether or not there is commonality in outcomes reported in these trials.

METHODS

Search and identification of studies

We followed a written protocol that was based on published guidelines⁹ and finalized prior to beginning the search. Specifically, we searched MEDLINE (1965–October 2008), EMBASE (1974–October 2008), CINHAL (1982–October 2008), and the Cochrane Library, using the search terms (cardiac arrest or heart arrest or post-resuscitation or post-cardiac arrest or post-cardiac arrest syndrome) and clinical trial. In order to identify potential unpublished data from clinical trials that have completed enrollment, we also (1) hand searched abstracts from the Resuscitation Science Symposium (American Heart Association Emergency Cardiovascular Care Committee) and the European Resuscitation Council Congress from 2006–2008; (2) reviewed published practice guidelines for post-cardiac arrest care¹⁰; and (3) searched websites containing details on clinical trials registration (National Library of Medicine – ClinicalTrials.gov and the World Health Organization – International Clinical Trials Registry Platform). Registered but unpublished trials were considered eligible for inclusion if the registration website indicated that enrollment in the clinical trial had been completed. We contacted the Principal Investigators of the unpublished studies identified above for clarification or data extraction as needed. Additionally, we consulted with two independent experts in the field of cardiac arrest to identify potential unpublished data. We also screened reference lists of the articles selected for inclusion to identify additional studies for potential inclusion.

Inclusion Criteria

We considered studies eligible for review regardless of language or publication type. We included experimental studies (randomized control trials) or quasi-experimental studies (prospective before-and-after trials, prospective controlled before-and-after trials, or interrupted time series analysis [not using historical controls]) of adult populations (age>17 years) with cardiac arrest treated with an experimental intervention after ROSC was achieved. We included studies with both (a) an intervention arm testing either a therapy or therapeutic strategy, and (b) a clearly defined control arm in which subjects received placebo or standard of care therapy. We excluded studies that were secondary reports of previously published trials, studies that tested therapies outside the acute phase (<24 hours) of post-cardiac arrest care (e.g. implantable defibrillators for secondary prevention), and studies with primary outcomes that were not patient-oriented (e.g. indices of feasibility of a therapeutic strategy, pharmacokinetic indices, etc.). We also excluded papers that were reviews, correspondence, editorials, and nonhuman studies; however, we screened the reference lists of review articles to identify further studies for inclusion. We attempted to contact corresponding authors for clarification of data extraction or quality assessment if not clear from the published article or abstract.

Study Selection and Data Abstraction

Two independent reviewers (ST and JHK) screened the titles and abstracts of identified studies for potential eligibility.¹¹ After the relevance screen the two reviewers compared their exclusion logs to determine whether there was disagreement and used the Kappa statistic to quantify the interobserver agreement. In cases of disagreement, a third reviewer (AEJ) assessed the abstract and a consensus was reached by conference between the three reviewers. All studies deemed potentially relevant were obtained and the full manuscripts were reviewed for inclusion. Two reviewers (BMF, BWR) independently abstracted data on all patient populations, interventions, and outcome measures using a standardized data collection form. Any disagreements in these processes were resolved by consensus with a third reviewer (ST).

Assessment of quality

We assessed the quality of the studies selected for inclusion using the Cochrane Collaboration’s tool for assessing the risk of bias in clinical trials evaluating four domains¹²

• Random sequence generation: Grade A = Sequence was adequately generated; Grade B = Sequence was inadequately generated; Grade C = Unknown

• Concealment of allocation: Grade A = Performed concealment allocation; Grade B = Did not use concealment allocation; Grade C = Unknown

• Blinding: Grade A = Investigators blinded to allocation; Grade B = Investigators not blinded to allocation; Grade C = Unknown. In cases where blinding was not feasible at the point of intervention (e.g. therapeutic hypothermia) we assigned a Grade A if the investigator collecting the primary outcome was blinded to the treatment allocation.

• Selective outcome reporting: Grade A = Free of suggestion of selective outcome reporting; Grade B = Suggestion of selective outcome reporting; Grade C = Unknown

Analysis

We performed a primarily qualitative analysis of the data in accordance with the recommended methodology for qualitative reviews published in the Cochrane Handbook.¹³ We collated and summarized all outcome measures, including primary and all secondary outcomes, in table format stratified by individual publication. We produced a summary result of how often each specific outcome was selected as a primary or secondary outcome measure. In order to determine which outcomes could potentially be comparable across trials, we also determined how often each outcome measure was reported anywhere in the results section of the included manuscripts. We performed a separate (pre-planned) secondary analysis of outcome measures among studies that were high quality (defined as receiving a Grade A in at least 3 of 4 risk of bias domains above).

This study was approved by the Institutional Review Board at Cooper University Hospital.

RESULTS

Search and selection

The comprehensive search yielded a total of 3851 potentially relevant publications. Details of the search and study selection are shown in Figure 1 and Table 1.

Figure 1.

Search, inclusion, and exclusion flow diagram.

Table 1.

Primary reasons for study exclusion.

	Number of reports
Excluded via relevance screen	3818
Excluded after full manuscript review
Not an experimental or quasi-experimental study	12
Wrong population	5
Secondary analysis of previously published study	2
Feasibility study only	1
TOTAL	20

Inclusion

After the relevance search (interobserver agreement κ=0.87) a complete manuscript review was performed on the remaining 33 articles. One full length manuscript in Spanish was translated to English, but failed to meet inclusion criteria. Thirteen studies were included in the final analysis with a total sample size of 1937 subjects.^{3, 4, 14–24} The included studies appear in Table 2.

Table 2.

Experimental or quasi-experimental clinical trials of post-cardiac arrest interventions.

FIRST AUTHOR	YEAR	n	DESIGN	THERAPY STUDIED	HIGH QUALITY RCT*	PRIMARY OUTCOME MEASURE	SECONDARY OUTCOME MEASURE(S)
Oksanen23	2007	90	RCT	Strict glycemic control	YES	Survival at 30 days	Biomarkers (serum NSE) at 24 and 48 hours
Kuisma20	2006	32	RCT	Normoxic ventilation [versus hyperoxic ventilation]	NO	Biomarkers (serum NSE and S-100 protein) at 24 and 48 hours	Systemic oxygenation indices
Laurent21	2005	61	RCT	High-volume hemofiltration	YES	Survival at 180 days	Good neurologic outcome (Cerebral Performance Category 1 or 2) at 180 days; Death by intractable shock
Damian16	2004	49	RCT	Coenzyme Q10	YES	Survival at 90 days	Survival at ICU discharge; Glasgow Outcomes Scale at 90 days; Biomarkers
Longstreth22	2002	300	RCT	Magnesium; Diazepam	YES	Awakening (speaking or following commands) at any time over 90 days	Independent function at any time over 90 days; Days to awakening or death
Bernard3	2002	77	RCT	Hypothermia	NO	Good neurologic outcome (discharged to home or rehabilitation facility)	Multiple physiologic indices (e.g. hemodynamic or biochemical effects of hypothermia)
HACA Study Group4	2002	275	RCT	Hypothermia	YES	Good neurologic outcome (Cerebral Performance Category 1 or 2) within 180 days	Survival at 180 days; Complications of hypothermia
Hachimi-Idrissi19	2001	30	RCT	Hypothermia	NO	Multiple physiologic indices (e.g. hemodynamic and biochemical effects of hypothermia)	Achievement of target temperature and time to achievement; Complications
BRCT II Study Group15	1991	516	RCT	Lidoflazine	YES	Cerebral Performance Category [analyzed as both ordinal and dichotomous (good=1 or 2) variable] at 1, 2, 3, 5, 7, 14, 30, 90, and 180 days	Best Cerebral Performance Category at any time over 180 days; Survival at 180 days
Gueugniaud18	1990	39	RCT	Nimodipine	NO	Cerebral perfusion pressure and intracranial pressure	--
Roine24	1990	155	RCT	Nimodipine	YES	Survival at 1 year; Glasgow Outcome Scale at 1 year	Cases of death attributed to anoxic brain injury at 1 year; Glasgow Coma Scale at 24
Forsman17	1989	51	RCT	Nimodipine	NO	Cerebral blood flow and cerebrospinal fluid pressure	Glasgow-Pittsburgh-Coma score at multiple time points; Time to awakening (following a command); Regained pre-arrest functional status at 120 days
BRCT I Study Group14	1986	262	RCT	Thiopental	NO	Cerebral Performance Category and Overall Performance Category [analyzed as both ordinal and dichotomous (good=1 or 2) variable] at multiple time points out to 1 year	Best Cerebral Performance Category at any time over 1 year

[RCT=randomized controlled trial; NSE=neuron specific enolase; ICU=intensive care unit; HACA=Hypothermia After Cardiac Arrest; BRCT=Brain Resuscitation Clinical Trial]

^*High quality was defined as a grade of “A” in at least three of four methodology domains using the Cochrane Collaboration’s tool for assessing risk of bias.¹²

Study characteristics

The 13 studies were published over 21 years (1986–2007). All of the studies were randomized controlled trials. Seven trials (54%) were multi-center and 6 (46%) were performed at a single center. Nine of the trials (69%) originated in Europe; two trials (15%) were multi-national including the United States and Europe; and the remaining trials were from Australia (1) and the United States (1). Ten trials (77%) exclusively enrolled out-of-hospital cardiac arrests; no trials exclusively enrolled in-hospital arrests; and 3 trials (23%) enrolled both (or did not specify). Five trials (38%) exclusively enrolled subjects with “shockable” initial rhythms (ventricular fibrillation or ventricular tachycardia); one trial (8%) exclusively enrolled “non-shockable” initial rhythms (pulseless electrical activity or asystole); and 7 trials (54%) enrolled both (or did not specify).

Study interventions

Seven (54%) of the trials tested pharmacologic interventions, including nimodipine (3), thiopental (1), lidoflazine (1), magnesium/diazepam (1), and coenzyme Q10 (1). The remaining six trials (46%) tested non-pharmacologic therapeutic strategies, including mild therapeutic hypothermia (3), strict glycemic control (1), normoxic (versus hyperoxic) post-ROSC ventilation (1), and high-volume hemofiltration (1). On methodological quality assessment, seven of the 13 trials (54%) were found to be high quality.^{4, 15, 16, 21–24}

Synthesis

The primary and secondary outcome measures selected in each clinical trial appear in Table 2. There was marked heterogeneity among outcome measures. Although the majority (10/13, 77%) of clinical trials selected some measure of survival or neurological/functional status, no two trials utilized exactly the same primary outcome measure. Time of measuring the outcomes (e.g. measured at hospital discharge versus 30, 90, 180, or 365 days) also varied widely.

We did identify two general themes in the data. First, of the nine (69%) trials that incorporated survival as an outcome measure, the majority of these trials also incorporated some type of neurological function measure in the outcomes (albeit heterogeneous types/times of neurological measurement). Only one trial used survival alone. Second, all of the studies that used an ordinal scale to assess functional neurological outcome (e.g. Cerebral Performance Category [CPC] 1 through 5) used a cut point to group the data into a dichotomous “good” versus “bad” outcome.

Table 3 displays the frequency with which each of the survival or neurological/functional outcomes from Table 2 was selected as a primary or secondary outcome measure, and the frequency of reporting each outcome anywhere in the results section of the manuscripts. The only suggestion of consistency that we found with the specific measures selected (or reported anywhere in the results) was found in the secondary analysis of high quality studies. Four of the seven (57%) high quality studies reported survival at 180 days, and three (43%) reported the incidence of good neurological outcome (CPC 1 or 2) at 180 days.

Table 3.

Patient-oriented outcome measures (i.e. survival and neurological/functional status) used in 13 randomized controlled trials of post-ROSC interventions. This table displays the outcome measures listed in Table 2, the frequency of use of each measure as the primary or secondary outcomes, and the frequency with which each measure was reported anywhere in the results section of the manuscripts.

OUTCOME MEASURE	#TRIALS USING THE OUTCOME (PRIMARY OR SECONDARY)	# TRIALS REPORTING THE OUTCOME ANYWHERE IN THE RESULTS
Neurological and functional status (n, %)
Cerebral Performance Category
Treated as ordinal variable
Serial (repeated measures over time)	2 (15)	2 (15)
Best score over 180 days	1 (8)	1 (8)
Best score over one year	1 (8)	1 (8)
Treated as dichotomous variable (good=1 or 2)
180 days	3 (23)	3 (23)
One year	1 (8)	1 (8)
Glasgow Coma Scale(repeated measures)	1 (8)	2 (15)
Glasgow-Pittsburgh Coma Score (repeated measures)	1 (8)	1 (8)
Awakening (speech or following commands)
90 days	1 (8)	1 (8)
Time to awakening	2 (15)	2 (15)
Overall Performance Category (or Glasgow Outcomes Scale)
Treated as ordinal variable
Serial (repeated measures over time)	1 (8)	1 (8)
90 days	1 (8)	1 (8)
One year	2 (15)	2 (15)
Treated as dichotomous variable (good=1 or 2)
One year	1 (8)	1 (8)
Assessment of functional independence
Able to be discharged to home/rehabilitation	1 (8)	1 (8)
Regained pre-arrest functional status
120 days	1 (8)	1 (8)
Independent functional status
90 days	1 (8)	1 (8)
Survival (n, %)
To intensive care unit discharge	1 (8)	1 (8)
To hospital discharge	1 (8)	4 (31)
30 days	1 (8)	2 (15)
90 days	1 (8)	3 (23)
180 days	3 (23)	4 (31)
One year	1 (8)	2 (15)

DISCUSSION

This systematic review found substantial heterogeneity in the outcome measures utilized in clinical trials of interventions for the post-cardiac arrest syndrome. Of the 13 randomized controlled trials published over more than two decades, no two trials used identical primary outcome measures. Most studies selected some outcome measure of survival and/or neurological or functional status, but the lack of uniformity in the specific measurements or when they were measured represents a systematic methodological shortcoming and important challenge for the current state of post-ROSC experimental trials.

Historically, clinical trials of resuscitation interventions for cardiac arrest (e.g. trials of new CPR or defibrillation strategies) have used outcome measures that are rather crude such as the achievement of ROSC or survival to hospital admission or discharge. This is most likely a function of the relatively poor likelihood of survival among cardiac arrest victims in whom ROSC has not yet been achieved. Recently, however, the post-cardiac arrest syndrome has emerged as a distinct clinical entity⁵ in which patients may not only have a relatively favorable prognosis, but also the trajectory of the disease can in fact be modulated by post-ROSC interventions. Therefore, more sensitive outcome measures that can differentiate degrees of functional outcome are especially important for clinical trials of interventions in the post-ROSC phase of therapy, and our findings highlight the need for standardization of outcome measures among these trials going forward.

The heterogeneity in outcomes demonstrated in this systematic review highlights many of the critical issues in the debate over optimal outcome measures for cardiac arrest clinical trials. Whether the main outcome should be survival alone or functional survival (e.g. Cerebral Performance Category) is a fundamental question for trial design. We found that only one trial utilized survival as an outcome without also incorporating some measure of neurological or functional status in the outcome measures. Therefore we conclude that there does appear to be consensus on this fundamental question – an assessment of neurological or functional status should be incorporated in the outcome measures in some fashion rather than survival alone. The next fundamental question for trial design is when to measure the outcome – short-term versus long-term. As post-ROSC interventions could potentially create more survivors to hospital discharge but not have a meaningful impact on long-term survival, especially for subjects discharged alive but with severe neurological injury, it seems reasonable to select outcome measures beyond the point of hospital discharge. However, we found no indication of consensus on when to measure outcomes, with short-term (hospital discharge, 30 days), intermediate (90 days), and long-term (180 days, one year) time frames similarly represented in the sample.

Another important decision for trial design and analysis is whether or not assessment of neurological or functional status, which is typically done with an ordinal scale, should be treated as a dichotomous variable (e.g. CPC 1 or 2 as a “good” outcome) or analyzed as an ordinal variable. The advantages of combining the points on these scales into two groups for an assessment of favorable versus unfavorable outcome is that it makes the statistical analysis fairly straightforward and the groups are likely clinically meaningful, both of which can make the research findings more understandable for the reader. Analysis of the raw ordinal data would allow for more granularity in the assessment of outcome, which may be optimal for the power to detect a difference between two treatment assignments, but small differences in analysis of an ordinal variable may not be as clinically meaningful. We found that no studies analyzed a neurological function scale as an ordinal variable without also analyzing the outcome based on a dichotomous cut point. Therefore there appears to be some level of consensus that grouping these categorical data into “good” versus “bad” outcome is advantageous. Another important question for trial design is whether or not quality of life assessment after hospital discharge should be incorporated. Interestingly, none of the 13 clinical trials to date have employed quality of life measures.

We recognize that the selection of pre-defined primary and secondary outcome measures is not the only important consideration. Collecting and reporting data on a wide range of clinically meaningful outcomes in manuscripts would permit identification of common denominators of outcome between trials (regardless of which outcome the investigators deemed to be the a priori outcome of interest) and would facilitate comparison or pooling (i.e. meta-analysis) of outcomes data from different trials. This systematic review also found marked heterogeneity of the outcomes that were reported in the results sections of the manuscripts (Table 3), due in large part to the fact that few trials reported other outcome variables beyond their pre-selected primary or secondary outcome measures. The only consistency that we found was the reporting of 180-day survival and neurological function (CPC 1 or 2 = favorable) in the majority of high quality trials.

It appears that more consistent outcomes reporting is needed. The most recent Utstein consensus guidelines for uniform reporting of cardiac arrest research²⁵ advocate the use of two outcome measures pertinent to clinical trials of post-ROSC interventions – survival to hospital discharge and functional survival (CPC 1 or 2 = favorable) at hospital discharge. Based on our findings in this systematic review and in order to build upon the only consistency in outcome reporting that currently exists in post-ROSC trials, perhaps the Utstein guidelines should also include a longer time frame for reporting survival and functional survival (180 days post-arrest) in addition to the time of hospital discharge.

We recognize important limitations in this systematic review. The main limitation is the fact that there have only been 13 experimental trials of post-ROSC interventions to date. Without an abundance of data to synthesize, we could only perform a qualitative analysis of the data and we were not able to perform a quantitative analysis to determine superiority (i.e. maximal responsiveness that represents a clinically important difference) of any particular measure. The small number of trials may be attributed to the fact that the importance of post-ROSC care was underappreciated until the landmark trials of therapeutic hypothermia were published.^3,4,5 However, the number of experimental studies of post-ROSC interventions is likely to increase sharply in the future due to recently renewed interest triggered by the hypothermia data.^{3, 4} That is why our systematic review of outcome measures may be valuable – to call attention to the need to achieve consensus before many more trials are designed and executed.

We acknowledge that some trials included in this systematic review may have had limited options for what outcome measures to select because of the pilot nature of the study or other funding constraints, and, importantly, this may have contributed to the heterogeneity that we found. We also recognize that this systematic review may not have wide applicability to clinicians. However, our results are extremely important to clinical scientists who are designing post-cardiac arrest trials. It is these trials that will ultimately result in changes of practice patterns. Therefore, analyzing outcomes with the aim of defining consensus on the pertinent and clinically relevant measures will undoubtedly impact clinicians who are implementing these research findings.

CONCLUSIONS

There is currently a lack of consensus on what outcome measures should be used in clinical trials of post-ROSC interventions and when they should be measured. We found only two commonalities: (1) clinical trials incorporated indices of neurological or functional status to measure functional survival rather than survival alone, and (2) ordinal scales of neurological function were collapsed into clinically meaningful groups of “good” versus “bad” outcome. Going forward, achieving consensus on optimal outcome measures and when to measure them would be an important advance for resuscitation science.