Differences between self-reported and verified adverse cardiovascular events in a randomised clinical trial

Abstract

Objectives

In clinical trials, adverse events are usually self-reported but may be adjudicated if serious or of particular interest. After adjudicating cardiovascular events for a 5-year calcium supplement trial, we observed discrepancies between self-reported and verified events. We systematically analysed those differences to assess their importance.

Design

Secondary analysis of adverse cardiovascular events in a 5-year, randomised, placebo-controlled trial of calcium supplementation (1 g calcium daily) in 1471 postmenopausal women (mean age 74 years).

Setting

Clinical research centre.

Methods

The participant's medical records were reviewed for all self-reported myocardial infarctions (MIs) or strokes, and the event independently adjudicated. Cause of death was obtained from hospital records or death certificates. To identify unreported events, the national hospital discharge database was searched and related hospital records were reviewed.

Results

45 women reported 64 MIs, of which 33 (52%) were verified after adjudication. An additional 25 MIs were identified: 1 during adjudication of other events, 21 from the hospital discharge database, 3 from death certificates. 68 women reported 86 strokes of which 50 (58%) were verified. An additional 13 strokes were identified: 7 during adjudication of reported transient ischaemic attacks, 5 from the hospital discharge database, 1 from death certificates. Therefore, 43% of verified MIs and 21% of verified strokes were not reported to investigators. For non-adjudicated discharge codes, 10% of MIs and 22% of strokes were not verified after adjudication. Nineteen per cent of verified MIs and 27% of verified strokes were not identified in discharge coding or death certificates. Neither the event source nor the level of adjudication altered the relationship between treatment allocation and cardiovascular events.

Conclusions

When adverse event accuracy is critical, researchers should consider adjudicating self-reported events and hospital discharge codes, and attempt to identify unreported events.

Trial registration

Australia New Zealand Clinical Trials registry: ACTRN 012605000242628.

Article summary.

Article focus

• The differences between self-reported and verified adverse cardiovascular events in a clinical trial.

Key messages

• Substantial proportions of self-reported myocardial infarctions (MIs) and strokes were not able to be verified, and substantial proportions of verified MIs and strokes were not reported by participants or were not identified in discharge coding or death certificates.

• When adverse event accuracy is critical to a clinical trial, unadjudicated self-reports or hospital discharge codes cannot be relied upon. Consideration should also be given to identifying unreported events.

Strengths and limitations of this study

• A rigorous search for adverse events occurred and all events were independently adjudicated.

• Cardiovascular events were secondary endpoints, and participants may have placed less value on reporting these events than the primary endpoints.

Introduction

In clinical research studies, participant self-report of adverse effects and outcome events are essential in determining the efficacy and tolerability of the intervention being studied. Participant self-reports may be accepted as accurate or may lead to independent adjudication, depending on the relevance to the study and the overall size and complexity of the study. In large clinical trials, adverse events are usually self-reported and not independently verified unless considered serious or of particular interest. However, the few studies that have specifically addressed the accuracy of self-reported medical events suggest relatively poor agreement between self-reports and medical records.^{1 2}

Previously, we completed a 5-year trial of the effects of calcium supplementation in healthy postmenopausal women in which we observed an unexpected increase in the rate of vascular events in women allocated calcium.³ All vascular events were initially self-reported, and then adjudicated by blinded study investigators. We also conducted a systematic search for valid events that were not reported by participants. We present the results of an analysis of the relationship between self-reported and adjudicated events, including unreported events. The current analyses were not planned in the original trial protocol or the subsequent protocol for the adjudication of vascular events. Ethical approval for the current analyses was not required.

Methods

Study design

The Auckland Calcium study was a 5-year, randomised, placebo-controlled trial of calcium supplementation in 1471 normal postmenopausal women (mean age 74 years), designed to assess the effects of 1 g daily calcium on fracture incidence.

Recruitment started in 1998 and was completed in 2006. Cardiovascular outcomes were a prespecified secondary endpoint. The methods have been described in full previously.⁴ In brief, women were recruited by advertisement and from mail-outs using electoral rolls, and were 5 years or more postmenopausal and aged 55 years or older. Women were ineligible if they were receiving therapy for osteoporosis, had other major ongoing disease, or had serum 25-hydroxyvitamin D<25 nmol/l. The study received approval from the regional Ethics Committee and the trial was registered with the Australia New Zealand Clinical Trials registry: ACTRN 012605000242628.

Cardiovascular event assessment

Participants were reviewed every 6 months. Adverse events were recorded at each visit but questions about specific symptoms or illnesses were not asked. A preplanned secondary analysis was a comparison between the groups in the frequencies of myocardial infarction (MI), stroke, transient ischaemic attack (TIA) and other cardiovascular events.³ During 2006–2007, cardiovascular events were adjudicated. The participant's medical records were reviewed when an MI, stroke or TIA was self-reported (or by family members for fatal events). For participants who died during the study, the cause of death was obtained from hospital records or the death certificate. Data for each event were compiled by a physician and then adjudicated by a cardiologist (MI) or neurologist (stroke or TIA). All were blinded to the treatment group of each participant. MI was defined in accordance with the Joint European Society of Cardiology/American College of Cardiology Committee criteria for acute, evolving or recent MI⁵ and stroke and TIA were defined in accordance with the WHO definition.⁶

Discharge codes

To identify events that were unreported, we searched the national database of hospital discharges for cardiovascular events (ICD-9 discharge codes 410 for MI and 430, 431, 433, 434 for stroke) that occurred during the study using each participant's unique National Health Index identifier. The hospital records related to these admissions were reviewed and adjudicated in the same manner as for self-reported events. We added all unreported, adjudicated hospital discharge events to the adjudicated self-reports to obtain a complete set of verified events. We compared this complete set of verified events with the events obtained solely from discharge ICD-9 codes or death certificates.

Statistics

Agreement between reported and verified events was assessed using the κ coefficient. The number of women experiencing an incident event in each treatment group was compared using Fisher's exact test. Differences between baseline characteristics were compared using t tests for continuous variables, and Fisher's exact test for categorical data. All statistical analyses were performed using the SAS software package (SAS Institute, Cary, North Carolina, USA V.9.1). p<0.05 was considered statistically significant.

Results

Myocardial infarction

Table 1 shows the number of self-reported, verified self-reported, verified unreported and total verified events. Thirty-three of 64 (52%) self-reported MIs were verified. Table 2 shows the final diagnosis from the medical records for the 31 MIs that were not verified. In 10 instances, there was no report of the event in the medical records, but in each case a verified MI had previously occurred during the study. For 9 of the remaining 21 events (43%), MI was specifically excluded with final diagnoses made of angina, unstable angina, or MI excluded. Twenty-five unreported events were identified: 1 during adjudication of other events, 21 from the national hospital discharge database search and 3 from death certificates. Thus, of 58 verified MIs in 52 women, only 33 (57%) were reported to study investigators and 25 (43%) were identified from other sources. The κ value for agreement between reported and verified MIs was 0.63 (95% CI 0.51 to 0.74).

Table 1.

Numbers of events by source and adjudication status

Source/adjudication	Myocardial infarction	Stroke
Total self-reported events	64 (45)	86 (68)
Verified self-reported events	33 (30)	50 (47)
Verified unreported events	25 (24)	13 (12)
Total verified events	58 (52)	63 (59)

Data are number of events (number of women).

Table 2.

Final diagnosis from medical records for self-reported, non-verified myocardial infarctions (n=31) and strokes (n=36)

Myocardial infarction		Stroke
Final diagnosis	N	Final diagnosis	N
Unstable angina	5	Transient ischaemic attack	6
Congestive heart failure	5	Fall	3
Angina	2	Acute confusional state	3
Sudden unexplained death	2	Sudden unexplained death	2
Atrial fibrillation	1	Benign positional vertigo	2
Palpitations	1	Dementia	2
Postural hypotension	1	Postural hypotension	1
Renal failure	1	Unsteady on feet	1
Shortness of breath cause unknown	1	Seizure	1
Myocardial infarction excluded	2	Head injury	1
No details of event in medical record	10	Blackout	1
		Vasovagal event	1
		Ischaemic retinal vein occlusion	1
		Stroke excluded	1
		No details of event in medical record	10

When only non-adjudicated events identified from the discharge database search were considered, there were 48 MIs in 41 women. When death certificate data were added, there were 52 MIs in 45 women. Five (10%) of the 48 MIs identified from the discharge database search were not verified. Another 11 verified MIs (19%) were identified from self-reports but not from hospital discharges or death certificates.

Stroke

Table 1 shows that 50 of 86 (58%) self-reported strokes were verified. Table 2 shows the final diagnosis for the 36 strokes that were not verified. In 10 instances, there was no report of the event in the medical records, but in each case a verified stroke had previously occurred during the study. For 8 of the remaining 26 events (30%), stroke was specifically considered and excluded with final diagnoses made of TIA, or stroke excluded. Thirteen unreported events were identified: 7 during the adjudication of reported TIA, 5 from the discharge database search and 1 from death certificates. Thus, of 63 verified strokes in 59 women, 50 (79%) were reported to study investigators as stroke, 7 (11%) as TIA and 10% were identified from other sources. The κ value for agreement between reported and verified stroke was 0.73 (95% CI 0.64 to 0.82).

When only non-adjudicated events identified from the discharge database search were considered, there were 50 strokes in 46 women. When death certificate data were added, there were 57 strokes in 48 women. Eleven of the 50 strokes (22%) identified from the discharge database search were not verified. Another 17 verified strokes (27%) were identified from self-reports but not from hospital discharges or death certificates.

The relationship between treatment allocation and MI or stroke for the different event sources and levels of event adjudication is shown in Otable 3. There were consistently increased relative risks for MI and stroke with calcium supplements, although the relative risks changed in some cases depending upon event source or level of adjudication, and the different numbers of events in each comparison led to more marked changes in p values.

Table 3.

Relationship between treatment allocation to calcium supplementation and risk of event by source and adjudication status

Event source and adjudication	Calcium* (N=732)	Placebo* (N=739)	Relative risk (95% CI)	p Value
Myocardial infarction
Total self-reported events	31	14	2.24 (1.20 to 4.17)	0.0099
Verified self-reported events†	21	10	2.12 (1.01 to 4.47)	0.047
Total hospital discharge coding	24	17	1.43 (0.77 to 2.63)	0.27
Total hospital discharge coding and death certificates	27	18	1.51 (0.84 to 2.73)	0.18
Verified self-reports/hospital discharge coding and death certificates	31	21	1.49 (0.86 to 2.57)	0.16
Stroke
Total self-reported events	40	28	1.44 (0.90 to 2.31)	0.14
Verified self-reported events‡	31	22	1.42 (0.83 to 2.43)	0.21
Total hospital discharge coding	26	20	1.31 (0.74 to 2.32)	0.37
Total hospital discharge coding and death certificates	27	21	1.30 (0.74 to 2.27)	0.38
Verified self-reports/hospital discharge coding and death certificates	34	25	1.45 (0.88 to 2.49)	0.15

*Data are number of women experiencing an incident event.

†Includes one incident myocardial infarction identified during the verification of other events.

‡Includes six incident strokes identified during the verification of transient ischaemic attacks.

Finally, we assessed possible relationships between baseline characteristics and discrepancies between self-reported and final verified events. Women in whom there were discrepancies were older (p<0.001), and were more likely to have reported at the baseline study visit having had a previous MI (p=0.013) or previous TIA or stroke (p=0.035) than women without such discrepancies. However, there were no differences in physical activity (p=0.7) or other comorbidities between the groups (p>0.1 for hypertension, dyslipidaemia, diabetes).

Discussion

We observed differences between self-reported MI or stroke and the final verified diagnosis obtained from the medical records or death certificates. Almost half of self-reported MIs (48%) and strokes (42%) were not able to be verified, and 43% of verified MIs and 10% of verified strokes were unreported, with a further 11% of verified strokes reported as TIA. Differences were also observed between events obtained solely from hospital discharge codes and final verified events. Ten per cent of MIs and 22% of strokes obtained from discharge codes were not verified, and 19% of verified MIs and 27% of verified strokes were identified from self-reports but not from discharge codes or death certificates. Relying solely on self-reports of MI or stroke, or solely on discharge codes in our clinical trial, would have led to significant numbers of participants being misclassified with regard to having experienced a significant adverse event. Despite the differences between self-reports, hospital discharge codes and verified events, neither the source of the event nor the level of adjudication substantially altered the relationship between treatment allocation and occurrence of either MI or stroke.

In most cases, the final diagnosis for the non-verified, self-reported MIs was related to a disorder of the heart, suggesting that miscommunication or misunderstanding between the participant and their physician led to the error. It has been suggested that the gastrointestinal side effects of calcium supplements might be misclassified as MI,⁷ but we found no evidence that this was the case. In at least 43% of non-verified cases, MI was specifically considered as a diagnosis and excluded. The most common final diagnosis for non-verified, self-reported strokes was TIA, suggesting that participants did not understand the distinction between these two conditions. This is not surprising as TIA is often described as a ‘mini-stroke’. In approximately one-third of non-verified, self-reported events (both MI and stroke), there were no details of any potentially related event in the medical records. All these events occurred in people who had experienced a previous verified event, suggesting that they reported the same event more than once, or they reported symptoms that were similar to their primary event for which they did not seek medical attention.

There are limited published data that specifically address differences between self-reported events and the medical records. A systematic review identified 15 studies that assessed the accuracy of questionnaires of medical history compared with the medical records.¹ The proportion of illnesses reported in questionnaires ranged from 30% to 53% of those listed in the medical record. Conversely, the medical record listed 36–70% of illnesses reported in questionnaires. Reporting of surgical procedures or hospitalisation in questionnaires appeared to be more accurate than reporting of non-surgical illness. The findings of this systematic review have been replicated in more recent studies.^{2 8–10} Further studies have addressed the accuracy of self-reported cardiovascular events. For MI and stroke, approximately 70–80% of self-reported events were confirmed by hospital record review,^{2 8–13} which the authors suggested is sufficiently accurate for use in research studies.^{8 9 11} For subtyping of stroke, review of hospital records adds useful data to self-reports and discharge coding.¹⁴

A comprehensive analysis of reporting of cardiovascular events that occurred during the Women's Health Initiative clinical and observational studies has been published.¹⁰ In these studies, local physician adjudicators reviewed self-reported data and patient medical records for all self-reported cardiovascular events. The local adjudicators were able to verify 68% of self-reported MIs and 72% of strokes. The local adjudicators could only verify MI in 78% of events with a discharge code for MI, and stroke in 81% of events with a discharge code for stroke, suggesting coding data from hospital admissions may not be accurate, a finding consistent with the results of our study. Finally, there was no complete agreement between local and central adjudicators. Eighty-one per cent of locally adjudicated MIs were verified by central adjudicators, and conversely 86% of centrally adjudicated MIs were classified as MI by local adjudicators. The issue of unreported events was not addressed.

Our findings are broadly consistent with this previous research, although it is uncertain whether self-reports in response to open-ended questions in a clinical trial are comparable to responses to questionnaires or medical records based on a mixture of open-ended and closed-ended questions. The lower accuracy of MI self-reports in our study compared with other studies might be because the participants in our trial were older, had more comorbidities, and had more cognitive impairment than participants in previous studies. Another contributing factor might be that although cardiovascular events were prespecified secondary endpoints, many of the participants might not have been aware of this, and may have placed less value on reporting these events than fractures, which were the primary endpoints of the study. In the UK, accuracy of diagnostic coding has improved substantially in recent years,¹⁵ although no similar data are available for New Zealand. It is possible that the magnitude of differences between adjudicated events and hospital discharge coding might have changed since our study was undertaken.

Inaccurate self-reports or underreporting of events would not be expected to affect the results of a study unless the study treatment introduced a systematic bias that produced differential rates of inaccurate self-reports or underreporting of events. However, when the event numbers are small, as in many clinical trials, chance differences can substantially impact the study results. In our study, the adjudication process led to marked changes in the degree of statistical significance, even though the relative risks of MI with calcium supplements remained elevated at each stage. The relationship between calcium supplements and occurrence of MI or stroke was similar regardless of whether the events were based solely on hospital discharge coding or self-reports or final verified events, a finding that is supported by the similar increase in vascular risk associated with calcium supplements across randomised trials that used various means of event identification.^{16 17}

A common approach in clinical trials is to adjudicate all significant adverse events centrally, although the need to do this remains uncertain.¹⁸ The advantages of central adjudication include systematically applying the definition of an event, reducing the possibility of differential misclassification of events, giving greater confidence in the validity of the study results, and by including suitable triggers, potentially identifying events that are missed by local investigators.¹⁸ However, central adjudication is unlikely to identify events that are not reported to the local investigators, adds significant cost and complexity to trials, and has not been shown to improve the ability to determine treatment effects in cardiovascular trials.¹⁸

In summary, our results suggest that when the accuracy of an event in clinical trials is critical, even for a relatively common and serious medical event such as MI or stroke, self-reports cannot be relied upon and should be independently verified from the medical records. However, even if independent verification occurs, our results also suggest that a substantial number of events will not have been reported. For adverse events of particular interest, additional steps, such as searches of hospital discharge databases, should be considered to attempt to identify unreported events. Relying solely on non-adjudicated hospital discharge coding will lead to similar inaccuracies as using self-reported data because some events will be missed and some events will be included that would not be verified by adjudication.