When Evidence Goes “Missing in Action”: Implications for Patient Management in Cardiac Surgery

Abstract:

Best-practice clinical decision-making for patient blood management (PBM) and transfusion in cardiac surgery requires high-quality, timely information. However, evidence will be misleading if published information lags too far behind evolving practice, or if trial results are biased, incomplete, or unreported. The result is that providers are deprived of accurate data, and patients will not receive best possible care. Publicly accessible trial registries provide information for structured audits of reporting compliance, and appraisal of evidence attrition and distortion. Trials related to blood management and transfusion in cardiac surgery and those registered in ClinicalTrials.gov were evaluated for relevance, reliability, transparency, timeliness, and prevalence of unreported trial results. Evidence was considered to have “disappeared” if no results were posted to the registry and no related PUBMED publications were available by July 2019. Data were summarized by descriptive statistics. A total of 181 registered trials were surveyed; 52% were prospectively registered. Most commonly reported primary outcomes were laboratory surrogate measures (34%). Patient- and practice-relevant outcomes—mortality/major morbidity (7%), transfusion (27%), and major bleeding (28%)—were less common. Only seven studies posted results to the registry within the mandated 12 months from study completion; median time to posting was 17 (interquartile range [IQR] 13, 37) months. Trial results for 58% were unreported 3–9 years after trial completion. A staggering amount of clinical trial evidence for PBM in cardiac surgery is missing from publicly accessible records and the literature. Investigators must be incentivized to promptly and completely report all results. Penalties for noncompliance are already in place and should be enforced. Simplified information linkage, centralized and routine audit cycles, and prioritization of robust “living” reviews may be more positive motivators. Implementation will require a sea change in the prevailing culture of research reporting, plus coordinated efforts of clinicians, applied statisticians, information technology specialists, and research librarians.

Patient blood management (PBM) is an integrated process of evidence-based clinical practice designed to deliver the most appropriate, effective, efficient, and safe care for patients. For cardiac surgery in particular, there is a clear need to produce accessible high-quality evidence to promote best-practice clinical decision-making (1). First, cardiac surgery patients account for a large proportion of the case mix of patients receiving transfusions. Although transfusion of blood and blood components in the United States has declined overall (2,3), transfusion rates range between 40 and 55% for coronary artery bypass grafting (CABG) surgery and may exceed 80% for combined CABG–valve procedures (4). Second, there is considerable (and unacceptable) variation between clinicians, institutions, and countries in transfusion practices and blood component use in CABG surgery, independent of case mix (5–8). Numerous studies have suggested that transfusions are associated with adverse patient outcomes, including increased mortality and major morbidity (5,9). Furthermore, costs of blood components and administration services are high, and the supply of blood is limited. These observations, coupled with evidence that a large proportion of transfusions may be unnecessary (7,10), indicate that much current transfusion practice for cardiac surgery patients may not in fact meet the “appropriate, effective, efficient, or safety” benchmarks for patient care.

Evidence-based recommendations for blood management practices are hierarchical, with systematic reviews and meta-analyses (SRMAs) considered the highest levels of evidence. In turn, it is assumed that SRMAs represent comprehensive aggregates of the highest quality studies. However, even systematic approaches to evidence grading may not reliably assess strength of recommendations. First, the sheer volume of published research and the apparent lack of utility of most published studies (11) will make complete assessment difficult or impossible (evidence is “hiding in plain sight”). Second, SRMAs have a number of disadvantages. Evidence compilation is slow and labor-intensive and cannot keep up with rapidly evolving challenges and standards for patient care. Therefore, SRMA-based recommendations may be based on outdated information. Observational studies are excluded from SRMAs, eliminating large amounts of potentially useful information. The quality of SRMAs may be compromised because of inappropriate methodology and distortion of the evidence base resulting from publication bias, selective outcome reporting, conflicts of interest, and outcome switching. Furthermore, only about half of clinical studies approved by ethical oversight committees are eventually published in the primary literature (12).

The purpose of this study was to evaluate gaps in reporting and compliance for clinical studies related to transfusion and PBM in cardiac surgery. Structured auditing of registered trials was performed to evaluate evidence distortion and disappearance. Trial registries such as ClinicalTrials.gov provide information for compliance audits and allow for assessment of rates of evidence attrition and distortion not always possible from surveys of published literature (13–15).

METHODS

Search Strategy

A structured audit of trials related to cardiac surgery, blood management, and modifiable risk factors (blood loss, anemia, and transfusion) was performed. ClinicalTrials.gov was used exclusively for this audit because it is the largest and most widely used publicly available register, and pertinent information (such as previous versions, outcome variables, completion dates, and publication links) is archived or otherwise easily available. Other registries examined (Health Canada Clinical Trial Database, EU Clinical Trials Register (EUCTR), Australian New Zealand Clinical Trials Registry, and WHO-ICTRP) did not provide all information items required for this survey. Trials were included if they were registered from January 2008 (following the U.S. Food and Drug Administration (FDA) Amendment Act of 2007 mandating trial registration and reporting) to August 2018 (to allow for the mandated 12 months for results reporting after trial completion) and if trial status was designated as “completed,” “terminated,” or “withdrawn.” Trials with “unknown” status (study had passed completion date and status was unverified for more than 2 years) were also included if the anticipated completion date was to have occurred before August 2018. Trials were extracted on the search terms (Cardiac surgery OR cardiopulmonary bypass) AND [identifiable elements of PBM] given as follows: (transfusion), (transfusion AND liberal OR restrictive), (transfusion AND trigger or threshold); (iron AND supplementation OR therapy); (anaemia OR anemia); blood management, PBM, blood conservation, blood storage, cell salvage, haemo* OR hemostatics; hemostasis AND (tranexamic acid, factors, fibrin, thrombin, sealant); (blood loss OR blood minimisation OR minimization OR reduction); (coagulation OR coagulopathy); (anticoag* OR heparin). In total, 2,386 trials were screened for eligibility, of which 181 met the inclusion criteria.

Evidence Performance Metrics

The evidence base is distorted when trial results are irrelevant, biased, or reported late, in part, or not at all (16). Each trial was assessed for relevance, reliability, promptness of reporting (timeliness), and transparency, according to the predetermined criteria (Table 1).

Table 1.

Working definitions for concepts and metrics used to assess reporting of clinical trials related to PBM strategies and therapies in cardiac surgery—information extracted from ClinicalTrials.gov.

Characteristic	Research Questions	Metrics
Relevant	Does the research address	A. Outcome classification
	important and unresolved questions?major knowledge or practice gaps?outcomes that are clinically informative, meaningful to patients?	1. Patient-oriented
		Mortality/major morbidity
		Transfusion
		Major bleeding
		2. Clinical
		3. Laboratory/device
		B. Outcome priority
		1. Primary
		2. Secondary
		C. Intervention
		1. Drug/biological
		2. Device
		3. Procedure
Reliable	Are trial resultsrelatively unbiased?reported consistently?	1. Study design: interventional, observational
		2. Bias minimization: randomization, blinding, blinding type, controls
		3. Sample sizes adequate (for mortality/major morbidity)
		4. Consistent terminology: “transfusion,” “major bleeding”
Transparent	Are all trial methods and results available and accessible?	1. Summary results posted to registry (Y/N)
		2. Prospective registration (Y/N)
		3. Retrospective registration (Y/N)
		4. Data sharing plan (Y/N)
		5. Relevant publications (Y/N)
Timely	Are trial results obtained and reported promptly?	1. Summary results reported within one year from trial completion (Y/N)
		2. Time between study completion and posting of results (months)

A study is relevant if it addresses important or unresolved clinical questions, major knowledge, or practice gaps and if outcomes are clinically informative and meaningful to patients (17). Relevance was based on knowledge gap priority areas for perioperative cardiac surgery management identified by National Blood Authority of Australia (NBAA) guidelines (18) and classified by drivers and outcomes for each trial entry (Figure 1). Drivers (“independent” or “grouping” variables) were categorized as management/delivery/point-of-care interventions, transfusion, perioperative strategies, and hemostatics, per the NBAA. Outcomes were classified as mortality and/or major morbidity, transfusion, major bleeding, clinical, or laboratory/device-related (19). Outcomes were further classified by type (primary or secondary) per registry classification. The primary outcome is considered by the investigator to be the most important for addressing study objectives and directs how the trial was powered.

Figure 1.

Knowledge gaps in PBM for cardiac surgery as identified by the National Blood Authority PBM guidelines (19).

Study reliability was assessed for interventional trials by reporting of bias minimization methods, consistency of terminology, and sample size adequacy. To minimize systematic, selection, and cognitive biases in participant allocation and outcome evaluation, interventional trials should have a control arm, and be randomized and blinded. Observational trials cannot be randomized and blinded, but factors contributing to bias can be controlled by judicious study planning and evaluation, and subject matching (20). Unfortunately, ClinicalTrials.gov does not provide information relevant to determining bias in observational studies.

Inconsistent terminology creates confusion and can result in miscommunication and misapplication. Standardized terminology for bleeding in cardiovascular research has been available for almost a decade (21). Terminology for “transfusion” and “bleeding” was compiled from investigator-reported descriptions in trial registry entries. Definition elements consisted of the variable label, any definitions provided, time frame for measurements, measurement units (if applicable), and application units (per patient, per group, per order, and total).

Sample size adequacy was assessed for studies evaluating mortality and major morbidity as either primary or secondary outcome. The minimum baseline event rate that would be required to detect minimal between-group differences was calculated with a power of .80 of 5, 10, and 30% lower relative risk, given the target enrollment numbers reported for each study. Calculations were performed with SAS proc power two-sample frequency test using the Farrington–Manning score for relative risk (SAS v.9.4, SAS Inc., Cary, NC).

If trials are to be nimble enough to meet rapidly evolving practice needs, then results should be reported promptly, with minimal lag between study completion and public availability of results. Trials were assessed for timeliness according to whether or not summary results were reported within one year from trial completion (as mandated) and the time interval between primary study completion and posting of results. If no results were posted, elapsed time was calculated as the time between primary study completion and July 30, 2019.

Transparency was assessed by whether trials were registered prospectively, if summary results were reported, if data sharing plans were reported, and if any peer-reviewed publications resulted from the trial (16). Although all trials, past and present, should be registered, prospectively registered trials are less likely to deviate from original protocols or switch outcomes (15). Per ClinicalTrials.gov, investigators indicate intention to share individual participant data (IPD) sets (yes, no, and undecided), together with identification of availability and locations of IPD and supporting information (study protocol, statistical analysis plan, clinical study report, analytic code, etc.). Corresponding peer-reviewed publications were located through automatic cross-references in ClinicalTrials.gov to PubMed entries based on the ClinicalTrials.gov identifier (NCT) and by manual searches in PubMed on keywords and principal investigators. Any articles identified by the manual search were back-referenced to the corresponding trial according to the study location, study start and completion dates, and primary and secondary outcomes. Evidence was considered to have “disappeared” if no results were posted to the registry and no related publications were available by July 2019.

RESULTS

A total of 181 studies representing data from approximately 54,000 patients met the inclusion criteria. Results are summarized in Table 2.

Table 2.

Summary data for 181 trials related to PBM strategies and therapies in cardiac surgery—information extracted from ClinicalTrials.gov.

	Trial Status				n (%)
	Completed	Terminated	Withdrawn	Unknown
n	121	20	8	32	181
Relevance
Drivers
Patient management, delivery, POC	31	12	1	8	52 (29)
Transfusion	21	3	3	5	32 ((18)
Perioperative strategies	49	4	2	16	71 (39)
Hemostatics, blood loss reduction	21	5	2	3	31 (17)
Drugs, biologicals	50	10	5	10	75 (41)
Devices	9	2	0	3	14 (8)
Procedures	17	4	2	6	29 (16)
Primary outcomes
1. Patient-oriented
Mortality/major morbidity	8	3	0	1	12 (7)
Transfusion	32	8	1	7	48 (27)
Major bleeding	31	8	1	11	51 (28)
2. Clinical	16	3	3	10	32 (18)
3. Laboratory	47	2	3	9	61 (34)
Secondary outcomes
1. Patient-oriented
Mortality/major morbidity	25	4	1	8	38 (21)
Transfusion	54	10	1	8	73 (40)
Major bleeding	40	6	1	7	54 (30)
2. Clinical	43	8	2	14	67 (37)
3. Laboratory	43	5	3	9	60 (33)
No outcomes reported	4	0	0	1	5 (3)
Reliability
Study design
Observational	48	3	1	9	61 (34)
Interventional	80	17	6	24	127 (70)
Bias minimization
Randomization	70	15	5	20	110 (61)
Blinding	58	10	3	15	86 (48)
Transparency
Summary results posted	29	6	n/a	0	35 (19)
Relevant publications	46	5	n/a	6	57 (31)
Both registry posting and publications	18	1	n/a	0	19 (10)
Neither registry or publication	64	10	n/a	27	101 (56)
Prospective registration	61	11	7	15	94 (52)
Retrospective registration	23	4	0	unknown	27 (15)
Individual patient data sharing?
Y	0	0	0	1	1 (1)
N	18	6	0	0	24 (13)
Undecided	3	0	0	2	5 (3)
Information not provided	100	14	8	30	152 (84)
Timeliness
Summary results reported ≤ 1 year from trial completion	5	2	n/a	0	7 (4)

See text for details of selection process.

Relevance

Less than one-third of surveyed studies (52/181; 29%) directly assessed patient and blood management and/or point-of-care delivery, and just over one-third (71/181, 39%) examined perioperative strategies of blood management. Direct evaluation of hemostatics and other blood loss reduction methods was reported by 17%. Transfusion and transfusion strategies were evaluated directly by 18%. The majority of interventions were drugs or biologicals (including blood and/or blood components; 41%).

Studies evaluating mortality and major morbidity outcomes were identified in 2012 as a major knowledge gap (18). However, fewer than 10% of studies (12/181; 7%) used these endpoints as a primary outcome and 21% (38/181) as a secondary outcome. “Transfusion” and “major bleeding” were included as an outcome variable in nearly one-half (86/181, 48%) and over one-third (71/181, 39%) of surveyed studies, respectively.

Reliability

Sixty-one percent (110/181) of surveyed trials were randomized interventional trials, and 46% (84/181) were both randomized and blinded. The remainder were single-arm or observational studies.

There was little consistency in working definitions for “transfusion” and “major bleeding” (Table 3). Only three trials referred specifically to international consensus guidelines (INTERMACS, CURE, and BARC) for defining “bleeding” (21). Recorded definitions varied in rigor, specificity, and duration for patient evaluation, and few met the consensus guideline criteria.

Table 3.

Definitions of “transfusion” compiled from 181 surveyed clinical trials related to PBM strategies and therapies in cardiac surgery—information extracted from individual trial reports registered in ClinicalTrials.gov.

Variable Label	Definitions	Measurement Units	Application Units	Time Frame
Transfusion	Not specified	Not specified	Not specified	Not specified
Transfusion “requirement”	pRBC only	“Amount”	Per patient	Perioperative
	Allogeneic blood products (components not specified)	“Total”		Postoperative
	Blood products (components and source not specified)	“Overall”		ICU
		≥2 U		1 hour
				POD 1, 2, 3, 4, 5, 7, 14, and 30
				Hospital discharge
Number of blood products	Allogeneic	Not specified	Not specified	POD 28
				Hospital discharge
Proportion of blood products	Components specified	Not specified	Not specified	Not specified
				Perioperative
				Postoperative
Number of patients requiring transfusion	Allogeneic blood products	>7 RBC U	Not specified	POD 1, 3, 7, and 30
	“needing transfusion” (components not specified)
	Components specified
Proportion/percentage of patients requiring transfusion	pRBC only	N/A	Not specified	Perioperative
	pRBC plus others			POD 5 and 30
	Components specified
Proportion/percentage of patients not requiring transfusion	No blood products	N/A	N/A	Not specified
				POD 1 and 90
Number of transfusions	Not specified	Not specified	Not specified	Not specified
	PRBC			Perioperative
Number of units	Not specified	N/A	Not specified	POD 1, 7, 10, 14, and 30
	Blood		Per patient	Hospital discharge
	pRBC only		Per group
	PLT only Combinations of specified blood components		Per order
	Combinations of blood components, not specified

Table 4.

Definitions of “bleeding” compiled from 181 clinical trials related to PBM strategies and therapies in cardiac surgery—information extracted from reports for individual trials registered in ClinicalTrials.gov.

Variable Label, Criteria	Definitions; Measurement Units	Time Frame
Bleeding per guideline definitions	CURE	Not specified
	INTERMACS	Not specified
	BARC	48 hours
“Bleeding”, “blood loss”	Not specified	Not specified
	Volume	1, 6, 12, and 24 hours
	mL	POD 5 and 7
	Amount	10 years
	Total	To chest closure
	Excessive	ICU
Chest tube drainage volume	Not specified	CPB, perioperative
	mL	Post-CABG
	mL/kg	Postoperative time: 6, 12, 20, and 24 hours
	>1.5 L	POD 7 and 30
	Total volume	In CCCU
	Per day	In ICU
	>5 mL/kg/h to 6 hours	Hospital discharge
	≥2 L/24 hours	–
Transfusion	≥2 U pRBC	Not specified
	>4 U pRBC	24 hours, 48 hours
	≥5 U pRBC/48 hours	POD 7, 10, and 30
	Number of units pRBC, FFP, and/or PLT	Hospital discharge
	Use of blood products	Death
	Decision to transfuse; blood loss requiring transfusion	–
	Above pre-ordered pRBC	–
	Blood products mL/hospital stay	–
Rate of surgical re-exploration	Not specified	24 hours
	Percent of patients	7 days
Laboratory values
Hb	Not specified	Not specified
	>2 g/dL decline	–
	Hb <8 g/dL	–
PLT	Counts	–
Hct	Absolute drop	–
Use of coagulation products	Not specified	Postoperative
	rFVIIa	–
Hemostasis time	Not specified	Duration of pump time to skin closure

Forty-five studies identifying mortality and major morbidity as either a primary or secondary outcome were evaluated for sample size adequacy. Median target enrollment was 200 (IQR 100, 508; minimum 16, maximum 7,000). Expected control group event rates for mortality and major morbidity would have to be in excess of 40–50% for studies of 200 or less enrolled subjects to detect a relatively large 30% reduction in relative risk (Figure 2).

Figure 2.

Targeted enrollment numbers for 45 registered clinical trials evaluating mortality and major morbidity as primary or secondary outcome. Median target enrollment was 200 (IQR 100, 508) patients. Relative risk to be detected (5, 10, 30%) is indicated by contour lines. Patient numbers required to detect a given relative risk are estimated by intersection of the relative risk line and a given baseline event rate for the comparator group. The dotted line indicates that a baseline event rate of 50% would be required to detect a 30% relative risk for at least 50% of the trials in this survey.

Timeliness

Eight trials were withdrawn before any patients were enrolled. Of the remaining 173 trials, only 35 (20%) posted summary results to the registry and only seven were within the mandated 12 months from trial completion. Median time to registry posting was 17 (IQR 13, 38) months for completed trials and 25 (IQR 10, 36) months for terminated trials. As of July 2019, median time that results remained unposted for completed, terminated, and unknown status trials was 65 (IQR 36, 93) months or over 5 years.

Transparency

Ninety-four trials (52%) were registered prospectively, 27 trials (15%) were registered after trial completion, and timing of registration could not be determined for 17 of 32 trials of “unknown” status. Thirty-eight trials (22%) were linked to one or more articles in the peer-reviewed literature without posting results to the registry, and 19 (11%) both posted results and were cross-indexed to one or more PubMed-listed articles. However, 58% (101/173) of trials did not either post results or produce a publication by July 2019 (“disappeared”).

DISCUSSION

This study suggests that an unacceptably large proportion of research evidence pertaining to transfusion and blood management in cardiac surgery is distorted or lost. Inadequate study designs, irrelevant outcomes, and inconsistencies in terminology and clinical definitions distort the evidence base. Trial results are neither accessible nor available if reported late or never reported at all. It is clear that efforts toward informed medical decision-making will be seriously compromised if providers cannot access relevant information in a timely way.

Both the Declaration of Helsinki and 2015 WHO position statement state that investigators have a “universal ethical obligation” to report the results of every clinical trial, regardless of whether or not trial results were “positive” and if a given trial is covered by the federal law or FDA trial requirements. Academic peer-reviewed publications are the traditional venue for reporting trial results. However, publication bias (selectively reporting only “positive” trial results), inconsistent reporting standards between journals, arbitrary publication decisions, and lengthy delays between trial completion and publication mean that dissemination of results though journal articles is relatively cumbersome and non-informative. In theory, more nimble results reporting with wider dissemination may be achieved by trial registration and prompt reporting of trial results in publicly accessible registries (such as ClinicalTrials.gov). Unfortunately, in practice, overall reporting compliance to date has been low, and penalties for poor compliance have never been enforced (13,14).

This study was limited, first, to the selection and assessment of registered trials that provided a completion date and only those trials registered in ClinicalTrials.gov. Second, an unknown number of trials may have been unregistered and, therefore, unavailable for audit. The 2007 FDA Amendments Act mandates registration of most non-phase I trials of FDA regulated drugs, biologicals, and medical devices but exempts other categories of National Institute of Health (NIH)-funded research trials, such as surgical, behavioral, or management procedures, or non-federally funded trials (22). It is likely that a large number of PBM/cardiac surgery–related trials may fall into this gray zone where investigators consider trial registration to be unnecessary. Therefore, results of this survey may not be entirely representative of all trials related to PBM and cardiac surgery conducted during the same time period. Third, it is possible that trial data may have been published in venues other than peer-reviewed journals indexed by PubMed, so some publications may have been missed. Nevertheless, the findings of this survey are congruent with those from other similar audits; e.g., less than half of NIH-funded trials were published in a peer-reviewed Medline–indexed journal within 30 months (22), and more than half of all trials registered with the EUCTR did not post results within 12 months of completion (23). Concerns about the large proportion of trials completed but with results that are unreported, suppressed, or reported late and trials that are apparently abandoned have been expressed for years, with little apparent change in investigator culture (13,23,24).

Investigators who perform clinical trials are accountable to patients, other healthcare providers, and study funders to publish their data in a timely and transparent way, regardless of whether findings are “positive” or “negative.” Accountability improvement measures have to date focused on the following four main initiatives. Prioritization of transparency items has been discussed previously and includes encouragement for prospective trial registration and reporting of summary results. Additional recommendations include increased accessibility to clinical protocols, supplementary trial materials (clinical study reports and statistical analysis plans), and IPD (25–27). Second, investigators must be incentivized to promptly and completely report all results. Penalties for noncompliance are already in place and should be enforced. Investigator or sponsor accountability could be increased by publicizing noncompliance (“named and shamed”) (23), followed by imposition and enforcement of sanctions on individual investigators for persistent noncompliance, such as denial of permissions to begin new trials and termination of funding until transparency and reporting requirements have been satisfied. Third, increased effort should be put into the design of more robust and relevant studies. Design elements include incorporation of bias minimization strategies in study design; more emphasis on clinically relevant, informative, and better specified and defined outcomes (28); and universal standardization of terms (such as “transfusion” and “massive bleeding”) with adoption in research protocols. A large proportion of potentially useful clinical evidence from observational trials and databases is overlooked because of potential bias and weaker evidentiary value. Hence, a long-term priority should be development of innovative statistical methods for combining and analyzing data from randomized controlled trials and non-randomized observational studies (29,30). Finally, the “power of the web” should be leveraged to enable increased linkage of trial information (27), incorporation of routine audit cycles (13,14), and development of “living” systematic reviews (31) for blood management.

There needs to be a massive sea change in the prevailing culture surrounding the design, conduct, and reporting of research results for transfusion and PBM-related studies. Implementation of recommendations described earlier will require the coordinated efforts of clinicians, applied and mathematical statisticians, information technology specialists, and research librarians. However, investigators themselves need to be reminded that failure to comply with best-practice standards for trial reporting results in harm to patients and is, therefore, both negligent and unethical.