Top Ten Tips Palliative Care Clinicians Should Know About Interpreting a Clinical Trial

Abstract

Evidence-based practice is foundational to high-quality palliative care delivery. However, the clinical trials that compose the evidence base are often methodologically imperfect. Applying their conclusions without critical application to the clinical practice context can harm patients. The tips provided can help clinicians infer judiciously from clinical trial results and avoid credulously accepting findings without critique. We suggest that statistical and mathematical expertise is unnecessary, but rather a keen curiosity about investigators’ rationale for certain design choices and how these choices can affect results is key. For a more comprehensive understanding of clinical trials, this article can be used with the authors’ corresponding ten tips article that focuses on designing a clinical trial.

Introduction

Clinical trials provide a reference framework for the practice of evidence-based medicine. Their data influence practices and guidelines, specifically data from randomized controlled trials (RCTs)—the gold standard for assessing an intervention’s efficacy. Though the number of clinical trials in palliative care is increasing, the applicability, relevance, and quality of the data can be poor and vary from trial to trial.¹ Given the heterogeneity of the palliative patient cohort, aspects of clinical applicability and generalizability can be particularly problematic within the narrow inclusion criteria of an RCT.

These limitations can pose challenges for palliative care clinicians, as how we interpret these data will invariably influence our clinical application of trial results. Reading between the lines of a trial can be daunting in this context, and subtleties in factors such as outcome measure validation, missing data, and selection bias (among others) can be confusing. This is further confounded when we consider varying levels of knowledge among multidisciplinary palliative care clinicians in critically assessing trial design or interpreting the validity of advanced methods (e.g., statistical modeling, rigorous qualitative or mixed methods analyses).

It is important to equip ourselves appropriately for this task. Here, our team has collaborated to demystify this process to aid clinicians in the interpretation of palliative care clinical trials. From our experiences and consensus opinion, we have delineated the following ten tips as a useful guide in this regard (Table 1). Though not exhaustive, these tips cover areas of particular importance or domains prone to misinterpretation in the process of trial appraisal. This article can be used in tandem with a previous article from our team that focused on the top ten tips palliative care clinicians should know about designing a clinical trial.²

Table 1.

Top Ten Tips Palliative Care Clinicians Should Know About Interpreting a Clinical Trial

Tip 1. Ensure the study is powered for detecting the outcome of interest by looking at the effect size estimation.

Tip 2. Ensure the primary outcome measure is valid.

Tip 3. Do not skim the methods section: translate patient characteristics to real-world practice.

Tip 4. Consider how missing data are managed and how this could influence results.

Tip 5. Consider if the study results are generalizable to your clinical practice.

Tip 6. Consider the funding source, their scientific reputation, and how they may have influenced presentation of results.

Tip 7. Rationalize the data: statistical significance does not mean clinical relevance.

Tip 8. Be cautious when inferring the safety of an intervention: Most studies are powered for the primary outcome measure, not adverse effects.

Tip 9. Be discerning about the negative and positive influences of trial design on outcomes.

Tip 10. Ensure the conclusion reasonably reflects the study results in the larger context of the overall study aims, design, implementation, and analysis.

Tip 1: Ensure the Study Is Powered for Detecting the Outcome of Interest by Looking at the Effect Size Estimation

Power provides the probability of correctly identifying the difference between two groups given the size of the sample and is usually set at 80%, which means 20% of the time, the investigator is willing to accept a type II error (failure to reject the null hypothesis when there is a true difference between groups or variable).³ In the book, Intuitive Biostatistics,⁴ Harvey Motulsky gives the following analogy: If you know you have a certain tool in your basement and you send your child down to get it, what is the chance the kid comes back with it? It depends on how long they spent looking, how large the tool is, and how organized the basement is. The elements of the analogy are analogous to the clinical trial’s sample size, the size of the effect, and how little scatter there is in the data.⁴ Overpowering a study by increasing the sample size also risks rejecting the null hypothesis when there are minimal or clinically trivial differences between groups.

The ideal study maintains power throughout the study. It avoids attrition, which underpowers the outcome, leading to an increasing risk of a false negative study or, by chance, in a minority of circumstances, a positive but exaggerated effect size (i.e., the winner’s curse phenomenon).⁵ The effect size is some measure of the treatment effect. It can be expressed in different ways. The winner’s curse phenomenon is a variation on the observation made by economists that the winner of an auction tends to overpay. Similarly, those studies that “win” publication tend to have exaggerated effect sizes (which is why they are published).

Pilot studies and first published underpowered studies often have an extreme bias in effect sizes (i.e., the Proteus phenomena) and have a low positive predictive value for a true effect size to the outcome.⁶ If the effect size of an underpowered pilot study is then used to power further studies to confirm the findings of the initial study, the replication studies will likely be underpowered and risk failing to replicate the pilot or finding a less impressive effect size than the pilot study. Pilot studies are not suitable to determine the effect size for subsequent studies. Finally, data in retrospect cannot be used to decide the power of a completed study from which the data are obtained. This means that larger subsequent studies following the same protocol should use an effect size based around a clinically meaningful difference, not the difference found in the pilot study.

In short, pilot studies are mainly for demonstrating a study protocol that is viable for a larger and more extensive study, but the results themselves should be taken with a very large grain of salt.

Tip 2: Ensure the Primary Outcome Measure Is Valid

When reading a clinical trial, it is critical to understand how the primary outcome is measured and whether this measurement is validated. For instance, in a study looking at different treatments for agitated delirium, the primary outcome measure might be the Richmond Agitation–Sedation Scale because it has been validated in severely ill patients.⁷ Many outcomes in palliative care are subjective in nature, such as patient-reported outcomes (e.g., pain scores, quality of life, and satisfaction) and clinician observations (e.g., performance status). Questionnaires are often used to assess patient-reported outcomes. Readers should look for details regarding the nature of questions (e.g., number of questions, how the questionnaire was administered), the answer choices (e.g., numeric scale or categorical), the time frame of assessment (e.g., “now” or “last week”), and instructions for interpretation (e.g., direction of scale, subscale scoring). Whenever possible, they should also describe the psychometric properties of the questionnaire, including reliability (i.e., reproducibility), validity (i.e., how well the questionnaire measures the intended outcome), linguistic validation (e.g., if the original version is in a different language), and responsiveness to change (e.g., a minimal clinically important difference).^8–10 Relevant references should be provided to substantiate the author’s claims.

As an evolving field, many outcomes in palliative care may not have been fully validated, and ad hoc questions such as “Do you feel at peace?” may be appropriate to assess some novel concepts.¹¹ The authors need to provide clear and transparent descriptions so that readers can be properly informed. Some studies may include several related measures to assess an outcome of interest. For example, terminal restlessness may be assessed with the Richmond Agitation–Sedation Scale as a primary outcome and both rescue medication use and caregiver observations as secondary outcomes.^12,13 The findings would be considered more robust if multiple complementary measures show consistent change.

Tip 3: Do Not Skim the Methods Section: Translate Patient Characteristics to Real-World Practice

RCTs work to minimize bias in the name of accurately measuring treatment efficacies. Populations reported in trials may lack the heterogeneity of patient demographics that we expect to see in the real-world clinical setting in terms of race, age, sex, and so on. Although stringent selection criteria are essential for assessing an intervention’s efficacy, they may raise the question of its real-world effectiveness outside of ideal conditions or in vulnerable patient groups.^14,15 If a population is restricted and does not reflect the palliative care population in your practice setting (in terms of diagnosis, age, performance status, and comorbidities), then be cautious about translating results into the clinical practice.

Understanding the interplay of internal and external validity in a trial aids translation to the clinical practice. What is true for the trial population may not hold when exposed to real-world clinical practice, for a variety of reasons. Of great import is a thorough assessment of a trial’s baseline demographics and screening criteria. The final population included in a trial assessment may only represent a fraction of the population accounted for by the eligibility criteria and may not represent the most symptomatic or at-risk patients.¹⁶ Geographical and societal factors, such as proximity to health services, socioeconomic status, or cultural beliefs regarding health care, are not often reported in RCT patient characteristics but can greatly affect participation, recruitment, outcomes, and, as such, the trial population.¹⁷

Assessing a trial population for heterogeneity of treatment effect (HTE) is also useful for translation to real-world practice. HTE refers to the variability of the magnitude of treatment effects that can be attributed to patient characteristics. Usually, HTE will be assessed in subgroup analyses contained in the RCT data, and as such, paying attention to treatment effects in these subgroups can aid extrapolation to real-world situations, for example, whether a novel adjuvant analgesic performs better on a neuropathic pain subgroup than on the entire study population. Factors to consider in interpreting this data include how confounding has been addressed, what statistical analyses have been used for interactions between variables, and how nonsignificant findings within these groups are reported.¹⁸

Tip 4: Consider How Missing Data Are Managed and How This Could Influence Results

Missing data are common in palliative care trials and may lead to bias. Large proportions of missing data may not only result in an underpowered trial, decreased precision, and type II error (false negative conclusions) but may also lead to type I error (false positive conclusions) if there is differential dropout favoring the intervention arm.

First, it is important to assess whether missing data were adequately reported, as well as the proportion of missing data and whether dropout differed by trial arm. There should be a Consolidated Standards of Reporting Trials (CONSORT) flow diagram showing the numbers of participants randomized, receiving the intended treatment, and analyzed for the primary outcome.¹⁹ Tables showing results should include denominators for each arm and endpoint, including for secondary outcomes.²⁰

Second, the pattern and reasons for missing data should be noted. Missing data for palliative care trials are generally not random. Thus, there is a risk of bias not only if rates of missing data differ by trial arm but also if dropout in the trial arms is for different reasons.²¹ For example, there will be bias in a trial of pain medication if participants in the intervention group withdraw because of side effects and those in the control group withdraw because of pain.

Finally, it should be assessed how missing data were handled and discussed by the research team. Although missing data often cannot be avoided in palliative care trials, sensitivity analyses should be performed to assess whether the result is robust across several different assumptions about the relationship between the missingness of the outcome and its true value.²¹ There should also be a discussion of limitations relating to the missingness of data, including the potential for bias and implications of the interpretation of the findings.²⁰

Tip 5: Consider If the Study Results Are Generalizable to Your Clinical Practice

When reading the results of a clinical trial, a clinician must consider the core question of generalizability. That is, “What does this mean for my practice and for my patients?”²² This question is particularly pertinent in palliative care research, whereby patients may differ across diagnoses, prognoses, and other clinical characteristics.²³ An approach to critical appraisal to assess the generalizability of a clinical trial can be found in Table 2. Using these outlined steps, it is possible to take a methodical approach toward reaching an informed decision about generalizing outcomes to the clinical practice.

Table 2.

Critical Appraisal Process to Assess Clinical Trial Generalizability

Guiding questions/steps	Appraisal considerations
Are the trial participants like my patients?	•Careful scrutiny of the eligibility criteria and descriptions of participants should occur including participants’ demographic details (e.g., gender, racial/ethnic diversity, social support, economic status, geography), clinical factors (e.g., diagnosis and disease stage and severity, comorbidities, performance status, timing of palliative care referral and of death), and caregiver factors (e.g., relationship, availability, and health status).22
	•Reviewing the trial’s CONSORT diagram will provide reasons for nonparticipation and for trial withdrawal, potentially highlighting systemic issues with implications for practice.35
Is the trial setting like my clinical setting?	•As the service structure, funding models, and policies of palliative care differ markedly, understanding the trial setting is essential.22 For example, if recruitment is drawn from a palliative care service, then referral policy will influence the study population.22
Are the intervention and its outcomes clearly defined and relevant?	•A clear, comprehensive description of the intervention is essential for it to be delivered as detailed into clinical practice. Furthermore, the benefits sought must be clinically significant (not the same as statistically significant) for patients.35
Be curious.	•The clinician appraiser should reflect on their own biases. Clinicians may have strong convictions about treatments, which may influence how they consider the evidence. Take a moment, reflect, and consider if this may be the case.

CONSORT, Consolidated Standards of Reporting Trials.

Tip 6: Consider the Funding Source, Their Scientific Reputation, and How They May Have Influenced Presentation of Results

Should you trust this article?

When you read an article, you make yourself vulnerable to the goals and values of its funders, as well as your expectation that the results will improve your knowledge or the hope that it will make you a better clinician, educator, researcher, or administrator. It is, therefore, important to consider the trustworthiness of any article, especially clinical trials with a higher potential to influence your thoughts and actions.

Unfortunately, medical literature is frequently a minefield of exaggerated or unsupported claims. There is always a risk that the article may be flawed because of failures of investigators’ rigor, competence, or integrity:

• •Failures of rigor are largely due to problematic designs. (See the authors’ corresponding article on the top ten tops palliative care clinicians should know about designing a clinical trial.)²
• •Failures of competence are mostly inadvertent and easier to detect. Several tips in this series cover frequent failures of competence in the design, execution, and interpretation of a clinical trial.
• •Failures of integrity are mostly intentional and much harder to identify. These relate mainly to financial conflicts of interest and academic fraud. Corporate funding is a major predictor of a study reporting positive findings.²⁴ Unfortunately, the source of funding is not always fully disclosed.

Some markers to increase trust in an article’s results:

• •Studies funded by granting agencies undergo more independent scrutiny.
• •More reputable academic institutions conduct a more thorough initial and continuing review of research studies and several layers of vigilance over data integrity.
• •Better quality journals conduct a more thorough review of manuscripts, including in many cases reanalysis of the results.
• •Look for investigators/authors with a strong track record of publications and grants (ideally from independent sources).

If a study’s results seem too good to be true, they likely are! Please consider that a single study should mostly only provide you with interesting information that needs further confirmation by other groups and is ideally independently funded.

Tip 7: Rationalize the Data: Statistical Significance Does Not Mean Clinical Relevance

Is this result clinically significant? That is, will it change your practice?

Only clinician judgment can confer clinical significance upon a result. Relying on statistical tests to justify clinical significance is a category error.

Consider a hypothetical, adequately powered trial in which the intervention produced a median improvement of 0.4 out of 10 compared with the comparator arm. The p-value was 0.01. For the sake of argument, assume the trial is otherwise sound. For instance, this could be a trial of medicinal cannabis for sleep-related disturbances.

What the above tells us is that if all of the assumptions about the statistical model including the null hypothesis were true, there is a 1% chance of observing a difference as large as 0.4 (or larger).²⁵ Our stated assumption that the trial is sound lets us accept this statement as true for the sake of this explanation. So, there is a 1% chance of observing a difference as large as 0.4 (or larger) if all of the statistical assumptions hold. (It is actually more complicated than this, as Calquohn convincingly argues. But this simplification is warranted for the present purpose.)

Yet, few palliative care clinicians will be enthused by such a small benefit. Our experience tells us this sort of improvement will not mean much by the bedside. Implied is the notion that there is some threshold value that reflects the minimum clinically important difference (MCID) and that an improvement of 0.4 on an 11-point numerical rating scale is likely below this.²⁶

The MCID has more recently been defined as “the smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate, in the absence of troublesome side effects and excessive cost, a change in the patient’s management.”²⁶ It is important to note that statistical significance is not enough, and the magnitude of difference between the two arms needs to be clinically meaningful; one way of formalizing the minimum threshold for this is the MCID concept.

Tip 8: Be Cautious When Inferring the Safety of an Intervention: Most Studies Are Powered for the Primary Outcome Measure, Not Adverse Effects

Although establishing efficacy is of clinical importance, clinical trials aim to determine net clinical benefit to help us understand who will benefit without harm from the study intervention. Determining how adverse events were assessed and reported in a trial is of utmost importance. Ideally, harms should be assessed on an intention-to-treat basis, considering grade and attribution, using internationally recognized systems.²⁷ In addition, adverse events of interest should be based on what is known about the intervention to date.²⁸ Evaluating if the clinical trial conforms with the standard of reporting outline in the CONSORT Harms statement is important to understand the level of confidence in the safety data presented.

The majority of clinical trials have a sample size and power calculation based on the primary outcome of a clinically relevant outcome related to clinical benefit. Clinical trials are typically statistically underpowered to detect a specific harm, and this can be compounded if the recruitment focuses on a lower-risk population not reflective of real-world clinical populations, does not include robust and focused methods to ascertain events, and if the adverse event is rare or has a long latency period.²⁹ For example, albumin replacement to avoid circulatory dysfunction for patients with malignancy-related ascites undergoing paracentesis is not supported by a lot of evidence, but this might be different for patients whose malignancy-related ascites is due to cancer-related portal hypertension and/or comorbid cirrhosis. Considering if the population in the trial reflects the population in whom you may consider giving the intervention can be helpful. Consideration of phase IV trials in the real world,³⁰ analysis of safety data collected over multiple trials using individual patient safety data,^27,31 and the use of novel methods of analysis such as Bayesian hierarchical models are approaches to be considered to better understand the safety of interventions.³¹

Tip 9: Be Discerning About the Negative and Positive Influences of Trial Design on Outcomes

Traditional RCTs are recognized to provide the highest quality information. These studies, also named explanatory trials, often form the foundation for research/trial decision-making strategies. However, the real-world experience may be different, and the proscriptive nature of RCTs may limit the degree to which the results can be generalized. A valid criticism of RCTs has been that design and inclusion criteria are restrictive (to minimize confounding) and can mean that any results are challenging to apply to a population seen in real-world clinical practice. This can mean that the clinician can be faced with a positive result that they struggle to fit into their practice or a negative result on trial participants that they do not think would apply to their patients.

Indeed, practical clinical trials (PCTs) (i.e., pragmatic clinical trials) can bridge the gap between research and clinical care as they compare different treatments under conditions in which they would be applied in practice. Although PCTs may have less power to detect a clinical effect, they provide data to increase generalizability to clinical practice settings, demonstrate improvement in health outcomes and cost-effectiveness, establish evidence-based guidelines, and enhance their adoption in clinical practice.³²

Tip 10: Ensure the Conclusion Reasonably Reflects the Study Results in the Larger Context of the Overall Study Aims, Design, Implementation, and Analysis

Interpretation of study results often requires a thorough review of the research objectives, study design and implementation, analysis, and presentation of results. The conclusion should summarize all the key findings of the study. In palliative care, there may be a very important research question answered using well-defined objectives and design, but the results may be limited by poor accrual, attrition, missing data, inability to reach the desired sample size, and limited funding. Therefore, interpretation of study results often requires a review of the entire study description in the research paper and not just the conclusion of the study, as it may not be a complete representation of the intervention effects.

To reasonably reflect the study results, the conclusion should accurately represent the outcome of an intervention on a validated measure as proposed in the hypothesis of the protocol and using a design to achieve the study objective. In other words, “Was the study able to negate the null hypothesis of the primary objective?” If the conclusion deviates from this description (for or against the hypothesis proposed), then it may not add to the strength of the evidence to support the proposed intervention. Excessive space dedicated to secondary outcome measures instead of a negative result for the primary outcome measure in the conclusion is an example. Similarly, the conduct and analysis of the study that deviates from the proposed design make interpretation of the study results challenging, and the conclusion should reflect the design change. In sum, various factors need to be considered for the conclusion to reasonably reflect the study results, which include “Does the conclusion reflect what was proposed in the hypothesis?” “Was the study’s conduct affected by attrition, missing data, and inability to attain the proposed sample size?”^33,34

Conclusion

Though not exhaustive, these ten tips cover core issues inherent to clinical trial interpretation. As the breadth and depth of palliative care research continue to expand, so too must our ability to discern its appraisal. Ultimately, the end goal of any palliative care research should be to improve the patient experience and outcomes. This goal should always be kept at the forefront when applying research to clinical practice. Maintaining curiosity and critical thinking in clinical trial appraisal can help in quality control, the detection and avoidance of harm, and when identifying new frameworks of care for our patients and their caregivers.

Author Disclosure Statement

Funding: DH is supported in part by grants from the National Cancer Institute (R01CA214960; R01CA225701; R01CA231471). WER is partially supported by the NIH/NCI comprehensive cancer center award number P30CA008748. SY – Grant to institution: Pfizer (cachexia study). TG, JH, PG – supported in part by Mater Foundation.

Funding Information

No funding was received for this article.