The three-minute appraisal of a prospective cohort study

INTRODUCTION

The prospective cohort study (PCS) is a valuable tool with important applications in epidemiological studies. The study involves the comparison of a cohort of individuals displaying a particular exposure characteristic, with a group of individuals without the exposure characteristic in the format of a longitudinal study.¹ PCSs offer researchers the advantage of measuring outcomes in the real world without the ethical and logistical constraints faced by randomized control trials (RCT). However, PCSs face concerns with internal validity due to the presence of selection bias and confounding variables. The purpose of this paper is to provide clinicians with guidelines for the critical appraisal of a PCS [Table 1].

Table 1.

A three-minute checklist for the critical appraisal of a PCS

KEY CRITERIA FOR CRITICAL APPRAISAL

Step 1: What are the comparison groups in the study?

Comparison groups in a study may include a control group compared to a treatment group; a treatment group compared to another treatment group; or a control group compared to a variety of treatment groups. The first step when assessing a study design is to determine the comparison made between groups.² Well-written studies should have clear guidelines for the selection of all groups. The next step involves assessing the applicability of the study's design to clinical populations.² For instance, if you are interested in assessing whether a drug prescribed for hypertension is better than other commercially available drugs, you would want to refer to a study comparing side effects of the drug of interest to other hypertension medication; rather than comparing it to a study examining effects of multiple drug classes. Finally, it is also important to evaluate study design for potential selection biases.² In order to minimize the presence of selection bias in studies, baseline characteristics should be relatively consistent between groups. Some sources of selection bias are quite evident for example, sending the treatment group patients to a specialist for surgery and control group or another treatment group's patients to a regular surgeon. Some sources such as channelling bias are subtle and may involve older patients being allocated to a specialist surgeon only, because they constitute a high-risk population. When reading an article, it is important to evaluate both sources and interpret effects on the results of the study

Step 2: How do confounding variables impact the study?

Confounding variables correlate positively or negatively with the independent or dependent variable. Confounding variables are viewed as the principal contributor to a false positive test. PCSs are vulnerable to both known and unknown confounders because patient allocation is not randomized.³ A literature review of material on the topic can help identify known confounding variables in study designs. When assessing a PCS for confounding variables, it is important to assess the information provided on confounders present in the intervention and comparison groups.³ Most of the papers provide this information in tabular format. One must also consider methods used to assess for differences in potential confounders between the two groups. Common tests used include the χ² test or t-test; however, the significance is sensitive to sample size which can make test results significant but not clinically meaningful.³ Another strategy is to use standardized differences to examine between group differences. This measure is not as sensitive to sample size as traditional tests, and provides a sense of relative magnitude of difference (differences greater than 0.1 are considered to be meaningful).

Step 3: What analytical strategy was used to assess results?

Finally, one must consider the analytical strategy used to assess results. A common analysis technique used is a regression analysis which looks at the relationship between an independent and dependent variable after adjusting for the effects of other independent variables.⁴ Another technique known as stratification involves dividing data into homogenous subgroups, followed by sampling for potential confounding variables among each subgroup. When assessing a study, it is important to look at the analytical strategy used, and which confounders were incorporated in the analytical model.⁴ It is also important to check the difference between the adjusted and unadjusted results. A large difference implies significant differences between baseline characteristics of the cohort subgroups, indicating a risk of selection bias. One must also consider is the credibility of the results, which can be assessed using a sensitivity analysis. The sensitivity analysis simulates the size and level of imbalance created by a potential confounder, which allows one to determine the extent to which confounders were incorporated in the creation of the study design.⁴ Lastly, one should consider the biological validity of the results. This is a relatively convoluted question, and the answer varies among different studies. However, readers can avoid confusion by comparing results to relatively similar cross-sectional studies.

Practical example

Article: Orosz GM, Magaziner J, Hannan EL, Morrison RS, Koval K, Gilbert M, et al. Association of timing of surgery for hip fractureand patient outcomes. JAMA 2004;291:1738-43.⁵

Study objective: To examine the association of timing of surgical repair of hip fracture with function and other outcomes.

Selection of comparison groups

1. Comparing function and other outcomes among patients eligible for early (≤24 hours) and late (>24 hours) surgery.

2. Incidence of hip fractures relatively high in the field of orthopedic surgery making this clinically relevant.

3. To lower the risk of selection bias, the investigators adjusted the analysesfor a range of variables used by clinicians; used propensity scoremethods to match cases of early and late surgery andrepeated the analyses by excluding patients who might not be appropriatecandidates for early surgery.

Impact of confounding variables

1. Previous studies yield conflicting results on relationship between early hip fracture surgery and mortality. No information about relationship with functional outcomes.

2. Propensity score had aC statistic of 0.68. Indicates no significant difference in baseline characteristics of both cohorts.

3. Two types of sensitivity analysis performed; one using propensity scores and one examining what happened if patients who were not candidates for early surgery were excluded.

Analytical strategy used

1. Least squares regression (for continuous outcomes),logistic regression (for binary outcomes), and Cox proportional hazards regression for main analyses. Controlled for 18 different variables (age, history of diabetes, hospital site, etc.).

2. Difference of 0.04-1.94 between adjusted and unadjusted results indicates consistent baseline characteristics between cohort subgroups.

3. Physicians should aim for early operative procedures to reduce patients′ pain and length of hospital stay.

CONCLUSION

Prospective cohort studies are vulnerable to selection bias and confounding factors, which can affect the validity of the results provided. When evaluating these studies, readers must use an organized approach to critically appraise the design and content of the study, as well as the applicability of the results to clinical populations [Table 2].

Table 2.

Do's and do not's when critically appraising a prospective cohort study