Study Types in Orthopaedics Research: Is My Study Design Appropriate for the Research Question?

Abstract

When performing orthopaedic clinical research, alternative study designs can be more appropriate depending on the research question, availability of data, and feasibility. The most common observational study designs in total joint arthroplasty research are cohort and cross-sectional studies. This article describes methodological considerations for different study designs with examples from the total joint arthroplasty literature. We highlight the advantages and feasibility of experimental and observational study designs using real-world examples. We illustrate how to avoid common mistakes, such as incorrect labeling of matched cohort studies as case-control studies. We further guide investigators through a step-by-step design of a case-control study. We conclude with considerations when choosing between alternative study designs.

Graphical abstract

Orthopaedic studies traditionally rely on arthroplasty registries, administrative databases or single-center surgical series to investigate concerns that arise from clinical practice. These studies use readily available data on a limited number of patients (eg, demographics, comorbidities) and surgical characteristics (eg, surgical approach, implant types, surgical complications). In recent years, there has been an increasing interest in understanding the role of new risk factors and outcomes but the traditional approaches to data collection and analyses typically fall short in addressing them, underscoring the need to adopt alternative study designs. The choice among alternative study designs is determined by feasibility constraints and the research question. In this article, we describe the different study designs with an emphasis on selecting the most efficient study design for investigating different types of exposures and outcomes of interest.

There are two broad categories of study designs: experimental and observational (Fig. 1). In experimental studies, researchers assign the exposure(s)/treatment(s) to a group of total joint arthroplasty (TJA) patients and follow them up over time to measure the effect of the exposure or treatment on the outcome(s) of interest. More often than not, experimental studies are not feasible in TJA research due to ethical, time, or financial constraints [1–3]. Since TJA is an elective procedure and outcomes of interest, such as time-to-revision, take years to properly measure, many research questions in TJA are investigated using observational rather than experimental studies. Furthermore, availability of large registries and databases make observational studies an efficient design choice when data are readily available. In observational studies, researchers do not assign patients to the exposure(s)/treatment(s) but rather identify TJA patients and observe their exposure status and outcome(s). The most common observational study designs are cohort, case-control, and cross-sectional studies. Studies in registry databases are typically cohort studies, are observational in nature, and are one of the cornerstones of TJA research.

Fig. 1.

Types of studies.

Observational studies are either prospective (ie, data are collected concurrently as they are measured) or retrospective (ie, registry-based studies where data are collected after they occurred). There are also circumstances in which prospective data are used in a retrospective analysis. This is the case, for instance, of carefully planned collection of research data that are not part of routine registry data collection. Prospective studies are stronger in their level of evidence since they are less prone to systematic bias [4]. Level of evidence is usually highest for randomized trials, followed by prospective cohorts, retrospective cohorts, retrospective case-control studies, and lastly cross-sectional studies [5].

The Magic of Randomization in Experimental Studies

Randomized controlled trials (RCTs) are considered the gold standard of study design. In RCTs, the exposure(s)/treatment(s) or placebo is randomly assigned so that the comparison groups are more likely to be balanced with respect to both known and unknown risk factors and their risks of any type of health outcome. By design, RCTs hypothetically eliminate confounding and minimize selection bias, thus, simpler statistical methods can be employed to achieve unbiased results [4,6]. Patients are followed prospectively and the outcome(s) of interest are measured as they occur at selected time points. RCTs are powerful study designs but uncommon in orthopaedics [7,8] because often they cannot be used for ethical, financial, and/or feasibility concerns [9,10]. Another difficulty in conducting RCTs in TJA research is the complexity of surgical interventions. Some of the challenges affecting the use of RCTs in TJA include patient and/or surgeon’s strong preference of surgical intervention, difficulties of blinding surgeons and patient/caregivers, surgeons’ differing capability with different surgical procedures, and the controversial use of placebo surgery [9–12]. Therefore, RCTs with medical interventions in TJA are easier to implement than surgical interventions, such as prophylactically assigning TJA patients to antibiotics or placebo following TJA surgery and observing their infection status, or different anti-coagulation strategies for prevention of thromboembolic events. Since TJA surgery itself and many associated surgical interventions cannot be ethically assigned via randomization parameters (eg, blood transfusion, antibiotic prophylaxis), much TJA research consists of observational studies.

RCTs are either explanatory or pragmatic [13]. Explanatory trials include a small number of patients and strict eligibility criteria with the primary purpose of testing the efficacy of a surgical or medical intervention in an ideal setting. In contrast, pragmatic trials aim to investigate the effectiveness in real-world practice and patients are randomized at the group level (eg, hospital, clinic, etc) rather than patient-level. In recent years, pragmatic trials are increasingly used as an alternative to observational studies for providing evidence on treatment effectiveness in a more generalizable setting, combining the rigorous process of RCTs with the real-world nature of observational studies [14].

Observational Studies

Cohort Studies

Registry-based studies in TJA are frequently performed as cohort studies. TJA population of interest is defined and followed over time to see if an outcome(s) occurs in various exposure subgroups. Cohort studies collect repeated measurements for each patient at various time points (1, 2, 5, 10, etc. years after surgery), allowing for longitudinal data analysis. With repeated measurements, research questions regarding the change in outcomes over time (eg, multiple revision surgeries, TJA in other joints) or the trajectory of outcomes (eg, functional status) can also be examined. Cohort studies typically collect information on multiple outcomes. Cohort studies are not subject to recall bias (ie, patients do not remember their previous exposures) but they are subject to selection bias because obtaining a TJA cohort that is representative of general TJA population of interest is challenging. For example, a TJA cohort study that is restricted to elderly Medicare patients is not representative of the younger TJA patients whose underlying surgical indications, implant types, or comorbidity profiles are different. Similarly, cohort studies in foreign registries with different implant types or surgical approaches will not be representative of TJA patients in the United States. A common way to overcome selection bias is to adjust for factors that are known to confound the relationship between exposure and outcomes. All confounding variables must be measured and adjusted for in analysis for results to be unbiased [4].

Confounding variables can be accounted for using different techniques including multivariable adjustment, propensity score matching, and propensity score weighting. Caution should be used when matching on propensity score, a score that summarizes all the covariates into one scalar. Propensity score can potentially increase imbalance relative to the original data. Therefore, it is always recommended to conduct diagnostics to assess balance in covariates after propensity score matching.

Cohort studies are usually overpowered to detect common TJA outcomes (eg, length of stay, discharge location, functional status), thus, associations of interest may have very small P values, even if the association is not clinically meaningful [15]. For example, in a large study of length of stay in the Medicare population, the reduction in length of stay over time can be as small as 10% but highly significant (P < .0001) due to the large sample size. Other limitations of cohort studies include a potential loss to follow-up and high costs associated with following up a large number of patients especially if required for long periods of time.

Arthroplasty registries contain prospectively collected information on individuals who undergo TJA, capturing data on only a limited number of patient characteristics (age, gender, and race), implants, and surgical details. Although valuable resources, registry-based cohort studies can present some methodological limitations which include large statistical power, incomplete mortality/follow-up data, and absence of important confounders and mediators (comorbidities, medications, and socioeconomic factors) [16]. Therefore, alternative study designs are needed to address different types of risk factors and outcomes in TJA.

Case-Control Studies

Case-control studies are efficient study designs that are used in the setting of rare outcomes that may take several years to develop (eg, cancer) or in hard to ascertain exposures such as family history or genotypes. When the outcome is rare (eg, outcomes that occur in less than 5% of patients), it is not practical to prospectively follow a large TJA cohort until enough patients experience the outcome. In this setting, TJA patients who experience the outcome (cases) are identified, ideally from an already existing TJA registry or database (Fig. 2). Then, TJA patients with similar characteristics who do not experience the outcome (controls) are selected. After identification of cases and controls, the exposure of interest is measured retrospectively. In the setting of difficult-to-ascertain exposures, such as genotypes, normally the exposure data are not collected on the entire registry cohort but are instead collected only in cases and controls. The selection of controls is one of the most challenging aspects of case-control studies since improper selection of controls can lead to large biases and/or results that are not generalizable. When the outcome is rare, the odds ratio calculated in a case-control study approximates the risk ratio that would be measured in a cohort study while using fewer resources.

Fig. 2.

Case-control study design.

In TJA research, matched cohort studies are sometimes incorrectly labeled as case-control studies. The adjective case-controlled is also incorrectly used for studies with a comparison group. If a study begins with the exposure and follows patients for a few years to measure outcomes (revision), it is a cohort study. By contrast, if a study begins with the outcome (revision), and looks back in time for an exposure, then that is a case-control study.

Consider a study where the researchers want to examine whether bisphosphonate use in total hip arthroplasty (THA) is protective for periprosthetic fractures. The exposure of interest is bisphosphonate use and the outcome is periprosthetic fractures. If the researchers decide to perform a matched cohort study, they would first identify THA patients who used bisphosphonates (exposure) at the time of their primary THA surgery, match them on age, gender, calendar year, surgical indications, and/or other factors to THA patients who did not use bisphosphonates at the time of their primary THA surgery, and follow-up both groups for a period of 2 years for occurrence of periprosthetic fractures (outcome). By contrast, if the researchers decide to perform a case-control study, they would first identify patients with periprosthetic fractures (cases), match them on age, gender, calendar year, and other factors to THA patients who did not experience periprosthetic fractures within 2 years of their primary THA surgery (controls), and review the medical records of all cases and controls to determine whether they had used bisphosphonates (exposure).

Here are the four steps to follow when designing a case-control study:

• Define explicitly the sampling frame and eligibility criteria (inclusions and exclusions) for cases. It is important to include all cases from the sampling frame and not just a convenience sample. Sampling frame could be a national TJA registry, single hospital or a population in a well-defined geographic area. Some case-control studies are population-based and some are hospital-based studies (ie, cases which are treated at a hospital during a period of time). Cases are preferably incident (new onset) cases rather than prevalent (both old and new) cases. Diagnosis patterns may change over time and therefore recently diagnosed cases are likely to be more consistent than those obtained over many years.

• Identify controls from the same source population or sampling frame as the cases. Make sure that the control selection is independent of the exposures of interest. The purpose of the control group is to determine the distribution of exposure in the source population. It is easier to perform case-control studies nested into registries because the registry population is a well-defined source population for control selection. Hospital-based case-control studies, especially at referral hospitals, are harder to perform because it is hard to define the source population for control selection. Cases are matched to one or more controls based on variables that are believed to be confounders to reduce the confounding effect of these factors. Matched case-control studies are more efficient than unmatched studies but matching on variables that are not confounders (ie, mediators, colliders [4]) may introduce bias. For example, while it might be tempting to match for smoking status in a study of periprosthetic fractures (because smokers have a higher risk of fractures), this would only be necessary if smoking status also influences the likelihood of using bisphosphonates (which is unlikely). It is therefore important to consider whether a variable is a true confounder before matching on it. Finally, identifying controls in hospital-based case-control studies is more challenging and in general to be avoided.

• If the study involves a survey or interviews to gather information about past exposures, make sure that the interviewers are well trained and also blinded to case control status of study subjects or at least the main hypothesis of the study.

Case-control studies are rare in orthopaedics but they have been used to assess risk factors for some rare TJA outcomes such as periprosthetic joint infections, dislocation, and revision surgery [17–19]. Case-control studies are a less efficient option when there is no well-defined sampling frame for identifying the cases and controls, when obtaining an exposure history is challenging, or when the frequency of exposure is low (Table 1). As a rule of thumb, cohort studies are more efficient than case-control studies when the incidence of an outcome is higher than the prevalence of exposure of interest. For example, the incidence of revision following total knee arthroplasty (TKA) is around 2% at 2 years, whereas the background prevalence of systemic lupus erythematous is around 0.1%. In this case, since the exposure is rare, a cohort study is more efficient than a case-control study when studying whether systemic lupus erythematosus is a risk factor for revision surgery. In contrast, the incidence of acute kidney injury following TKA is <2%, whereas the prevalence of nonsteroidal anti-inflammatory drugs use is at least 25% in TKA candidates. In this case, giving the small incidence of the outcome, a case-control study is more efficient than a cohort study when studying whether nonsteroidal anti-inflammatory drugs use is a risk factor for acute kidney injury following TKA surgery.

Table 1.

Advantages and Disadvantages of Cohort Versus Case-Control Studies.

Cohort Studies	Case-Control Studies
It is suited for studying rare exposures.	It is suited for studying rare outcomes.
It allows to study multiple exposures and multiple outcomes.	It allows to study multiple exposures.
It allows computation of risk ratio and rate ratio.	It allows estimation of risk ratio from odds ratio but it is not feasible to estimate incidence.
It is possible to show that the outcome follows the exposure because subjects are disease-free at the beginning of the observation period when exposure status is defined.	It is difficult to determine temporality between exposure and outcome because the ascertainment of exposure is done after the outcome.
The nonexposed group is easier to identify.	The selection of the control group might be biased if control subjects are not truly representative of the population that produced the cases.
Larger sample size is required because the rate of outcome is usually smaller than the prevalence of the exposure.	Smaller sample size is needed.
It is more expensive if done prospectively as a prolonged follow-up period is needed for the outcomes to occur.	It is less expensive. It does not require a prolonged follow-up period because the outcome has already occurred.
It can be affected by loss-to-follow-up which may lead to bias.	It is not affected by loss-to-follow-up because there is no follow-up.
It is fairly easy to understand.	It can be more difficult to understand.
The study duration is longer.	The study duration is shorter.

Cross-Sectional Studies

In cross-sectional studies, exposure(s) and outcome(s) are measured at the same time (Fig. 3), and as a result, a different sample of people is observed at every time point of interest. This means that the temporal relationship between an exposure and outcome is unknown. In a cross-sectional study, investigators may describe the characteristics of the population, assess the prevalence of a condition, or test for a cross-sectional association between an exposure and outcome, but when these are measured at the same point of time, results should generally be interpreted with much caution. Because of the nature of cross-sectional study design, limitations that include the inability to determine incidence and difficulty in making casual inference can arise. An example of a cross-sectional study in TJA research is the evaluation of the association between bisphosphonate use and radiological findings in TJA patients. For example, although bone mineral density may be better in bisphosphonate users, conclusions cannot be made regarding causation.

Fig. 3.

Cross-sectional study design.

Observational studies that explore the association between outcome and exposure at the population level rather than at the individual level are called ecological studies. These are typically used to measure trends in incidence using aggregate data instead of subject-level measurements. Examples of ecological studies in orthopaedics include the assessment of the geographic variation in arthroplasty rates and associated factors such as area-level measures of socioeconomic deprivation or surgeon supply [20].

Guidelines for Researchers and Reviewers

To decide the type of study that answers specific research questions, investigators should consider the following items:

• What is the primary hypothesis of the study? Defining the primary purpose of the study will guide plausible study designs. If the goal of the study is to explore a rare exposure, such as effects of HIV-infection in TJA patients, a cohort study is more appropriate than a case-control study.

• Am I evaluating a novel intervention for which there are no available retrospective data? If that is the case, an experimental design, with appropriate considerations of ethical, financial, and safety aspects, may be the right way to go.

• What is the study budget in terms of money, time, and other resources? Clinical trials are expensive as they require time, money, and the involvement of several individuals to collect, manage, and analyze the data. Clinical trials are also more expensive than cohort studies. Aims and overall study design may need to be adjusted based on budget considerations.

• Is a clinical trial being reanalyzed to study an exposure that the patients were not randomized on? If this is the case, the new study may need to be treated as an observational study.

• Outside of clinical trials, an effort should be put toward collecting data on known confounders of the exposure-outcome relationship to reduce bias. Minimizing missing data is also important since data missing conditional on other factors biases the results if not accounted for properly.

Conclusion

In orthopaedic research, study design is even more important than data analysis because it directly impacts the level of evidence of the study’s conclusion. Although RCTs are the gold standard, they often are not feasible in orthopaedic research. Many orthopaedic studies are cohort studies using existing data from large TJA registries and databases. In any research study, consideration should be given to efficiency of study design and potential clinical significance of the results.