Previous articles in this series discussed observational study designs, including cross-sectional, cohort, and case-control studies. This paper focuses on understanding causal inferences and methods to improve them for observational studies. For example, an observational study might show an association (correlation) between a predictor variable (independent variable) and an outcome (dependent variable); however, the results may not represent cause and effect (Hulley, 2013). Causal inference implies an intervention, e.g., treatment or behavior was the ‘cause’ of the effect (or outcome). Understanding causal inferences between predictor(s) and outcome(s) can provide insights to understanding the etiology of a disease, identify methods to prevent or reduce disease (or occurrence) and potentially initiate the development of treatments (Hulley, 2013). For example, are eating carrots associated with improved eye health?
It is important to note that some associations found in an observational study do not represent cause and effect. However, there are well-recognized explanations for associations between a predictor and outcome in such studies (Hulley, 2013). These occur by chance (random error), bias (systematic error), and confounding variables. The remainder of this paper will describe each item and methods to minimize these factors.
Explanations: Chance/Random Error
Chance or random error creates associations between a predictor and an outcome from an observational study sample which is erroneous (Hulley, 2013). In other words, the association does not exist in the population. For example, let’s say that there is no association between sedentary behavior and obesity among HIV+ participants seeking care from a specific clinic, wherein 35% are sedentary based on a physical activity questionnaire. If 20 HIV+ participants who are obese (body mass index (BMI) 30 or greater) and 20 HIV+ participants who are normal weight (BMI less than 24.9) were selected for a study, it would be expected that 7 participants per group (35% of 20) would be sedentary, using a physical activity questionnaire.
However, by chance, 12 sedentary participants are in the obese group and 4 in the normal weight group. Thus, the observed outcome is an erroneous (spurious) association between physical activity and obesity if this occurs. Chance is also called random error because it does not have an explanation. However, if the association from a random error is statistically significant, it is called a type I error. This error is made by rejecting the null hypothesis, a statement about the predicted relationship between variables that states no relationship exists between the variables under study (Polit, 2022). In other words, the variables do not affect each other. See Table 1 for calculations.
Table 1.
Example 1: Odds Ratio Calculations
| Outcome |
Exposed Obese (Body Mass Index ≥ 30) |
Unexposed Non-Obese (Body Mass Index < 24.9) |
Total |
|---|---|---|---|
|
Disease Sedentary *(Low Activity Level) |
12 a |
4 b |
16 (a + b) |
|
No Disease Not Sedentary *(Moderate to High Activity Level) |
8 c |
16 d |
24 (c + d) |
| Total | 20 (a + c) |
20 (b + d) |
40 (a + b + c + d) |
measured by the Physical Activity Questionnaire
- a = exposed participant and acquires the outcome of interest
- b = unexposed participant and acquires the outcome of interest
- c = exposed participant and does not acquire the outcome of interest
- d = unexposed participant and does not acquire the outcome of interest
Odds Ratio = ad/bc = 12×16/4×8 = 192/32 = 6
- OR = 1 Exposure did not effect the odds of the outcome
- OR > 1 Exposure is associated with the higher odds of outcome versus nonexposed group
- OR < 1 Exposure is associated with lower odds of outcome verus exposed group
- Upper 95% CI = e ^ [ln(OR) + 1.96 sqrt(1/a + 1/b + 1/c + 1/d)] = 1.4483
- Lower 95% CI = e ^ [ln(OR) - 1.96 sqrt(1/a + 1/b + 1/c + 1/d)] = 24.6868
- P value = 0.0130
Methods to Reduce Chance/Random Error
Methods to reduce random error can be minimized at the research design phase (Hulley, 2013). First, investigators can select instruments (e.g., questionnaires, equipment) to measure the study variables (Bernadette Capili, 2021a). For example, for the HIV obesity and sedentary example study, select a validated questionnaire to assess physical activity and body weight. Next, review the psychometric properties of the tool (please review previous articles in this series Measurement in Nursing Research (Curtis & Keeler, 2021) and Selection and Implementation of Outcome Measurement (Bernadette Capili, 2021a)) and develop an operations manual to properly ensure correct usage of equipment and calibration such as a weight scale. Additionally, the manual should detail how to implement the study, including administering questionnaires and collecting specimens (Bernadette Capili, 2021a).
During the analysis phase, the investigators can calculate the p- values (probability value or the level of statistical significance) and confidence intervals (CI) to determine the degree of the observed association (the data the investigator found) compared with what could have occurred by chance (Hulley, 2013). The p-value number describes the likelihood that the collected data could have happened by random chance; that is, the null hypothesis was true (no relationship exists between the variables). The p-value or level of statistical significance is expressed as a number between 0 and 1, which the investigator selects the value they will accept as significant. Usually, a p-value ≤ 0.05 is considered statistically significant, meaning less than a 5% probability that the null hypothesis is correct. Therefore, the investigator should reject the null hypothesis. Alternatively, a p-value > 0.05 is not statistically significant. Therefore, it indicates that the investigator does not reject the null hypothesis (i.e., a relationship does not exist between the variables) (Majid, 2018). It is important to note that an investigator does not accept the null hypothesis but only rejects the null or fails to reject the null hypothesis. Therefore, a statistically significant result does not imply with 100% certainty that the finding is correct. Furthermore, it is critical to be aware that statistical significance does not signify that a random error did not occur. Usually, when results are published with a statistically significant finding, an investigator might write, “Our results provide support for our findings that HIV+ participants who are sedentary have a greater likelihood for obesity.” A future article will focus on inferential statistics.
Now let’s review CI. CI is often more useful than p-values in determining statistical significance (Hulley, 2013) because it measures the precision of the association. In addition, a narrower CI suggests a more precise (more robust) estimate of the relationship between variables. Larger sample sizes also impact the width of the CI, providing a more precise estimate. Using our example (see Table 1), we found an odds ratio (OR) of 6 with a 95% CI [1.4583, 24.6868] and p-value = 0.0130. The interpretation would support that those obese HIV participants were nearly six times more sedentary than non-obese participants. The six (6) is the point estimate obtained from this example, and the finding is statistically significant (p-value < 0.05). It is important to note that the sample size was small, generating a wide CI, indicating a lower level of precision. If this study were repeated, the point estimate would differ, with some studies obtaining less than or a greater value than 6 OR. In other words, the interpretation might conclude that a point estimate could range from as low as 1.483 to as high as 24.6868. Please see the article on Cross-Sectional Studies (B Capili, 2021) for additional information about odds ratio and CI.
Explanations: Bias/Systematic Error
Bias is difficult to avoid altogether in research. A well-designed study can minimize bias, but complete elimination is not guaranteed (Siddiqi, 2011). Nevertheless, the investigator’s awareness of potential bias at all phases of research increases the likelihood of minimizing bias and improving the overall study quality. Bias (systematic error) can occur during participant selection (selection bias) and data collection (measurement bias). Systematic bias can result in an erroneous association where no relationship exists or masks a true association between variables (Malone, Nicholl, & Tracey, 2014). In addition, bias (systematic error) is a factor that can lead to an incorrect conclusion caused by the research design or implementation of the study and persists despite increasing the sample size (Malone et al., 2014).
Selection bias arises when participants are not representative of the target population due to improper procedures selecting the sample and issues that influence study participation duration (study dropout) (Alexander, 2015a). Sources of selection bias include a) healthy worker effect, b) hospital patient bias, c) selective survival and loss to follow-up, and d) volunteer and non-response bias (Alexander, 2015a).
The healthy worker effect relates to the notion that working individuals are generally healthier because they are not on leave from work due to a disability. The worker effect is significant if evaluating occupational exposure studies. For example, assessing for environmental noise and hearing impairment, all participants should be working to accurately evaluate the association between exposure (environmental noise) and the health outcome (hearing impairment). Otherwise, the association between exposure and the health outcome could be biased towards the null (no difference) (Alexander, 2015a).
The hospital patient bias can occur with case-control studies when selected participants are from a pool of hospitalized patients (Review Case-Control paper in this series). To illustrate this concept, let’s use a case-control study assessing sodium intake and hypertension among adults. First, investigators assembled the control group from hospitalized patients who were not admitted for hypertension. Then, suppose a portion of the hospitalized patients (controls) have other conditions that caused them to reduce their sodium intake (i.e.history of stroke, history of myocardial infarction). In that case, the control group would not represent the target population from which hypertension arose. Additionally, most patients with hypertension are not managed in the hospital except for hypertensive emergencies. Therefore, the cases are also not representative of individuals with hypertension.
The selective survival and losses to follow-up bias occur when the association between a risk factor (predictor variable) and a health outcome (dependent variable) differs among the participants who drop out or stay in the study. For example, clinical research studies usually experience participant dropouts before completing the study. However, the selective survival and losses to follow-up bias can introduce a false survival advantage among healthier participants than participants who do not survive long enough to receive treatment (Malone et al., 2014). In other words, participants screened for a disease/condition for prevention (i.e., routine colonoscopy) may present with less severe disease than participants who present to a provider with symptomatic conditions. Therefore, there is an assumption that earlier diagnosis through routine screening leads to more prolonged survival due to earlier diagnosis.
Volunteer and non-response bias occurs when individuals who join a research study differ in characteristics from individuals who do not join (Alexander, 2015a). Additionally, individuals who join studies usually do better than individuals with similar conditions, only seen in routine medical care (McCarney et al., 2007). This effect is called the Hawthorne effect. Additionally, individuals who join a study may differ from the general target population regarding health awareness and other issues. Therefore, bias is introduced if the association between risk factors and health outcomes differs between study participants who join and nonresponders.
Measurement bias, also known as information bias, can occur due to inaccurate measurements of crucial study variables (Alexander, 2015b). For example, an investigator uses a poorly constructed questionnaire to collect data about physical activity that does not adequately address the level of physical activity or is challenging to understand is a potential source of measurement bias. Additionally, unclear interview questions or lack of follow-up probing questions to clarify responses are possible measurement bias sources. For instance, if the interview question was ‘what does physical activity mean to you?’ the probe phrase might be, ‘is this what you said?’ to clarify the response. Another potential measurement bias (information bias) may occur when the exposure to disease (as in a cohort study) or study health outcome is misclassified (Alexander, 2015b). The misclassification might happen when investigators use dichotomous or categorical variables to classify an exposure or an outcome. Such errors can occur when the exposure or outcome (cohort studies) has subjective elements. For example, how is the level of physical activity defined? Does house cleaning count as a physical activity? Or if determining if a participant is more or less active than someone of similar age. These examples demonstrate subjective elements which can result in misclassifications. Finally, recall and reporting bias is another type of measurement bias. Recall bias is an error due to differences in the accuracy of reporting events or exposures from the past (Alexander, 2015b). This type of bias can arise with retrospective studies, mainly if disease status influences recall to past exposure. To illustrate, participants in a cardiac study focused on myocardial infarction might better remember their smoking history. In contrast, participants in an Alzheimer’s study might not remember smoking in the past. Reporting bias occurs when participants under-report behaviors or conditions considered sensitive or socially undesirable, such as the history of partner violence, sexually transmitted infections, or use of illicit substances (i.e., drugs, alcohol) (Malone et al., 2014).
Methods to Reduce Bias/Systematic Error
Methods to reduce selection and measurement bias should occur during the study design and implementation phases. For selection bias, develop eligibility criteria and use existing literature to guide the development (Bernadette Capili, 2021b). Additionally, when reporting results, include demographic data and information on the rate of non-participation, e.g., screen failure, declines, follow-up loss, withdrawals, etc. We can learn from understanding why someone joins or not join. As per recommendations in reducing the random error section, select the best instrument to measure the variables under study (independent and dependent variables). Use validated instruments with published psychometric properties (validity, internal consistency, reliability) and tested the population under investigation (Bernadette Capili, 2021a). Before starting a study, enlisting a patient for their input is another helpful method to reduce measurement bias. They are the target population and can provide invaluable feedback on study materials (for clarity and burden), help identify potential barriers to study implementation, etc. These can be incorporated before the study if errors or concerns are identified.
Again, to reduce misclassification, develop clear eligibility criteria to participate in a research study (review the article Selection of Study Participants, (Bernadette Capili, 2021b). Additionally, create guidelines for data classification (Malone et al., 2014). Also, if possible, blind the data collectors regarding exposure or health outcome status. Blinding (masking) is the process of ensuring participants and or investigators are unaware of the study group assignment to reduce subjectivity in evaluations leading to possible biased results (Day & Altman, 2000). However, for some observational studies, such as the case-control, potential participants are interviewed to determine the history of a disease, making blinding challenging with smaller study teams. To tackle this challenge, develop standardized data collection procedures and train data collectors to obtain data in a standardized manner. In other words, create step-by-step instructions for the study team, conduct mock study sessions, and provide refresher training regularly (quarterly, bi-annually, etc.).
To decrease recall bias, verify the information from multiple sources such as medical records, spouse, or family member if permitted (Malone et al., 2014). In addition, to assist investigators in how to talk to patients about sensitive topics to increase the reliability of participant self-report (reduce reporting bias), the National Institute on Drug Abuse (NIDA) has a *presentation available on their website to teach healthcare professionals (NIDA, 2010).
Explanations: Confounding
Confounding is also a form of bias. Confounding occurs when a variable (a third factor) influences the dependent and independent variables creating a distorted association (Bernadette Capili, 2021a). In other words, the study outcome (results) ‘confounds’ the actual relationship between the variables. A confounding variable is also associated with the predictor (independent variable) of interest and the cause of the outcome (dependent variable) (Hulley, 2013). Confounding is often a likely explanation for cause-effect in observational studies and is a critical component to rule out. However, identifying confounders is challenging, and reducing them involves judgment and prior experience with the data from published research. Understanding the etiology (biologic, physiologic, etc.) of the condition/disease is always the best solution to reduce confounding (Hulley, 2013).
Methods to Reduce Confounding
Methods to reduce confounding should occur during the study design and analysis phases. First, it is essential to note some unmeasured confounding variables or possible confounding variables that the investigators may not consider. However, a strategy to identify potential confounders is to list variables such as age, activity level, blood pressure that could be associated with the predictor variable and also cause the outcome (Hulley, 2013). Two design approaches, specification, and matching can be used for cohort and case-control studies.
For specification, apply this method during the development of the eligibility criteria, define or specify the value of the potential confounding variable, and exclude participants with a different value (Hulley, 2013). Let’s continue with our MI and sedentary behavior study to illustrate this concept. To apply specification to this study, the investigator could exclude participants with a BMI ≥ 30 (exclude obese participants). Therefore, if an association between MI and sedentary behavior is found, it would not be due to obesity. Specification is a helpful method, but it has shortcomings. First, it reduces the available number of eligible participants, causing difficulties for study recruitment. Secondly, variables that have been excluded or restricted can not be evaluated. Lastly, specification limits the study’s generalizability and compromises the ability to generalize to obese participants, as in our example.
For matching, investigators recruit similar study participants concerning possible confounding variables such as BMI. Matching is used to prevent confounding by recruiting participants (cases and controls) with the same values of the confounding variable (Hulley, 2013) and is often used in case-control studies. Matching is also called pair-wise matching and is completed individually. For example, to control for obesity in a study of sedentary behavior as a predictor of MI, each case (a participant with an MI) is matched to one or more control participants with a similar BMI (i.e., a case with BMI 31, a control with BMI 31). The physical activity of each case is compared with the physical activity of the matched controls.
An option for pair-wise matching is matching in groups, called frequency matching (Hulley, 2013). First, for each level of BMI, the cases with a similar level of BMI and controls are selected. Then, the investigator decides the make-up of the frequency matching, such as a 1:2 ratio case to controls. To illustrate this concept, the investigator uses two controls per case for the sedentary behavior and MI study, controlling for BMI. For this example, the investigator recruits 10 cases who are obese (BMI 30–34) and recruits 20 controls in a similar BMI range (BMI 30–34), matched as a group to the 10 cases.
Matching is an effective method to control confounding variables, but it has several disadvantages. First, matching is time-consuming in recruiting participants and costly because the investigator must search a larger pool of potential participants. Second, it limits the sample size, and the decision to match must be conducted at the start of the study. Lastly, factors (variables) that were matched can not be evaluated.
The stratification and multivariate analysis approach was used during the analysis phase to reduce confounding bias. Stratification is a technique to control confounding variables by separating the participants into levels (strata) according to the level of potential confounders and analyzing the association between the predictor variable and outcome separately in each level (Hulley, 2013). To demonstrate this concept, let’s say we conducted a case-control study to assess the association between myocardial infarction (MI) (case) and sedentary behavior (outcome) among obese and non-obese study participants. In the literature, obesity has been linked with MI and sedentary behavior. See Figure 1 and see Table 2 for the odds ratio calculation. The odds ratio for our example was 2.25, combing both the obese and non-obese participants in the analysis. The result indicates that sedentary participants were 2.25 times more likely to have a history of MI, regardless of body mass index (BMI). Next, we stratify our participants by the categories of obese and non-obese participants to assess whether we see a difference based on BMI. Finally, we compare only the obese participants against obese participants (sedentary or not sedentary) and MI history. Similarly, we did the same for the non-obese participants. See Tables 3A and 3B for the odds ratio calculations.
Figure 1.
Confounding Variable
Table 2.
Example 2: Odds Ratio Calculations
| Combined Obese and Non-Obese | |||
|---|---|---|---|
| Outcome |
Case
MI |
Control
No MI |
Total |
|
Disease Sedentary *(Low Activity Level) |
90 a |
60 b |
150 (a + b) |
|
No Disease Not Sedentary *(Moderate to High Activity Level) |
60 c |
90 d |
150 (c + d) |
| Total | 150 (a + c) |
150 (b + d) |
300 (a + b + c + d) |
measured by the Physical Activity Questionnaire
- a = case participant acquires the outcome of interest
- b = control participant and acquires the outcome of interest
- c = case participant and does not acquire the outcome of interest
- d = control participant and does not acquire the outcome of interest
Odds Ratio: ad/cb = 90 × 90/60 × 60 = 2.25. The odds ratio for MI associated with sedentary behavior.
- OR = 1 odds of exposure among cases are the same as or equal to the odds of exposure among controls
- OR > 1 odds of exposure among cases are greater than the odds of exposure among controls
- OR < 1 odds of exposure among cases are less than the odds of exposure among controls
Table 3.
Example 3: Odds Ratio Calculations
| 3A) Obese (Body Mass Index ≥ 30) | |||
|---|---|---|---|
| Outcome |
Case
MI |
Control
No MI |
Total |
|
Disease Sedentary *(Low Activity Level) |
80 a |
40 b |
120 (a + b) |
|
No Disease Not Sedentary *(Moderate to High Activity Level) |
20 c |
10 d |
30 (c + d) |
| Total | 100 (a + c) |
50 (b + d) |
150 (a + b + c + d) |
| Odds Ratio: ad/cb = 80 × 10/20 × 40 = 1. Tde odds ratio for MI associated witd sedentary behavior among obese participants. | |||
| 3B) Non-Obese (Body Mass Index < 24.9) | |||
| Outcome |
Case
MI |
Control
No MI |
Total |
|
Disease Sedentary *(Low Activity Level) |
10 a |
20 b |
30 (a + b) |
|
No Disease Not Sedentary *(Moderate to High Activity Level) |
40 c |
80 d |
120 (c + d) |
| Total | 50 (a + c) |
100 (b + d) |
150 (a + b + c + d) |
| Odds Ratio: ad/cb = 10 × 80/40 × 20 = 1. The odds ratio for MI associated with sedentary behavior among non-obese participants. | |||
Based on the results, the odds ratio for the stratified samples (obese or non-obese) was equal to 1, showing that MI was not associated with sedentary behavior among obese or non-obese participants. However, the stratified results for BMI (obese or non-obese) could indicate confounding in that sedentary behavior is not associated with obese or non-obese participants.
To further review the relationship and identify the potential confounder, the calculations in Tables 4A and 4B were completed. For table 4A, the participants with and without a history of MI were combined and showed that participants with sedentary behavior were 16 times more likely to be obese. Additionally, for table 4B, the participants who were and were not sedentary were combined, indicating that participants with a history of MI were four times more likely to be obese. Thus, obesity is a confounder based on the defining criteria and the results because it is strongly associated with sedentary behavior and MI.
Table 4.
Example 4: Odds Ratio Calculations
| 4A) MI and No MI Combined | |||
|---|---|---|---|
| Outcome |
Case Sedentary *(Low Activity Level) |
Control Not Sedentary *(Moderate to High Activity Level) |
Total |
|
Disease Obese (Body Mass Index ≥ 30) |
120 a |
30 b |
150 (a + b) |
|
No Disease Non-Obese (Body Mass Index < 24.9) |
30 c |
120 d |
150 (c + d) |
| Total | 150 (a + c) |
150 (b + d) |
300 (a + b + c + d) |
| Odds Ratio: ad/cb = 120 × 120/30 × 30 = 16. The odds ratio for sedentary behavior associated witd obesity. | |||
| 4B) Sedentary and Not Sedentary Combined | |||
| Outcome |
Case
MI |
Control
No MI |
Total |
|
Disease Obese (Body Mass Index ≥ 30) |
100 a |
50 b |
150 (a + b) |
|
No Disease Non-Obese (Body Mass Index < 24.9) |
50 c |
100 d |
150 (c + d) |
| Total | 150 (a + c) |
150 (b + d) |
300 (a + b + c + d) |
| Odds Ratio: ad/cb = 100 × 100/50 × 50 = 4. The odds ratio for MI history associated with obesity. | |||
A strength of stratification is that the investigator can select which variables to stratify after collecting data. However, a weakness of this approach is the limited number of variables controlled at the same time. For instance, potential variables confounding our sedentary behavior and MI case-control study might also include age, blood pressure, cholesterol, daily caloric intake, and smoking. To stratify for these five variables with three strata (levels) for each would involve 35 = 243 per strata (example strata for smoking: 1–10 cigarettes per day, 11–20 cigarettes per day, 21–30 cigarettes per day). The limitation is that some strata will not have any cases or controls, and the strata can not be used. Additionally, the study’s sample size limits the number of levels (strata).
Multivariate analysis is a complex statistical (mathematical) model and usually requires a statistical software program. However, this method helps adjust (take into account) for confounding variables since many outcomes can be affected by more than one variable (factor). In addition, multivariate analysis can simultaneously assess the relationship of several variables on an outcome (Pourhoseingholi, Baghestani, & Vahedi, 2012). In other words, this technique can evaluate and adjust the impact of one or more confounding variables. Finally, multivariate analysis can assess the independent effect of each variable on the outcome. While multivariate analysis is a powerful technique, the variables used (as in our sedentary behavior and MI study: age, blood pressure, smoking, caloric intake, cholesterol level), the statistical model can only account for differences in the confounding variables measured in the study. Therefore, the variables selected for the model should be biologically plausible and clinically meaningful. Please consult a statistician for additional information. Please see table 5 for a summary of recommendations.
Table 5.
Summary of Methods to Improve Causal Inference
| Methods to Reduce Chance/Random Error |
|
| Methods to Reduce Bias/Systematic Error |
|
| Methods to Reduce Confounding |
|
Conclusion
Understanding the cause for potential bias and methods to improve causal inferences for observational studies can increase awareness and reduce potential bias in designing, implementing, and analyzing research data. Consumers of research can also critically review research conclusions with the awareness of potential bias and confounders.
Acknowledgments
This manuscript is supported in part by grant # UL1TR001866 from the National Center for Advancing Translational Sciences (NCATS), National Institutes of Health (NIH) Clinical and Translational Science Award (CTSA) program.
Footnotes
References
- Alexander L, Lopes B, Richetti-Masterson K, Yeatts KR. (2015a) Selection Bias. In, (2nd Ed ed.). Durham, NC. [Google Scholar]
- Alexander L, Lopes B, Richetti-Masterson K, Yeatts KR. (2015b) Sources of Systematic Error or Bias: Information Bias. In, (2nd Ed ed.). Durham, NC. [Google Scholar]
- Capili B (2021). Cross-Sectional Studies. American Journal of Nursing, 121(10), 59–62. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Capili B (2021a). Selection and Implementation of Outcome Measurements. AJN The American Journal of Nursing, 121(8). Retrieved from https://journals.lww.com/ajnonline/Fulltext/2021/08000/Selection_and_Implementation_of_Outcome.30.aspx [DOI] [PMC free article] [PubMed] [Google Scholar]
- Capili B (2021b). Selection of the Study Participants. AJN The American Journal of Nursing, 121(1). Retrieved from https://journals.lww.com/ajnonline/Fulltext/2021/01000/Selection_of_the_Study_Participants.22.aspx [DOI] [PMC free article] [PubMed] [Google Scholar]
- Curtis AC, & Keeler C (2021). Measurement in Nursing Research. AJN The American Journal of Nursing, 121(6). Retrieved from https://journals.lww.com/ajnonline/Fulltext/2021/06000/Measurement_in_Nursing_Research.27.aspx [DOI] [PubMed] [Google Scholar]
- Day SJ, & Altman DG (2000). Blinding in clinical trials and other studies. BMJ, 321(7259), 504. doi: 10.1136/bmj.321.7259.504 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hulley S, Cummings SR, Browner WS, Grady DG, Newman TB (Ed.) (2013). Designing Clinical Research (4th ed.). Philadelphia, PA: Wolters Kluwer/Lippincott Williams & Wilkins. [Google Scholar]
- Majid U (2018). Research Fundamentals: Study Design, Population, and Sample Size. Undergraduate Research in Natural and Clinical Science and Technology (URNCST) Journal, 2. doi: 10.26685/urncst.16 [DOI] [Google Scholar]
- Malone H, Nicholl H, & Tracey C (2014). Awareness and minimisation of systematic bias in research. Br J Nurs, 23(5), 279–282. doi: 10.12968/bjon.2014.23.5.279 [DOI] [PubMed] [Google Scholar]
- McCarney R, Warner J, Iliffe S, van Haselen R, Griffin M, & Fisher P (2007). The Hawthorne Effect: a randomised, controlled trial. BMC Medical Research Methodology, 7(1), 30. doi: 10.1186/1471-2288-7-30 [DOI] [PMC free article] [PubMed] [Google Scholar]
- NIDA, N. (2010). Talking to Patients About Sensitive Topics: Communication and Screening Techniques for Increasing the Reliability of Patient Self-Report. Retrieved from https://www.drugabuse.gov/nidamed-medical-health-professionals/health-professions-education/centers-excellence/curriculum-resources-overview/resources-by-topic
- Polit D, Beck CT. (2022). Essentials of Nursing Research: Appraising Evidence for Nursing Practice (10th Edition ed.). Philadelphia, PA: Wolters Kluwer. [Google Scholar]
- Pourhoseingholi MA, Baghestani AR, & Vahedi M (2012). How to control confounding effects by statistical analysis. Gastroenterology and hepatology from bed to bench, 5(2), 79–83. Retrieved from https://pubmed.ncbi.nlm.nih.gov/24834204 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4017459/ [PMC free article] [PubMed] [Google Scholar]
- Siddiqi N (2011). Publication bias in epidemiological studies. Cent Eur J Public Health, 19(2), 118–120. doi: 10.21101/cejph.a3581 [DOI] [PubMed] [Google Scholar]