Understanding sources of bias in research

Critically evaluating the evidence, in particular research evidence, which underpins practice, is central to quality care and service improvements. Systematically appraising research includes assessing the rigour with which methods were undertaken and factors that may have biased findings. This article will outline what bias means in relation to research, why it is important to consider bias when appraising research and describe common types of bias across research processes. We will also offer strategies that researchers can undertake to minimise bias.

What is bias in relation to research, and why is understanding bias important?

The Critical Appraisal Skills Programme (CASP) describes bias in research as ‘systematic errors that can occur at any stage of the research process’ and can have a ‘significant impact on the reliability and validity of the findings’ that may lead to a distortion of the conclusions.¹ Understanding research bias is important for several reasons. First, bias exists across research designs and approaches and, while difficult to eliminate, should be accounted for. Second, bias can occur at each stage of the research process, from study design, participant selection, data collection and analysis, and the interpretation and reporting of findings. Third, bias impacts the validity and reliability of study findings, and misinterpretation of data can have important consequences for practice. The seminal example of the consequences of bias is the controversial study that suggested a link between the measles-mumps-rubella vaccine and autism in children.² A rare retraction of the published study occurred because of media reports that highlighted significant bias in the research process.³ Bias occurred on several levels: the process of selecting participants was misrepresented; the sample size was too small to infer firm conclusions from the analysis of the data; and the results were overstated, which suggested caution against widespread vaccination and an urgent need for further research. However, in the time between the original publication and later research refuting the original findings, the uptake of the measles-mumps-rubella vaccine in Britain declined, resulting in a 25-fold increase in measles in the 10-year period following the original publication. 20 years on, measles vaccination rates have continued to fall year on year, with ongoing concerns about the vaccine’s safety.

What are common types of bias in research?

Although different study designs have specific methodological challenges and constraints, bias can occur at each stage of the research process. Examples of potential sources of bias across research processes in relation to study design, participant selection, data collection and analysis, reporting of findings and publication bias are presented in table 1. In quantitative research, the validity and reliability are assessed using statistical tests that estimate the size of error in samples and calculate the significance of findings (typically, p values or CIs). The tests and measures used to establish the validity and reliability of quantitative research cannot be applied to qualitative research. However, in the broadest context, these terms are applicable, with validity referring to the integrity and application of the methods and the precision with which the findings accurately reflect the data and reliability referring to the consistency within the analytical processes.⁴

Table 1. Types of research bias.

Design bias	Poor study design and incongruence between aims and methods increase the likelihood of bias. For example, exploring weight loss programmes using a survey is unlikely to obtain in-depth rich data about individuals’ experiences, which might include reasons why people dropped out of the programme and the impact on their daily lives.Bias can occur when a researcher’s personal beliefs influence the choice of research question and methodology. For example, a researcher working for a pharmaceutical company may choose a research question that supports the usefulness of the drug being investigated.
Selection/participant bias	Selection bias relates to both study inclusion criteria and the process of recruiting participants. Bias can occur if the intended population are unlikely to participate which results in a non-representative sample. Successful research begins with recruiting participants who meet the study’s aims. For example, recruitment bias could occur if participants were invited to participate in a study that required access to a computer, but the population of interest is unlikely to use technology, which would exclude them from the study and therefore not capture the population of interest.Inclusion bias in quantitative research typically relates to selecting participants that are representative of the study population, and where applicable allocation of participants to ensure similarity between comparison groups. In addition, accounting for the differences between people who remain in a study and those who withdraw may be important in some study designs. For example, an evaluation of an exercise programme may be affected by participant withdrawal; participants who become disillusioned, disinterested or cannot find the time to participate may drop out, which may bias the findings towards more favourable results.Confounding bias can also occur because of an association between ‘cause’ and ‘effect’. For example, comparing treatment outcomes for similar conditions between general and specialised centres may find higher mortality rates at specialised centres yet patients referred to these centres are more likely to have high risk factors and more complex needs.In qualitative research, it is usual to recruit participants with a range of experiences in relation to the topic being explored: therefore, accounting for biases in relation to the sampling strategies is essential. For example, recruiting participants from a weight-loss programme is likely to be biased towards females because men are less likely to attend weight-loss programmes and the findings are unlikely to represent both male and female perspectives.
Data collection bias and measurement bias	Data collection bias can occur when a researcher’s personal beliefs influence the way information or data is collected.In quantitative studies, measurement bias can occur if a tool or instrument: (1) has not been assessed for its validity or reliability, for example, using a shared decision-making tool that measures patient satisfaction rather than decision-making; (2) is not suitable for the specific setting or patient groups, for example, using a patient assessment tool for use in an intensive care setting in a maternity setting; and (3) an instrument not calibrated properly may consistently measure inaccurately, for example, weighing participants with poorly calibrated scales.In retrospective studies, participants may not remember and report events accurately. For example, completing questionnaires about the experience of pain, which relies on recall, may not reflect actual pain experiences.In qualitative research, interviewing is a commonly used method of data collection; how questions are asked will influence the information elicited. For example, a leading question, ‘Would you like to die at home?’, is likely to receive a closed yes or no response and not gain insights into participants’ experiences and could be replaced with; ‘Please describe where you would like to die and why?’
Analysis bias	When analysing data, the researcher may naturally seek data that confirm their hypotheses or personal experience, overlooking data inconsistent with personal beliefs. During the analysis, the researcher may emphasise or discount certain data in favour of a particular result, which often aligns with the researcher’s personal viewpoint leading to distortion of the findings. For example, when researching the impact of alcohol on young people, the researcher focuses on the negative findings such as antisocial behaviour and discounts more positive outcomes, such as developing social skills and peer support groups.
Bias in reporting findings	Reporting bias refers to including or excluding specific study findings and is similar to analysis bias but typically is associated with ensuring a study is accepted in academic journals: typically journals are more likely to publish positive or statistically significant results,10 which can lead to a misreporting of the findings.
Publication bias	Published studies nearly always have some degree of bias. Journals and associated editors may choose to publish only positive results and omit studies that show no effect. For example, in quantitative research, studies are more likely to be published if reporting statistically significant findings than those with negative or non-significant findings.10Non-publication in qualitative studies is more likely to occur because of a lack of depth when describing study methodologies and findings are not clearly presented.11

How is bias minimised when undertaking research?

Bias exists in all study designs, and although researchers should attempt to minimise bias, outlining potential sources of bias when reporting studies enables greater critical evaluation of the research findings and conclusions.⁵ Researchers bring to each study their experiences, ideas, prejudices and personal philosophies, which, if accounted for in advance of the study, enhance the transparency of possible research bias. Clearly articulating the rationale for and choosing an appropriate research design to meet the study aims can reduce common pitfalls in relation to bias.

Ethics committees have an important role in considering whether the research design and methodological approaches are biased and suitable to address the problem being explored. Feedback from peers, funding bodies and ethics committees is an essential part of designing research studies and often provides valuable practical guidance in developing robust research. Similarly, journals have a role in ensuring the quality of studies published. The widespread use of guidelines such as the Consolidated Standards of Reporting Trials Statement for reporting randomised controlled trials, Preferred Reporting Items for Systematic Reviews and Meta-analysis and Consolidation criteria for Reporting Qualitative Research in improving the quality of research publication, led by the Enhancing the Quality and Transparency of Health Research network (https://www.equator-network.org/about-us/equator-network-what-we-do-and-how-we-are-organised/). However, over 60 guidelines have been identified, with concerns raised about the development rigour and implementation during article peer review processes.⁶ Studies that are registered prior to the start of data collection and the analysis are likely to be protected from publication bias, as unfavourable results will be disclosed and likely offer a more lucid depiction of the impact that treatment has on individuals.⁷

In quantitative studies, selection bias is often reduced by the random selection of participants and, in the case of clinical trials, randomisation of participants into comparison groups. However, not accounting for participants who withdraw from the study or are lost to follow-up can result in sample bias or change the characteristics of participants in comparison groups.⁸ In qualitative research, purposeful sampling has advantages when compared with convenience sampling in that bias is reduced because the sample is constantly refined to meet the study aims. Premature closure of the selection of participants before analysis is complete can threaten the validity of a qualitative study. This can be overcome by continuing to recruit new participants into the study during data analysis until no new information emerges, known as data saturation.⁹

In quantitative studies, having a well-designed research protocol explicitly outlining data collection and analysis can assist in reducing bias. Feasibility studies are often undertaken to refine protocols and procedures. Bias can be reduced by maximising follow-up, and where appropriate in randomised control trials, analysis should be based on the intention-to-treat principle, a strategy that assesses clinical effectiveness because not everyone complies with treatment, and the treatment people receive may be changed according to how they respond. Qualitative research has been criticised for lacking transparency in relation to the analytical processes employed.⁴ Qualitative researchers must demonstrate rigour, associated with openness, relevance to practice and congruence of the methodological approach. Although other researchers may interpret the data differently, appreciating and understanding how the themes were developed is an essential part of demonstrating the robustness of the findings. Reducing bias can include respondent validation, constant comparisons across participant accounts, representing deviant cases and outliers, prolonged involvement or persistent observation of participants, independent analysis of the data by other researchers and triangulation.⁴

In summary, minimising bias is a key consideration when designing and undertaking research. Researchers have an ethical duty to outline the limitations of studies and account for potential sources of bias. This will enable health professionals and policymakers to evaluate and scrutinise study findings and consider these when applying findings to practice or policy.