Matched case-control studies: a review of reported statistical methodology

Abstract

Background

Case-control studies are a common and efficient means of studying rare diseases or illnesses with long latency periods. Matching of cases and controls is frequently employed to control the effects of known potential confounding variables. The analysis of matched data requires specific statistical methods.

Methods

The objective of this study was to determine the proportion of published, peer-reviewed matched case-control studies that used statistical methods appropriate for matched data. Using a comprehensive set of search criteria we identified 37 matched case-control studies for detailed analysis.

Results

Among these 37 articles, only 16 studies were analyzed with proper statistical techniques (43%). Studies that were properly analyzed were more likely to have included case patients with cancer and cardiovascular disease compared to those that did not use proper statistics (10/16 or 63%, versus 5/21 or 24%, P = 0.02). They were also more likely to have matched multiple controls for each case (14/16 or 88%, versus 13/21 or 62%, P = 0.08). In addition, studies with properly analyzed data were more likely to have been published in a journal with an impact factor listed in the top 100 according to the Journal Citation Reports index (12/16 or 69%, versus 1/21 or 5%, P ≤ 0.0001).

Conclusion

The findings of this study raise concern that the majority of matched case-control studies report results that are derived from improper statistical analyses. This may lead to errors in estimating the relationship between a disease and exposure, as well as the incorrect adaptation of emerging medical literature.

Introduction

Case-control studies provide a quick and efficient means of studying diseases with long latency periods or with low incidence in the population.¹ Given their utility, it is not surprising that a Medline search of the English language literature with the words “case-control” in the title revealed more than 1000 articles published in 2010. Although they are convenient and common, there are several important considerations in the design of case-control studies.^1,2 One consideration is the decision to match cases to controls and the subsequent selection of statistical techniques that are appropriate for the matched data. The analysis of matched (dependent) data is different from unmatched (independent) data and is described in detail by Breslow and Day.³ In their text on the analysis of case-control studies, they describe the use of paired t-tests for measured outcomes with 1:1 matching and with a symmetrical distribution, the Wilcoxon signed ranks test for measured outcomes with 1:1 matching as a non-parametric alternative, model-based options such as the appropriate linear regression to handle modification and additional potential confounding, and matched sets other than 1:1 matching. For dichotomous outcomes, they describe McNemar’s test, a Mantel–Haenszel matched-pairs analysis, and additional methods to handle matched sets other than 1:1 matching. They also describe conditional logistic regression to handle all forms of matching as well as the consideration of modification and other potential confounding.³

We recently published a study wherein matching was employed to control for known potential confounding variables.⁴ During the analysis, we noticed that many of the peer-reviewed publications that describe the use of matched data, employ inconsistent statistical methods of analysis. Data analyses that employ incorrect statistical methods will commonly result in inappropriate conclusions. Therefore, the current study was designed to evaluate the statistical methodology in a collection of matched case-control studies.

Methods

A literature review was conducted using PubMed from January 1, 2010 to December 1, 2010 with the goal of identifying articles that employed a matched case-control design. The search-term was “case-control” (found in the article title only) and the search was limited to human studies published in the English language literature in PubMed’s defined subset of “core clinical journals” (see Appendix 1 for a list of these core clinical journals). This search strategy was chosen to yield a representative sample of case-control studies in a variety of subject areas, published in peer-reviewed, mainstream journals. One of the authors screened all the abstracts for relevance and appropriate full-length articles were subsequently retrieved for appraisal. To maintain relative homogeneity among the final collection of articles, it was decided, a priori, to exclude articles that focused on subjects in the pediatric age group (under 18 years of age). We also excluded studies that used matching methods other than simple, individual criteria-based matching, ie, frequency-matching, and propensity-matching, as this type of data is analyzed with different statistical methods.⁵

Appendix 1.

List of “core clinical journal” subset of PubMed journals

Academic Medicine: journal of the Association of American Medical Colleges

AJR: American Journal of Roentgenology

American Family Physician

American Heart Journal

The American Journal of Cardiology

The American Journal of Clinical Nutrition

American Journal of Clinical Pathology

The American Journal of Medicine

The American Journal of Nursing

American Journal of Obstetrics and Gynecology

American Journal of Ophthalmology

American Journal of Pathology

American Journal of Physical Medicine and Rehabilitation/Association of Academic Physiatrists

The American Journal of Psychiatry

American Journal of Public Health

American Journal of Respiratory and Critical Care Medicine

American Journal of Surgery

The American Journal of the Medical Sciences

The American Journal of Tropical Medicine and Hygiene

Anaesthesia

Anesthesia and Analgesia

Anesthesiology

Annals of Emergency Medicine

Annals of Internal Medicine

The Annals of Otology, Rhinology, and Laryngology

Annals of surgery

The Annals of Thoracic Surgery

Archives of Dermatology

Archives of Disease in Childhood

Archives of Disease in Childhood, fetal and neonatal edition

Archives of Environmental and Occupational Health [continues Archives of environmental health]

Archives of General Psychiatry

Archives of Internal Medicine

Archives of Neurology

Archives of Ophthalmology

Archives of Otolaryngology, head and neck surgery

Archives of Pathology and Laboratory Medicine

Archives of Pediatrics and Adolescent Medicine

Archives of Physical Medicine and Rehabilitation

Archives of Surgery (Chicago, Ill : 1960)

Arthritis and Rheumatism

BJOG : an international journal of obstetrics and gynaecology [continues British journal of obstetrics and gynaecology]

Blood

BMJ (Clinical research ed)

Brain: a journal of neurology

The British Journal of Radiology

The British Journal of Surgery

CA: A cancer Journal for Clinicians

Cancer

Chest

Circulation

Clinical Orthopedics and Related Research

Clinical Pediatrics

Clinical Toxicology: the official journal of the American Academy of Clinical Toxicology and European Association of Poisons Centers and Clinical Toxicologists [continues Journal of toxicology. Clinical toxicology]

Clinical Pharmacology and Therapeutics

CMAJ: Canadian Medical Association Journal/Journal de l’Association Medicale Canadienne

Critical Care Medicine

Current Problems in Surgery

Diabetes

Digestive Diseases and Sciences

DM: Disease-a-month

Endocrinology

Gastroenterology

Gut

Heart and Lung: the journal of critical care

Heart (British Cardiac Society)

Hospital Practice (1995) [Formed by the union of: Hospital practice (Family practice ed.); Hospital practice (Hospital ed.); and Hospital practice (Office ed). No issues published between 2001 Sep 15;36(9) and 2009 Dec;37(1)]

Hospitals and Health Networks/AHA

JAMA: Journal of the American Medical Association

The Journal of Allergy and Clinical Immunology

The Journal of Bone and Joint Surgery, American volume

The Journal of Bone and Joint Surgery, British volume

The Journal of Clinical Endocrinology and Metabolism

The Journal of Clinical Investigation

Journal of Clinical Pathology

The Journal of Family Practice

Journal of Immunology (Baltimore, Md : 1950)

The Journal of Infectious Diseases

The Journal of Laryngology and Otology

The Journal of Nervous and Mental Disease

Journal of Neurosurgery

The Journal of Nursing Administration

Journal of Oral and Maxillofacial Surgery: official journal of the American Association of Oral and Maxillofacial Surgeons

The Journal of Pediatrics

Journal of the American College of Cardiology

Journal of the American College of Surgeons

Journal of the American Dietetic Association

The Journal of Thoracic and Cardiovascular Surgery

The Journal of Trauma

The Journal of Urology

The Journals of Gerontology. Series A, Biological sciences and medical sciences

The Journals of Gerontology. Series B, Psychological sciences and social sciences

Lancet

Mayo Clinic Proceedings

The Medical Clinics of North America

The Medical Letter on Drugs and Therapeutics

Medicine

Neurology

The New England Journal of Medicine

The Nursing Clinics of North America

Nursing Outlook

Nursing Research

Obstetrics and Gynecology

The Orthopedic Clinics of North America

Pediatric Clinics of North America

Pediatrics

Physical Therapy

Plastic and Reconstructive Surgery

Postgraduate Medicine

Progress in Cardiovascular Diseases

Public Health Reports (Washington, DC : 1974)

Radiologic Clinics of North America

Radiology

Rheumatology (Oxford, England) [continues British Journal of Rheumatology]

Southern Medical Journal

Surgery

The Surgical Clinics of North America

Translational Research: the journal of laboratory and clinical medicine [continues The Journal of Laboratory and Clinical Medicine]

The Urologic Clinics of North America

Each full length article was independently reviewed in detail by two of the authors. The goal of this review was to evaluate the appropriateness of the statistical methodology. Disagreements between the reviewers were resolved by the independent evaluation of a senior biostatistician. Inter- observer agreement was quantified using the kappa statistic, wherein a kappa value of 0.61 to 1.0 indicates substantial agreement.⁶ Statistical methods were appropriate for matched data if they were consistent with those described by Breslow and Day.³ Each article was appraised using the following analysis scoring system: (1) continuous outcomes analyzed using the paired t-test, or Wilcoxon signed ranks test, or others, as described above and dichotomous outcomes analyzed using McNemar’s test, a Mantel–Haenszel matched-pairs analysis, conditional logistic regression, or other, as described above; (2) investigators failed to analyze continuous outcomes with a paired t-test or the Wilcoxon signed ranks test, or did not use McNemar’s test, a Mantel– Haenszel matched-pairs analysis, or conditional logistic regression for dichotomous outcomes; and (3) the authors did not use any of the aforementioned statistical methods for continuous and dichotomous outcomes.

Following the review of their statistical methodology, the collection of matched case-control articles were reviewed a second time for factors that may be associated with the use of statistical methods appropriate for matched data. These factors included items that form common issues in the design of matched case-control studies, namely the case population definition, the number of matching variables, and the control-to-case ratio. In addition, we used the 2010 Journal Citation Reports (JCR) index to determine whether the appropriateness of the statistical methodology was associated with the impact factor of the publishing journal. Using the JCR index, we determined that an impact factor of at least 12.245 was required for a journal to be listed in the top 100 major journals. Data was analyzed using Stata (v11.0; Stata Corp, College Station, TX) and statistical significance was set at P ≤ 0.05.

Results

The initial search strategy yielded 74 articles (Figure 1). Upon review of these abstracts, 36 articles were excluded for reasons outlined in Figure 1. The remaining 38 articles were reviewed in detail.^7–44 After reviewing the articles, two of the authors identified one study that employed frequency-matching rather than individual patient-matching.²¹ Exclusion of this study from further statistical evaluation left 37 studies for the overall analysis. Table 1 provides a summary of these 37 studies. The two authors reviewing the studies agreed on the appropriateness of the statistical methods in 36 of the 37 studies (97%) and the inter-observer agreement, as measured by the kappa statistic, was 0.94. Sixteen of the selected studies were analyzed with correct statistical methods (analysis score of 1, 43%), and 21 were analyzed with at least one incorrect statistical method (sum of analysis score 2 and 3, 57%).

Figure 1.

Search strategy flow.

Table 1.

Details of studies included in statistical methodology analysis

Study	Case population	Cases (n)	Controls (n)	Matching variables	Analysis scorea
Ances et al7	Patients with human immunodeficiency virus (HIV)	10	20	Gender, education	3
Antonelli et al8	Patients with systemic lupus erythematosus	153	918	Gender, age	3
Carnaby-Mann et al9	Patients with dysphagia who entered the McNeill Dysphagia Therapy Program	8	16	Age, gender, primary medical diagnosis	2
Dubois et al10	Patients with previous thoracic radiation therapy undergoing coronary stenting	41	82	Gender, stented vessel, drug-eluting stent use, available duration of follow-up, unstable coronary disease, renal insufficiency, diabetes mellitus, bifurcation disease, stent length and size, ejection fraction	1b
Etminan et al11	Patients who had a coronary revascularization procedure between 1995–2004 with a discharge diagnosis of epilepsy	217	2170	Age, cohort entry day	1
Friedland et al12	Patients with cochlear implants	28	28	Pre-cochlear implant score on Hearing In Noise Test–Quiet, duration of deafness	3
Garg et al13	Patients 18–65 years old with migraine headaches onset before age 50	144	144	Age, gender	1
Green et al14	Patients ≥ 40 years old with incident esophageal, gastric, or colorectal cancer	15 613	77 750	Age, gender, general practice, observation time	1
Jenab et al15	Patients with incident colorectal cancer	1248	1248	Age, gender, study center, time of day at blood collection, duration of fasting at blood collection; women were further matched by menopausal status, phase of menstrual cycle at time of blood collection, use of hormone replacement therapy	1
Kermani et al16	Incident cases with giant cell arteritis	204	407	Age, gender, length of medical history	3
Koscieiny et al17	Patients with claudication or critical ischemia undergoing surgery for occlusion of the superficial femoral artery (SFA)	28	28	Occluded SFA, patent popliteal artery not amenable to endovascular intervention, patent profunda femoris artery with less than 50% stenosis before the first branch, ≥1 patent crural outflow artery, American Society of Anesthesiologists Class I, II, or III, patient mobile before surgery, or expected to have postoperative mobility, on aspirin and a statin	3
Lang et al18	Patients with HIV found to have an incident myocardial infarction	289	884	Age, gender, clinical center	2c
Marshall et al19	Patients with HIV and Kaposi’s sarcoma	21	14	Unclear	3
Martinez et al20	Patients with sudden cardiac death, or near death on antidepressants	568	14 812	Date of incident antidepressant prescription, year of birth, gender, indication for antidepressant prescription	1
Nickel et al22	Females with interstitial cystitis/painful bladder syndrome	207	117	Age, partner status, education	3
O’donnell et al23	Patients with first acute stroke	3000	3000	Age, gender, ± ethnic origin	3d
Parker et al24	Patients with incident venous thromboembolism	25, 532	89, 491	Age, calendar time, gender, family practice	1
Persoon et al25	Patients with limb shaking associated with internal carotid artery occlusion	34	68	Age, gender	3
Pouwels et al26	Patients with incident admission for a hip fracture	6,763	26,341	Year of birth, gender, geographic location, index admission date	1
Renoux et al27	Patients with incident diagnosis of stroke and on hormone replacement therapy	15,710	59,958	Age, general practice attended, year of joining the practice	1
Ripatti et al28	Patients with coronary artery disease	2,101	3,914	Gender, birth year	1e
Roder et al29	Patients with total hip arthroplasty and signs of cup failure	809	3,986	Hospital site of total hip arthroplasty, date of surgery; a follow-up exam within 6 months of the follow-up interval of the case; cup design, size, and material	1
Schaer et al30	Patients with atrial fibrillation	4661	18 642	Age, gender, general practice, calendar time	1
Schillaci et al31	Patients aged 18–50 years with migraine headaches	60	60	Age, gender, blood pressure	2
Siriwardena et al32	Patients with incident myocardial infarction	16, 012	62, 694	Age, gender, general practice, calendar time	1
Talving et al33	Patients with severe traumatic brain injury receiving erythropoiesis stimulating agent	89	178	Age, gender, mechanism, Glasgow Coma Scale, hypotension, injury severity score, abbreviated injury scale, presence of anemia	3
Tammemagi et al34	Never smokers or former smokers with adult-onset chronic rhinosinusitis	306	306	Age, gender, race/ethnicity	2
Taran et al35	Women with a diagnosis of cellular meiomyomas	99	198	Surgeon, surgical procedure, year of surgery	2
Telem et al36	Patients with anastamotic leak following colorectal surgery	90	180	Procedure, surgeon	3
Trifiro et al37	Elderly patients receiving an anti-psychotic drug prescription with an incident episode of community-acquired pneumonia	258	1689	Year of birth, gender, index date	1
Vickers et al38	Patients with prostate cancer	126	373	Date of birth, date of baseline blood tests	1
Vlaar et al39	Patients with transfusion-associated lung injury admitted to an intensive care unit	109	327f	Age, gender, admission diagnosis	3
Warensjö et al40	Consecutive patients with incident myocardial infarction	444	556	Gender, age, date of health survey, and geographic region	2
Wassenaar et al41	Patients with acromegaly undergoing screening colonoscopy	107	214	Age, gender	3
White et al42	Patients who discontinued zonisamide due to psychiatric or cognitive adverse events	59	118	Zonisamide initiation date	2
Wohl et al43	Patients older than 20 years of age with a diagnosis of pemphigus	255	509	Age, gender	3
Yates and James44	Physicians with a finding of serious professional misconduct	59	236	Year of graduation from medical school	1

Notes:

^aSee methods section for an outline of the statistical scoring system;

^bCategorical variables assessed using the Cochran–Mantel–Haenszel test with the matched pair as a stratification variable;

^cUsed univariate conditional logistic regression to perform hypothesis test on baseline continuous variables;

^dAuthors mentioned deliberately analyzing the data as if it were not matched;

^eBased on analysis of the nested COROGENE matched case-control sample;

^fthree separate control groups of 109 patients each.

Given the low number of studies with correctly analyzed data, each of the 37 articles was reviewed for factors that may be associated with the use of correct statistical tests (Table 2). Studies that were properly analyzed were more likely to have included case patients with cancer and cardiovascular disease compared to those that did not use proper statistics (10/16 or 63%, versus 5/21 or 24%, P = 0.02). Furthermore, properly analyzed studies were more likely to have matched multiple controls for each case (14/16 or 88%, versus 13/21 or 62%, P = 0.1).

Table 2.

Study design and publication characteristics

Factor	Correct analysisa (n = 16)	Incorrect analysisb (n = 21)	Pc
Case population, n (%)
Cancer or cardiovascular diseased	10 (63)	5 (24)	0.02
HIV	0 (0)	3 (14)	0.1
Other	6 (38)	13 (62)	0.1
Basic science topic, n (%)	1 (6)	1 (5)	0.8
Medical topic, n (%)	13 (81)	15 (71)	0.5
Surgical topic, n (%)	2 (13)	5 (24)	0.4
>2 matching variables, n (%)	11 (63)	12 (57)	0.5
>1:1 control-to-case ratio, n (%)	14 (88)	13 (62)	0.1
Publishing journal in top 100 according to JCR, n (%)	11 (69)	1 (5)	<0.0001

Notes:

^aCorrect statistical analysis: received a statistical analysis score of “1”;

^bIncorrect statistical analysis: received a statistical analysis score of “2” or “3”;

^cProportions compared via Fisher’s exact test;

^dcardiovascular disease was defined as any disease of the coronary vascular, peripheral vascular, cerebrovascular, and cardiac electrical system.

Abbreviations: HIV, human immunodeficiency virus; JCR, journal citation reports.

Table 3 presents the data on each publishing journal’s impact factor, in addition to the association between the previously described study design characteristics and correct statistical methodology. From this table, it is clear that matched case-control articles published in the British Medical Journal (BMJ) were consistently analyzed with correct statistical techniques. Furthermore, the BMJ was responsible for publishing the greatest number of articles in this series of matched case-control studies (7/37 or 19%). This is in contrast to articles published in Archives of Otolaryngology- Head and Neck Surgery, Journal of Clinical Endocrinology and Metabolism, and Neurology wherein the data was frequently analyzed incorrectly. Moreover, matched case-control studies published in Lancet were notably inconsistent in their statistical methodology. From this table, it is also evident that more studies in the correctly analyzed collection were published in a journal with an impact factor within the top 100 listing on JCR (11/16 or 69%, versus 1/21 or 5%, P ≤ 0.0001). The median (interquartile range [IQR]) impact factor among studies that were correctly analyzed was 13.471 (8.516–13.950), compared to 6.495 (4.231–8.017) for those with incorrect statistical methodology (Wilcoxon rank sum test, P = 0.0009).

Table 3.

A list of the publishing journals and impact factors among the collection of articles reviewed in this study

Journal	Study(s)	Impact factora
Analyzed correctly
Journal of Bone and Joint Surgery, American Volume	Roder et al29	2.967
Heart	Dubois et al10	4.706
Journal of Clinical Endocrinology and Metabolism	Pouwels et al26	6.495
Neurology	Etminan et al11	8.017
Canadian Medical Association Journal	Siriwardena et al32	9.015
British Medical Journal	Green et al,14 Jenab et al,15 Martinez et al,20 Parker et al,24 Renoux et al,27 Vickers et al,38 Yates and James44	13.471
Circulation	b Garg et al13	14.429
Annals of Internal Medicine	Schaer et al,30 Trifiro et al37	16.729
The Lancet	Ripatti et al28	33.633
Analyzed incorrectly
Archives of Otolaryngology, Head and Neck Surgery	Friedland et al,12 Tammemagi et al34	1.571
Archives of Physical Medicine and Rehabilitation	Carnaby-Mann et al9	2.254
American Journal of Obstetrics and Gynecology	Taran et al35	3.313
Journal of Urology	Nickel et al22	3.862
Archives of Dermatology	Wohl et al43	4.231
British Journal of Surgery	Koscieiny et al17	4.444
Archives of Surgery	Telem et al36	4.500
Critical Care Medicine	Vlaar et al39	6.254
Journal of Infectious Diseases	Marshall et al19	6.288
Journal of Clinical Endocrinology and Metabolism	Antonelli et al,8 Wassenaar et al41	6.495
American Journal of Clinical Nutrition	Warensjö et al40	6.606
Annals of Surgery	Talving et al33	7.474
Neurology	Ances et al,7 Schillaci et al,31 White et al42	8.017
Arthritis and Rheumatism	Kermani et al16	8.435
Brain	Persoon et al25	9.230
Archives of Internal Medicine	Lang et al18	10.639
The Lancet	O’donnell et al23	33.633

Notes:

^aImpact factor as listed on Journal Citation Reports for 2010;

^bExcellent example of the analysis of matched data.

Discussion

To our knowledge, this is the first study to assess the appropriateness of the statistical methodology used in a published series of matched case-control studies. From this structured review of studies published in a number of diverse mainstream, peer-reviewed journals it is clear that matched case-control studies are not consistently analyzed using appropriate statistical methods. More than 50% of the articles reviewed in this study present data that was analyzed with improper statistics. For many of these studies, this may simply change the strength of the association between the disease and exposure of interest; however, for studies with small numbers of discordant sets, the use of appropriate statistical methods may alter the significance of the findings. Unfortunately, none of the articles reviewed in this study with incorrect statistical methodology provided the data in a format whereby the magnitude of the difference between a proper and improper analysis could be assessed. This is important, as it is clear from recent reviews of the literature that the use of statistics in medical literature is increasing over time.^45,46 However, if the analyses are being performed incorrectly, this increased use of statistics does not equate to an improvement in study quality and may lead to the adoption of incorrect medical literature.

Although this study appears to be the first to evaluate the quality of the statistical methods employed in a series of matched case-control studies, it is not the first study to review the quality of statistical methods in medical journals. A number of these studies were published in the 1980’s and 1990’s at a time that coincided with a rapid rise in the use of statistics in medical research. They consistently found that a minority of studies reported unacceptable methods of data analysis and concluded that this is likely due to the fact that individuals leading these medical publications may not have a solid grasp on basic statistical concepts.^45,47 This may not be any different in today’s medical literature and is likely further compounded by the fact that statisticians are not consistently involved in the peer review process,⁴⁸ a step that often improves the quality of the statistical methodology of accepted articles.⁴⁹ Furthermore, the advent of sophisticated statistical software has allowed the novice researcher to perform complex statistical analyses in a matter of seconds, whereas these complicated analyses were previously performed solely by statisticians as they required careful thought and complex mathematical formulae.

The study design characteristics and their relation to the proper analysis of matched data as shown in Table 2, generate a few interesting hypotheses. First, studies involving case populations with cancer or cardiovascular disease were more likely to employ statistical techniques that account for dependent data than studies involving other case definitions. The reason for this is not clear but it may be a reflection of the rigor with which these studies were designed. Second, a greater number of studies in the incorrectly analyzed collection focused on a surgical topic, compared to studies in the correctly analyzed collection. Although the reasons for this finding are unclear, one potential explanation might be that the use of inferential statistics in surgical studies is a more recent development when compared to studies focusing on other medical topics. Older surgical research involved smaller sample sizes and very few of these studies employed inferential statistics.⁴⁶ The observation that studies with multiple controls matched to each case were more likely to use statistics that were appropriate for matched data was not surprising. The decision to match more than one control per case may increase the power of case-control studies,¹ which, in turn, increases the strength of the study and reflects a thoughtful, systematic approach to the study design. This same thought was likely extended to the analysis phase and resulted in the correct application of statistical methodology.

Furthermore, the observations made with regard to the publishing journal’s impact factor were also not surprising. Recent studies have shown that there is a correlation with the strength of study design (including the use of recommended statistical reporting) and journal publication characteristics including the journal’s impact factor.^50,51 Lee and colleagues reviewed 243 randomly selected articles from the general internal medicine literature published between January 1, 1999 and December 31, 1999 to determine if there was a link between methodological quality and journal characteristics.⁵⁰ The authors found significant associations between quality scores and higher citation rates, higher impact factors, higher circulation, and lower manuscript acceptance rates. Similarly, Kuroki et al investigated the potential link between research methodology and statistical reporting in medical journals with a high impact factor compared to moderate-impact-factor obstetrics and gynecology journals. ⁵¹ The high-impact-factor medical journals included: Journal of the American Medical Association, The Lancet, and the New England Journal of Medicine; whereas the moderate-impact-factor group included American Journal of Obstetrics and Gynecology, British Journal of Obstetrics and Gynecology, and Obstetrics and Gynecology. The authors found that the majority of studies included in the high-impact-factor group were randomized controlled trials (35%) and had high compliance with recommended statistical reporting (84% compared to 65%, P = 0.002). Therefore, our finding of an increased impact factor in the correct statistical analysis group aligns with trends observed in similar studies.

This study has a number of pertinent strengths and limitations that warrant discussion. First, this study is novel; as to our knowledge, this is the first review of the appropriateness of the statistical methodology employed in a collection of matched case-control studies. Second, the studies selected for review focused on a broad range of topics in a number of different peer-reviewed journals, so, although we reviewed only 37 studies, this sample is representative of the much larger population of available articles. Furthermore, many of these articles were published in mainstream journals read by individuals from a variety of backgrounds such as the British Medical Journal, the Canadian Medical Association Journal, and the Lancet. The major limitation of this study is that the methodology of the articles was not assessed with a validated scoring system. To our knowledge, such a scoring system does not exist for matched case-control studies, and other validated scoring systems were not applicable to the current study’s objectives.⁵² In spite of this limitation, the inter-observer agreement between the authors reviewing the studies was high. This suggests that the scoring system used in this study was applied in a consistent manner, and our conclusion regarding inconsistent use of proper statistical methods in matched case-control studies is valid. Another important limitation is the inability to determine whether the use of proper statistics would change the conclusions presented by studies that used improper statistical methods. This is due to the fact that very few case-control studies presented tables outlining the number of discordant sets. This limitation notwithstanding, it is possible that the use of proper statistical methods will at least decrease the strength of the association between the outcome and exposure variables when compared to that obtained from improper statistical methodology.

Conclusion

The majority of matched case-control studies reviewed in this investigation used improper statistical methods. Although matching cases to controls provides a means of controlling for known potential confounding variables, it is a complicated process that requires a great deal of thought in order to be effective. Improper application of this methodology can distort a study’s power and possibly lead to the reporting of incorrect disease-exposure associations. The acceptance of invalid conclusions and subsequent adaptation into medical practice may lead to the inappropriate use of resources and even worse, harm to individuals. This is why guidelines such as CONSORT (Consolidated Standards of Reporting Trials) and STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) were created.^53,54 These documents contain templates that are designed to create homogeneity in the reporting of randomized clinical trials and observational studies, respectively. The results of the current study suggest that although the STROBE checklist includes recommendations for outlining the matching methodology when reporting a matched case-control study, these comprehensive epidemiologic guidelines may require an additional section that outlines the proper statistical techniques to be employed when conducting a matched case-control study.