Skip to main content
PMC home page
Journal of Epidemiology and Community Health logoLink to Journal of Epidemiology and Community Health
. 1998 Jun;52(6):377–384. doi: 10.1136/jech.52.6.377

The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions

S H Downs, N Black
PMCID: PMC1756728  PMID: 9764259

Abstract

OBJECTIVE: To test the feasibility of creating a valid and reliable checklist with the following features: appropriate for assessing both randomised and non-randomised studies; provision of both an overall score for study quality and a profile of scores not only for the quality of reporting, internal validity (bias and confounding) and power, but also for external validity. DESIGN: A pilot version was first developed, based on epidemiological principles, reviews, and existing checklists for randomised studies. Face and content validity were assessed by three experienced reviewers and reliability was determined using two raters assessing 10 randomised and 10 non- randomised studies. Using different raters, the checklist was revised and tested for internal consistency (Kuder-Richardson 20), test-retest and inter-rater reliability (Spearman correlation coefficient and sign rank test; kappa statistics), criterion validity, and respondent burden. MAIN RESULTS: The performance of the checklist improved considerably after revision of a pilot version. The Quality Index had high internal consistency (KR-20: 0.89) as did the subscales apart from external validity (KR-20: 0.54). Test-retest (r 0.88) and inter-rater (r 0.75) reliability of the Quality Index were good. Reliability of the subscales varied from good (bias) to poor (external validity). The Quality Index correlated highly with an existing, established instrument for assessing randomised studies (r 0.90). There was little difference between its performance with non-randomised and with randomised studies. Raters took about 20 minutes to assess each paper (range 10 to 45 minutes). CONCLUSIONS: This study has shown that it is feasible to develop a checklist that can be used to assess the methodological quality not only of randomised controlled trials but also non-randomised studies. It has also shown that it is possible to produce a checklist that provides a profile of the paper, alerting reviewers to its particular methodological strengths and weaknesses. Further work is required to improve the checklist and the training of raters in the assessment of external validity.

 

Full Text

The Full Text of this article is available as a PDF (149.7 KB).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. BADGLEY R. F. An assessment of research methods reported in 103 scientific articles from two Canadian medical journals. Can Med Assoc J. 1961 Jul 29;85:246–250. [PMC free article] [PubMed] [Google Scholar]
  2. Bailey K. R. Generalizing the results of randomized clinical trials. Control Clin Trials. 1994 Feb;15(1):15–23. doi: 10.1016/0197-2456(94)90024-8. [DOI] [PubMed] [Google Scholar]
  3. Black N. A., Downs S. H. The effectiveness of surgery for stress incontinence in women: a systematic review. Br J Urol. 1996 Oct;78(4):497–510. doi: 10.1046/j.1464-410x.1996.01422.x. [DOI] [PubMed] [Google Scholar]
  4. Colditz G. A., Miller J. N., Mosteller F. How study design affects outcomes in comparisons of therapy. I: Medical. Stat Med. 1989 Apr;8(4):441–454. doi: 10.1002/sim.4780080408. [DOI] [PubMed] [Google Scholar]
  5. Davis C. E. Generalizing from clinical trials. Control Clin Trials. 1994 Feb;15(1):11–14. doi: 10.1016/0197-2456(94)90023-x. [DOI] [PubMed] [Google Scholar]
  6. DerSimonian R., Charette L. J., McPeek B., Mosteller F. Reporting on methods in clinical trials. N Engl J Med. 1982 Jun 3;306(22):1332–1337. doi: 10.1056/NEJM198206033062204. [DOI] [PubMed] [Google Scholar]
  7. Gardner M. J., Machin D., Campbell M. J. Use of check lists in assessing the statistical content of medical studies. Br Med J (Clin Res Ed) 1986 Mar 22;292(6523):810–812. doi: 10.1136/bmj.292.6523.810. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Harth S. C., Thong Y. H. Sociodemographic and motivational characteristics of parents who volunteer their children for clinical research: a controlled study. BMJ. 1990 May 26;300(6736):1372–1375. doi: 10.1136/bmj.300.6736.1372. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Landis J. R., Koch G. G. The measurement of observer agreement for categorical data. Biometrics. 1977 Mar;33(1):159–174. [PubMed] [Google Scholar]
  10. Lionel N. D., Herxheimer A. Assessing reports of therapeutic trials. Br Med J. 1970 Sep 12;3(5723):637–640. doi: 10.1136/bmj.3.5723.637. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Marubini E., Mariani L., Salvadori B., Veronesi U., Saccozzi R., Merson M., Zucali R., Rilke F. Results of a breast-cancer-surgery trial compared with observational data from routine practice. Lancet. 1996 Apr 13;347(9007):1000–1003. doi: 10.1016/s0140-6736(96)90145-2. [DOI] [PubMed] [Google Scholar]
  12. Moher D., Jadad A. R., Nichol G., Penman M., Tugwell P., Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995 Feb;16(1):62–73. doi: 10.1016/0197-2456(94)00031-w. [DOI] [PubMed] [Google Scholar]
  13. Peto R. Clinical trial reporting. Lancet. 1996 Sep 28;348(9031):894–895. doi: 10.1016/S0140-6736(05)64769-1. [DOI] [PubMed] [Google Scholar]
  14. Rosenberg W., Donald A. Evidence based medicine: an approach to clinical problem-solving. BMJ. 1995 Apr 29;310(6987):1122–1126. doi: 10.1136/bmj.310.6987.1122. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Sackett D. L. Bias in analytic research. J Chronic Dis. 1979;32(1-2):51–63. doi: 10.1016/0021-9681(79)90012-2. [DOI] [PubMed] [Google Scholar]
  16. Sacks H. S., Berrier J., Reitman D., Ancona-Berk V. A., Chalmers T. C. Meta-analyses of randomized controlled trials. N Engl J Med. 1987 Feb 19;316(8):450–455. doi: 10.1056/NEJM198702193160806. [DOI] [PubMed] [Google Scholar]
  17. Schulz K. F., Chalmers I., Hayes R. J., Altman D. G. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995 Feb 1;273(5):408–412. doi: 10.1001/jama.273.5.408. [DOI] [PubMed] [Google Scholar]
  18. Thomson M. E., Kramer M. S. Methodologic standards for controlled clinical trials of early contact and maternal-infant behavior. Pediatrics. 1984 Mar;73(3):294–300. [PubMed] [Google Scholar]

Articles from Journal of Epidemiology and Community Health are provided here courtesy of BMJ Publishing Group

RESOURCES