Skip to main content
NCBI home page
Canadian Respiratory Journal logoLink to Canadian Respiratory Journal
letter
. 2012 Sep-Oct;19(5):331. doi: 10.1155/2012/683602

The rule of thumb in validity analysis of a test

Siamak Sabour 1
PMCID: PMC3473009  PMID: 23061079

To the Editor,

I was interested to read the article by Lacasse et al (1) published in the March/April 2012 issue of the Canadian Respiratory Journal. The authors aimed to determine whether the principal diagnoses of chronic obstructive pulmonary disease (COPD) made in hospitalized patients and recorded in a large administrative database were valid. They reported only positive predictive value (PPV, 50.4% [95% CI 47.7% to 53.3%]), with 372 patients (30.5%) classified as ‘indeterminate’.

To the best of my knowledge, out of seven statistical tests for validity analysis, only PPV is greatly dependent on the prevalence of the outcome (ie, COPD) which is one of the weaknesses of PPV for validity analysis. The higher the prevalence, the higher the PPV! Why did the authors not use positive likelihood ratio (LR, true positive/false positive) and negative likelihood ratio (false negative/true negative) as well as odds ratio (true results/false results – preferably more than 50), which are among the best tests to evaluate the validity (accuracy) of a single test compared with a gold standard. Considering the range of LR+ (1 – infinity) and LR– (0 – 1), these test can correctly assess validity (2–4).

As the authors point out in their conclusion, routine ascertainment of the validity of diagnoses should be considered before using administrative databases in clinical and health services research. It is obvious that the accuracy of a large administrative database is usually lower than the gold standard; however, depending on the quality of the database and, of course, the definition of the outcome, sometimes there is no need to consider validity analysis.

REFERENCES

  • 1.Lacasse Y, Daigle JM, Martin S, Maltais F. Validity of chronic obstructive pulmonary disease diagnoses in a large administrative database. Can Respir J. 2012;19:e5–9. doi: 10.1155/2012/260374. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Jeckel JF, Katz DL, Elmore JC. Epidemiology, Biostatistics and Preventive Medicine. 1st edn. Philadelphia: Saunders Elsevier; 2008. [Google Scholar]
  • 3.Rothman K, Greenland S. Modern Epidemiology. 3rd edn. Philadelphia: Williams Lippincott and Wilkins; 2010. [Google Scholar]
  • 4.Szklo M, Nieto FJ. Epidemiology Beyond the Basics. 2nd edn. Sudbury: Jones and Bartlett; 2007. [Google Scholar]
Can Respir J. 2012 Sep-Oct;19(5):331.

Letters to the Editor

Yves Lacasse 1, François Maltais 1

Re: S Sabour. The rule of thumb in validity analysis of a test

To the Editor,

To estimate likelihood ratios (both positive and negative), we should have included patients with and others without a discharge diagnosis of COPD. We studied only patients with such a discharge diagnosis. We did not need more because our objective was to determine the proportion of valid discharge diagnoses of COPD in the MedEcho database, ie, diagnoses corroborated by clinical history (including smoking status) and pulmonary function tests. In itself, this proportion corresponds to the positive predictive value.

Several validation studies of administrative databases have reported sensitivity and specificity of coding algorithms for identifying patients with COPD (1,2), often with disappointing results. We would argue that the validation of administrative databases for the identification of patients with a specific condition cannot be based on probabilities. Researchers must be very confident that the patients they identify truly have the condition under study. Had we included a control group with a discharge diagnosis other than COPD, we would not have reported likelihood ratios. We would have preferred reporting a statistic of concordance between the database diagnoses and our gold standard (such as kappa [3]), which is another way of expressing validity.

We reiterate that validation studies are mandatory before using databases for research purposes; the decision to use an administrative database should be made on validity thresholds determined a priori, and the criteria of quality must be very high. Furthermore, validation studies must be regularly updated if the database is to be used in several occasions over time. Unfortunately, administrative database research infrequently use validated diagnostic and procedural codes (4). Without proper validation studies clearly demonstrating the quality of the data, projects that use a database for research should be abandoned.

REFERENCES

  • 1.Stein BD, Bautista A, Schumock GT, et al. The validity of International Classification of Diseases, Ninth Revision, Clinical Modification diagnosis codes for identifying patients hospitalized for COPD exacerbations. Chest. 2012;141:87–93. doi: 10.1378/chest.11-0024. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Gershon AS, Wang C, Guan J, et al. Identifying individuals with physician diagnosed COPD in health administrative databases. COPD. 2009;6:388–94. doi: 10.1080/15412550903140865. [DOI] [PubMed] [Google Scholar]
  • 3.Kramer MS, Feinstein AR. Clinical biostatistics. LIV. The biostatistics of concordance. Clin Pharmacol Ther. 1981;29:111–23. doi: 10.1038/clpt.1981.18. [DOI] [PubMed] [Google Scholar]
  • 4.Van Walraven C, Bennett C, Forster AJ. Administrative database research infrequently used validated diagnostic or procedural codes. J Clin Epidemiol. 2011;64:1054–9. doi: 10.1016/j.jclinepi.2011.01.001. [DOI] [PubMed] [Google Scholar]

Articles from Canadian Respiratory Journal : Journal of the Canadian Thoracic Society are provided here courtesy of Wiley

RESOURCES