Abstract
Background:
Database research is being used in orthopedic literature with increased regularity. The main limitation of database research is the absence of diagnosis and treatment verification afforded by medical chart review. This absence may limit the accuracy of some conclusions and recommendations produced by database research.
Hypothesis/Purpose:
The purpose of this study is to describe the accuracy of one database (Rochester Epidemiology Project) used in orthopedic research to detect isolated anterior cruciate ligament (ACL) tears and to discuss the limitations of database research. Our hypothesis is that diagnostic codes alone are unlikely to be accurate in identifying patients with ACL tears.
Study Design:
Historical geographic cohort study, Level IV
Methods:
A population-based historical cohort study was performed using the Rochester Epidemiology Project (REP) database. All subjects identified had International Classification of Diseases (ICD9) diagnosis codes consistent with anterior cruciate ligament tears between January 1, 1990 and December 31, 2010. The medical records of all subjects were reviewed in detail to confirm the accuracy of diagnosis and gather data on injury type, laterality, associated meniscal injuries, magnetic resonance imaging (MRI) findings and treatment details.
Results:
A total of 3,494 patients had codes consistent with ACL tears and 2,288 of them were confirmed through chart review to have an isolated ACL tear (65.5%). Among these were 1,841 patients (52.7%) with an ACL tear within one year of injury and an additional 447 patients (12.8%) with an ACL tear greater than one year from injury. Thirty-nine patients (1.1%) were diagnosed with a partial ACL tear on MRI, 48 patients (1.4%) with an isolated PCL tear, and 22 patients (0.6%) with a combined ACL and PCL injury. 24 patients (0.7%) had had ACL reconstruction prior to the study period. The remaining 1,073 patients (30.7%) had diagnostic codes consistent with an ACL tear but did not have a cruciate ligament injury.
Conclusion:
This study demonstrates low accuracy when using diagnostic codes alone to identify an ACL tear. Database studies offer unique benefits to medical literature but the inherent limitations should be taken into account when using this data to counsel patients, dictate clinical management, or make healthcare policy decisions. Information from a healthcare database is most accurate when accompanied by verification of diagnosis, treatment, and outcomes with medical chart review.
Introduction
Database research is being used in orthopedic literature with increased regularity.10–14, 19,21 The benefit of this type of research is the ability to quickly identify a large cohort of patients with a given condition and evaluate treatment outcomes. This process is usually accomplished using diagnostic or procedural codes from large health care administrative databases.5–7
Numerous applications of database research are seen throughout the orthopedic literature especially when looking for changes in treatment patterns.1,4,8 For example, one study described an increase in the annual incidence of either primary or revision ACL reconstruction (ACLR) surgery in a large population of privately-insured patients.4 An additional study used the PearlDiver Patient Record Database to demonstrate an increase in the number of meniscal repairs performed in the United States over a seven year period, although it is uncertain if all cases represented primary meniscal repairs.1
The main limitation of database research is the absence of diagnosis and treatment verification afforded by medical chart review. This absence ultimately may limit the accuracy of some conclusions and recommendations produced by database research. The purpose of this study was to describe the accuracy of one database (Rochester Epidemiology Project) used in orthopedic research and to investigate potential limitations of database research. The hypothesis tested was that diagnostic codes alone may overestimate the number of patients with ACL tears.
Methods
A population-based historical cohort study was performed using the Rochester Epidemiology Project (REP) database in Olmsted County, MN which had a population of 144,260 in 2010. Briefly, the REP is a medical record linkage system which provides access to the complete medical records for all residents of Olmsted County, regardless of the medical facility in which the care was delivered.15 This information is derived directly from physician-determined diagnostic codes and compiles comprehensive diagnostic and procedural information from all medical centers in Olmsted County into one database. This population-based setting allows essentially complete ascertainment and follow-up of all clinically diagnosed cases of ACL tears in a geographically-defined community and provides the ability to access original medical records for confirmation of diagnosis and treatment. Due to the geographical isolation of Olmsted County from other large urban centers and availability of health care providers, most residents receive care within Olmsted County which allows uninterrupted natural history studies.15,18
All subjects identified were residents of Olmsted County, MN and had International Classification of Diseases (ICD9) diagnosis codes consistent with anterior cruciate ligament tears (844.2, 717.83) between January 1, 1990 and December 31, 2010. This search identified a total of 3,494 potential subjects that had diagnostic codes positive for an ACL tear. The medical records of all subjects were reviewed in detail to confirm the accuracy of diagnostic code. There were two possible outcomes from chart review. First, subjects had positive ACL diagnostic codes and were confirmed to have an ACL tear. Second, patients had positive ACL diagnostic codes but were found not to have an ACL tear. Magnetic resonance imaging (MRI) was available in 95% of cases and was read by a musculoskeletal-trained radiologist or general radiologist and the images were reviewed by the primary author. An isolated ACL tear was defined as not occurring with a concomitant ligament injury that required surgery; however, ACL tears with medial collateral ligament (MCL) sprains treated non-operatively were included. ACL tears with concomitant meniscus or articular cartilage injury were also included. Institutional Review Board (IRB) approval was obtained for this study from all medical institutions in Olmsted County including Mayo Clinic and Olmsted Medical Center.
Results
The final study cohort consisted of 2,288 isolated ACL tears during the study period which represents a diagnostic accuracy (positive predictive value) of 65.5% (2,288/3,494). There were 1,841 patients (52.7%) diagnosed with an ACL tear within one year of injury and an additional 447 patients (12.8%) diagnosed with an ACL tear greater than one year after injury.
Thirty-nine patients (1.1%) were diagnosed with a partial ACL tear on MRI and 24 patients (0.7%) had an ACL reconstruction prior to the study period but had a diagnostic code consistent with an ACL tear. Forty-eight patients (1.4%) were diagnosed with an isolated PCL tear and 22 patients (0.6%) had a combined ACL and PCL injury. Although these patients did not have an isolated ACL tear, their ACL coding was accurate. Therefore, inclusion of all patients with an ACL tear combined with PCL tear, previous ACL reconstruction, or ACL equivalent (partial ACL tear) increased the diagnostic accuracy to 67.9% (2,373/3,494). Diagnoses of all patients in our study can be seen below in Table 1.
Table 1:
Diagnoses of All Identified Patients
| Isolated ACL Tear within 1 yr of injury greater than 1 yr from injury |
2,288 (65.5%) 1,841 (52.7%) 447 (12.8%) |
| Partial ACL Tear | 39 (1.1%) |
| Isolated PCL Tear | 48 (1.4%) |
| Combined ACL & PCL Tear | 22 (0.6%) |
| ACLR Prior to Study Period | 24 (0.7%) |
| No Cruciate Ligament Injury | 1,073 (30.7) |
| TOTAL | 3,494 (100%) |
The remaining 1,073 patients (30.7%) who had diagnostic codes consistent with an ACL tear did not have a cruciate ligament injury. This included 21 patients who were diagnosed with a fracture of the tibia or femur. The remaining patients were inappropriately coded with an ACL tear and instead were ultimately diagnosed with knee contusions/sprains (65%), isolated meniscus tear (15%), isolated collateral ligament injury (10%), patellar dislocation (5%), knee synovitis (2%), tibia eminence/plateau fractures (1%), extensor mechanism injury (1%), or femur fracture (1%) (Table 2).
Table 2:
Diagnoses of Incorrectly Coded ACL Tears
| Injury | Number of Events (%) |
|---|---|
| Contusion/Knee Sprain | 694 (65%) |
| Isolated Meniscus Tear | 161 (15%) |
| Isolated Collateral Ligament injury | 109 (10%) |
| Patellar Dislocation/Subluxation | 57 (5%) |
| Knee Synovitis | 18 (2%) |
| Tibia Eminence/Plateau Fractures | 14 (1%) |
| Extensor Mechanism Injury | 13 (1%) |
| Femur Fracture | 7 (1%) |
| TOTAL | 1,073 |
The majority of these incorrect diagnosis codes were entered by primary care physicians as well as a substantial number by emergency medicine and PMR providers. Incorrect diagnosis codes were also entered by Orthopedic surgeons (Table 3).
Table 3:
Numbers and Types of Incorrect Diagnoses by Provider Type
| Injury | Events | Emergency Medicine | Primary Care | PMR | Orthopedic Surgery |
|---|---|---|---|---|---|
| Contusion/Knee Sprain | 694 (65%) | 73 | 452 | 160 | 9 |
| Isolated Meniscus Tear | 161 (15%) | 19 | 77 | 35 | 30 |
| Isolated Collateral Ligament | 109 (10%) | 14 | 49 | 20 | 26 |
| Patellar Dislocation / Subluxation | 57 (5%) | 27 | 25 | 5 | 0 |
| Knee Synovitis | 18 (2%) | 3 | 10 | 5 | 0 |
| Tibia Eminence / Plateau Fractures | 14 (1%) | 12 | 1 | 1 | 0 |
| Extensor Mechanism | 13 (1%) | 11 | 2 | 0 | 0 |
| Femur Fracture | 7 (1%) | 7 | 0 | 0 | 0 |
| TOTAL | 1,073 | 166 | 616 | 226 | 65 |
Discussion
The most significant finding of this study was the relatively low accuracy of diagnostic codes alone for identification of an ACL tear and likely indicates that some ACL tears are not always a conspicuous diagnosis. In this cohort, an ACL tear was correctly identified in only two-thirds of all cases. The most commonly encountered scenario for inaccurate coding was an initial diagnosis made by a primary care physician. However, misdiagnosis also occurred by emergency medicine providers prior to advanced imaging, orthopedic surgeons and other providers who specialize in the musculoskeletal physical examination. These errors also occurred after an initial consultation and prior to MRI scanning. The presence of a knee effusion and patient guarding during the physical exam likely make standard exam maneuvers (Lachman, anterior drawer, pivot shift) less sensitive for an ACL tear. Therefore physicians often made an initial diagnosis based on mechanism of injury and clinical history rather than confirmatory physical exam findings. This database includes diagnostic codes for all physicians regardless of specialty or familiarity with musculoskeletal injuries likely making it prone to a high rate of inaccurate coding from an initial encounter.
Database studies have often relied on ICD codes to identify patient cohorts.11,20 One administrative database study reported a significant increase in the incidence of ACL reconstruction between 1994 and 2007 using ICD-9 diagnosis codes.11 However, the authors noted that they could not directly differentiate between anterior and posterior cruciate ligament reconstructions as CPT codes were not available.11 Thus, their findings likely represented an overestimation of the true ACL reconstruction incidence. Likewise, multiple studies found a substantial increase in ACLR over a two to three decade period but also noted that using ICD diagnosis codes alone may have overestimated the true incidence of ACLR.3, 10, 12 A similar study using ICD-10 (Australian Modification) diagnosis codes also reported the potential overestimation of ACLR incidence due to the lack of specificity of ICD codes to the anterior cruciate ligament alone.22 The most common reasons for overestimating the incidence of ACL injury are inaccurate diagnosis (often based on initial presentation prior to advance imaging), incorrect entry into the database, and the lack of specificity of ICD codes for ACL injury. Based on the results of this study, using diagnostic codes alone would have resulted in a significant overestimation of the incidence of ACL injury.
In contrast to this, use of ICD codes for determining incidence may also underestimate true values. One study utilized the National Electronic Injury Surveillance System to identify patients presenting to the emergency room after a patellar dislocation. This study reported that the annual incidence of patellar dislocation is 2.9 per 100,000 person-years in the United States.20 However, an additional study that identified patients in a geographically-determined population reported an incidence of patellar dislocation to be 23.2 per 100,000 person-years (10-times higher) after verification of the diagnosis with medical chart review.15 Bedard et al examined the accuracies of ICD code searches in multiple common databases including the National Surgical Quality Improvement Programs (NSQIP), the Nationwide Inpatient Sample (NIS), the Medicare Standard Analytic Files (MED), and the Humana Administrative Claims database (HAC). They found significant variability in the prevalence of surgical complications following orthopedic procedures across the various databases and especially noted multiple underestimations of NIS when compared to other databases.2 It is important to note that the current study is unable to detect these potential underestimates. Database information may underestimate outcomes or diagnoses when the ability to capture information is limited. For example, limiting data entry to certain specialties of providers or different definitions of postoperative complications can underestimate true values.
Additional database studies have used CPT codes to identify patients who received surgical treatment.4,7,8 An administrative database study of insured patients demonstrated a significant increase in the incidence of ACL reconstruction between 2001 and 2005.4 A similar study using the PearlDiver database reported a significant increase in the incidence of ACL reconstruction between 2004 and 2009. The value of these studies is that they demonstrate an increase in the number of ACL reconstructions performed each year in the United States which may potentially be helpful for national resource allocation. However, in the absence of medical chart review, these studies cannot differentiate between a primary ACLR, a revision ACLR, and a contralateral ACLR which are all included as primary events. Similarly, these studies cannot determine if ACLR occurred after an acute ACL tear or in a patient with chronic ACL deficiency. The magnitude at which this limitation affects the results reported is unknown. Additionally, these studies provide no information on the incidence of ACL injury. One study of a geographically-determined population demonstrated that although the rate of ACLR increased significantly, the incidence of ACL injury decreased slightly in males and remained relatively unchanged in females over a 21 year observation period.15
Database information is likely most accurate when direct access to medical records is available. One series that reported complications among 784 patients after surgical treatment of distal biceps ruptures identified patients using specific diagnosis search terms (instead of ICD or CPT codes) in a large healthcare database.6 These authors confirmed the complications with manual chart review, although they did not mention the accuracy of the diagnosis search terms used in the study to identify patients.6 George et al studied the growth in obesity among patients following primary total knee (TKA) or total hip arthroplasty (THA) using ICD-9 codes. They found that using diagnostic codes alone overestimated the growth of obesity in THA/TKA by 5 to 8 times compared to reviewing the BMI in medical records.9 The authors concluded that studies using large databases should be interpreted with caution due to the low coding accuracy found in their review. Additionally, the Rochester Epidemiology Project is a population-based healthcare database that allows researchers to directly review patient medical records and has been used in a variety of orthopedic related research.17 Conclusions made from database investigation should be validated by other research designs. Additionally, statistical differences reported from large samples may not be reflective of clinically significance.
Researchers should recognize that each healthcare database has unique attributes which may be well suited to answer a specific type of study question. For example, large administrative databases without the ability to review medical records, such as the PearlDiver database, are likely to be effective in reporting national trends in surgical treatment patterns and healthcare resource utilization.1,4 In contrast, the Kaiser-Permanante database is a large administrative database that has the potential for medical chart review and may be better suited to report the outcomes and complications of surgical treatment.6 Finally, the REP is a population-based healthcare database (also with the potential for medical chart review) and may be well suited to help answer epidemiologic questions with the ability to study a geographically-defined population. Researchers should continue to improve database methodology and ensure accurate recording of information which will improve the utility for orthopedic surgeons.
The results from this study should be taken with the following limitations. The accuracy for identification of ACL tears in this cohort is specific to the REP and may not be reflective of the diagnostic accuracy of other healthcare databases. The injury patterns in this geographic region may not be generalizable to other populations. Additionally, MRI verification was not available in 5% of cases. Limiting the study inclusion to only patients with an MRI scan would likely increase the diagnostic accuracy; however, it would not accurately capture the true coding patterns in this database. Patients with ACL tears who were not appropriately identified by a physician were not captured in this study and limit our ability to record false negative diagnoses. The accuracy of diagnostic codes presented applies only to ACL tears in a single database and may not be representative of the coding accuracy of other orthopedic injuries. In addition, patients move in and out of Olmsted County which may affect the outcomes of this study. Despite these, the population-based design and verification of diagnosis by chart review allowed this study to identify true anterior cruciate ligament injuries with a high degree of accuracy.
Conclusion
This study demonstrates low accuracy when using diagnostic codes alone to identify an ACL tear. Database studies offer unique benefits to medical literature but limitations of these studies should be taken into account when using this data to counsel patients, dictate clinical management, or make healthcare policy decisions. Information from a healthcare database is most accurate when accompanied by verification of diagnosis, treatment, and outcomes with medical chart review. The results of this study can be useful to draw meaningful conclusions from database research.
What is known about this subject: Numerous applications of database research are seen throughout the orthopedic literature especially when looking for changes in treatment patterns. The main limitation of database research is the absence of diagnosis and treatment verification afforded by medical chart review.
What this study adds to existing knowledge: The most significant finding of this study was the relatively low accuracy of diagnostic codes alone for identification of an ACL tear and likely indicates that some ACL tears are not always a conspicuous diagnosis. The results of this study suggest that database information is likely most accurate when direct access to medical records is available.
Acknowledgment:
This study was made possible by the Rochester Epidemiology Project which is supported by the National Institute on Aging of the National Institutes of Health under award number R01AG034676. The findings and conclusions of this report are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health.
References
- 1.Abrams G, Frank R, Gupta A, Harris J, McCormick F, Cole B. Trends in meniscus repair and meniscectomy in the United States, 2005–2011. Am J Sports Med. 2013;41(10):2333–2339 [DOI] [PubMed] [Google Scholar]
- 2.Bedard N, Pugely A, McHugh M, Lux N, Oter J, Bozic K, Gao Y, Callaghan J. Analysis of outcomes after TKA: Do all databases produce similar findings? Clin Orthop Relat Res. 2018;476(1):52–63. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Buller L, Best M, Baraga M, Kaplan L. Trends in anterior cruciate ligament reconstruction in the United States. Orthop J Sports Med. 2014;3(1). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Csintalan R, Inacio M, Funahashi T. Incidence Rate of Anterior Cruciate Ligament Reconstructions. Perm J. 2008;12(3):17–21 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Cvetanovich G, Yanke A, McCormick F, Bach B, Cole B. Trends in meniscal allograft transplantation in the United States, 2007–2011. Arthroscopy. 2015;31(6):1123–1237 [DOI] [PubMed] [Google Scholar]
- 6.Dunphy T, Hudson J, Batech M, Acevedo D, Mirzayan R. Surgical treatment of distal biceps tendon ruptures: An analysis of complications in 784 surgical repairs. Am J Sports Med. 2017. doi: 10.1177/0363546517720200 [DOI] [PubMed] [Google Scholar]
- 7.Erickson B, Basques B, Griffin J, Taylor S, O’Brien S, Verma N, Romeo A. The effect of concomitant biceps tenodesis on reoperation rates after rotator cuff repair: A review of a large private-payer database from 2007–2014. Arthroscopy. 2017;33(7):1301–1307. [DOI] [PubMed] [Google Scholar]
- 8.Erickson B, Nwachukwu B, Rosas S, Schairer W, McCormick F, Bach B, Bush-Joseph C, Romeo A. Trends in medial ulnar collateral ligament reconstruction in the United States: A retrospective review of a large private-payer database from 2007–2011. Am J Sports Med. 2015;43(7):1770–1774 [DOI] [PubMed] [Google Scholar]
- 9.George J, Newman J, Ramanathan D, Klika A, Higuera C, Barsoum W. Administrative database can yield false conclusions – An example of obesity in total joint arthroplasty. J Arthroplasty. 2017:32(9):86–90. [DOI] [PubMed] [Google Scholar]
- 10.Leathers M, Merz A, Wong J, Scott T, Wang J, Hame S. Trends and demographics in anterior cruciate ligament reconstruction in the United States. J Knee Surg. 2015;28(5):390–394 [DOI] [PubMed] [Google Scholar]
- 11.Leonard B, Best M, Baraga M, Kaplan L. Trends in Anterior Cruciate Ligament Reconstruction in the United States. Orthop J Sports Med. 2015. January;3(1) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Mall N, Chalmers P, Moric M, Tanaka M, Cole B, Bach B, Patella G. Incidence and trends of anterior cruciate ligament reconstruction in the United States. Am J Sports Med. 2014;42(10):2636–2370. [DOI] [PubMed] [Google Scholar]
- 13.McCormick F, Harris J, Abrams G, Frank R, Gupta R, Hussey K, Wilson H, Bach B, Cole B. Trends in the surgical treatment of articular cartilage lesions in the United States: an analysis of a large private-payer database over a period of 8 years. Arthroscopy. 2014;30(2):222–226 [DOI] [PubMed] [Google Scholar]
- 14.McCormick F, Nwachukwu B, Kiriakopoulos E, Schairer W, Provencher M, Levy J. In-hospital mortality risk for total shoulder arthroplasty: A comprehensive review of the medicare database from 2005 to 2011. Int J Shoulder Surg. 2015;9(4):110–113 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Sanders T, Maradit Kremers H, Bryan A, Larson D, Dahm D, Levy B, Stuart M, Krych A. Incidence of anterior cruciate ligament tears and reconstruction: A 21-year population-based study. Am J Sports Med. 2016;44(6):1502–1507 [DOI] [PubMed] [Google Scholar]
- 16.Sanders T, Pareek A, Hewett T, Stuart M, Dahm D, Krych A. Incidence of first-time lateral patellar dislocation: A 21-year population-based study. Sports Health. 2017; doi 10.1177/1984768117725055 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.St Sauver J, Grossardt B, Leibson C, Yawn B, Melton L, Rocca W. Generalizability of epidemiological findings and public health decisions: An illustration from the Rochester Epidemiology Project. Mayo Clin Proc. 2012;87:151–160. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Truntzer J, Hoppe D, Shapiro L, Abrams G, Safran M. Complication rates for hip arthroscopy are underestimated: A population-based study. Arthroscopy. 2017;33(6):1194–1201 [DOI] [PubMed] [Google Scholar]
- 19.Waterman B, Belmont P, Owens B. Patellar dislocation in the United States: role of sex, age, race, and athletic participation. J Knee Surg. 2012;25(1):51–57 [DOI] [PubMed] [Google Scholar]
- 20.Weinreb J, Yoshida R, Cote M, O’Sullivan M, Mazzocca A. A review of databases used in orthopedic surgery research and an analysis of database use in Arthroscopy: The journal of Arthroscopic and Related Surgery. Arthroscopy. 2017;33(1):225–231 [DOI] [PubMed] [Google Scholar]
- 21.Zbrojkiewicz D, Vertullo C, Grayson J. Increasing rates of anterior cruciate ligament reconstruction in young Australians, 2000–2015. Med J Aust. 2018. [Epub ahead of print] [DOI] [PubMed] [Google Scholar]