Comparison of SNOMED CT versus Medcin Terminology Concept Coverage for Mild Traumatic Brain Injury

Abstract

Background:

Traumatic Brain Injury (TBI) is a “signature” injury of the current wars in Iraq and Afghanistan. Structured electronic data regarding TBI findings is important for research, population health and other secondary uses but requires appropriate underlying standard terminologies to ensure interoperability and reuse. Currently the U.S. Department of Veterans Affairs (VA) uses the terminology SNOMED CT and the Department of Defense (DOD) uses Medcin.

Methods:

We developed a comprehensive case definition of mild TBI composed of 68 clinical terms. Using automated and manual techniques, we evaluated how well the mild TBI case definition terms could be represented by SNOMED CT and Medcin, and compared the results. We performed additional analysis stratified by whether the concepts were rated by a TBI expert panel as having High, Medium, or Low importance to the definition of mild TBI.

Results:

SNOMED CT sensitivity (recall) was 90% overall for coverage of mild TBI concepts, and Medcin sensitivity was 49%, p < 0.001 (using McNemar’s chi square). Positive predictive value (precision) for each was 100%. SNOMED CT outperformed Medcin for concept coverage independent of import rating by our TBI experts.

Discussion:

SNOMED CT was significantly better able to represent mild TBI concepts than Medcin. This finding may inform data gathering, management and sharing, and data exchange strategies between the VA and DOD for active duty soldiers and veterans with mild TBI. Since mild TBI is an important condition in the civilian population as well, the current study results may be useful also for the general medical setting.

Background

The current study examines the adequacy of concept coverage for a case definition of mild traumatic brain injury (TBI) among two leading health care terminologies, SNOMED CT^® used by the Department of Veterans Affairs (VA) and Medcin^® used by the Department of Defense (DOD).

TBI has been labeled a “signature” injury of the Operation Enduring Freedom-Afghanistan (OEF) and Operation Iraqi Freedom (OIF) campaigns in which the United States (U.S.) military participate.[1–3] Since 2001, more than 1.7 million U.S. troops have been deployed under OEF /OIF.[4] During the nearly ten year history of the wars in Iraq and Afghanistan, an estimated 20% or more of U.S. service members have sustained TBI, making it one of the most common injuries among our military personnel.[4–7] In the current military campaigns, the frequent use of improvised explosive devices (IEDs) and other events leading to explosion-related injuries contribute to the relatively high incidence of TBI in deployed personnel.[6,8]

TBI severity is defined by symptoms at the time of injury.[9] Mild TBI, commonly referred to as “concussion”, is by far the most frequent type of TBI sustained during deployment to OEF/OIF, and by civilians.[6,9] Initial symptoms of the injury may be brief alteration of consciousness or loss of consciousness lasting only seconds to minutes. In most cases, the sequelae of mild TBI resolve in days to a few months. However, in some patients, post-concussive symptoms persist and may lead to long-term functional limitations.[4,6] Symptoms are more likely to be persistent among those people who experience multiple TBIs.[6]

Despite understanding that mild TBI is the most commonly sustained form of TBI, reports estimating prevalence of mild TBI, or prevalence of mild TBI as a subset of all TBI, vary in the literature. Reporting discrepancies are in part due to variation in the definitions used for mild TBI versus moderate and severe TBI, and in part due to variations in clinical terminology used to describe mild TBI.[10] Hoge et al [1] reported an overall incidence of mild TBI of nearly 15% in a study of 2,525 soldiers returning from Iraq. Jackson et al [7] reported estimates that almost 50% of soldiers injured in combat return with some form of TBI, and further noted that in the general population an estimated 80% of individuals who sustain TBI are classified as mild. In a 2003 report to Congress, the CDC Mild Traumatic Brain Injury Work Group noted that mild TBI as accounting for at least 75 percent of all traumatic brain injuries in the U.S.[11]

While our study intended to develop and assess terminology concept coverage for a diagnosis of mild TBI in OEF/OIF soldiers and veterans, most signs and symptoms of mild TBI are the same as those manifested by civilians who sustain mild TBI. The experience of the general medical community in diagnosing, managing, and reporting mild TBI has informed the current management of mild TBI in soldiers and veterans.[12] In turn, military and media attention on mild TBI as a prevalent injury among our OEF/OIF soldiers has caused renewed attention to the importance of the disorder in the U.S. general population, a concern well known to our nation’s emergency room personnel. 1.7 million traumatic brain injuries occur annually in the U.S., accounting for more than 1.3 million emergency room visits, the vast majority being for mild TBIs.[13–14] Common causes of TBI in the general population include falls, motor vehicle accidents, and “struck by” injuries such as may occur in sports-related activities. Documented sports-related concussions alone account for an estimated 300,000 cases of mild TBI annually.[12] In tandem with our military population, there is heightened awareness of sequelae of mild TBI among professional, college, and high school athletes who participate in sports such as football that carry high risk of head injury.[14–18]

Evaluations such as objective evidence of brain injury on brain scan or neurological examination and objective deficits on neuropsychological testing often are diagnostic for moderate and severe TBI, but frequently are not useful in the diagnosis of mild TBI.[9] Discussions regarding mild TBI, particularly in the U.S. Veteran population, have appeared in recent medical literature with varying criteria for detection and diagnosis.[1, 19–22] The civilian and military medical communities have worked together to gain consensus on a diagnostic definition of mild TBI [6,7,12,22–28], to close “knowledge gaps” regarding TBI [27], and to understand how best to assess the true prevalence and relative importance of various symptoms that comprise the sequelae of the condition.[4,6,29–33] A clinical practice guideline for mild TBI was recently released jointly by the VA and DOD.[9]

Mild TBI has been referred to as the “silent epidemic” precisely because the consequences that arise from it often are not overtly visible [11], and because some of the symptoms (e.g., fatigue or headache) are seen commonly due to other causes in both the general and military populations. At the time of injury, the absence of external injury or immediate disabling symptoms may allow an injured soldier to continue daily duties with only brief clinical evaluation or without seeking any medical attention at all. Over time, in soldiers and veterans in particular, symptoms of mild TBI may overlap with symptoms of Post Traumatic Stress Disorder, posing a significant challenge to accurately diagnosing mild TBI.[5,26,34] The importance of screening vigorously for TBI among soldiers and veterans only recently has received priority attention.[9,7,22,35]

The DOD electronic health record (EHR) system uses Medcin, a clinical terminology designed to support medical documentation entry. Introduced in 1978, Medcin now contains 280,000 concepts.[36] Following recommendations of the Consolidated Health Informatics Council and the National Committee on Vital and Health Statistics, the VA is committed to using SNOMED CT for problem lists and other functions of its EHR. Licensed for U.S.-wide use in 2003 and evaluated in 15 Medline indexed studies in 2006, SNOMED CT is maintained and distributed by the International Health Terminology Standards Development Organisation (IHTSDO).[36 IHTSDO] SNOMED CT includes approximately 370,000 concepts with greater than 1 million synonyms.

The current study seeks to understand which of these health care terminologies provides the most appropriate concept coverage for mild TBI, a diagnosis of increasing importance among our soldiers and veterans.

Methods

To evaluate how well mild TBI case definition terms could be represented by SNOMED CT and Medcin, the current study first used consensus methods[40] to develop a clinical case definition of mild TBI, and then applied methods previously described [41–43] to measure coverage for mild TBI concepts by the two terminologies. We performed additional analysis based upon whether the concepts were rated by a TBI expert panel as having High, Medium, or Low importance to the definition of mild TBI.

For the first step in our study, we convened a workgroup of 15 clinicians, researchers, and informaticists, including 6 TBI content experts from five institutions nationally. Using published literature endorsed by the convened workgroup, we developed a working case definition of mild TBI. The case definition we developed comprehensively includes criteria from the Brief Traumatic Brain Injury Screen (BTBIS) [22,23], the Neurobehavioral Symptom Inventory [23], the ICD-10 Criteria for Postconcussional Syndrome [23, the Mayo TBI Severity Classification System [28], and the CDC (Centers for Disease Control) TBI definitions and symptoms.[38,39] Our goal in engaging an expert panel to pool content from all of the aforementioned definitions was to approach an exhaustive list of terms relevant to mild TBI, and to make sure that the list included all terms important to the presentation of mild TBI in the military and veteran populations. Jones and Hunter have provided a basic justification for the use of consensus methods in health services research, including the “expert panel” (nominal group technique).[40] Expert panels and reviewers have been used extensively to develop and validate terminological systems.[41–46]

For the second step in our study, we engaged another panel of six TBI clinical experts to review our mild TBI case definition. The specialty areas of these experts include clinical neuropsychology/cognition, brain injury/polytrauma, neurosciences/head trauma, physical medicine/war trauma, and trauma surgery. Specifically, the expert panel rated 60 clinical terms from the case definition based on how much they agreed or disagreed that the term should be included in a working case definition of mild TBI, using a 5 point Likert scale (5 = Strongly Agree, 4 = Agree, 3 = Neither Agree nor Disagree, 2 = Disagree, 1 = Strongly Disagree).

Additionally, a team of 3 clinicians (2 physicians, 1 nurse) searched for variations on mild TBI terms by performing detailed human review of 10 VA Compensation and Pension (C&P) examination records of OIF/OEF era veterans who had been evaluated for possible mild TBI. These records were content rich with mild TBI terms expressed specifically in the records of veterans and soldiers. The 3 clinical experts identified 8 commonly occurring terms that represented synonyms felt by the clinicians to be important to the definition of mild TBI in soldiers and veterans, and these terms were added to our case definition. For example, our original clinical case definition included the term “Blast Injury”. We added the term “Explosion Injury” to the final working definition to account for the likely appearance of the term in a recorded clinical evaluation of a soldier or veteran. By including symptoms and terms specific to mild TBI among veterans of OIF/OEF military “theatre”, our case definition both encompasses and extends the criteria most commonly used to diagnose mild TBI in the general population.

In all, the authors identified 68 terms that represent clinical symptoms or findings of mild TBI (Table 1). Of these 68 terms, 60 were the clinical terms from our working case definition and 8 were the terms that appeared commonly in review of actual veteran clinical records.

Table 1.

Clinical Terms for Mild TBI Case Definition

Fragment Injury	Shrapnel Injury	Bullet Injury
Vehicular Injury	Airplane Injury	Helicopter Injury
Humvee Injury	Fall Injury	Blast Injury
Explosion Injury	Blast Injury due to Improvised	Blast Injury due to Rocket
	Explosive Device	Propelled Grenade
Blast Injury due to Grenade	Blast Injury due to Land Mine	Dazed
Confused	Seeing stars	Blacked out
Amnesia for event	Loss of Consciousness	Knocked out
Losing consciousness for less than a minute	Losing consciousness for 1 to 20 minutes	Losing consciousness for longer than 20 minutes
Symptoms of concussion after injury	Headache after injury	Dizziness after injury
Irritability after injury	Head Injury	Head Trauma
Skull fracture	Dizziness	Balance problems, Imbalance
Ringing in the ears, Tinnitus	Poor coordination	Clumsy
Headaches	Nausea	Vision problems, Trouble seeing
Blurring, Blurry Vision	Sensitivity to light	Hearing difficulty
Sensitivity to noise	Numbness in parts of body	Tingling in parts of body
Change in taste	Change in smell	Increased appetite
Decreased appetite	Poor concentration, Cannot pay attention, Easily distracted	Forgetfulness, Cannot remember things
Difficulty making decisions	Slowed thinking	Difficulty getting organized
Cannot finish things	Fatigue, Loss of energy, Getting tired easily	Difficulty falling asleep
Difficulty staying asleep	Feeling anxious	Feeling tense
Feeling depressed	Feeling sad	Easily annoyed, Irritability
Poor frustration tolerance	Feeling easily overwhelmed by things	Reduced tolerance to alcohol
Problems with emotional control, Emotional dyscontrol, Disinhibition	Not remembering the injury, Cannot recall injury, Amnesia for injury

For the next step of the study, and applying a method employed and described in prior investigations [41–44], we used both manual and automated processes to map the 68 terms from our final mild TBI case definition separately to SNOMED CT and to Medcin. First, we used an automated natural language processor, the Multi-threaded Clinical Vocabulary Server (MCVS) [43], to map the 68 terms to each terminology. The MCVS is a set of tools that facilitates health vocabulary indexing, and provides automatic concept mappings from free-text phrases to clinical terminologies.[43,44] The results of the automated mappings were given to two independent physician reviewers who then used the MCVS terminology browser manually to search SNOMED CT and Medcin prior to declaring any concept a “non-match” in either terminology. A third independent physician reviewer adjudicated a final decision whenever the first two reviewers disagreed.

A term was considered valid for coding in a medical terminology if all reviewers agreed that it was a medically reasonable and unambiguous term. For each term being evaluated, the reviewers judged SNOMED CT or Medcin as “positive” when the terminology contained concepts represented by the study terms, and “negative” when the terminology did not contain concepts represented by the study terms. Reviewers judged each terminology’s coverage as being: “True Positive” (TP) if the term was a reasonable medical entity and the terminology contained a concept to represent the term; “False Positive” if the term was not felt to be a valid medical entity but the terminology contained a concept that matched the term; “True Negative” if the term was not a medical entity and the terminology did not map to any term; and “False Negative” when the term was a reasonable medical entity but the terminology failed to have a concept to represent the term appropriately. The schema for reviewer determinations is depicted in Figure 1.

Figure 1.

Reviewer Determinations schema

Finally, we compared concept coverage between SNOMED and Medcin. The SNOMED and Medcin classifications provided matched data. To account for matching and compare SNOMED and Medcin performance, McNemar’s chi square was used to test SNOMED with Medcin sensitivity for the overall clinical case definition of mild TBI. We performed additional analysis based upon the importance rating given by our expert panel. Since no term was rated below a weighted score of 3, we divided the range of ratings into 3 scoring classifications: Low (score <= 3.67), Medium (score > 3.67 and <= 4.33), and High (score > 4.33). Sensitivity and positive predictive value of SNOMED and Medcin were calculated for each of the 3 categories of import.

Results

Concept coverage and classification of results are shown in Figure 1. We found 67 of the 68 terms (Table 1) in our mild TBI case definition to be valid for attempting to code in a medical terminology. The term “seeing stars” was not coded because our independent manual reviewers determined that the term was too ambiguous (e.g., seeing stars after sustaining head trauma, versus literally seeing stars upon sky gazing).

SNOMED CT’s sensitivity (recall) as a candidate terminology for our case definition of mild TBI was 90%, and positive predictive value (PPV) (precision) was 100%. By contrast, Medcin’s sensitivity was 49%, with a PPV of 100%. The superior performance of SNOMED CT over Medcin was statistically significant with a p-value of <0.001 (using McNemar’s chi square). Table 2 contains detailed results of the performance of each terminology.

Table 2.

Analysis of SNOMED CT vs Medcin Content Coverage for entire set of Mild TBI Clinical Concepts (n=67). PPV: positive predictive value. NPV: negative predictive value.

	TP	TN	FP	FN	Sensitivity (recall)	Specificity	PPV (precision)	NPV
SNOMED CT	60	1	0	7	90%	100%	100%	13%
Medcin	33	1	0	34	49%	100%	100%	3%

Of the 67 terms considered valid, our TBI expert panel had rated 59 terms for their relative importance to the clinical definition of mild TBI (as described in the Methods section, the final 8 terms were added to the definition following the expert panel completing its ratings). We performed sub-group analysis based on the three categories of import (Table 3). SNOMED CT outperformed Medcin in every category of import.

Table 3.

Subgroup analysis of Content Coverage based on Expert Evaluation of Clinical Import of 59 TBI Terms

	Total (n = 67)		High Import (n = 18)		Medium Import (n = 30)		Low Import (n = 11)
	Sensitivity	Specificity	Sensitivity	Specificity	Sensitivity	Specificity	Sensitivity	Specificity
SNOMED	90%	100%	83%	100%	93%	na	100%	na
Medcin	49%	100%	35%	100%	57%	na	55%	na

Discussion

DOD and VA operate two of the nation’s largest health care systems, providing health care to service members and veterans often in a shared capacity. Currently, DOD and VA are engaged in collaborative initiatives, including development of the Virtual Lifetime Electronic Health Record (VLER), that are dependent on their ability to fully share electronic health information.[47] The VA and DOD each have well-established but quite separate patient electronic health records (EHRs). The ability to retrieve and apply structured, codified data from their EHRs can promote data sharing activities, and enhance important secondary uses such as health services research, quality improvement activities and surveillance, and population health. In very recent years, in an effort to better characterize, understand, and treat the condition, both agencies have increased specific screening for mild TBI.[5,7,9,11,23,34,48] Demonstrating how well mild TBI can be represented using current terminology systems is a first step toward ensuring that accurate data may be extracted for this purpose from the EHR both for individual clinical and population health purposes.

SNOMED CT, the terminology supporting components of the VA’s EHR system, was significantly better able to represent TBI concepts than Medcin, the terminology underpinning of AHLTA, the DOD’s EHR system. SNOMED CT outperformed Medcin in overall sensitivity for concepts in a comprehensive case definition of mild TBI (90% vs 49%, p<0.001), and in sensitivity for concept coverage independent of import rating by our TBI experts. In a prior study, SNOMED CT also outperformed Medcin as a reference terminology and as an interface terminology for representation of a VA general medical examination template.[41]

In the current study, SNOMED CT has a sensitivity or recall of 90% which is similar to the 92.3% sensitivity found by Elkin et al for clinical problems.[43] Many of the concepts necessary for TBI surveillance fall under the Disorders and Findings Hierarchy, which was the definition of clinical problems used in Elkin’s earlier study.

The difference in performance of the terminologies in this study is due, in part, to size. SNOMED CT (370,000 concepts) is a larger terminology than Medcin (280,000 concepts). Therefore the content coverage of SNOMED CT when used as a pre-coordinated terminology would exceed that of Medcin. In addition, SNOMED CT was able to match many terms with compositional expressions (22 terms), while Medcin is not compositional. We allowed Boolean combinations of Medcin concepts and using this technique we matched only 2 terms linked by Boolean operators (i.e. And, Or, Not). SNOMED contains more reusable atomic concepts than Medcin, as well as linking semantics that can be used to create post-coordinated expressions. Medcin by contrast contains many pre-coordinated terms, but lacks formal mechanisms for post-coordination. The performance difference between the two terminologies may reflect the flexibility and power of post-coordination for content representation. Medcin has been described by the company that created and maintains it as “a robust database of medical concepts with methods for presenting and documenting relevant symptoms, history, physical findings, tests, diagnoses and therapy for virtually any patient presentation.”[36] Until we performed our study, we could not know whether Medcin’s population of pre-coordinated terms would be sufficient to represent the terms important to a case definition of mild TBI.

Lastly, our reviewers noted that Medcin contained some terms that were more granular than would be required for our case definition, but Medcin more often than SNOMED CT failed to have appropriate organizing concepts to cover the term. Example clinical terms and the differences in the ability of SNOMED CT versus Medcin to cover the term are shown in Table 4.

Table 4.

Terminology Mapping Example

Clinical Term	SNOMED CT Concept	MEDCIN Concept
Bullet Injury	Bullet, device (physical object) [86122002] [K] Traumatic injury (disorder) [417746004][K]	‘Bullet’ – not matched. TRAUMA [33713][K]
Difficulty falling asleep	Initial insomnia (disorder) [59050008] [K]	Initial insomnia (disorder) [59050008] [K]
Difficulty staying asleep	Middle insomnia (disorder) [67233009] [K]	No match

Andrews et al [49] pointed out challenges in achieving consistency in coding among experts specifically when utilizing SNOMED CT for clinical research concepts, but urged that “Determining the extent and consistency to which concepts embedded in clinical research can be represented by SNOMED CT will help illuminate unmet needs and inherent complexities that may impede semantic interoperability and effective clinical research data management.”[49] Our study results are consistent with Andrews’ ultimate opinion that “SNOMED CT is a viable and appropriate data standard for clinical research”. In our study, SNOMED CT appears a viable and appropriate data standard for surveillance of mild TBI in particular.

Study Limitations

The main limitation of this manuscript is that it is a study of the content coverage of mild TBI-related surveillance terms. As next steps we would need to validate the model against real data to determine the level of accuracy that our model affords us with respect to biosurveillance for mild TBI. This validation stage may produce other important concepts that require coverage in order to have built a sensitive and specific generalizable model

Another limitation of our study concerns the size of our panel of six TBI clinical experts who reviewed our case definition to rate how much they agreed or disagreed that each term should be included in a case definition of mild TBI. The panel size was not large enough for the panel’s ratings to have statistical power (which would only serve to strengthen study results). The contribution of our expert panel’s ratings nonetheless is supported by justification offered by Jones and Hunter for use of consensus methods in health services research, including the expert panel.[40], and by prior studies that have used expert panels and reviewers to develop and validate terminological systems.[41–46].

A third study limitation was that the human review of medical records that was performed to identify synonyms particular to the presentation of mild TBI in the records of soldiers and veterans, was performed on only a small number of records (due to the labor intensity of the task), and the 8 terms identified by record review were not included in the expert panel ranking of the import of our other terms to the clinical definition of mild TBI. Therefore, although our case definition was quite comprehensive with respect to validated sources from the clinical literature, mild TBI terms important to veterans’ clinical histories may have been missed, and we cannot know how either terminology would have covered terms that might have been identified had review of a larger number of medical records been conducted.

Conclusion

Overall, SNOMED CT outperformed Medcin in concept coverage for a comprehensive case definition of mild Traumatic Brain Injury. Medcin has many useful precoordinated terms. We believe that Medcin would benefit by having more atomic concepts and by providing a mechanism to support post-coordination. Our results help inform the VA and DOD regarding the utility of their respective tools for mapping terms relevant to a clinical diagnosis that is of great concern to both entities, and more generally may help inform the two agencies as they address data sharing challenges in the context of their current activities to share electronic health record data.

While the current study focuses on a case definition of mild TBI that would be certain to contain terms relevant to the condition in U.S. soldiers and veterans, mild TBI also is of concern in the general civilian population, and we believe that our case definition contains the terms relevant to the disorder in the civilian population. Examples include many participants in contact sports and a high proportion of unbelted motor vehicle accidents. TBI is a serious disorder, and mild cases are routinely missed.[10] An accurate case definition has the potential to improve biosurveillance among both the military and civilian populations. Improved detection, early detection, and treatment offer the potential to lessen the impact of mild TBI sequelae on vulnerable populations.

Future research should validate this model against a population at risk for mild TBI.

Figure 2.

Classification scheme and results