Validation of Administrative Measures of Social and Behavioral Risk in Veterans Affairs Medical Records

INTRODUCTION

A growing body of research indicates that social and behavioral risk factors, such as social isolation and marginal housing, influence health outcomes.¹ Unfortunately, obtaining such risk factors from administrative records remains a challenge.² We assessed the fidelity of administrative coding of six social and behavioral risk factors (lives alone, marginal housing, alcohol and drug abuse, use of substance abuse services, and violence) through chart validation of hospitalized patients within the Veterans Health Administration (VA).

METHODS

We identified older Veterans (≥ 65) who were hospitalized in the VA from 2009 to 2012 with either: congestive heart failure, acute myocardial infarction, or pneumonia. Using national VA data, we created six measures of social and behavioral risk using International Classification of Disease, Ninth Revision (ICD-9) diagnosis codes, VA clinic description codes, and laboratory data (Table 1). After determining the prevalence of each risk factor within our cohort, we randomly identified 50 patient charts that met identification criteria (i.e., contained applicable ICD-9, clinical description codes, or laboratory data) and 50 that did not. Two reviewers independently reviewed all clinical notes in the year prior to discharge from an index admission for evidence of each respective risk factor. A kappa statistic was performed on 25% of cases for each measure. Positive (PPV) and negative predictive values (NPV) for each risk factor were calculated and recorded.

Table 1.

Social and Behavioral Risk Factors Administrative Diagnostic Codes and Definitions used for Chart Review

Risk factors	Data source(s) used to identify patients in electronic health records	Definition used for chart review
Lives alone	ICD-9#	Evidence that a patient does not live with either a family member or a roommate on a consistent basis
Marginal housing	ICD-9##	An individual who lacks a fixed, regular, and adequate nighttime residence, or who lives in temporary or transitional housing; or who lives in a place not meant for human habitation**
Alcohol abuse	Clinical and Laboratory Data###	Evidence of continuous or impaired control over alcohol intake and use of alcohol despite its adverse consequences with evidence of alcohol use despite efforts to quite, mentions of the negative impacts of alcohol use on friends, family, interpersonal relationships, housing and employment opportunities***
Drug abuse	ICD-9####	Compulsive drug seeking and use despite harmful consequences with evidence of continued use despite efforts to quite, mentions of the negative impact of drug use on family, friends, interpersonal relationships, housing or employment opportunities, or (+) urine drug screen for methamphetamine, phencyclidine, cocaine or opiate or benzodiazepine without evidence of prescription****
Use of substance Abuse services	Clinic description codes	Evidence of patient utilizing substance abuse services in the year prior to discharge from the index hospitalization.
Violence	ICD-9###### and clinic description codes^	Evidence of physical action causing bodily harm to self or others as reported by the patient or care provider

**Adapted from the Housing and Urban Development (HUD) definition for homelessness;

***Adapted from the National Council on Alcoholism and Drug Dependence and the American Society for Addiction Medicine’s definitions of alcoholism;

****Adapted from the National Institute of Drug Abuse

ICD-9: Administrative Diagnostic Codes

^#ICD-9 Codes: v60.3, v60.4.

^##ICD-9 Codes: v60.0, v60.1.

^###Logical Observation Identifier Names and Codes (LOINC):

^####ICD-9 Codes: 965.00, 965.01, 965.02, 965.09, E850.0, E850.1, E850.2, E935.0, E935.1, E935.2, 304.xx, 305.x, 965.0x, 967.0, 969.4, 969.6, 969.7, E850.0, E850.1, E850.2, E854.1, E854.2, E854.3.

^#####VA Clinic Description Codes: 513, 514, 519, 545, 547, 548, 560, 593, 595, 596, 597, 598, 599.

^######ICD-9 Codes: E985.x, E986, E960.x, E962.0, E962.1, E962.2, E963, E965.x, E966, E967.x, E968.x, E970, E971, E972, E973, E974, E975, E976, E977

^VA Clinic Description Codes: 130, 131

RESULTS

Among 600 reviewed charts, “drug abuse” (4.4%) and “use of substance abuse services” (3.2%) had the highest prevalence, followed by “living alone” (2.9%) and “marginal housing” (2.9%). Of the six assessed risk factors, patients with administrative measures for “violence” (PPV: 0.94), “alcohol” (PPV: 0.88), and “use of substance abuse services” (PPV: 0.88) had the highest PPV. Comparatively, administrative measures for “living alone” (PPV: 0.60) and “marginal housing” (PPV: 0.78) were limited in their predictive abilities. All risk factors except for “lives alone” (0.72) had NPV > 0.85 (Table 2). The kappa-statistic between pairs of reviewers ranged from 0.66 to 1.0 on a per-measure basis.

Table 2.

Chart Validation Characteristics for Social and Behavioral Risk Factors

Social risk factor	No. (%) of patients identified in EHR	Positive predictive value	Negative predictive value
Lives alone	3472 (2.9%)	0.60	0.72
Marginal housing	3509 (2.9%)	0.78	0.94
Use of substance abuse services	3925 (3.2%)	0.88	1.0
Alcohol abuse	2874 (2.4%)	0.88	0.86
Drug abuse	5325 (4.4)	0.82	1.0
Violence	136 (0.11%)	0.94	0.96

EHR, Electronic Health Record

DISCUSSION

In this chart validation of VA administrative diagnostic codes for social and behavioral risk factors known to effect health outcomes, we found excellent PPV for codes specific for “violence,” “alcohol abuse,” and “use of substance abuse services”—but limited when assessing for “living alone” and “marginal housing.” Given the growing understanding of how social risk factors effect health outcomes,³ risk-adjustment,⁴ and quality of care, the extent to which such factors can be extracted from administrative data is becoming an increasingly important issue.

A recent study found 22 social characteristics not included in either VA or Medicare’s current risk-adjustment methods (e.g., social isolation and smoking status) were predictive of hospital readmissions—suggesting that extraction and inclusion of such factors may be important in risk-adjustment—and by extension, subsequent readmission penalties.⁴ Furthermore, despite the recognized difficulty in obtaining social risk factors from the medical record, the National Quality Forum recently issued a report stating that social risk factors should be included in risk-adjustment models as a means to improve risk adjustment.² Unfortunately, extracting social risk factors from administrative data with adequate validity and efficiency remains a complex endeavor.⁵ Our study may be helpful as we describe a unique extraction approach that is not limited to ICD coding alone, but rather uses a variety of administrative data to assess for the presence of social risk factors. Despite the low prevalence of each risk factor, the use of multiple data sources likely contributed to the favorable performance of the administrative measures.

Our study has limitations. First, because a natural ambiguity of some social risk factors (e.g., marginal housing, and violence) exists, this may lead to under-coding, thus missing patients who may be affected by these risk factors. Second, given the specificity of our cohort to older hospitalized patients, generalizability to other populations may be limited. Finally, we were restricted to the use of ICD-9 diagnostic codes given the time-span of our cohort.

In total, this study indicates that VA administrative codes for social and behavioral risk factors such as violence, alcohol and drug abuse, and use of substance abuse services are accurate indicators of the presence of these risk factors. Further work in assessing other social and behavioral risk factors that may affect health outcomes is needed and will assist in optimizing care for at-risk individuals.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.