Using Electronic Medical Records to Determine the Diagnosis of Clinical Depression

Abstract

Objective

To investigate the validity of using electronic medical records (EMR) database in a large health organization for identifying patients with clinical depression.

Method

The Massachusetts General Hospital EMR system was used to generate a sample of primary care patients seen in the primary care clinic in 2007. Using this sample, the validity of using certain fields in the EMR database (i.e., billing diagnosis, problem list, and medication list) to identify patients with clinical depression was compared to primary care physician (PCP) assessment by a written questionnaire. Based on this standard, the sensitivity, specificity, positive predictive value, negative predictive value, and the areas under receiver operating characteristic curve (AUC) of three specific EMR fields – individually and in combination - were calculated to identify which EMR field best predicted PCP classification.

Results

The EMR fields “billing diagnosis,” “problem list,” and antidepressant in “medication list,” were all able to identify patients’ diagnosis of depression by their PCPs reasonably well. Having one or more “billing diagnosis” of depression had the highest sensitivity and highest AUC (77% sensitivity, 76% specificity, AUC 0.77) among any of the fields used alone.

Conclusion

The AUC for “billing diagnosis” of depression performed the best of the three single fields tested, with an AUC of 0.77, corresponding to a test with moderate accuracy. This analysis demonstrates that specific EMR fields can be used as a proxy for PCP assessment of depression for this EMR system. Limitations to our analysis include the physician response rate to our survey as well as the quality of the data, which is collected primarily for administrative and clinical purposes. When using administrative and clinical data in mental health studies, researchers must first assess the accuracy of choosing specific fields within their EMR system in order to determine the level of accuracy for them to be used as proxies for clinical diagnoses.

Introduction

With the advent of the electronic medical record (EMR), academic medical centers are taking advantage of computer technology to facilitate clinical research [1]. Through the EMR, large sets of patient data are now available for researchers to identify cohorts of patients based on demographic or illness characteristics to address research questions. Mental health services researchers, for example, have looked to the EMR to answer questions regarding diagnostic and prescribing patterns in depressive disorders [2]. Although the type of data included within the EMR varies by institution, common administrative and clinical data fields include billing diagnoses, patient symptoms, past medical histories, lifestyle choices, physical examinations, medical diagnoses, tests and procedure results, treatments, medications, referrals to specialists, and inpatient discharge information [3].

However, these medical records are primarily designed for administrative and clinical purposes; because data from EMR and not collected specifically for research, thus their use for research could be problematic. Many studies have been based on EMR data with the assumption that EMR data accurately reflect the provider’s assessment of the patient [2, 3, 4], but the accuracy of that assumption has not yet been fully evaluated [5]. Less is known about how accurately medical records can be used as a substitute for validated scales administered in controlled research settings, or even how well medical records reflect clinical presentation at the time of the clinician’s EMR entry [5, 6]. Because the EMR is an administrative and clinical tool primarily used for documentation, data may be affected by individual physician behavior patterns and financial necessity [7]. In a study exploring the validity of using the EMR to identify cases of new-onset depression, discrepancies found between using criteria in the EMR versus manual chart review were likely due to factors such as clinician uncertainty about diagnosis, stigma related to the inclusion of a mental health diagnosis in the EMR, or a focus on other co-morbid disorders requiring more immediate attention [2]. Furthermore, relying on billing codes to identify patients with depression may also have limitations: Primary care physicians (PCPs) may not be able to bill for depression because such codes may not be included on their billing forms or because such codes may not be reimbursable by insurance, and some patients may prefer that their mental health diagnoses go undocumented [7].

The purpose of this study was to evaluate how well EMR data can be used to identify patients who likely have clinical depression. As our standard of diagnosis, we used PCP assessment of patients through a written survey. Data from three different EMR fields were evaluated separately and in combination: administrative data from “billing diagnosis” codes, as well as clinical data from patients’ “problem lists” and antidepressant medications listed on patients’ “medication lists”. To our knowledge, this is the first attempt to validate the use of specific administrative and clinical EMR fields as proxies of clinical depression diagnosis as compared to PCP assessment.

Methods

To determine whether PCPs would diagnose specific patients with depression, a written survey of a convenience sample of PCPs at the Massachusetts General Hospital was conducted in January 2009. First, a database of all patients seen by MGH primary care in 2007 was created using the Research Patient Data Registry (RPDR). The RPDR is a research database synthesizing data from a variety of administrative and clinical databases; through the RPDR, investigators can generate a patient database of interest using specific criteria [8, 9]. A convenience sample of 82 PCPs with diverse populations was selected from all PCPs whose practices included at least 5 Asian, 10 Latino, and 10 African American patients with depression (defined below).

For each PCP, fifteen patients were randomly selected from their practices to evaluate in a written survey. Five patients were selected from each of the following three groups: 1) patients with a depression diagnosis and receiving treatment: these patients’ records included evidence of a depression diagnosis (their records contained depression-related billing codes, included depression in the problem list, or were prescribed antidepressant medications), and they were prescribed antidepressant medications; 2) patients with a depression diagnosis but not receiving treatment: these patients’ records included evidence of a depression diagnosis as above, but they were not prescribed any antidepressant medications; and 3) patients without depression: these patients’ records contained no evidence of a depression diagnosis or treatment.

Study materials were mailed directly to PCPs offices via interoffice mail and included the following: a letter describing the study and procedures; a questionnaire on 15 patients described above; and a $10 gift card to thank them for their time. On the survey questionnaire, PCPs were asked to indicate whether they believed that each of the selected patients was clinically depressed in the year 2007. PCPs were allowed to consult the EMR when answering these questions. Patients were excluded from analyses if the PCP responded that the patient was either not his/hers or felt insufficient information was available to provide a classification of depression.

PCP responses were compared with the patient database to determine which elements of the EMR best predicted PCP classification of depression. Of the standard administrative and clinical EMR fields available, three were selected a priori as the most promising indicators of the presence of clinical depression: 1) depression-related billing codes from administrative databases, including International Classification of Diseases, 9^th Revision (ICD-9) codes and corresponding hospital-based billing codes; and from patient clinical databases, 2) diagnoses of depressive disorders in the patient problem list, and 3) antidepressant medications associated with patient medication lists in the EMR. For field #3, patients were not considered depressed if they were receiving antidepressant medications but also had a diagnosis related to pain or anxiety (see Table I). All study procedures were approved by the MGH Institutional Review Board.

Table I.

Electronic medical records fields used to establish clinical depression

Billing Diagnoses	290.13  Presenile dementia with depressive features 290.21  Senile dementia with depressive features 290.43     Arteriosclerotic dementia with depressive features 296.2    Major depressive disorder, single episode1 296.3    Major depressive disorder, recurrent episode1 296.82     Atypical depressive disorder 296.9    Unspecified affective psychosis 296.99  Other specified affective psychoses 298    Depressive type psychosis 300.4    Neurotic depression 301.1    Affective personality disorder, unspecified 305.8    Antidepressant type abuse, unspecified use 305.81     Antidepressant type abuse, continuous use 309    Adjustment reaction with brief depressive reaction 309.1    Adjustment reaction with prolonged depressive reaction 311       Depressive disorder, not otherwise specified 969    Poisoning by antidepressants E939.0    Antidepressants causing adverse effects in therapeutic use V79.0   Screening for depression2 LPA101   Depression-LMR 1012 LPA1593 Depression-LMR 15932 LPA751   Dysthymia-LMR 7512 YJSD3     Reactive depression-Oncall2 YJSJ7      Grief reaction-Oncall2 YHAJ9    Dysthymia-Oncall2 YJSN1  Depression-Oncall2 E854.0  Accidental poisoning by antidepressants2 E939.0  Antidepressants causing adverse effects in therapeutic use2 426   Depressive neuroses2
Patient Problem List	Any mention of “Depression,” “Depressed,” “Major Depression” on patient problem lists
Medication List3	The following antidepressants were included:
	Amitriptyline Bupropion Citalopram Clomipramine Desipramine Doxepin Duloxetine Escitalopram	Fluoxetine Fluvoxamine Imipramine Maprotiline Mirtazapine Nefazodone Nortriptyline Paroxetine	Phenelzine Protriptyline Sertraline Selegiline patch Tranylcypromine Trimipramine Venlafaxine
	If patients were taking antidepressants but had the following diagnoses, they were not considered depressed:  300.00 Anxiety disorder not otherwise specified  300.01 Panic disorder without agoraphobia  300.02 Generalized anxiety disorder  300.09 Other anxiety states  309.81 Post-traumatic stress disorder  388 Any pain syndrome

¹Includes depression subtypes

²Hospital-specific billing codes

³Including brand and generic names; generic names included here for simplicity

Data Analysis

The performance of each EMR field was assessed against the standard of the PCPs’ assessment in the survey. We conducted analyses on the sensitivity, specificity and area under the receiver operating characteristic curve, and provide 95% confidence intervals for each of three fields. In our analyses, sensitivity is defined is defined as the proportion of patients known to have depression based on physician review who also test positive for the criteria of interest, and specificity is the proportion of patients known to not have depression based on physician review who test negative for the criteria of interest [10]. A receiver operating characteristic (ROC) curve was used to investigate the statistical model’s ability to separate positive from negative cases. ROC curves are generally used to evaluate classification models for diagnosis in biomedical informatics research [11]. ROC curves were obtained by calculating the sensitivity and specificity of the test at every possible cut-off point, and plotting sensitivity against 1-specificity [12]. The area under the curve (AUC) of the ROC serves as a single measure, independent of prevalence, that summarizes the discriminative ability of a test across the full range of cut-offs [13]. The AUC of the ROC is a reflection of how good the test is at distinguishing between patients with disease and those without disease. The greater the AUC, the better the test; in general the closer the AUC to 1, the better the overall diagnostic performance of the test, and the closer it is to 0.5, the poorer the test. All statistical analyses were performed using Stata [14].

Results

Eighty-two PCPs at MGH were mailed surveys. Fifty of the 82 PCPs surveyed returned questionnaires, yielding a response rate of 61% (Figure 1). Of the 750 patients whom the responding PCPs were asked to classify, 323 were excluded: 219 (29%) because the PCP indicated the patient was “not my patient” and 104 (14%) because the PCP reported that there was “not enough information.” Of the 427 patients who were retained for analysis, half (n=212, 50%) were classified as depressed by the PCP (see Table II).

Figure 1.

Classification of Patients by their Primary Care Physician

Table II.

Patients with depression-related billing diagnostic codes

Billing Diagnostic Code	Patients with a Depression-related Billing Diagnostic Code in the Electronic Medical Record (EMR)
	All patients in survey sample n=1230	Patients included in analysis n=427	Of patients included in the analysis, Primary Care Provider (PCP) classified as:
			Depressed n=212	Not Depressed n=215
	n (%)1	n (%)2	n (%)3	n (%)4
290.13 Presenile dementia with depressive features	1 (<1)	0 (0)	-	-
290.21 Senile dementia with depressive features	4 (<1)	1 (<1)	0 (0)	1 (100)
296.2 Major depressive disorder, single episode	22 (2)	12 (3)	10 (83)	2 (17)
296.3 Major depressive disorder, recurrent episode	113 (9)	40 (9)	34 (85)	6 (15)
296.90 Unspecified affective psychosis	53 (4)	21 (5)	16 (76)	5 (24)
296.99 Other specified affective psychoses	1 (<1)	1 (<1)	1 (100)	0 (0)
300.4 Neurotic depression	57 (5)	28 (7)	22 (79)	6 (21)
305.81 Antidepressant type abuse, continuous use	1 (<1)	0 (0)	-	-
309.0 Adjustment reaction with brief depressive reaction	18 (1)	6 (1)	4 (67)	2 (33)
309.1 Adjustment reaction with prolonged depressive reaction	48 (4)	17 (4)	14 (82)	3 (18)
311 Depressive disorder, not otherwise specified	392 (32)	139 (33)	112 (81)	27 (19)
Hospital-specific codes	154 (13)	70 (16)	52 (74)	18 (26)
TOTAL number of patients with any diagnosis	567 (46)	216 (51)	164 (76)	52 (24)

¹This percentage is defined as the number of patients with a specific diagnosis divided by the number of patients in the survey sample (n=1230).

²This percentage is defined as the number of patients with a specific diagnosis divided by the number of patients included in the analysis (n=427).

³This percentage is defined as the number of patients PCP classified as depressed divided by the number of patients included in the analysis that carried the specific diagnosis listed in the row of interest.

⁴This percentage is defined as the number of patients PCPs classified as NOT depressed divided by the number of patients included in the analysis that carried the specific diagnosis listed in the row of interest.

The use of information from a single EMR field, including depression in “billing diagnosis,” depression in “problem list,” and antidepressant in “medication list,” were all able to identify patients’ diagnosis of depression by their PCPs with moderate accuracy as illustrated by the AUC (Table III). The “billing diagnosis” of depression performed the best of the individual EMR fields, with the highest sensitivity (77%) and AUC (0.77) as compared to the other two fields. The difference in AUC was statistically significant when comparing this field to the “problem list” field (p<0.001); there was a trend toward significance when comparing the AUC for “billing diagnosis” to the AUC for “medication list” (p=0.056). Overall, 864 of the 1230 patients included in our sample (70%) were classified with clinical depression using “billing diagnosis”; the most common ICD-9 code was for Depressive disorder not otherwise specified (ICD-9 code 311), followed by the corresponding hospital-specific internal code. Of the patients given a diagnostic code of 311, 81% (n=112) were classified as depressed by the PCP and 19% (n=27) were classified as not depressed. Table II presents the numbers and percentages of patients with each diagnostic code in the EMR. Percentages of specific diagnoses for all patients in the survey sample (n= 1230) as compared to those patients included in the analysis (n=427) were similar, as were percentages of specific diagnoses in the excluded group.

Table III.

Performance of electronic medical records (EMR) fields in predicting primary care physician diagnosis of clinical depression

Fields	Sensitivity (%)	Specificity (%)	Area under the Curve (AUC) (95% CI)	Positive Predictive Value (%)	Negative Predictive Value (%)
One EMR field
Billing diagnosis of depression	77	76	0.77 (0.73–0.81)1	76	77
Depression on patient problem list	49	78	0.63 (0.59–0.68)	68	61
Antidepressants on medication list	56	88	0.72 (0.68–0.76)	83	67
Combinations of EMR fields
Billing diagnosis OR antidepressant on medication list	85	73	0.79 (0.76–0.83)2	76	84
Billing diagnosis OR problem list	93	60	0.76 (0.73–0.80)	69	90
Billing diagnosis OR problem list OR antidepressant on medication list	93	60	0.76 (0.73–0.80)	69	90
Billing diagnosis AND antidepressant on medication list	48	91	0.69 (0.65–0.73)3	83	64
Billing diagnosis AND problem list	33	94	0.64 (0.60–0.67)3	85	59
Billing diagnosis AND problem list AND antidepressant on medication list	25	96	0.61 (0.57–0.64)3	86	57

¹The AUC for this field is statistically significant as compared to the AUC for Depression on patient problem list (p<0.001); there is a trend towards significance when compared to the AUC for Antidepressants on medication list (p=0.056).

²The AUC for this combination is statistically significant as compared to the AUC for Billing diagnosis of depression (p<0.01).

³The AUC for Billing diagnosis of depression is statistically significant as compared to the AUC of these combinations (p<0.001)

Given that the “billing diagnosis” field appeared to perform better than either the “problem list” or “medication list” fields, we conducted an additional analysis using the “billing diagnosis” field in various combinations with the two other fields to explore the sensitivity, specificity, ROC curves, positive predictive values (PPV) and negative predictive values (NPV) for these combinations (Table III). Results from this analysis showed that broadening the depression definition by the presence of depression on 1) “billing diagnosis” or “problem list,” 2) “billing diagnosis” or “medication list,” and 3) “billing diagnosis” or “problem list” or “medication list” resulted in higher sensitivity but correspondingly lower specificity for each combination. Only the AUC for the combination of “billing diagnosis” or “medication list,” was statistically significant (p<0.01) as compared to the AUC of “billing diagnosis” alone. This combination achieved higher sensitivity (85% versus 77%) and a slightly higher AUC (0.79 versus 0.77), although with a corresponding decrease in specificity (73% versus 76%). In contrast, tightening the definition by the presence of depression on 1) “billing diagnosis” and “problem list,” 2) “billing diagnosis” and “medication list,” and 3) “billing diagnosis” and “problem list” and “medication list” resulted in higher specificity but correspondingly lower sensitivity and lower AUC for each combination; the AUC for “billing diagnosis” was statistically significantly better than the AUC for each of these combinations.

Discussion

Our analysis demonstrates that certain administrative and clinical EMR fields may be used to predict PCP assessment of clinical depression with moderate accuracy. Having one or more “billing diagnosis” of depression performed the best of the three single fields tested, with an AUC of 0.77; having one or more “billing diagnosis” of depression or having an antidepressant on the “medication list” improved the AUC, but only minimally, to 0.79. These AUCs correspond to a test with moderate accuracy, as a test with an AUC greater than 0.9 is considered to have high accuracy, while an AUC in the range of 0.7–0.9 indicates moderate accuracy, with 0.5–0.7 corresponding to low accuracy, and 0.5 a chance result [13]. These findings suggest that EMR data in our system can be used as a proxy for physician assessment of depression diagnosis with moderate accuracy. Although AUC results could vary across different institutions, particularly across those using different EMR systems, these results are informative in light of growing concerns that rates of medical diagnoses and prescription use in the EMR may not accurately reflect clinician assessment [4].

Compared to using the single field “Billing diagnosis”, the combination of “billing diagnosis or medication list,” had higher sensitivity (85% versus 77%), slightly lower specificity (73% versus 76%), but a very small added overall accuracy (AUC 0.79 versus 0.77, a 2.5% increase). Other combinations of selected EMR fields to predict PCP assessment of depression did not add substantial value compared to using single fields alone. One can argue that the single “billing diagnosis” field could be a practical choice as a proxy for the diagnosis of depression, depending on the nature of the research to be performed using the electronic database. The better performance of “billing diagnosis” field, compared to the other single fields, may be that this administrative data field captures depression diagnosis billing by both PCPs and providers in mental health practices (e.g., psychiatrists, psychologists, social workers). In contrast, clinical data from problem list fields may be less complete, reflecting diagnoses only by providers who choose to use this field. Similarly, even though antidepressants documented in the “medication list” field are a part of a list generated from electronic prescriptions created by providers, this field could be another clinical data field in the electronic medical record that might be influenced by individual provider preference in documentation. Finally, for both problem list and medication data, it was not possible to determine whether depression ever was included on the problem list or if patients were given antidepressants in 2007, even if it were subsequently resolved. It is possible that the low sensitivity of a problem list data in particular was due to the fact that the depressive episode resolved and therefore depression was removed from the problem list.

There exist certain limitations to our analysis. First, only 62.5% PCPs responded to our survey. Compared to physicians who failed to respond, those physicians who responded may differ from those who did not. For example, they may have more interest in recognizing and treating depression in their practices. In addition, providers were surveyed in January 2009 regarding the diagnostic status of depressive symptoms reported by patients in 2007, which could have limited the accuracy of their assessments in this survey. However, PCPs were encouraged to review their visit notes from patients’ 2007 electronic medical records when completing the survey to decrease recall bias. That 14% of the patients PCPs chose not to classify stating there was “not enough information” provided in the EMR to determine a depression diagnosis suggests that PCPs were thoughtful in making their diagnoses, limiting their answers only to those patients for whom they were reasonably sure they could make an accurate diagnosis. PCPs were asked about which patients met the criteria for “clinical depression”; no additional listing of depression diagnoses were given for their perusal, which may have left the definition up to interpretation for some clinicians. However, although some clinicians in our survey contacted the study team to inform us of certain patients for whom there was “not enough information,” no clinicians contacted us with concerns to clarify the meaning of this term.

Two potential biases that may arise from analyses that report sensitivity and specificity to a gold standard assessment are spectrum and verification biases. In the case of spectrum bias, our study drew patients preferentially from a limited portion of the disease spectrum, i.e. patients known to have or not have specific diagnostic codes for depression [15]. This bias reflects the inherent variation in test performance among population subgroups, and thus caution must be made in generalizing our results to a general population sample [16]. Verification bias may have occurred in this study when applying the gold standard test of physician review to cases with and without depression in a manner that is disproportionate to prevalence of disease in the general population [17]. In our analyses, we included 50% of patients with clinical depression versus 50% patients without clinical depression, whereas, in a general population, the 12-month prevalence of depression is much lower, on average around 5% [18, 19]. Although these biases are not necessarily large, we acknowledge their impact on the generalizability of our analysis to a larger population sample.

Finally, we were not able to evaluate the deliberate use by PCPs of alternate diagnostic codes for depression. Data from Rost et al. [7] suggests that the deliberate substitution of alternative diagnostic codes is common in primary care practices due to provider uncertainty related to the depression diagnosis, insurance reimbursement policies and fear of endangering patients’ future eligibility for health insurance. Fatigue, malaise, insomnia and headaches accounted for 59.8%, 43.9% and 28.0%, respectively, of the alternative codes used by primary care physicians for major depressive disorder in that study [7]. In this study, we did not include those nonspecific diagnoses as depression-related billing codes and did not evaluate how well they corresponded to PCP diagnoses of depression.

Conclusion

Although the EMR has been designed for primarily for administrative and clinical applications, its structure can be adapted for use in research. This has been the case with our institution, where the RPDR can cull data from various administrative and clinical sources, with great potential to create powerful databases for research purposes. Our analysis suggests that specific administrative EMR fields can be used as reasonable proxies for clinical depression diagnosis as compared to PCP assessment. However, as our analysis demonstrates, prior to using administrative and clinical data in mental health studies, researchers must consider the validity of choosing specific fields within their EMR system in order to define clinical diagnoses.

Summary Table.

What is already known

Electronic medical records (EMR) are a powerful tool to facilitate clinical research The EMR has been used to answer questions regarding diagnostic and prescribing patterns in depressive disorders However, less is known about how well medical records reflect clinical presentation at the time of the clinician’s EMR entry

What this study added to our knowledge

Our analysis demonstrates that certain administrative and clinical EMR fields may be used to predict PCP assessment of clinical depression with moderate accuracy. Although the EMR has been designed for primarily for administrative and clinical applications, its structure can be adapted for use in research. When using administrative and clinical data in mental health studies, researchers must consider the validity of choosing specific fields within their EMR system in order to define clinical diagnoses.