The Concordance of Electronic Health Record Diagnoses and Substance use Self-Reports Among Reproductive Aged Women Enrolled in a Community-Based Addiction Reduction Program

Abstract

Substance use disorders among reproductive aged women are a major public health issue. There is little work investigating the validity and reliability of electronic health record (EHR) data for measuring substance use in this population. This study examined the concordance of self-reported substance use with clinical diagnoses of substance use, substance abuse and substance use disorder in EHR data. Reproductive age women enrolled in the Community-Based Addiction Reduction (CARE) program were interviewed by peer recovery coaches (PRC) at enrollment. That survey data was linked with EHR data (n = 102). Concordance between self-reported substance use and clinical diagnoses in the EHR was examined for opioids, cannabis/THC, and cocaine. Cohen’s kappa, sensitivity, and specificity were calculated. The survey captured a higher number of women who use substances compared to the EHR. The concordance of self-report with EHR diagnosis varied by substance and was higher for opioids (17.6%) relative to cannabis/THC (8.8%), and cocaine (3.0%). Additionally, opioids had higher sensitivity (46.2%) and lower specificity (76.2%) relative to cannabis/THC and cocaine. Survey data collected by PRCs captured more substance use than EHRs, suggesting that EHRs underestimate substance use prevalence. The higher sensitivity and lower specificity of opioids was due to a larger number of women who had a diagnosis of opioid use in the EHR who did not self-report opioid use in the self-report survey relative to cannabis/THC and cocaine. Opioid self-report and diagnosis may be influenced by research setting, question wording, or receipt of medication for opioid use disorder.

• What do we already know about this topic?

• Substance use disorders among reproductive aged women are a major public health issue in the United States, but there is little work investigating the validity and reliability of electronic health record (EHR) data for measuring substance use in this population.

• How does your research contribute to the field?

• To our knowledge, this study is the first to examine the concordance of self-reported substance use with clinical diagnoses of substance use in EHR data among reproductive aged women.

• What are your research’s implications toward theory, practice, or policy?

• Electronic health records may underestimate substance use prevalence among reproductive age women and opioid self-report and diagnosis may be more influenced by setting (clinical vs research) or question wording relative to cannabis/THC and cocaine.

Introduction

Substance use is an ongoing public health issue in the United States, with a growing number of overdose related deaths in recent years.^1,2 Reproductive aged women, who make up approximately 20% of the total population, ³ are no exception to this trend, and over half of the opioid overdose deaths among women have occurred in women of childbearing age (ages 15-44). ² Even if death does not occur, among women who used opioids for extended periods, there is a risk of damage to the heart, respiratory depression, and some types of cancer. ⁴

We have also seen an increase in maternal and fetal morbidity due to substance abuse during the pregnancy period.^5,6 The most common substances used during pregnancy are cocaine and tetrahydrocannabinol (THC), ⁷ and opioid use during pregnancy is a growing issue. ⁸ Cocaine use during pregnancy is associated with infant preterm birth ⁹ and low birthweight ⁹ as well as maternal hypertension ¹⁰ and myocardial infarction. ¹⁰ THC use during pregnancy is associated with increased stillbirth, ¹¹ low birth weight ¹² and behavioral issues. ¹² Opioid use during pregnancy is associated with neonatal opioid withdrawal syndrome (NOWS), a cluster of symptoms such as tremors, diarrhea, seizures and hyperirritability, ¹³ and subsequent developmental delays as the child matures. ¹⁴ In addition to the risk posed to the fetus if the woman is pregnant, women are of special interest to addiction researchers. Women progress more quickly into addiction with opioids, cocaine and THC relative to men ⁴ and are more likely to have comorbid psychological conditions exacerbated by use. ⁴ This requires additional consideration and attention when treating substance use disorders among this group.

Capturing substance use prevalence is helpful for establishing trends in recreational and disordered substance use as it relates to overdoses and negative birth outcomes. Obtaining accurate estimates of substance use can be difficult for many reasons: stigma, lack of trust, fear of legal consequences, etc.^15,16 Thus, self-report has significant issues with accuracy, but drug testing within the clinical setting may also fail to appropriately capture use due to the short toxicological window of many substances. ¹⁷ Electronic health records (EHR) have been used to study substance use by tracking overdose diagnoses ¹⁸ as well as prescriptions for opioid agonists to combat overdoses ¹⁹ and medication for opioid use disorder. ²⁰ EHR data are a rich source of health services information, but they have been shown to have low sensitivity in identifying substance use.^21,22 However, these studies examined samples of both men and women with a large spread of ages and did not target reproductive aged women specifically.^21,22

It is unclear what the concordance of self-reported substance use with EHR diagnoses would be for reproductive aged women because reproductive aged women, and especially pregnant women, have unique incentives to not disclose substance use to providers. They may be afraid of judgment by their providers and if they are pregnant, legal consequences and potential involvement with Child Protective Services.^{23 -25} On the other hand, many pregnant people reduce or discontinue substance use during pregnancy, ²⁶ indicating a desire to prioritize the health of the infant, which could change reporting behaviors. Women who use substances have also been shown to be more motivated to seek treatment during pregnancy, ²⁷ which may actually improve concordance between self-reports and EHR records.

Understanding the accuracy of EHR records for assessing drug use among reproductive aged women and during pregnancy is essential for researchers as the incidence of drug related fetal morbidity continues to rise. ⁵ We know that substance use is increasing, ²⁸ and often has fatal consequences, but we do not know if reproductive aged women are being accurately identified as someone who uses substances, abuses substances, and/or has a substance use disorder (SUD) by the health system. Identifying substance use, substance abuse, and SUD, and providing appropriate treatment as soon as possible is also essential for improving the health of infants since the majority of substance use in reproductive aged women begins before the pregnancy period and continues post-conception,^5,29 which can result in serious complications. ³⁰ The USPST task force does recommend routine screening of substance use disorders, including among pregnant women, when appropriate follow-up care can be provided. ³¹ However, it does not specify the exact frequency of this screening. ³¹ Even when screening is done, the receipt of an EHR diagnosis of substance use, substance abuse, or SUD is often dependent upon the patient experiencing negative drug related consequences rather than simply acknowledging use. ³² Therefore, the concordance between self-reports of drug use and EHR diagnoses of substance use, substance abuse and SUD is understandably low in previous studies.^21,22 However, this relationship has not been explored in a sample of reproductive aged women who have specified that they struggle with substance use and are seeking care.

Utilizing a survey administered by a peer recovery coach (PRC) as our benchmark,^33,34 this study assessed the sensitivity and specificity of cannabis/THC, cocaine and opioid use self-report with ICD-10 EHR diagnoses among reproductive aged women. The sample was comprised of reproductive aged women with an acknowledged substance use issue who enrolled in a behavior change intervention. Given the acknowledged substance use and willingness to seek care, there was reason to believe that this population would have relatively high concordance of substance use self-reports and EHR diagnoses. However, EHR diagnoses still have substantial limitations in terms of how often women are screened and who meets the criteria for a medical diagnosis.^31,32 so the level of concordance for this population is unclear.

Data and Methods

The data used in this study were collected from women enrolled in the Community-Based Addiction Reduction (CARE) program, a behavioral change intervention targeting reproductive aged women with a known or self-declared substance use disorder. Women in CARE were matched with a peer recovery coach (PRC) who had a personal history of substance use disorder and recovery. The PRC mentored the CARE participant and connected her to resources within the healthcare system and beyond (food assistance, free diapers, etc.) ³⁵ CARE survey data used in this paper were collected at the time of enrollment in the study. Participants were enrolled in CARE from April 2020 to February 2022. More information on the CARE intervention and the recruitment process is reported elsewhere. ³⁵

A total of 184 women enrolled in the CARE program. Nine enrollees withdrew from the study, and information on 1 respondent was dropped due to missing information in key study variables. Of the remaining enrollees, 170 consented to link their survey data to the EHR and 163 of those consented were successfully linked to EHR from the 2 largest health networks in the county where the CARE intervention took place as well as The Indiana Patient Care Network (INPC). INPC is a statewide initiative to collect the health records of patients across Indiana from a variety of health institutions.^36,37 In this way, we were able to capture as many “touches” with the healthcare system as possible. We also included diagnoses of substance use that were classified as, “uncomplicated,” and “in remission,” since these encounter diagnoses would indicate that a conversation regarding substance use occurred in that visit. Substance use disorder that is, “in remission,” does not necessarily indicate that the person abstains completely from the substance. ³⁸ It simply means that they no longer meet the criteria for dependence. ³⁸ Given the high specificity and low sensitivity of EHR records with substance use in previous studies,^21,22 the fact that the women in our sample have all struggled with substance use issues, and that our survey asks about use rather than dependence, we determined that inclusion of these ICD codes was the most conservative approach. The study sample is limited to the 102 women who had at least 1 primary care or emergency room visit within the 3 months prior to enrollment.

The dependent variable was self-reported substance use taken from the CARE survey question, “In the past 3 month, how often have you used?” The response categories were: never, once or twice, monthly, weekly, daily, or almost daily. The responses were dichotomized so that 1 indicates any use within the past 3 months (once or twice, monthly, weekly, daily, or almost daily) and 0 indicates not using the substance. Participants were able to self-report race with the following categories, “Black,” “White,” “Asian,” or “Other.” They were also able to self-report ethnicity as “Hispanic,” or “Not Hispanic.” All patients chose either, “Black” or “White” and only 3 respondents (1 Black respondent and 2 White respondents) indicated a Hispanic ethnicity. This was not large enough for a separate category, so we included only racial categories of Black and White. Sensitivity analyses with these women omitted had similar results. No other racial categories were chosen by the participants. Self-reported age was reported continuously and coded as “<25,” “25 to 30,” “31 to 35,” “36 to 40” and “41 plus.” Women were also asked “Which 1 of these comes closest to your own feelings about your household’s income these days?” The possible responses were, “Living comfortably on present income,” “Getting by on present income,” “Finding it difficult on present income,” and “Finding it very difficult on present income.” Women were asked their pregnancy status at the time of enrollment and the response categories were, “Pregnant,” “Postpartum,” or “Neither.” EHR recorded substance use, substance abuse, or SUD was obtained by pulling all health records for each woman for the 3 months prior to her enrollment and defining diagnoses of substance use, substance abuse, and SUD using the phenotypes listed in appendix A. All patient encounters (outpatient, inpatient and emergency room) were included in the analysis. We did not use diagnoses listed in the patient’s problem list, which is a list of chronic health issues that is included in the patient’s health record. For this study, all references to “EHR data” should be interpreted as diagnoses entered into a patient’s health record from an encounter that occurred in the 3 months prior to enrolling in the CARE study. This was done to ensure that all diagnoses were related to that specific encounter.

The analysis includes descriptive statistics with chi-squares along with Cohen’s kappa that we calculated to assess agreement between self-reported substance use in the CARE survey and diagnoses of substance use, substance abuse, and SUD in the EHR records. Kappa statistic is based on the difference between observed agreement and agreement expected to be present by chance alone. The values of the kappa statistic lie on a scale of −1 to 1, with higher values corresponding to a higher degree of agreement. ³⁹ Kappa values were interpreted using Landis and Koch categorization as follows: almost perfect (>0.80), substantial (0.61-0.80), moderate (0.41-0.60), fair (0.21-0.40), slight (0.00-0.20) and poor agreement (<0.00). ⁴⁰

In addition to kappa, sensitivity and specificity of self-reports were estimated using self-reported survey data as the benchmark.^33,34 Sensitivity estimated a proportion of self-reported use that was also identified as such by electronic health records. Specificity estimated a proportion of self-reported negative patients who were correctly identified as non-users by EHR. All analyses were performed using Stata version 14. ⁴¹

We compare diagnoses of substance use, substance abuse, and SUD in the EHR that were recorded in the 3 months prior to enrollment in CARE to match the time frame specified in the enrollment survey. As a robustness check, we also ran the analysis on the diagnoses of substance use, substance abuse, and SUD given in the 12-months and 24-months before enrollment in CARE. The results were not substantially different. Utilizing EHR data before the time of enrollment was done so that (a) the EHR time frame would match the time frame asked about in the survey and (b) to avoid potential confounding effects of treatment in the CARE program. Data are available upon request.

Results

More than half of the participants were White (59%) while 41% were Black (Table 1). The average age of the participants was 32.2 years at the time of enrollment. Approximately 13% of the women included in this analysis were under the age of 25, 56% were between the ages of 25 and 35, and 31% were above 35 years of age. A total of 43 women (42.2%) in the sample were pregnant or postpartum at the time of enrollment. Only 1 in 10 participants reported living comfortably on present income, 90% of the sample reported having some degree of financial difficulties.

Table 1.

Description of the Study Population (n = 102).

Characteristics	n	%
Race
White	60	58.8
Black	42	41.2
Age (at the time of survey)
<25	13	12.8
25-30	29	28.4
31-35	28	27.5
36-40	18	17.7
41+	14	13.7
Income
Living comfortably on present income	10	9.8
Getting by on present income	33	32.4
Finding it difficult on present income	19	18.6
Finding it very difficult on present income	38	37.3
Missing values	2	2.0
Pregnancy status at the time of enrollment
Pregnant	21	20.6
Postpartum	22	21.6
Neither	51	50.0
Missing	8	7.8
Self-reported substance use in CARE Survey
Opioid
Yes	39	38.2
No	63	61.8
Cannabis/THC
Yes	44	43.1
No	58	56.9
Cocaine
Yes	19	18.6
No	83	81.4
Diagnosis of substance use, substance abuse, or substance use disorder in the Electronic Health Record (EHR)*
Opioid	33	32.4
Cannabis/THC	12	11.8
Cocaine	6	5.9

Note. Substance use is reported for 3 months prior to the survey in both the CARE survey and EHR.

^*The diagnostic codes for opioids, cannabis/THC, and cocaine are available in Table A1.

In the self-reported CARE survey, 38% of women reported opioid use during the 3 months prior to their enrollment in the study and 32% of women had a diagnosis of opioid use in the EHR (Table 1). Another 43% of women reported using cannabis/THC at least once during 3 months prior to the survey, whereas only 12% had a diagnosis of cannabis/THC use in the EHRs. The prevalence of cocaine use during the same period was 19% as reported in the CARE survey. However, only 6% had cocaine use recorded in their EHR during 3 months prior to their enrollment in the CARE program. When the self-reported substance use data are broken down by type of drug, we see that the use of these drugs was not significantly different by race, age, pregnancy status, or income, except for cocaine. Cocaine use which was higher among Black women compared to White women (29%vs 12%) (P < 0.05) (Table 2).

Table 2.

Descriptive Statistics by Reported Substance Use.

Sample characteristics	Opioid use reported in survey (n = 102)		p-value	Cannabis/THC use reported in survey (n = 102)		p-value	Cocaine use reported in survey (n = 102)		P-Value
	No n (%)	Yes n (%)		No n (%)	Yes n (%)		No n (%)	Yes n (%)
n	63	39		58	44		83	19
Race			0.981			0.115			0.031
White	37(61.7)	23(38.3)		38(63.3)	22(36.7)		53(88.3)	7(11.7)
Black	26(61.9)	16(38.1)		20(47.6)	22(52.4)		30(71.4)	12(28.6)
Age (at enrollment)			0.418			0.876			0.294
<25	4(38.5)	8(61.5)		6(46.2)	7(58.9)		13(100.0)	0(0.0)
25-30	19(65.5)	10(34.5)		17(58.6)	12(41.4)		24(82.8)	5(17.2)
30-35	17(60.7)	11(39.3)		15(53.6)	13(46.4)		23(82.1)	5(17.9)
35-40	12(66.7)	6(33.3)		11(61.1)	7(38.9)		13(72.2)	5(27.8)
40+	10(71.4)	4(28.6)		9(64.3)	5(35.7)		10(71.4)	4(28.6)
Income			0.306			0.833			0.969
Living comfortably	9(90.0)	1(10.0)		7(70.0)	3(30.0)		8(80.0)	2(20.0)
Getting by	18(54.6)	15(45.5)		19(57.6)	14(42.4)		27(81.8)	6(18.2)
Finding it difficult	13(68.4)	6(31.6)		9(47.4)	10(52.6)		15(78.9)	4(21.1)
Finding it very difficult	22(57.9)	16(42.1)		22(57.9)	16(42.1)		31(81.6)	7(18.4)
Missing	1(50.0)	1(50.0)		1(50.0)	1(50.0)		2(100.0)	0(0.0)
Pregnancy status at the time of enrollment			0.331			0.841			0.407
Pregnant	11(52.4)	10(47.6)		12(57.1)	9(42.9)		18(85.7)	3(14.3)
Postpartum	12(54.6)	10(45.5)		14(63.6)	8(36.4)		18(81.8)	4(18.2)
Neither	36(70.6)	15(29.4)		27(52.9)	24(47.1)		39(76.5)	12(23.5)
Missing	4(50.0)	4(50.0)		5(62.5)	3(37.5)		8(100.0)	0 (0.0)

Note. Reference period is 3 months prior to enrolling in CARE survey.

P-Values are from chi-square test of independence.

Figure 1 presents the Venn diagram of concordance between self-reported drug use in the CARE Survey and EHR. Out of the 102 women, 39 women self-reported opioid use during 3 months prior to their enrollment in CARE, compared to 33 women whose diagnosis included opioid use in their EHRs for the same time window. Only 18 women had a concordant response about their opioid use in both the self-reported survey and the EHRs. For cannabis/THC, 44 women self-reported use in the CARE survey compared to 12 women whose diagnosis included cannabis/THC use in their EHRs. Only 9 women had a concordant response for cannabis/THC use. Nineteen women reported cocaine use in the survey compared to 3 women whose diagnosis included opioid use in their EHRs. Only 3 women had a concordant response about their cocaine use in both the self-reported survey and the EHRs.

Figure 1.

Concordance in substance use reporting in the CARE survey and diagnoses of substance use, substance abuse, or SUD in the Electronic Health Records during the 3 months prior to the enrollment in the CARE program.

The kappa statistic indicated none to slight agreement between self-reported substance use in the Care survey and EHRs for all substances (k ≤ 0.23) (Table 3). The sensitivity of self-reported opioid use was higher compared to cannabis/THC and cocaine use. The sensitivity of self-reported use of opioids was at 46.2% compared to 20.5% for cannabis and 15.8% for cocaine. The specificity of self-reports for cannabis and cocaine was high (94.8% and 96.4%). However, the specificity was lower for opioid use (76.2%) than for cannabis and cocaine.

Table 3.

Measures of Agreement Between Self-Reported Substance Use and Diagnoses of Substance Use, Substance Abuse, or SUD in the Electronic Health Records (3-months prior to the survey) (N = 102).

Substance	Reported use n (%)				Measures of agreement	Measures of validity
	Both survey and EHR	Survey only	EHR only	Neither	Kappa	Sensitivity (%)	Specificity (%)
Opioid	18 (17.6)	21 (20.6)	15 (14.7)	48 (47.1)	0.23 [0.03,0.42]	46.2 [36.5,55.8]	76.2 [67.9,84.5]
Cannabis/THC	9 (8.8)	35 (34.3)	3 (3.0)	55 (53.9)	0.17 [0.02,0.37]	20.5 [12.6,28.3]	94.8 [90.5,99.1]
Cocaine	3 (2.9)	16 (15.7)	3 (2.9)	80 (78.4)	0.17 [−0.06,0.39]	15.8 [8.7,22.9]	96.4 [92.8,100.0]

Note. Confidence intervals are presented in parenthesis.

Discussion

Assessing the accuracy of EHR diagnoses of substance use, substance abuse, and SUD is essential for clinicians to provide appropriate treatment to all individuals who may be suffering from a SUD or engaging in recreational substance use that could escalate into dependence. It is especially important among reproductive aged women who are currently pregnant or could become pregnant due to the increased risks that substance use poses to the fetus. It is also critical for researchers who use these data to study substance use and birth outcomes. Our study found that the EHR diagnostic records captured a much smaller number of women using substances than the self-report survey. This is unsurprising given the potential barriers to substance use, substance abuse, and SUD reporting in the clinical setting such as stigma, lack of trust and fear of legal consequences.^15,16 It should also be noted that recreational substance use is not the same as substance abuse or substance use disorder. Thus, the patient could have disclosed recreational use and not received a SUD diagnosis, or the patient could have chosen not to disclose since they do not view their use as disordered. This result also aligns with previous work on cannabis/THC use, ²² which found low specificity, but high sensitivity of EHR diagnoses. Thus, respondents who self-reported cannabis/THC use were not likely to have a corresponding EHR diagnosis, but if a respondent did have an EHR diagnosis, they were very likely to respond yes to the self-report survey. A similar pattern was seen for cocaine.

The more surprising result is the relatively higher sensitivity (46.2%) and relatively lower specificity (73.4%) of the opioid use measure compared to the other substances. Even though 38% of women self-reported using opioids prior to their enrollment into the CARE Program and 32% women had a diagnosis of opioid use in their electronic health records. Only 18% had both an EHR diagnosis and a positive self-report in the survey. It appears that the EHR records are capturing 1 group of women who use opioids, and the self-report is capturing a different group of opioid users. We know from previous research that EHR diagnoses of substance use, substance abuse, and SUD are more accurate for frequent users ²¹ and that women are more likely to disclose use when they are also being assessed for drug use via a toxicology screen. ⁴² Concomitantly, we know that individuals are more likely to disclose use when interviewed by researchers relative to being questioned by clinicians or given a questionnaire. ⁴² However, it’s not clear why this would be true for opioids but not cannabis/THC or cocaine. Other studies have reported a low sensitivity of EHR diagnoses for opioids relative to cocaine as well as a higher level of disagreement in opioid self-reports relative to other substances when verified by urine test. ⁴³ Thus, opioids may be different from other substances in important ways.

First, screening for opioid use requires asking questions about both prescription and illicit opioids. ⁴⁴ While we do not know how the question was phrased in the clinical setting for this study, we do know that the data gathered from the women participating in the CARE study did not distinguish between illicit and prescription opioid use. Thus, if a woman is abusing a prescription opioid, such as codeine or oxycodone, she may not classify that as “substance use,” the same as an illicit opioid such as heroin. If the woman was asked about both prescription and illicit opioid use in the clinical setting, that may have influenced their response. We also examined differences by reporting status (self-report, EHR, or both) and found no significant differences by demographics or pregnancy status (Appendix B).

Second, opioid use disorder (OUD) is unique in that the treatment for this condition often involves a prescription for a controlled dose of the opioid to be given to the patient to ease cravings and withdrawal symptoms. ⁴⁵ Thus, if a patient has an OUD diagnosis, it is likely that the patient also has a prescription for a medication to treat opioid use disorder (MOUD). ⁴⁵ Unfortunately, it is not clear from the literature if individuals receiving MOUD would self-report this prescription as “opioid use.” This could result in a diagnosis of OUD, including a diagnosis of OUD in remission, but no reported use in the survey. Given that many of the women with an OUD diagnosis may have received prescriptions for MOUD, it is possible that a lower proportion of women with an EHR diagnosis of OUD are reporting opioid use, because they are taking MOUD and refraining from using illicit opioids. However, it is unclear from the data if that is the full story. Given that there is still overlap in EHR diagnoses and opioid use self-reports the following scenarios are possible: (1) Some of the women with prescriptions for MOUD reported their MOUD prescription as “use.” (2) Some women continued to use illicit opioids alongside their MOUD. (3) Some women did not receive an MOUD prescription despite their OUD diagnosis and they were using other types of opioids. Additional work examining how MOUD is conceptualized by women and how it affects self-reports of use must be done to clarify this relationship.

Further, fentanyl, a common opioid that is abused, is strongly associated with polydrug use. ⁴⁶ It is possible that individuals are exposed to fentanyl alongside other drugs such as cocaine. Women exposed to fentanyl as result of polydrug use may not consider themselves to be opioid users if fentanyl is not the primary drug they use. This means that the way a question is asked, and whether or not it is followed up by a toxicology screen could greatly impact reporting in both the clinical and research space. ⁴² Given the unique usage patterns of opioids, future studies should investigate how opioid use self-report is impacted by question wording, polydrug use and MOUD use.

The study has some important limitations. Women in the CARE program were only asked about substance use in the past 3 months and if they had ever used the substance. We were able to match the length of time referenced in the survey question with patient encounters that occurred in that same time frame. However, there may have been greater concordance if the survey had asked about a longer time frame, such as 12 months. We also did not engage in chart review to examine notes taken during patient encounters that may have documented substance use, substance abuse, and SUD without a diagnosis in the patient’s EHR, or used diagnoses listed in the patient’s problem list. Therefore, we are unable to comment directly on how well substance use, substance abuse, or SUD is noted in the patient’s full clinical record. This study can only provide insight into the data captured by encounter level diagnoses recorded in the EHR, which are commonly used for research purposes. It is possible that the problem list would have contained additional information on substance use, but the problem lists are not always updated at every visit and may accurately capture whether or not a conversation regarding substance use took place at the encounters we examined in our analysis. Therefore, we chose to focus on encounter level diagnoses to better match the questions regarding recent substance use “in the past 3 months,” in the intake survey.

The results from this study must interpreted with the knowledge that recreational substance use is not the same as substance abuse or substance use disorder, and this may account for some of the discrepancy between reported use in the survey and EHR diagnoses. This is a substantial limitation and may account for a portion of the discrepancy in the EHR diagnoses and the survey responses. A more accurate comparison would, perhaps, be a diagnostic interview asking similar questions that would be asked in a clinical setting. The survey responses and the EHR diagnoses may be estimating different types of substance use (recreational use vs clinically relevant substance use, abuse, or SUD)– both of which are of interest to researchers but are not necessarily comparable. It is important that researchers who utilize both self-report and clinical diagnoses of SUD keep this in mind when estimating substance use prevalence.

It is also possible that CARE subjects wanted to provide interviewers with responses that aligned with the study, and this response bias artificially inflated the responses for the enrollment interview. The research subjects may also have simply felt more comfortable disclosing use in the intake survey relative to disclosing in a clinical setting. Another important consideration is that study was conducted from April 2020 to February 2022, which contains the shutdowns for Covid that occurred in March of 2020. ⁴⁷ We know that drug overdoses increased during Covid, ⁴⁸ but we do not know how it might have impacted substance use in this population or EHR diagnoses of substance use, substance abuse, and SUD if women avoided seeing a physician for primary care due to fear of Covid infection. Our results align with work that did not occur during the pandemic, ²² which gives us confidence that the results are generalizable outside of this time, but this bias must be acknowledged.

In addition, this study population was specifically targeted to women of reproductive age with a known or self-declared substance use disorder, which may limit the generalizability of the results. However, to our knowledge, this is still the first study to examine concordance of self-reported drug use in a survey with EHR records among reproductive age women. These results highlight a disconnect between substance use, substance abuse, and SUD identification in a clinic versus substance use identification in a non-clinical setting. As overdose mortality continues to rise, ⁵ we must begin to think more critically about identification of substance use, substance abuse, and SUD in the EHR and the role of the health care system in prevention and treatment of SUD. Furthermore, researchers who utilize EHR data to examine the impact of substance use on other health issues must recognize the limitations of a clinical substance use diagnosis as it does not capture most substance use in the population.

Conclusions

This study examined the concordance of self-reported substance use with EHR diagnoses of substance use, substance abuse, and SUD. The findings contribute to a growing body of literature showing that clinical diagnoses of substance use, substance abuse, and SUD only capture a small portion of actual substance users,^21,22,49 but those diagnoses are usually true positives. We also found that opioids have relatively higher sensitivity and lower specificity compared to cannabis/THC and cocaine. This may mean that disclosure of opioid use is more sensitive to question wording, or other factors such as clinical versus research setting. Opioid use self-reports may also be influenced by the receipt of MOUD prescriptions, but additional research is needed to clarify the nature of this relationship. Given the nature of the study sample, which was women with an acknowledged substance use issue who were willing to seek care, the low concordance of self-reports and EHR diagnoses is striking. This suggests that researchers using EHR records to study substance use should be very careful in any conclusions they draw and that clinicians may want to think more critically about how we document substance use in medical records.

Appendix Table A.

ICD-10 Codes Used for Identification of Diagnoses of Substance Use, Substance Abuse, and SUD in the Electronic Health Records.

Drug type	ICD-10 code used in EHR
Opioids	F11, F111, F1110, F1111, F1112, F11120, F11121, F11122, F11129, F1114, F1115, F11150, F11151, F11159, F1118, F11181, F11182, F11188, F1119, F112, F1120, F1121, F1122, F11220, F11221, F11222, F11229, F1123, F1124, F1125, F11250, F11251, F11259, F1128, F11281, F11282, F11288, F1129, F119, F1190, F1192, F11920, F11921, F11922, F11929, F1193, F1194, F1195, F11950, F11951, F11959, F1198, F11981, F11982, F11988, F1199
Cannabis/THC	F12, F121, F1210, F1211, F1212, F12120, F12121, F12122, F12129, F1215, F12150, F12151, F12159, F1218, F12180, F12188, F1219, F122, F1220, F1221, F1222, F12220, F12221, F12222, F12229, F1223, F1225, F12250, F12251, F12259, F1228, F12280, F12288, F1229, F129, F1290, F1292, F12920, F12921, F12922, F12929, F1293, F1295, F12950, F12951, F12959, F1298, F12980, F12988, F1299
Cocaine	F14, F141, F1410, F1411, F1412, F14120, F14121, F14122, F14129, F1414, F1415, F14150, F14151, F14159, F1418, F14180, F14181, F14182, F14188, F1419, F142, F1420, F1421, F1422, F14220, F14221, F14222, F14229, F1423, F1424, F1425, F14250, F14251, F14259, F1428, F14280, F14281, F14282, F14288, F1429, F149, F1490, F1492, F14920, F14921, F14922, F14929, F1494, F1495, F14950, F14951, F14959, F1498, F14980, F14981, F14982, F14988, F1499

Appendix Table B.

Concordance in Opioid Use Reporting Between CARE Survey and Electronic Health Records During the 3 Months Prior to the Enrollment in the CARE Program by Characteristics of the Respondents.

Sample characteristics	Reporting status						P-Value
	Survey only		EHR only		Both survey and EHR
	n	%	n	%	n	%
Race
White	12	33.3	13	36.1	11	30.6	0.149
Black	9	50.0	2	11.1	7	38.9
Age (at enrollment)							0.307
<35	14	36.8	9	23.7	15	39.5
35+	7	43.8	6	37.5	3	18.8
Income							0.710
Living comfortably or getting by	9	39.1	7	30.4	7	30.4
Finding it difficult or very difficult	12	40.0	8	26.7	10	33.3
Missing	0	0.0	0	0.0	1	100.0
Pregnancy status at the time of enrollment							0.267
Pregnant	4	28.6	4	28.6	6	42.9
Postpartum	7	53.9	3	23.1	3	23.1
Neither	10	50.0	5	25.0	5	25.0
Missing	0	0.0	3	42.9	4	57.1
Total	21	38.9	15	27.8	18	33.3