Automating Assessment of Lifestyle Counseling in Electronic Health Records

Abstract

Background

Numerous population-based surveys indicate that overweight and obese patients can benefit from lifestyle counseling during routine clinical care.

Purpose

To determine if natural language processing (NLP) could be applied to information in the electronic health record (EHR) to automatically assess delivery of counseling related to weight management in clinical health care encounters.

Methods

The MediClass system with NLP capabilities was used to identify weight management counseling in EHR encounter records. Knowledge for the NLP application was derived from the 5As framework for behavior counseling: Ask (evaluate weight and related disease), Advise at-risk patients to lose weight, Assess patients’ readiness to change behavior, Assist through discussion of weight loss methods and programs and Arrange follow-up efforts including referral. Using samples of EHR data in 1/1/2007-3/31/2011 period from two health systems, the accuracy of the MediClass processor for identifying these counseling elements was evaluated in post-partum visits of 600 women with gestational diabetes mellitus (GDM) compared to manual chart review as gold standard. Data were analyzed in 2013.

Results

Mean sensitivity and specificity for each of the 5As compared to the gold standard was at or above 85%, with the exception of sensitivity for Assist which was measured at 40% and 60% respectively for each of the two health systems. The automated method identified many valid cases of Assist not identified in the gold standard.

Conclusions

The MediClass processor has performance capability sufficiently similar to human abstractors to permit automated assessment of counseling for weight loss in post-partum encounter records.

OBJECTIVE

Large studies such as the Diabetes Prevention Program¹have shown that appropriate weight management can significantly delay the progression to diabetes for persons at high risk. Furthermore, numerous population-based surveys suggest that overweight and obese patients can benefit from counseling during routine clinical care.^2-10 Obese patients who received counseling and weight management advice from physicians were significantly more likely to undertake weight management programs than those who did not receive such advice, underscoring the value of increased counseling during routine clinical encounters.¹⁰ From data such as these, a consensus to provide weight management counseling has emerged as a standard for practice in primary care.^1,12

In 1998, components of the 5As behavioral treatment model for smoking cessation were incorporated into NIH guidelines for obesity management in primary care.^13-15 The 5As model includes 1) Asking (through objective measurement and assessment), 2) Advising about the need to achieve a desired outcome and the benefits of doing so, 3) Assessing patient readiness to change behavior, 4) Assisting in establishing appropriate intervention and 5) Arranging follow-up on these behavioral change efforts. Behavior change counseling based on the 5As model may benefit the treatment of diabetes and obesity.¹⁶ A controlled study found that Internal Medicine residents who were trained in 5As counseling provided a higher quality of care to obese patients than their non-trained counterparts.¹⁷ The 5As can provide a framework to assess the degree to which counseling for specific behaviors is provided in clinical practice. However, many elements of behavioral counseling are not adequately represented in coded data of the EHR, requiring manual review of free-text notes in clinical encounter records to identify delivery of the components of 5As counseling.^18-20 Natural language processing (NLP) offers an appealing alternative to manual review for efficient identification of counseling interventions in electronic health records (EHRs), and has been shown to have acceptable accuracy for identifying smoking cessation counseling in EHRs.¹⁸ Classifying the health status and treatment of obese and diabetic patients using NLP can identify which patient groups are receiving what types of weight-loss interventions. It may also reveal insights into treatment processes and identify mediators of health outcomes. For example, an evaluation of primary care records for 30,000 patients with diabetes showed that lifestyle counseling (identified using NLP) was strongly associated with faster achievement of A1C, blood pressure, and LDL cholesterol control.²¹

Women who develop gestational diabetes mellitus (GDM), a fairly common condition during pregnancy,^22,23 have increased risk for developing type 2 diabetes.^24,25 Many women who develop GDM are overweight or obese²⁶ and their weight confers an additional increased risk for developing type 2 diabetes over time. There is limited information on what weight management counseling women with GDM receive in the early post-pregnancy period. To obtain this information, the study team developed an automated processor to detect weight-loss counseling 5As in primary care encounter records for women with a recent diagnosis of GDM and assessed it’s validity in comparison to manual chart review.

RESEARCH METHODS

Setting and Study Participants

Data for this study were collected from two regions of Kaiser Permanente (Southern California and Colorado) that together deliver care to over 4 million members. The Institutional Review Board in each region as well as the Kaiser Northwest region, where these analyses were conducted, approved the study protocol for this work, which was part of the larger Surveillance, Prevention, and Management of Diabetes Mellitus (SUPREME-DM) Study.²⁷ Clinical data for individual patients were drawn from EHRs from these two regions. Each record included progress notes, nursing notes, patient instructions (after visit summary) or letters attached to the encounter record for outpatient primary care or obstetrical visits in the first year post-partum. Data analysis took place during 2013.

Operationalizing the 5As

Building from prior work that had operationalized the 5As for identifying smoking cessation counseling in primary care,^16,18,28 the 5As for weight-loss counseling were defined as follows: 1) Ask (defined as weight and diabetes assessment), 2) Advise (defined as directive statements by clinician to patient to encourage weight loss, improve diet, and increase exercise), 3) Assess (defined as psychological assessments of a patient’s readiness or motivation to alter behavior to achieve these goals), 4) Assist (defined as a broad array of counseling activities for achieving weight loss) and 5) Arrange (defined as explicit follow-up efforts to provide services or referrals for weight management or diabetes prevention programs). Previous work established the face validity of these specifications of the 5As for weight loss in primary care, and identified significant prevalence of these counseling activities in primary care records of the EHR in one health system.²⁹

To refine specifications for the 5A’s in the context of postpartum visits of women with GDM, a set of encounter records from the project sites (a pilot sample) were reviewed for documentation of weight management, weight, exercise, physical activity, diet, diet change, diabetes risk, diabetes, and GDM. This pilot sample included women diagnosed with GDM who had a live singleton birth in 2009 and were members of the participating health plan for at least 12 months after delivery. It is recommended that women with GDM have a glucose test (fasting plasma glucose, oral glucose tolerance test, etc.) during their postpartum visit, which routinely occurs from 6-12 weeks postpartum. Therefore, the sample of 100 women was stratified to include 25 women with no postpartum testing, 25 women who received postpartum testing and had normal results, and 50 women who received postpartum testing and had abnormal results. From this preliminary work, a classification scheme to define the 5As of counseling for weight loss was developed (Table 2).

Table 2.

The 5As adapted for weight loss counseling: The SUPREME-DM Study

5A step	Definition	Example statements in free text section of EMR	Source in EMR
Ask	Physical Assessment. Documentation of weights, heights, or BMI, including observational and qualitative statements about weight, weight-loss activities, diabetes generally or GDM specifically.	“has gained significant weight over last year”, “negative for weight loss”, “exercises regularly”, “still working on diet”, “GDM”, “Obesity (ICD 278.00E)”	Progress Note
Advise	Generic statements advising patient to lose weight, increase exercise, improve diet, avoid/prevent diabetes. Typically indicated by imperative or “one-way” statements by clinician.	“pt needs to lose wt”, “suggest you continue with current exercise”, “advised to begin walking program ”, “increase exercise”, “watch diet”, “lower weight to avoid onset of diabetes”, “get active, lose weight, eat healthy”	Progress Note Patient Ed Materials
Assess	Psychological Assessment. Statements reflecting patient’s readiness to lose weight, increase exercise, improve diet, concern for diabetes.	“pt has tried to lose wt many times”, “pt wants to improve diet”, “pt is unable to lose wt”, “pt wants to lose wt”, “pt plans to inc exercise”, “pt thinking about losing weight”, “pt. here to discuss obesity”, “pt concerned about diabetes”	Progress Note
Assist	Statements addressing a method to achieve weight loss/exercise/diet/DM prevention goals, invoking any of the following: self- help or educ. literature resources, referral resources, in-office counseling about avoiding certain behaviors/situations or promoting others, treatment options, or weight loss medication assistance.	“disc. pts challenges losing wt”, “starting adipex”, “his diet falls apart when family gets together”, “encouraged patient to continue with wt. loss”, “counseled on diet and exercise”, “given handout on diabetes prevention”	Progress Note Patient Ed Materials
Arrange	Statements documenting plans to follow-up with the patient regarding weight loss/diet/exercise/diabetes.	“follow up re: wt loss next visit”, “schedule appt in 4 weeks, obesity review”, “referred to nutritionist”, “referred to diabetes educator”	Progress Note Patient Ed Materials

Development and Validation Datasets

Two samples of patients’ records were obtained from each project site—creating a Development Dataset (total n=400) and a Validation Dataset (total n=600)—for development and testing of an NLP processor to identify the 5As of weight loss counseling.

The Development Dataset was employed to build the automated processor by ensuring inclusion of records likely to contain documentation of weight loss counseling from clinical visits. From each of the two KP regions contributing data (Colorado and Southern California), 200 encounter records (N = 400 total) were randomly selected from the population of women who met the following criteria: 1) ≥ 18 years of age at the time of delivery and members of the participating health plan for at least 6 months before and at least 12 months after delivery, 2) had a live singleton birth in 2007-2010 and were seen for a primary or obstetrical care visit within 6-12 weeks post-partum and were determined to have GDM based on either: a) oral glucose tolerance test (OGTT) results indicative of GDM during the pregnancy or b) a diagnosis code for GDM (ICD9 = 648.8), 3) had a fasting plasma glucose (FPG), an OGTT, or a A1c result within 6 months after their delivery date and 4) a BMI > 25 recorded in the 9- 24 months before delivery OR in the 2-12 months after delivery.

The Validation Dataset was used to evaluate the finalized automated processor, as reported below in the Results section. Total sample size of this dataset was selected to maximize measurement precision for the evaluation study, while minimizing the cost and risks to patient confidentiality posed by manual chart review of sensitive progress notes data. A sample size of 600 records affords reasonable precision for estimating sensitivity and specificity greater than 0.80 when identifying events whose prevalence in the data is greater than 5%. Therefore, 300 women (N = 600 total) were randomly selected from each KP region’s population who met criteria 1-2 (above). If included women also met criterion 3, then the visit on their date of testing was the one included in the sample. For all other included women (i.e., those without a postpartum glucose test) the first visit following delivery was included in the sample. Manual coding of the Validation Dataset was used to create a gold standard, providing a method to measure the performance of the automated data processor.

Chart Review

Primary care encounters from the Development and Validation Datasets were coded by 3 trained chart reviewers to identify the 5As of weight loss counseling using the classification scheme (Table 2). Encounter records, formatted as XML files, were uploaded into a secure server where reviewers annotated them with a program called “Manual Coder”—a component of the CER Hub (www.cerhub.org) suite of informatics tools. Users of Manual Coder step through encounter records and have the ability to highlight text elements representative of study measures (in this instance, components of the 5As for weight loss) and then select the code appropriate to the highlighted text from a list. The applied codes, as well as the highlighted content from the encounter record, can be extracted from the database for analysis. During the course of the coding process, questions arising from project staff on specific data elements were communicated to the lead investigator (BH). Responses were used to update the written instructions and communicated by email. The manual coding process was completed in two months. The coded datasets were reviewed by the lead trainer (SK), who checked for and adjudicated discrepancies with the coding instructions. A gold standard was created from the 5As classifications decisions made by the three abstractors on the Validation Dataset. For each encounter record, there are 5 classification decisions made by each reviewer, indicated by a code applied or not for each of the 5As. The gold standard included all cases of unanimity (all 3 either applied the code or did not within the record). All other positive cases (one or two but not all three reviewers applied the code to the encounter) were reviewed and the discrepancies adjudicated by the lead investigator (BH).

Development of the Automated Data Processor

The automated data processor was built as a study-specific application of the MediClass technology.³⁰ MediClass applications map the contents of an encounter to study-specific clinical concepts based on phrases detected in free-text sections and codes detected in structured sections of the EHR. In this study, only text data elements (progress notes and patient education materials) are included in the encounter records. Classifications are performed by context-sensitive rules that identify Boolean combinations of clinical concepts related to weight loss counseling. Knowledge encoded into the MediClass processor identifies discussions, referral activities and motivational assessments documented by the care provider. The processor is developed through an iterative process that begins with designing rules to capture the phrases identified by the chart reviewers in coding the Development Dataset and refined until performance improvements compared to the manually coded Development Dataset were minimized or non-existent.

The processor was run against the Validation Dataset and its performance compared to gold standard was assessed for sensitivity (the proportion of true positives found by the data processor to the total positives in the gold standard), specificity (the proportion of true negatives found by the data processor to the total negatives in the gold standard), positive predictive value (the proportion of true positives to the total positives found by the data processor), and negative predictive value (the proportion of true negatives to the total negatives found by the data processor). 95% confidence intervals for these proportions were calculated according to the efficient-score method (corrected for continuity) as described by Newcombe.³¹

RESULTS

In chart review of the Validation Dataset, reviewers predominantly agreed on the presence of each of the 5 As (Table 3). During adjudication, all but three positive cases of majority opinion (i.e., 2 of 3 reviewers applied the code to the encounter) were accepted into the gold standard. Additionally, of the 63 cases where a single reviewer applied the code to an encounter (called a “single”) a total of 44 were found to be correct on adjudication and accepted into the gold standard. The above findings indicate that chart abstractors were seldom “wrong” in what they coded, but they often missed relevant events.

Table 3.

Gold standard creation from 3 independent manual chart reviews of validation dataset for weight loss counseling.

		Ask	Advise	Assess	Assist	Arrange
Site 1 (n=300) a	Majority (2 or 3, of 3)b	184	116	2	1	0
	Majority Overruled	0	0	1	0	0
	“singles” added	0	15	4	4	0
	“singles” left out	0	2	1	4	0
Site 2 (n=300) a	Majority (2 or 3, of 3)b	137	42	2	3	0
	Majority Overruled	0	0	1	1	0
	“singles” added	0	17	1	3	0
	“singles” left out	0	1	2	8	0

^aOnly positive decisions for each of the “A’s” are shown as counts in the table

^bAll unanimous decisions were included in the final gold standard and all other positive decisions were reviewed and the discrepancies adjudicated. Final Gold Standard inclusion = Majority − Majority Overruled + “singles added”

Application of the data processor to the Validation Dataset also produced 5As classification decisions. The sensitivity and specificity of these classifications is shown in comparison to the gold standard (Table 4). The processor obtained similar results across both health systems. Sensitivity averaged 98% (Ask), 85% (Advise), 90% (Assess), and 50% (Assist) across the two sites. Because there were no Arrange positive cases in the gold standard, sensitivity could not be computed for this A. Specificity averaged 85% (Ask), 85% (Advise), 97% (Assess), 86% (Assist), and 98% (Arrange) across the two sites.

Table 4.

Comparison of the automated data processor results to the gold standard for weight loss counseling.

Site 1a,b	Ask	Advise	Assess	Assist	Arrange
FP	15	23	11	44	2
FN	5	7	1	3	0
TN	101	146	284	251	298
TP	179	124	4	2	0
Sens	0.97 (0.93,0.99)	0.95 (0.89, 0.98)	0.80 (0.30, 0.99)	0.40 (0.07,0.83)	-
Spec	0.87 (0.79,0.92)	0.86 (0.80,0.91)	0.96 (0.93,0.98)	0.85 (0.80,0.89)	0.99 (0.97, 1.0)
PPV	0.92 (0.87,0.95)	0.84 (0.77,0.90)	0.27 (0.08,0.55)	0.04 (0.01,0.16)	0.0 (0.0, 0.80)
NPV	0.95 (0.89,0.98)	0.95 (0.90,0.98)	1.0 (0.98, 1.0)	0.99 (0.96,1.0)	1.0 (0.98,1.0)

Site 2a,b	Ask	Advise	Assess	Assist	Arrange
FP	28	37	4	39	5
FN	2	14	0	2	0
TN	135	204	294	256	295
TP	135	45	2	3	0
Sens	0.99 (0.94,1.0)	0.76 (0.63,0.86)	1.00 (0.12,1.0)	0.60 (0.17,0.93)	-
Spec	0.83 (0.76,0.88)	0.85 (0.79,0.89)	0.99 (0.96,1.0)	0.87 (0.82,0.90)	0.98(0.96,0.99)
PPV	0.83 (0.76,0.88)	0.55 (0.44,0.65)	0.33 (0.06,0.76)	0.07 (0.02, 0.21)	0.0 (0.0,0.54)
NPV	0.99 (0.94,1.0)	0.94 (0.89,0.96)	1.0 (0.98, 1.0)	0.99 (0.97,1.0)	1.0 (0.98, 1.0)

^aTP is “True Positive”, FP is “False Positive”, TN is “True Negative”, FN is “False Negative” of data processor result compared to the gold standard;

^bSens is sensitivity = TP/(TP+FN); Spec is specificity = TN/(TN+FP); PPV is positive predictive value = TP/(TP+FP); NPV is negative predictive value = TN/(TN+FN); These proportions are shown with 95% confidence intervals.

For the classification of Assist, which had substantially lower sensitivity and positive predictive values in the performance analysis, the data processor generated 44 false positives in Site 1 data and 39 false positives in Site 2 data (Table 5). Review of the data responsible for these false positive classification decisions by the data processor showed that a majority were not incorrect classifications but rather were missing positive Assist decisions in the gold standard (i.e., they were missed by the manual coders). In particular, of the false positive Assist decisions made by the automated data processor, 37 of 44 at Site1 and 25 of 39 at Site 2 were considered correct decisions on review by the investigators (Table 5).

Table 5.

False positives for “Assist” that were subsequently deemed to be correctly identified by the processor but missing in the gold standard.

SITE 1	n	Text Responsible for Data Processor Classification as Assist
	8	At this class, you will learn about healthy lifestyle habits that can help prevent diabetes This class is taught by an expert Registered Dietitian at various clinic locations or an on line webinar from the comfort of your home. You may register for the class by calling XXX-XXX-XXXX. If you are interested in the webinar format, go to XXXXX.org to register.
	6	PLAN: Postpartum instructions, plans including family planning. Postnatal Depression Scale completed. As a result of her Edinburgh Questionnaire, shared decision for her care includes but is not limited to: Score 5-9:Hemoglobin and TSH labs drawn, support, exercise and monitor
	7	PLAN: Patient counseling: Normal postpartum counseling Pain management counseling Contraception counseling: yes Natural Family Planning Other counseling performed: Lactation counseling, Diet counseling,
	6	Find a class for new mothers and new babies that has an exercise time.
	10 Single instances (all from different records)	exercise classes that you can join. Kaiser has many programs available to help you get healthier and lose weight. You may call XXX-XXX-XXXX if interested in more information a good fitness program at stollerfit.com XXXXX.org for our award-winning weightloss program recommend that you attend the pre-Diabetes class. Disc GDM RISK GDM teaching GDM counseling GDM reviewed Discussed return to normal activities and exercise
SITE 2
	9	discussed diet and exercise
	5	WE ALSO OFFER THE FOLLOWING HEALTH EDUCATION CLASSES: - DIABETES (4 SESSION SERIES THAT INCLUDES DIABETES NUTRITION AND TESTING YOUR BLOOD SUGAR) - PRE-DIABETES: PREVENTING DIABETES
	2	WEIGHT LOSS, HEALTHY EATING AND PHYSICAL ACTIVITY BY CHECKING THE XXXXXXXXX.ORG WEBSITE
	9 Single instances	XXXXXXX HAS A WEIGHT MANAGEMENT DEPARTMENT, WHICH OFFERS A VARIETY OF WEIGHT AND HEALTH COURSES OR PROGRAMS PATIENT CAN CALL 1-XXX-XXX-XXXX FOR HEALTHY LIVING HELP LINE WEIGHT LOSS EDUCATION PROGRAMS. XXXXXX NURSE CAN GIVE OR MAIL PATIENT A LOW CHOLESTEROL DIET HANDOUT. PATIENT CAN ALSO CHECK INTERNET HTTP: // WWW.NHLBI.NIH.GOV/CHD / LIFESTYLES.HTM REVIEWED WITH PATIENT THAT GIVEN HER HISTORY OF GESTATIONAL DIABETES MELLITUS IN PREGNANCY, SHE IS AT INCREASED RISK OF DEVELOPING TYPE II DIABETES MELLITUS LATER IN LIFE Literature given, may return to normal activities COUNSELING: family planning, weight and exercise Low cholesterol diet handout Diet and exercise discussed. Weight loss classes offered Discussed need for weight management in detail Weight management class available through health education, call…

DISCUSSION

Our findings demonstrate that an automated NLP-based processor of EHRs can identify weight loss counseling activities in the progress notes, visit summaries and patient education materials from the postpartum clinical visits of women with GDM. Although sensitivity and specificity compared to the gold standard exceeded 85% in most cases, the data processor also produced many classifications that were not identified by the chart reviewers. In fact, 86% of the data processor’s disagreement with the gold standard across all of the As (95 cases at Site 1 and 113 cases at Site 2, see Table 4) was accounted for by false positives (i.e., counseling events found by the automated processor but not by manual chart review).

Possible explanations for the discrepancy include the fact that some chart notations may be difficult to spot for the human chart reviewers or are interpreted as instances of text that are provided through “templates” or text automatically generated by the EHR and thus not representing authentic patient counseling. In reviewing these results, the research team decided that often this documentation in the EHR (see Table 5 for the case of Assist) did indeed represent counseling to achieve weight loss in the clinical setting. All false positives were subsequently reviewed and, across all of the A’s, approximately 50% were found to be correctly identified by the automated processor even though missing from the gold standard chart review. Adjusting the quantitative results for this subsequent review is not warranted but does indicate that these results significantly understate the actual performance of the automated processor’s sensitivity and positive predictive values for the counseling activities.

There are several limitations to using NLP to assess counseling for weight loss in clinical practice. First, the processor is only as good as the quality of knowledge encoded into it. To ensure quality, the processor was developed using data drawn from diverse care settings involving a wide range of providers and women with GDM. Second, although much of what the tool does may be generalized to other related high risk groups (e.g. “prediabetes”) or other populations, there are always nuances to EHR implementations, and the clinical content these generate, that must be accommodated. These accommodations, in the form of refinements to the processor, must be determined empirically. Additionally, automated assessment of EHR visits only works when the care is documented in the record. Obviously, the method has no access to undocumented discussions between provider and patient. Furthermore, when counseling activities are documented it does not always follow that they accurately reflect what was discussed with or understood by the patient. Finally, the current study addresses care delivered in two Kaiser Permanente regions, which may not adequately represent all primary care counseling and documentation practices. New developments being pursued to overcome these limitations include tools to accelerate development and validation of NLP solutions such as the one described in this paper (see also, www.cerhub.org). These tools will allow for simplified refinement and reuse of NLP solutions, making them more easily adapted to diverse and rapidly changing EHR implementations and documentation practices.

Automated assessment of EHR data for evidence of counseling to encourage behavior change in those at high risk provides a powerful research and operations tool to determine the groups of patients that are receiving counseling (and potentially the source of the counseling) and also allows for determining how this counseling (and which specific parts of counseling) affects clinical outcomes. Applying the NLP processor to the GDM population in the SUPREME-DM diabetes prevention study will help us to understand the role of lifestyle counseling by providers in this population at high risk for diabetes. In the future, this tool could also be transferable to other populations and practice settings involving patients at high risk for disease for which weight loss is known to reduce risk.

CONCLUSIONS

The MediClass data processor has a performance capability sufficiently similar to trained medical records abstractors to permit automated assessment of counseling for weight loss in EHR encounter records from patients receiving post-partum primary care following a delivery complicated by GDM. With the rising prevalence of GDM²³ and overweight and obesity in the childbearing age population,³² both of which may contribute to adverse pregnancy outcomes and increased risk of type 2 diabetes in future years, increased emphasis on the care of women in the postpartum period is warranted. Use of NLP to identify care delivery activities in EHRs provides the opportunity to describe the care received by this high risk group of women to assess which health care interventions may lead to women adopting or sustaining behaviors to mitigate this risk.

Table 1.

Women included in the Validation Dataset (total n=600).

	Site 1 (n=300)	Site 2 (n=300)
Age (mean, sd)	32.5 yrs, 5.2 yrs	32.7 yrs, 5.3 yrs
BMI (%)
Less than 25	33	20
25 – 29.99	31	23
30 – 39.99	28	27
40 or greater	5	8
Missing	3	23
Race/Ethnicity (%)
African-American	6	4
Asian/Pacific  Islander	11	24
Native American	1	1
Non-Hispanic White	51	17
Hispanic	22	52
Other/multiple	1	2
Unknown	8	0