Predicting the readability of physicians’ secure messages to improve health communication using novel linguistic features: Findings from the ECLIPPSE study

Abstract

Background:

Low literacy skills impact important aspects of communication, including health-related information exchanges. Unsuccessful communication on the part of physician or patient contributes to lower quality of care, is associated with poorer chronic disease control, jeopardizes patient safety and can lead to unfavorable healthcare utilization patterns. To date, very little research has focused on digital communication between physicians and patients, such as secure messages sent via electronic patient portals.

Method:

The purpose of the current study is to develop an automated readability formula to better understand what elements of physicians’ digital messages make them more or less difficult to understand. The formula is developed using advanced natural language processing (NLP) to predict human ratings of physician text difficulty.

Results:

The results indicate that NLP indices that capture a diverse set of linguistic features predict the difficulty of physician messages better than classic readability tools such as Flesch Kincaid Grade Level. Our results also provide information about the textual features that best explain text readability.

Conclusion:

Implications for how the readability formula could provide feedback to physicians to improve digital health communication by promoting linguistic concordance between physician and patient are discussed.

Introduction

The purpose of the current study is develop an automated readability formula of physicians’ secure messages (SMs) to patients based on advanced linguistic features to better understand what elements of physicians’ written text make them more or less difficult to understand (i.e. a readability formula for physician messages). We focus specifically on digital communication (i.e. written electronic communications) between physicians and type 2 diabetes (DM2) patients in the form of SMs sent within online patient portals. We focus on DM2 patients because asynchronous SMs appear especially important for patients that require ongoing self-management on the part of the patient, as well as associated counseling, guidance and coordination of care on the part of the physician [1]. SM is frequently used in diabetes care to discuss such issues as emerging symptoms, medication concerns, test results, responses to treatment, patient education, appointment and referral requests and processes, and other administrative concerns [2].

To develop our linguistic formula to assess readability of physicians’ writing, we use natural language processing (NLP) indices that capture the construct of text readability in terms of lexical sophistication, syntactic complexity, sentiment and cognition variables, and text cohesion. Our readability criterion is developed from expert ratings of text difficulty that categorize the SMs as being easier or more difficult to understand by struggling readers. We measure the performance of this formula by training it on a subset of SMs and extending the resulting model to a testing subset. Finally, we compare the performance of this new formula to a classic readability measure, the Flesch-Kincaid grade level formula (FKGL; [3]), as a reference.

Text readability

It is estimated that one third of American adults are struggling readers that read at basic or less than basic levels [4,5]. Low literacy skills impact important aspects of communication, including health-related information exchanges as found in written medical texts. Health literacy (HL) can include a number of demographic and individual difference factors including education level, culture, access to resources, socioeconomic status, and age, among others. HL also includes a patient’s ability to obtain, process, comprehend, and communicate basic health information [6,7] and is highly correlated with literacy skills [8]. Limited HL is common in healthcare contexts and undermines healthcare communication [9], especially among patients with chronic diseases such as DM2 [10]. Limited HL is a strong predictor of poorer overall health, reduced access to care, lower quality of care, worse disease control and complications, unfavorable healthcare utilization patterns, and higher mortality rates [11,12]. A growing body of evidence demonstrates that reducing the literacy demands of health communications directed to patients may reduce HL-related health disparities [1,13,14].

In terms of linguistic factors, medical texts that are linguistically simple enough to allow readers below the 6th grade level to process them are generally considered easier to read, while texts simple enough to allow readers at the 7^th-9^th grade levels to process them are considered of average difficulty. Texts at the 9^th grade level and above are considered more difficult to read [15,16]. These guidelines have led agencies such as The National Institutes of Health and the American Medical Association to suggest that patient-oriented health texts be written at around the 6th to 8^th grade levels [17,18]. However, with so many adults reading at lower levels, it is estimated that about 89 million people in the United States do not have strong enough HL skills to understand and process most health-related materials, even if written at these grade levels [19].

Medical texts that are composed above patients’ reading skills can lead to negative health outcomes and health-related behaviors, as well as increased health care costs [16]. Providing patients with more comprehensible texts could lead to greater uptake and recall of medical information [20]. Several recommendations have been made to achieve this objective, and in particular, to improve the readability of patient-directed text materials [17].

Assessing text readability

In a meta-analysis of 155 previous studies, Wang et al [21] reported that six classic readability formulas based on linguistics features are commonly used in medical text analysis: Dale-Chall [22], Flesch Reading Ease (FRE; [23]), Flesch-Kincaid Grade Level (FKGL, [3]), Frequency of Gobbledygook [24], Simple Measure of Gobbledygook (SMOG, [25]), and Fry [26]. The most commonly used formula has been the FKGL, which was used in almost 60% of studies reviewed by Wang et al. [21]. However, research into the effectiveness of these classic readability formulas in medical domains [21,27,28] and text and discourse disciplines [29–34] have reported concerns about their performance and validity.

Assessments of health-related texts using classic readability formulas generally indicate the texts are written at levels too difficult for many adults to successfully comprehend. For instance, Walsh and Volsko [20] examined 100 consumer-health-information articles using three readability formulas and reported that most of the selected articles were written above the 7th-grade reading level. Other studies have found that a vast majority of diverse medical texts are too difficult to read for the intended audience including patient education brochures [35], health-related websites [18,36,37], consumer education materials, health newsletters, clinical trial records, clinical reports, journal articles, [38], patient education documents [39], letters sent from physicians to patients [40], certified web-based educational materials and resource clearinghouses for diabetes patients [41].

Notably, numerous problems in terms of prediction and reliability have been reported for classic readability formulas. As an example, Wang et al. [21] used six different readability formulas to assess depression-related health texts and found that each formula could vary by up to six reading grade level estimates for the same text. In another study, Wu et al. [27] used three readability formulas to compare the readability medical texts, finding that that readability formulas classified referral letters as the most difficult to read in comparison to the clinician notes (whereas the reverse should have been the case). Lastly, Zheng and Yu [28] examined generalist texts (i.e. Wikipedia articles) and specialist texts (i.e. electronic health records) using classic readability formulas and found, paradoxically, that electronic health records were estimated to be easier to read than the Wikipedia articles.

Finally, studies that have examined the effects of improving readability of health-related texts on patient comprehension using classic readability formulas have shown mixed results, with most having only minimal to no effects [2,42,43]. These results indicate that simply changing the length of sentences or introducing shorter words do not have strong effects on text comprehension.

Natural language processing and text readability

One theoretical explanation for the absence of stable effects in classic readability formulas is because the linguistic features of these formulas (e.g. word and sentence length) do not strongly map onto linguistic constructs predictive of reading comprehension [29–34]. These concerns have led many researchers to test different linguistic features and seek alternative readability formulas based on more advanced NLP features.

In the medical domain, researchers have developed models of readability using structural length measures similar to those found in classic readability formulas including word length, sentence length, characters per word, and sentences per paragraph [44–47]. However, these researchers have also supplemented these features with more advanced natural language processing (NLP) measures including lexical sophistication indices such as lexical diversity [44], word familiarity, and word frequency [46,47], semantic features such as average term and concept familiarity scores [45], syntactic features including epistemic modals, relative clauses, passive structures [44], and part of speech tags [45,46], and cohesion features such as word overlap ratios [46]

These newer NLP informed models of readability have shown better performance than classic readability formulas. For instance, Kim et al. [45] reported that their readability formula was more successful at categorizing electronic health records than classic readability formulas. Subsequent studies also showed that Kim et al’s [45] formula better captured the readability of three medical documents types (referral letters, non-referral letters, and health articles) than classic readability formulas [27]. Additionally, the readability model developed by Zeng-Treitler et al. [46] was shown to be a better predictor than classic readability formulas for levels of readability (i.e. easy-to-read and hard-to-read medical texts), the difficulty of medical texts as rated by experts, and of medical documents types for which readability was calculated using cloze testing procedures. Similar results for more advanced NLP readability models have been reported outside of medical domains as well (e.g. [48–51])

Method

Tailoring written communication to patients is a difficult process, especially when little is known about what makes a medical text more difficult to comprehend linguistically, when existing readability formulas are problematic, and when physician training is absent. In addition, as medical communication moves online via electronic patient portals, emerging technologies may lead to a greater need to address digital literacy. In spite of this, little research has examined the readability of physicians’ messages to patients, despite the fact that linguistic concordance in communication exchanges between physicians and patients is an important pathway to achieve shared meaning, especially for patients with limited health literacy (HL, [9,52]).

To make advances in establishing linguistic concordance, we must first understand what linguistic elements of physicians’ secure messages (SM)s make them more or less difficult to understand (i.e. develop a readability formula specifically for physicians’ message to patients). Thus, the purpose of the current study is to harness SMs derived from a patient portal, authored by a large sample of physicians working in a large health system to identify the linguistic features of physicians’ SMs to patients that are associated with text complexity. To do so, we examine relations between linguistic features and a gold standard of expert ratings of physician text complexity.

Corpus

The SMs that comprise our corpus comes from the Diabetes Study of Northern California (DISTANCE) derived from electronic communication exchanges in Kaiser Permanente Northern California (KPNC), a nonprofit, fully integrated healthcare delivery system. DISTANCE has been described in detail elsewhere [53,54]. Briefly, our study sample is drawn from the DISTANCE [54], and includes data from a subset of 10,504 diabetes patients who had sent one or more English-language SMs to their primary care physicians between July 1, 2006 – Dec. 31 2015. The SMs sent by physicians to patients include messages about lab results, and patient and physician questions regarding scheduling requests, medication refills, and test results, all which match previous analyses of similar data sets [55].

Within DISTANCE, 1,136 primary care physicians sent SMs to patients. We aggregated those SMs by patient that included at least 150 words (the selected threshold for accurately analyzing the text complexity of the SMs, based on prior research; [56]) to create SM threads. Since our goal is to eventually examine concordance between physician and patient language, we used a limited random sampling technique for this study by only including SMs from physicians that had sent SMs to specific patients whose SMs were used in a previous analysis [54]. The previous analysis modeled patient HL and we wanted to collect matching physician complexity data to eventually model concordance. Our final sample comprised 724 unique message threads from 592 individual physicians sent to 486 unique patients. From this sub-sample, these physicians sent SMs to an average of 1.23 patients; 112 of these 592 physicians messaged at least two different patients.

Upon reading the de-identified SMs threads, it was apparent that some physicians employed automated text available in the electronic health record’s portal to explain common symptoms, ailments and treatments. To estimate the frequency of automated text, two expert raters examined each message thread. The raters agreed that 220 of the SM threads contained some form of automated text embedded within the SM (involving ~30% of the SMs). We elected to retain SM threads that included automated text in the message threads for our subsequent linguistic analyses because (a) such text is representative of the language used by physicians when messaging patients and (b) it proved difficult to reliably exclude automated text segments using natural language processing (NLP) or other machine learning approaches.

To control for the fact that excess text length might influence the human ratings of readability, SM threads were randomly trimmed to contain approximately ~300 words. When trimming, no individual messages contained in the SM threads was truncated (i.e. we kept SMs intact). The average text length for the SM threads in the final corpus was 294.9 words (SD = 139.5). Among the SMs threads, 359 were composed by female and 365 by male physicians. The average age of the physicians was 53.8 years. The majority of SMs were written by Asian physicians (n = 335) followed by White (n = 291) and Black (n = 38) physicians. Twenty-two of these physicians identified themselves as Hispanic. Of the physicians, 136 reported the ability to write in a language other than English and 201 reported they spoke at least one additional language beyond English.

Expert ratings

The SM threads from the physicians to the patients were evaluated for understandability by two expert raters in terms of how difficult they were for ‘struggling readers’ to process and comprehend [6,7]. The evaluations thus focused on cognitive aspects of reading and did not focus on demographic or individual differences related to HL including education, culture, access to resources, socioeconomic status, or age. Each rater had an advanced degree, experience teaching literacy-based classes, expertise in medical discourse, and prior experience rating medical text samples. The raters were first provided with a definition of a struggling reader as:

Individuals who struggle with reading expend considerable cognitive effort and attention in translating print into language, which expends cognitive resources that could better be utilized in constructing, interpreting, and evaluating meaning. They struggle with learning to map the writing system (i.e. the printed visual symbols individually and in combination) to the spoken form of the language (i.e. the phonetics, phonology, and morphology). Their reading behavior is characterized by slow, effortful processing of text. This can impact not only their recognition of individual words, but also ability to build meaning from sentences and paragraphs of text.

The raters were then instructed to use a scoring rubric for each SM thread based on a five-point Likert scale. The rubric asked raters to judge ‘How easy would it be it for a struggling reader to understand this message?’ The five point-Likert scale was:

1. Very Easy

2. Somewhat Easy

3. Neither easy nor difficult

4. Somewhat Difficult

5. Very Difficult

This is similar to the approach used by Kandula and Zeng-Treitler [38], who had expert raters judge the readability of health texts using a 1–7 scale (1 = can be understood by anyone with basic literacy; and 7 = can be understood only by someone with professional education in a health domain). The raters in our study were first trained on a subset of 30 physician SMs not included in the final corpus. After training on the 30 texts, the raters demonstrated an inter-rater reliability (Cohen’s Kappa) of 0.61. Raters then independently assessed each physician SM thread. After independent ratings, any disagreements that reflected a deviation of >1 point on the Likert scale were adjudicated between the two raters. There were 34 such disagreements in the corpus (4.7% of SM threads). Inter-rater reliability after adjudication of all SM threads (Cohen’s Kappa) was .57. Final scores for understandability of a message were based on an average score between the two raters. Distribution of the scores was normal.

Linguistic features

We next employed natural language processing (NLP) tools to calculate linguistic features for each SM thread to develop the Model of Text Readability in Physicians (MoTeR-P) using the expert ratings. The selected linguistic features overlapped with previously developed and validated text complexity measures associated with theories of reading and included text cohesion, lexical sophistication, sentiment analysis and syntactic complexity features. Lexical sophistication and semantic features were calculated using the Tool for the Automatic Analysis of Lexical Sophistication (TAALES; [57]) and the SEntiment ANalysis and Cognition Engine (SEANCE; [58]). Syntactic complexity was calculated using the Tool for the Automatic Analysis of Syntactic Sophistication and Complexity (TAASSC; [59]) and Coh-Metrix [60]. Text cohesion was calculated using the Tool for the Automatic Analysis of Cohesion (TAACO; [61]). Finally, we used Coh-Metrix to calculate traditional readability scores for each text (i.e. Flesch-Kincaid Grade Level and Flesch Reading Ease). A more detailed account of these tools and the linguistic features they report are provided in Dascalu [62], Crossley, Kyle, and McNamara [61], [58, 59], and Kyle and Crossley [57]. We provide a brief overview below.

SEANCE

SEANCE [58] is a sentiment analysis tool that relies on a number of pre-existing sentiment, social positioning, and cognition dictionaries. SEANCE can provide semantic information about the words in a text, which may help readers link common themes and ideas. SEANCE contains a number of pre-developed word vectors developed to measure sentiment, cognition, and social order. These vectors are taken from freely available source databases. For many of these vectors, SEANCE also provides a negation feature (i.e. a contextual valence shifter) that ignores positive terms that are negated (e.g. not happy). Some specific indices provided by SEANCE include component scores that measures the number of words related to fear and disgust, the number of words related to objects and to the touching and moving objects, the number of words related to action, the number of words related to joy, and the number of words related to friends and family.

TAALES

TAALES [57] calculates over 200 indices related to the lexical sophistication of words in a text. Measure of lexical sophistication correlate with faster processing and more accurate word decoding. TAALES measures simple lexical information (i.e. number of word and n-gram types where n-grams refer to multi-word units), word and n-gram frequency (i.e. how many times a lexical item or combination of lexical items occur in a larger, reference corpus), word range (i.e. how many documents in a reference corpus that a lexical item appears), word properties (e.g. familiarity, concreteness, meaningfulness), and strength of association between words (e.g. are two words strongly associated like ebb and flow).

Frequency and range indices in TAALES are calculated from available corpora such as SUBTLEXus [63] and the Corpus of Contemporary American English (COCA; [64]). Beyond frequency, bigram (two-word phrases) and trigram (three-word phrases) indices also measure proportion scores (i.e. the proportion of common n-grams found in a reference corpus). Word property information indices come from the Medical Research Council (MRC) psycholinguistic database [65] or other freely available databases such as age of acquisition scores (i.e. at what age people think words are learned) reported by [66].

TAACO

TAACO [61] reports on over 150 indices related to text cohesion. Sensitivity to cohesion structures in a text can help readers process and understand paragraphs and larger discourse segments [43,67]. For many cohesion indices, TAACO integrates a part of speech tagger found in the Stanford Parser [68] and synonym sets reported by the WordNet lexical database [69]. TAACO provides lexical overlap at the sentence (local cohesion) and paragraph (global cohesion) level for function words (e.g. pronouns) content words, and arguments (i.e. similar words such as familiar, family, and familiarity) and part of speech tags (e.g. nouns and verbs) as well for synonym overlap.

TAASSC

TAASSC ([59, 70]) measures both coarse and fine-grained clausal and phrasal complexity, both of which are related to syntactic complexity. Greater syntactic complexity can make it more difficult to organize main ideas within a text and assign thematic roles, leading to difficulties in parsing sentences effectively [71,72]. At the more granular level, TAASSC reports on indices of clausal complexity, phrasal complexity, and verb argument constructions. These indices include the frequency of verb-argument constructions and average number of direct object dependencies (i.e. the number of dependents per direct object be they adjectives or prepositional phrases).

Coh-Metrix

Coh-Metrix calculates many syntactic features not available in TAASSC. For this study, we were interested in syntactic measures that calculate the mean number of words before the main verb, the number of higher level constituents per phrases, and syntactic similarity between adjacent sentences (i.e. overlap in the structure and part of speech tags between sentences). Coh-Metrix also calculates a number of measure of lexical diversity (i.e. the variety of words produced) including D [73] and Maas [74]. Lastly, Coh-Metrix calculates Flesch Kincaid Grade Level [3]. The Flesch-Kincaid grade level formula is a recalculation of the Flesch Reading Ease formula [23] which calculated a text’s readability based on sentence length and number of syllables per word. The formula is

$$\text{Flesch Kincaid Grade Level}=+\left(11.8\times \text{number of syllables}/\text{number of words}\right)-15.59$$

Statistical analysis

The SM threads were dichotomized as being of either ‘low’ or ‘high’ complexity. Secure message (SM) threads scored by expert raters from 1–3 were classified as easier to understand (n = 471), while SM threads scored from 4–5 were classified as more difficult to understand (n = 253). Prior to statistical analyses, the linguistic variables were pruned. First, we removed those variables that were non-normally distributed. Our thresholds for normal distribution were skewness and kurtosis values of ±2 ([75]). Second, we removed all variables that did not report at least a small effect size (d = ~ .200) with the dependent variable (i.e. the low/high classification). Third, variables were checked for multicollinearity (defined as r> ±.700). If variables were multicollinear, the variable with the highest effect size was retained and the other variables were removed. After variable pruning, 85 variables remained.

These 85 variables were then used as variables in a linear discriminant analysis (LDA) to predict expert ratings of complexity and develop Model of Text Readability in Physicians (MoTeR-P). We tested the accuracy of the training model on an independent testing set that was not part of model building (a 66% set of SMs, n = 478). The model reported by this LDA was then used to predict group membership for the remaining 34% SMs (the test set, n = 246). Our data was not evenly balanced across easy and difficult categories because many texts were scored as 3 (neither easy nor difficult). To account for the imbalance in the data and its potential effects on the LDA algorithm, we set probability weights such that messages with an LDA probability weight of more than .55 were assigned to the easy category, instead of the default 0.50 probability. We also controlled for suppression effects in the algorithm by removing variables that showed flipped signs between their co-efficient scores and their t values. Lastly, to control for over-fitting, we set a threshold for a maximum number of predictors to inform the training model that corresponded to 1 predictor for every 20 items (i.e. a maximum of 24 predictors). We report results from extending the LDA to the testing set in terms of chi-square values, kappa scores, overall accuracy, specificity, and sensitivity. A separate LDA meant to provide a comparative analysis to predict expert ratings of text complexity was also conducted using Flesch Kincaid Grade Level.

Results

MoTeR-P

The final linear discriminant analysis (LDA) model for Model of Text Readability in Physicians (MoTeR-P) contained 24 features (see Table 1 for descriptive and statistical details for these features). The LDA model correctly classified 184 of the 246 messages in the test set, X² = 50.977, p < .001, for an accuracy of .749 (see Table 2 for the confusion matrix for the LDA model based on the Complexity Profile). The measure of agreement between the expert raters’ secure message (SM) categorization type and that assigned by the model produced a Cohen’s Kappa of 0.455, demonstrating moderate agreement. Sensitivity for the model was .674 while specificity was .788.

Table 1.

Descriptive statistics for variables included in LDA model.

Feature	t	p	d	Mean (SD): Difficult messages	Mean (SD): Easy messages	LDA co-efficient
Word range score, CW, SUBTLEXus	−7.897	< .001	−0.758	3.196 (0.172)	3.303 (0.123)	3.682
Word age of acquisition scores, AW, Kuperman	7.074	< .001	0.679	5.496 (0.39)	5.284 (0.26)	−0.291
Lexical diversity (D)	5.555	< .001	0.533	89.827 (23.659)	77.259 (23.543)	−0.002
FW overlap, paragraphs	4.854	< .001	0.466	8.225 (4.016)	6.379 (3.936)	−0.066
FW frequency, COCA academic	4.164	< .001	0.399	16831.399 (3494.824)
15496.32 (3257.931)	−0.001
Lexical diversity (Maas)	−4.022	< .001	−0.386	0.021 (0.003)	0.023 (0.004)	17.125
Bigram association strength (delta p), COCA academic	3.988	< .001	0.383	0.045 (0.012)	0.041 (0.01)	−7.369
Terms related to action	−3.474	< .001	−0.333	0.609 (0.133)	0.655 (0.142)	1.073
Syntactic similarity score	−3.429	< .001	−0.329	0.104 (0.031)	0.116 (0.039)	1.861
Word familiarity, AW, MRC	−3.408	< .001	−0.327	593.239 (4.894)	594.622 (3.825)	0.001
Fear and disgust words	3.275	< .001	0.314	0.118 (0.067)	0.097 (0.07)	−1.606
Verb overlap (binary), paragraphs	3.225	< .001	0.309	0.764 (0.257)	0.674 (0.311)	−0.890
Average number of direct object dependencies	3.051	< .010	0.293	1.253 (0.368)	1.148 (0.355)	−0.043
Words related to friends and family	−3.037	< .010	−0.291	0.191 (0.081)	0.217 (0.095)	2.099
Argument overlap (binary), paragraphs	2.853	< .010	0.274	0.579 (0.197)	0.524 (0.203)	−0.150
Words related to joy	−2.849	< .010	−0.273	0.617 (0.501)	0.788 (0.68)	0.225
Argument overlap, sentences	−2.708	< .010	−0.26	0.219 (0.078)	0.241 (0.09)	0.989
Trigram range, COCA fiction	2.697	< .010	0.259	0.028 (0.008)	0.026 (0.007)	−13.702
Trigram frequency, COCA newspaper	2.606	< .010	0.25	0.429 (0.118)	0.401 (0.111)	−0.985
Pronoun overlap, paragraphs	2.479	< .050	0.238	1.354 (0.66)	1.199 (0.648)	−0.175
Construction frequency, SD, COCA fiction	2.242	< .050	0.215	650503.988 (135651.74)	620837.079 (139117.53)	−0.001
Word concreteness, AW, MRC	−2.183	< .050	−0.209	2.642 (0.136)	2.669 (0.126)	0.910
Incidence of words related to objects	−2.086	< .050	−0.2	0.134 (0.053)	0.145 (0.061)	2.552
Argument overlap, paragraphs	−1.941	< .050	−0.186	0.383 (0.159)	0.413 (0.164)	1.177

AW = All words; CW = content words; FW = function words; SD = standard deviation; M = mean.

Table 2.

Confusion matrix for LDA results (test set).

	Difficult	Easy
Difficult	58	28
Easy	34	126

The coefficients for the linguistic features indicated that SMs judged as more difficult to understand contained more sophisticated lexical features including more difficult words (i.e. words that occurred in fewer texts, words with higher age of acquisition scores, less concrete words, and less familiar words) and displayed greater lexical diversity. Conversely, more difficult texts included more frequent tri-grams and tri-grams that occurred in a greater number of texts. Messages that were more difficult to understand also contained more frequent academic function words. Syntactically, SMs judged harder to understand had less syntactic overlap across sentences and had sentences that contained more dependencies. In terms of cohesion, more difficult texts had a greater number of function words (including pronouns) and verbs that overlapped across paragraphs. In terms of arguments (i.e. nouns), more difficult texts had less overlap at the sentence and paragraph level. In terms of sentiment, messages that were more difficult to understand contained more words related to fear and disgust, and fewer words related to joy and friends and family. The cognitive variables indicated that SMs that were more difficult to understand contained fewer words related to actions and objects (which correlates with word concreteness).

Comparative analyses

Significant differences were reported for Flesch-Kincaid Grade Level (FKGL) scores between easy to understand SMs (M = 7.248 grade level, SD = 2.153) and difficult to understand SMs (M = 8.144 grade level, SD = 2.376), t = 4.722, p < .001, d = .388. An LDA model based on the FKGL score correctly classified 160 of the 246 messages in the test set, X² = 6.560, p = .010, for an accuracy of .650 (see Table 3 for the confusion matrix for the Flesch Kincaid Grade Level model). The measure of agreement between the expert raters’ message SM categorization type and that assigned by the Flesch Kincaid model produced a Cohen’s Kappa of 0.154, demonstrating only weak agreement. Sensitivity for the model was .302 while specificity was .838.

Table 3.

Confusion matrix for Flesch-Kincaid Grade Level results (test set).

	Difficult	Easy
Difficult	26	60
Easy	26	134

Discussion

Limited health literacy (HL) is common in healthcare contexts and can undermine medical communication between physicians and patients. Importantly, unsuccessful communication on the part of physician or patient can have important effects on health outcomes [76–78]. One approach to this problem is to better tailor communication between physicians and their patients with limited HL in the hopes of improving health outcomes and reducing health-related disparities [1,13,14]. As a first step, this study seeks to measure the readability of physicians’ secure messages (SMs) sent to patients in an electronic health record in order to understand what linguistic aspects of the SMs make them easier or more difficult to understand by struggling readers.

The study provides further evidence that the difficulty of health-related written materials, in this case physicians’ SMs, can be assessed using linguistic features beyond those captured by traditional readability formulas such as Flesch-Kincaid Grade Level (FKGL) and provides support that the ease of comprehending written text is impacted by a complex and wide array of linguistic features [27,44–46,48,49] that can be better captured by natural language processing (NLP) tools. Effectively predicting text difficulty can provide a foundation for understanding what linguistic elements of a text may make it more or less difficult to understand for struggling readers, like those patients with low HL. This foundation can inform our understanding of lower HL and could eventually be used by physicians, administrators, and healthcare systems to guide text production in a manner that improves health communications by meeting the needs of populations with a range of HL, using HL strategies in interpersonal communications to confirm understanding, providing easier access to health information/services, designing and distributing content that is easy to understand and act on, and preparing workforces to be health literate [79]. Determining if models like Model of Text Readability in Physicians (MoTeR-P) can be used to help compose more comprehensible SMs so as to better match patients’ reading levels is an important next step. If greater linguistic concordance can be reached, future research can evaluate whether such concordance enhances patient portal communications, thereby promoting favorable patterns of adherence, patient safety and healthcare utilization [16,76,80–83].

Overall, this study found that physician SM difficulty was predicted by challenging words in the text. While FKGL also indexes word difficulty in its formula, it does so by relying solely on the average number of syllables in words, a proxy for word difficulty. In this study, we find that reading challenges in the SMs are predicted by a multitude of lexical indices including word frequency as well as age of acquisition, familiarity, concreteness, lexical diversity, and multi-word phrases (i.e. tri-grams). Similar findings have been reported in terms of lexical diversity [44] word familiarity, and multi-word phrase frequency [46] and word frequency [47]. As such, the use of more advanced NLP approaches not only affords more precise calculations of reading accuracy but also provides greater information about the lexical features in the text that lead to greater reading difficulty.

Our results also support the notion that physician SMs were harder to read if they contained more syntactic dependencies and less syntactic overlap across sentences. Neither of these features would be captured by the syntactic measures found in classic readability formulas, which focus solely on sentence length. Similar findings have been reported in previous studies of health text readability in that more readable texts contain fewer relative clauses (i.e. fewer embedded clauses) and fewer passive clauses [44].

A more principled problem with classic readability formulas is that they do not consider text cohesion features, which are important elements of text readability that logically connect sections of text [84–86]. At least one previous medical readability formula included a measure of cohesion, but this measure did not distinguish between texts categorized as easier or more difficult to read [46]. The findings from our study did find significant differences in cohesion features between easy and difficult texts. Specifically greater noun overlap increase text readability whereas function word and verb overlap decreased text readability. Intuitively this makes sense because the core meaning of many non-narrative texts lies more in the items discussed than the actions or the structural components surrounding those items. Thus, we would expect that greater overlap of nouns between sentences and paragraphs would increase cohesion whereas verb and function word overlap may not. It is also important to note that greater pronoun overlap was predictive of less readable text. Because pronouns have no explicit meaning and instead depend on readers’ ability to determine the pronoun’s previous referent (i.e. anaphoric resolution), it makes sense that struggling readers may find that greater overlap of pronouns between text segments makes the text less readable.

Lastly, classic readability formulas do not consider affective or cognitive variables. To our knowledge, no research has indicated that affective variables are important indicators of text readability. However, we included them because SMs between physicians and patients – where medical decisions related to personal health are concerned – may rely on positive affect on the part of the physician to reduce anxiety for the reader and, thus, indirectly make a text more readable for the patient. The readability formula developed in this study supports this assertion in that more readable texts contained fewer words related to fear and disgust and more words related to joy. In addition, since friends and family are an important component of any support network, especially medical support networks where friends and family members are more likely to be involved in decision making, the use of more words related to friends and family may help ground and internalize the message, making it more understandable. Additionally, mentions of friends and families may personalize messages, making them less objectionable and more readable. The cognitive variable related to object terms contained in the model overlap with our lexical variable of concreteness, providing additional support that more concrete/object-related words help produce a more readable text. The action word feature indicated that the production of more words related to actions (i.e. verbs) helps with readability. While difficult to interpret, at least one previous study [46] found that a greater number of verbs was associated with ease of text readability. It may be that more verbs help readers construct more accurate semantic relationships and/or greater temporal cohesion [87].

While we expected MoTeR-P to outperform FKGL, there are still interesting elements of the FKGL analysis to discuss. Specifically, unlike many other previous studies [5,18,36–39], our study found that the grade levels reported (7th grade or lower for the easy to understand texts and 8^th grade or higher for the more difficult texts) were aligned with what is considered average text difficulty and related text difficulty thresholds [15,16]. This may be a result of the specific domain (email-type SMs) or may reflect active strategies on the part of physicians to attempt to make their SMs understandable for an average reader. Notably, sensitivity was low while specificity was high such that FKGL was more likely to classify text, regardless of level, as easy. These results show that while the FKGL scores were in the average range for difficulty, the formula performed poorly at classifying texts as more or less difficult to read.

We recognize that achieving readability is only one step in a complex process of attempting to overcome a range of determinants of, and barriers to, optimal healthcare that tend to co-occur with limited HL. A recent conceptual model has illustrated the complex ways in which HL interacts with pathways connecting the social determinants of health to health disparities [88]. This model indicates that patient HL is situated not only as a mediator or effect modifier in a causal model, but also as an important factor in an ecological model that also includes characteristics and attributes of the health care system that define the contexts in which patients and physicians operate. We believe that the contribution of our current study is to provide health systems with an automated tool to begin to measure and systematically address one of the contextual factors that distinguish those healthcare organizations that are ‘health literate’ from those that are not: the ease with which their clinicians can be understood by patients who struggle with limited HL.

While we introduce and provide preliminary validation for a new readability formula for health-related text, we also recognize that text comprehension is greater than the sum of its linguistic parts. It is possible that we could increase the algorithm’s prediction accuracy with the addition of specific linguistic factors that may be especially important to HL, including the difficulty of medical acronyms and medical terms [44,47] and the quality of both explanatory or elicitative content that may be contained in physician text [7]. Additionally, the linguistic features that we examined may be insufficient to address the needs of struggling readers in diverse clinical settings; linguistic indicators of empathy, interactivity and shared meaning, for example, could be tested in future research. Beyond linguistic features, it would also be important to incorporate other variables relevant to HL, including patients’ background knowledge, age, overall reading ability and socio-cultural background [44]. Analyzing visual aspects of text, such as figures, charts, white space, and layout design [17], and measuring the content and function of the SMs (i.e. if the function of the SM was a scheduling request, a medication refill response, a report of test results, advice around self-management, or a response to a patient query) may influence the extent to which a secure message can be easily understood and acted on. While some of these non-linguistic features could prove feasible to collect and include in future readability formulas (e.g. demographic information, non-linguistic textual information), developing automated techniques to examine the content and function of SMs may be more challenging. However, incorporating such techniques has been the subject of recent research [89]. For instance, topic models could be developed using word embedding techniques such as latent Dirichlet allocation (LDA). It is possible that some topics would be more difficult for struggling readers than others. While assessing the function of a SM may prove more difficult, recent research in academic writing indicates success in teasing apart genre, discourse, and rhetorical moves that help indicate the function of writing [90,91]. Such approaches might be adapted to physicians’ SMs to develop future iterations and further enrich the work we have presented here.

It should be noted that our reading criterion relied on expert ratings of what struggling readers would perceive as understandable. We are currently collaborating with diverse type 2 diabetes (DM2) patients recruited from a public hospital system – some of whom are struggling readers – to examine whether expert ratings of SM complexity are concordant with patient ratings of these same messages. Relatedly, while the use of patient portals is rapidly expanding among all racial and socioeconomic subgroups, there is some evidence that SM may be less accessible to lower literacy readers. As such, our category of ‘easy to read’ may not be generalizable to all struggling readers and may explain the relatively low inter-rater reliability reported for this study. Future work to replicate our model using SMs obtained from public-sector health systems that care for a disproportionately larger number of struggling readers may lead to modifications in linguistic indices included or adaptations to cut-points to better serve the needs of the lowest level readers. We also recognize that a model that generates binary classifications may not best represent all applications and, while MoTeR-P could be applied to health-related text beyond SMs, its applicability in such settings is unknown. Finally, the work to date is limited to primary care physicians and their SMs. Future work should explore MoTeR-P with other members of the healthcare team who engage in SMs, from medical assistants to nurses to sub-specialists. We would like to include background information of the healthcare team members as predictors of readability given the available data and the use of appropriate machine learning approaches.

Advances in computational linguistics will likely enable the development of even more comprehensive measures that build on our new readability measure to assess the likelihood that patients who are struggling readers understand physicians’ communications. Future iterations might incorporate automated indicators of the content and function of emails, the quality of elicitative and explanatory content, the visual attributes of the communications, and the emotional tone employed. In parallel, the next steps in our research are first to apply MoTeR-P to the entire DISTANCE sample of physician SMs sent to patients in order to examine potential differences in SM readability based on physician characteristics, as well as differences both within and across physicians with respect to the extent to which they tailor SM complexity to meet the HL of individual patients. Second, because complex physician communication could lead to poorer patient adherence and worse health outcomes, we also intend to examine whether greater concordance between physician SM complexity as measured by MoTeR-P and patient HL readability formulas is predictive of a range of diabetes-related health outcomes. Third, we plan to explore whether physicians have characteristic ‘signatures’ with respect to achieving concordance across their patients so as to determine whether overall communication style is an important factor in predicting health outcomes above and beyond individual-level concordance. For example, it will be important to determine whether the patient outcomes of physicians whose communications reflect a ‘universal precautions’ pattern (wherein easily comprehensible language is employed regardless of a patient’s HL, [92]) differ from those of physicians whose communications reflect a ‘universal tailoring’ pattern (wherein the complexity of a physician’s language tends to match that of the patient’s HL level). Fourth, depending on our findings, we plan to use MoTeR-P to inform physician training and feedback efforts as part of pre-graduate, post-graduate, and continuous medical education efforts. Fifth, we plan on exploring the degree to which the expert ratings of SMs align with DM2 patients’ ratings of the same text, as a means to further validate MoTeR-P. Finally, an additional application of MoTeR-P is to develop a readability tool that could be practically implemented within health systems. Such a tool would provide real-time feedback to physicians about the readability of their SMs, which could lead to greater SM readability and improve patient comprehension and related health outcomes. Such a system not only could provide summative feedback about the readability of a SM but also specific suggestions about how texts could be modified to make them more comprehensible. Early work at manually simplifying medical texts has demonstrated that linguistic modifications can make texts more readable [39], and other researchers have sought automatic methods to modify texts [46,93]. This work will provide a foundation for scaling readability formulas into a practical health system-wide application to both optimize care for patients of all HL levels, as well as help reduce HL-related health disparities.

The ECLIPPSE Project set out to harness secure messages sent by diabetes patients to their primary care physician(s) to develop literacy profiles that can identify patients with limited health literacy in an automated way that avoids time-consuming and potentially sensitive questioning of the patient. Given the time and personnel demands intrinsic to current health literacy instruments, measuring health literacy has historically been extremely challenging. An automated literacy profile could provide an efficient means to identify subpopulations of patients with limited health literacy. Identifying patients likely to have limited health literacy could prove useful for alerting clinicians about potential difficulties in comprehending written and/or verbal instructions. Additionally, patients identified as having limited health literacy could be supported better by receiving follow-up communications to ensure understanding of critical communications, e.g. medication instructions, and promote adherence and increased shared meaning [29]. As such, our research to develop automated methods for health literacy assessment represents a significant accomplishment with potentially broad clinical and population health benefits in the context of health services delivery.