Skip to main content
BMJ Open logoLink to BMJ Open
. 2019 Sep 13;9(9):e029357. doi: 10.1136/bmjopen-2019-029357

General practitioners’ predictions of their own patients’ health literacy: a cross-sectional study in Belgium

Hannelore Storms 1,, Bert Aertgeerts 2, Frank Vandenabeele 1,3, Neree Claes 1,4
PMCID: PMC6747646  PMID: 31519674

Abstract

Objectives

To support patients in their disease management, providing information that is adjusted to patients’ knowledge and ability to process health information (ie, health literacy) is crucial. To ensure effective health communication, general practitioners (GPs) should be able to identify people with limited health literacy. To this end, (dis)agreement between patients’ health literacy and GPs’ estimations thereof was examined. Also, characteristics impacting health literacy (dis)agreement were studied.

Design

Cross-sectional survey of general practice patients and GPs undertaken in 2016–17.

Setting

Forty-one general practices in two Dutch-speaking provinces in Belgium.

Participants

Patients (18 years of age and older) visiting general practices. Patients were excluded when having severe impairments (physical, mental, sensory).

Main outcome measures

Patients’ health literacy was assessed with 16-item European Health Literacy Survey Questionnaire. GPs indicated estimations on patients’ health literacy using a simple scale (inadequate; problematic; adequate). (Dis)agreement between patients’ health literacy and GPs’ estimations thereof (GPs’ estimations being equal to/higher/lower than patients’ health literacy) was measured using Kappa statistics. The impact of patient and GP characteristics, including duration of GP–patient relationships, on this (dis)agreement was examined using generalised linear logit model.

Results

Health literacy of patients (n=1375) was inadequate (n=201; 14.6%), problematic (n=299; 21.7%), adequate (n=875; 63.6%). GPs overestimated the proportion patients with adequate health literacy: adequate (n=1241; 90.3%), problematic (n=130; 9.5%) and inadequate (n=4; 0.3%). Overall, GPs’ correct; over-/underestimations of health literacy occurred for, respectively, 60.9%; 34.2%; 4.9% patients, resulting in a slight agreement (κ=0.033). The likelihood for GPs to over-/underestimate patients’ health literacy increases with decreasing educational level of patients; and decreasing number of years patients have been consulting with their GP.

Conclusions

Intuitively assessing health literacy is difficult. Patients’ education, the duration of GP–patient relationships and GPs’ gender impact GPs’ perceptions of patients’ health literacy.

Keywords: general practice, health literacy, doctor-patient communication, literacy, health disparities, health knowledge


Strengths and limitations of this study.

  • This is an extensive study, with a large number of patients participating across several general practices.

  • Paper-based 16-item European Health Literacy Survey Questionnaires were used to encourage patients to fill out the health literacy survey.

  • Voluntary general practitioner and patient participation was potentially selective due to the relatively small research window, survey weariness or reluctance to disclose difficulties regarding health communication.

Introduction

Healthcare is facing enormous challenges. Care is shifting towards managing rather than curing diseases due to the ageing of the population and the rising prevalence of chronic diseases and multi-morbidity. As a result, doctor–patient interactions are transforming into partnerships in which patients’ needs, values and beliefs influence the course of the care process.1 2 Besides being responsive to patients’ needs, the complexity of healthcare forces healthcare professionals to also take into account patients’ health literacy (HL).

HL is defined as ‘one’s knowledge, motivation and competences to access, understand, appraise, and apply health information in order to make judgments and take decisions in everyday life concerning healthcare, disease prevention and health promotion to maintain or improve quality of life during the life course’.3 With the focus on patients managing their care, patients’ knowledge, motivation, understanding and skills with regard to health information are important prerequisites to guarantee high quality care.

HL impacts on accessibility and utilisation of care, doctor–patient communication, self-care and, subsequently, health outcomes.4–7 Research in eight European countries demonstrated that 47% of the population experiences difficulties processing health-related information, hence, has problematic or inadequate HL.8–14 Since the development of the conceptual framework of the European Health Literacy Survey (HLS-EU) consortium, HL research in Europe—and Belgium—is expanding.3 In Belgium, the most renowned studies have focused on the prevalence, with 40% of the Belgian population being low health literate; HL as an intermediary for tobacco use, health status, physical activity and the pricing of medication9 and on the use of healthcare services and the associated costs.14 This research of Vandenbosch et al. found no significant effect between someone’s HL and the use of healthcare services such as the emergency room or general practitioner (GP) consultations.14 However, people with lower HL levels were found to have significantly more hospitalisations and more GP visits at home than those with adequate HL.14

Overall, people with limited HL may find it difficult to understand medication instructions, have poorer medication adherence,7 10 15 16 use preventive services less,7 17 have low self-efficacy,18–20 struggle with self-managing chronic diseases7 and have a worse health status than those with adequate HL.7 21 Most at risk are non-native speakers, the elderly and those with limited education.10 13 22 23 To ensure patients receive appropriate care, healthcare professionals should make sure patients are (being) informed, have sufficient understanding, are given the opportunity to discuss treatment options and are involved in decision making.2

Awareness of patients’ understanding of health information and their ability—and willingness—to be involved in (decisions about) their care are essential to tailor information and guide patients through the healthcare system.24 25 GPs in particular are well-placed to ensure patients receive the care that meets their needs. By communicating effectively and facilitating patients’ involvement in their care process, GPs may contribute to improving health outcomes.4 7 24 However, this requires GPs to be able to identify patients with limited HL.

To date, there are few studies investigating healthcare professionals’ abilities to identify people with limited HL. Moreover, the Northern American studies that exist, demonstrated healthcare professionals’ inability to identify people with limited HL, predominantly overestimating HL.26–28 To assess HL, these studies relied on so called ‘objective’ (health) literacy tools (Newest Vital Sign (NVS),29 Rapid Estimate of Adult Literacy in Medicine (REALM),30 31 Rapid Estimate of Adult Literacy in Medicine-Revised (REALM-R)).32 Their sample sizes were relatively small, with the number of participants ranging from 65 to 182.26–28 Despite differences in study design, non-academic primary care physicians, residents and nurses all overestimated patients’ HL. They estimated HL to be adequate for, respectively, 74% patients; 90% patients; 68% patients.26–28 The corresponding κ (0.19 and 0.09) demonstrated the ‘little’ agreement between healthcare professionals’ estimations and patients’ actual HL.

The aim of this research was to explore the agreement between patients’ HL and GPs’ HL estimations thereof, as well as to examine characteristics impacting on this HL (dis)agreement.

Methods

Study design

Cross-sectional study with paper-based questionnaires (patients and GPs).

Recruitment

To recruit GPs, the researcher (HS) contacted 79 GPs working in two Belgian provinces of interest (Vlaams-Brabant and Limburg) (figure 1). Five GPs were in the direct network of the research team and immediately agreed to participate. In addition, 73 out of 122 GPs responsible for organising regional quality meetings for their peers, were contacted through phone calls and emails. This resulted in 19 invites for the researcher to attend these regional quality meetings. During these meetings, the researcher informed GPs about the HL conceptual framework, the purpose and study design of this research. Ultimately, 98 GPs expressed their intent to participate (with a varying number of participating GPs per general practice). The number of participating GPs reduced to 88 because GPs declined to participate before the research was set up (n=7) or because they were unable to carry out the research (n=3). Subsequently, GPs were excluded from data analysis when fewer than four questionnaires per GP were returned to the researcher (zero questionnaires (n=4), between one and three (n=3) and loss of completed questionnaires (n=1)) (figure 1). Recruitment and start-up took place between October 2016 and December 2017.

Figure 1.

Figure 1

Recruitment of general practitioners.

Sample size

This study used purposive sampling for the recruitment of GPs and patients. It was aimed to reach 2000 patients. Compared with the general population of the Dutch-speaking part of Belgium, women and people aged 65 years and older were over-represented, with 63% women (compared with 51% in the adult population of the Dutch-speaking region), and 31% people aged 65 years and older (compared with 25% in the adult population of the Dutch-speaking region).33

Setting and participants

In each general practice, the research window was set to be minimal 1 month. Participation (GPs and patients) was voluntary, written informed consent from the latter was obtained prior to medical consultation. Men and women, 18 years of age and older, consulting with a participating GP, were eligible for inclusion. Patients were excluded when having a severe cognitive impairment, sensory disability, psychological or psychiatric disorder. This exclusion was documented by the participating GPs.

Measures

Health literacy

HL of patients was assessed with 16-item European Health Literacy Survey Questionnaire (HLS-EU-Q16). Derived from the 47-item European Health Literacy Survey Questionnaire (HLS-EU-Q47),3 this 16-item version measures 11 of 12 sub-dimensions of HL, as defined by the conceptual model developed by the HLS-EU consortium.34 These dimensions result from integrating three health relevant domains (healthcare, disease prevention, health promotion) and four competencies relevant for the processing of health information (access, understand, appraise, apply).34 HLS-EU-Q16 highly correlates with the 47-item version, but it does not allow statements on the sub-dimensions of HL.34–36 The Dutch HLS-EU-Q16 has been used in Belgium9 and in the Netherlands.37 38

Items were formulated as questions (‘How easy would you say it is to find information on treatments of illnesses that concern you?’, ‘How easy would you say it is to understand your doctor’s or pharmacist’s instruction on how to take a prescribed medicine?’) Each question was rated on a 4-point Likert scale (very difficult; difficult; easy; and very easy). These scores were dichotomised, by coding responses 0 = ‘(very) difficult’ and 1 = ‘(very) easy’. After summing the answers, a score between 0 and 16 was obtained. Consequently, patients were categorised as having inadequate HL (scoring 0–8 points), problematic HL (scoring 9–12 points), adequate HL (scoring 13–16 points).

Where a patient marked two adjacent Likert scores, the lower (the one referring to experiencing difficulties) score was registered. Where a patient marked two Likert scores within two points of each other, the middle value was registered (eg, if 2 and 4 marked, 3 was registered).

As opposed to patients’ HL assessment, GPs’ estimations of their patients’ HL were restricted to indicating either inadequate; problematic; adequate HL on a simple scale. To this end, GPs were educated on the HL concept and the associated HLS-EU questionnaires, at least twice. Among others, GPs were informed about how categories of HL were determined, allowing them to scale their patients’ HL into one of the three categories.

Other variables

Patients self-reported their gender (male; female), age (continuous) and educational attainment (no formal education; primary education; secondary education; higher education). The duration of GP–patient relationship was documented by GPs, based on the number of years patients have been visiting them (<1 year; 1–5 years; 6–10 years; >10 years). To determine if patients were consulting their usual GP or not, GPs indicated whether they were substituting for a colleague (yes; no).

GPs’ gender (male; female), age and years since graduation (both continuous) were registered independently from patients’ surveys.

Data collection

Prior to participation, GPs were educated about HL. At that moment, the research in general and the GP survey in particular were presented. Subsequently, GPs could agree to participate. When participating GPs received the surveys, the concept HL was explained once again and instructions on how to fill out the GP survey were repeated. Boxes labelled with a participating GP’s name were set out in the waiting area. These boxes contained white envelopes, each with a consent form and both a patient and a GP survey, labelled with a unique number. Leaflets with pictogram instructions were distributed throughout the waiting area to inform patients about the research.

In the waiting room, prior to their consultation with a participating GP, patients would voluntarily choose to fill out the patient survey (patient characteristics and HLS-EU-Q16). During consultation, patients would give their GP the GP survey. Subsequently, at the end of a participating patient’s medical consultation, GPs registered patient’s age, duration of GP–patient relationship and their own HL estimation of that patient, in a separate GP survey. Patient and GP surveys were collected in separate envelopes, not only to encourage patients to answer truthfully, but also to ensure GPs had no access to patients’ responses.

Statistical analyses

Demographic data and HL (of patients and estimations by GPs) were analysed using descriptive statistics. Kappa statistics were calculated to measure agreement between patients’ HL and GPs’ estimations of these patients’ HL.39 A generalised linear logit model was used to assess if patient and GP characteristics impacted on GPs estimating HL higher/lower (over-/underestimation). GPs’ correct estimation (equal to patients’ HL) was used as the reference group. Patient variables included gender, age and education, the duration of GP–patient relationship. GP variables included gender, age, years graduated and substituting for a colleague. Missing values on the initially considered variables were excluded from analyses. Finally, analysis on variables with significant results encompassed only three variables (patients’ education, the duration of GP–patient relationship and GPs’ gender). Statistical significance was assessed as p<0.05. Data were analysed with R.40

Ethical approval and consent

Prior to participation, participants received a full explanation of the purpose of the study, their rights as participants, anonymity and confidentiality of the data collected.

Patient and public involvement

The study design of this research was discussed in the feasibility study on the Dutch HLS-EU-Q16, prior to this study.41 Based on these participants’ recommendations, adjustments were made to the patient survey’s layout. Patients did not interpret results nor did they contribute to the writing of the manuscript.

Results

1833 surveys were filled out and returned. The number of participants reduced to 1469 (and 80 GPs across 41 general practices) due to the exclusion of questionnaires (not eligible (n=161); data on HL were missing (n=203)) (figure 2). The exclusion of questionnaires because of missing HL data predominantly resulted from a lacking HL estimation by a GP. Consequently, 148 patients were excluded although patients’ HL was available. HL of these patients was inadequate (n=26; 17.6%), problematic (n=27; 18.2%) and adequate (n=95; 64.2%). Characteristics of excluded patients can be found in the online supplementary file.

Figure 2.

Figure 2

Flow diagram of excluded questionnaires.

Supplementary data

bmjopen-2019-029357supp001.pdf (451.1KB, pdf)

To analyse data using the generalised linear logit model, patients with missing values were excluded (14 and 70 missing values for patients’ gender and education; 14 for the duration of GP–patient relationship; 11 for GPs substituting for a colleague). Analyses were performed on a data set of 1375 patients (tables 1–3).

Table 1.

Patient characteristics

Overall sample Generalised logit model Patient’s HL
Inadequate Problematic Adequate
% (N) % (N) % (N) % (N) % (N)
Gender
 Female 63 (919) 63.9 (878) 60.7 (122) 63.2 (189) 64.8 (567)
 Missing values 1 (14) 0 (0) 0 (0) 0 (0) 0 (0)
Age
 18–24 years 4 (57) 3.9 (54) 5.5 (11) 5.0 (15) 3.2 (28)
 25–34 years 11 (154) 10.4 (143) 12.4 (25) 12.7 (38) 9.1 (80)
 35–44 years 13 (192) 13.2 (182) 12.9 (26) 13.4 (40) 13.3 (116)
 45–54 years 20 (294) 20.2 (278) 13.4 (27) 14.0 (42) 23.9 (209)
 55–64 years 21 (311) 21.5 (296) 25.4 (51) 19.7 (59) 21.3 (186)
 65–74 years 19 (283) 19.3 (265) 18.4 (37) 22.1 (66) 18.5 (162)
 75–84 years 10 (153) 9.7 (134) 9.0 (18) 11.4 (34) 9.4 (82)
 85–104 years 2 (25) 1.7 (23) 3.0 (6) 1.7 (5) 1.4 (12)
 Mean age (SD) 54.8 (16.5) 54.6 (16.4) 54.1 (17.4) 54.7 (17.7) 54.7 (15.7)
 Missing values 0 (0) 0 (0) 0 (0) 0 (0) 0 (0)
Educational attainment
 No formal education 3.9 (54) 3.9 (54) 7.0 (14) 5.0 (15) 2.9 (25)
 Primary education 7.6 (107) 7.6 (104) 11.9 (24) 8.0 (24) 6.4 (56)
 Secondary education 40.2 (563) 40.3 (554) 42.8 (86) 42.5 (127) 39.0 (341)
 Higher education 48.2 (675) 48.2 (663) 38.3 (77) 44.5 (133) 51.8 (453)
 Missing values 5 (70) 0 (0) 0 (0) 0 (0) 0 (0)
Duration of GP–patient relationship
 <1 year 9.8 (143) 9.5 (130) 11.9 (24) 9.0 (27) 9.0 (79)
 1–5 years 29.9 (435) 30.0 (412) 30.3 (61) 34.8 (104) 28.2 (247)
 6–10 years 15.4 (225) 15.5 (213) 14.4 (29) 14.0 (42) 16.2 (142)
 >10 years 44.8 (652) 45.1 (620) 43.3 (87) 42.1 (126) 46.5 (407)
 Missing values 1 (14) 0 (0) 0 (0) 0 (0) 0 (0)
Total N 1469 1375 201 299 875

GP, general practitioner; HL, health literacy.

Table 2.

GP characteristics

Overall sample Generalised logit model Patient’s HL
Inadequate Problematic Adequate
% (N) % (N) % (N) % (N) % (N)
Gender
Female 51.1 (750) 51.4 (668) 47.3 (95) 50.5 (151) 52.7 (461)
Missing values 0 (0) 0 (0) 0 (0) 0 (0) 0 (0)
Age
25–34 years 28.9 (425) 28.5 (392) 29.4 (59) 31.1 (93) 27.4 (240)
35–44 years 19.5 (287) 20.0 (275) 18.9 (38) 19.7 (59) 20.3 (178)
45–54 years 25.1 (368) 25.2 (346) 21.9 (44) 24.7 (74) 26.1 (228)
55–64 years 20.8 (305) 20.4 (281) 20.4 (41) 19.4 (58) 20.8 (182)
65–74 years 5.7 (84) 5.9 (81) 9.5 (19) 5.0 (15) 5.4 (47)
Missing values 0 (0) 0 (0) 0 (0) 0 (0) 0 (0)
Substituting for a colleague
yes 5.6 (82) 5.4 (74) 9.0 (18) 3.7 (11) 5.1 (45)
Missing values 0.7 (11) 0 (0) 0 (0) 0 (0) 0 (0)
Total N 1469 1375 201 299 875

GP, general practitioner; HL, health literacy.

Table 3.

GP HL estimation

Overall sample Generalised logit model Patient’s HL
Inadequate Problematic Adequate
% (N) % (N) % (N) % (N) % (N)
GP HL estimation
 Inadequate 0.3 (5) 0.3 (4) 1.0 (2)ο 0 (0) 0.2 (2)
 Problematic 9.5 (139) 9.5 (130) 17.9 (36)Δ 9.4 (28)ο 7.5 (66)
 Adequate 90.2 (1325) 90.3 (1241) 81.1 (163)Δ Δ 90.6 (271)Δ 92.2 (807)ο
Agreement between patients’ HL and GPs’ estimations thereof
 Overestimation 34.6 (508) 34 (470) 99 (199) 91 (271) N/A
 Equal estimation 60.7 (891) 61 (837) 1 (2) 9 (28) 92 (807)
 Underestimation 4.7 (70) 5 (68) N/A 0 (0) 8 (68)
Total N 1469 1375 201 299 875

GPs estimate HL equal toο or one level lower/one level higherΔ or two levels lower /two levels higherΔ Δ than patients’ actual HL.

GP, general practitioner; HL, health literacy.

Sample characteristics

Patients were on average 54.6 years old (SD 16.4); 63.9% were female; 48.2% were highly educated (table 1). Regarding the duration of GP–patient relationship, approximately 10% of the patients have been consulting their GP for less than 1 year, 45.1% for more than 10 years (table 1).

The mean age of GPs was 42.8 years old (SD 13.5); 53% of GPs were female (n=80). GPs were graduated on average 17.3 (SD 13.3) years and were working on average 14.3 (SD 13.4) years in their current practice, with a maximum of 48 years (table 2). Fifteen GPs were working solo, whereas the remaining GPs were working in group practices employing three to seven GPs. Six practices were located in four different regions with high population density, whereas ten practices were located in five multicultural regions.

Health literacy agreement

Patients’ HL was adequate, problematic and inadequate in, respectively, 63.6% (n=875), 21.7% (n=299) and 14.6% (n=201) of 1375 patients, whereas GPs overestimated the proportion of patients with adequate HL, while underestimating the proportion patients with problematic and, in particular, inadequate HL: adequate (n=1241; 90.3%), problematic (n=130; 9.5%) and inadequate (n=4; 0.3%) (figure 3). Correct estimation of HL by GPs, across all categories, occurred for 837/1375 (60.9%) of patients. HL levels were overestimated for 199+271/1375 (34.2%) of all patients, respectively, among patients with inadequate and problematic HL. HL levels were underestimated for 68/1375 (4.9%) of all patients; in fact, this only concerned patients with adequate HL (table 3). Based on these data, there was slight agreement between patients’ HL and GPs’ estimations thereof, κ=0.033 (95% CI, 0.00124 to 0.0648), p<0.05.

Figure 3.

Figure 3

General practitioner's (GP's) health literacy (HL) estimations and patient's HL (n=1375).

Characteristics impacting on health literacy (dis)agreement

Only three variables significantly impacted on the (dis)agreement between GPs’ estimations of and patients’ actual HL: patients’ education, the duration of GP–patient relationship and the gender of GPs.

Patients’ educational level

HL is more likely to be underestimated when a patient has no formal education (OR 5.58, 95% CI: 1.60 to 19.50); primary education (OR 14.13, 95% CI: 6.54 to 30.54); secondary education (OR 5.05, 95% CI: 2.65 to 9.61), compared with patients with higher education (table 4). GPs are more likely to overestimate patients’ HL in patients with primary education (OR 2.02, 95% CI: 1.22 to 3.13); secondary education (OR 1.34, 95% CI: 1.04 to 1.73), compared with patients with higher education (table 4).

Table 4.

ORs and 95% CIs for patient and GP characteristics impacting GPs’ estimations of patients’ HL (p<0.05)

HL disagreement * OR 95% CI
Patients’ education (referent category: higher education)
 No formal education Underestimation 5.58 1.60 to 19.50
 Primary education Underestimation 14.13 6.54 to 30.54
Overestimation 2.02 1.22 to 3.13
 Secondary education Underestimation 5.05 2.65 to 9.61
Overestimation 1.34 1.04 to 1.73
Duration of GP–patient relationships (referent category: >10 years)
 <1 year Underestimation 6.7 1.10 to 3.70
 1–5 years Underestimation 4.81 2.79 to 16.09
Overestimation 1.51 1.17 to 1.94
 6–10 years Underestimation 3.70 2.53 to 9.14
GPs’ gender (referent category: female)
 Male Underestimation 2.02 1.69 to 8.09

*GPs’ estimations versus patients’ HL, with reference group: GPs’ estimations of patients’ HL= patients’ HL.

†Underestimation: GPs’ estimations of patients’ HL < patients’ HL.

‡Overestimation: GPs’ estimations of patients’ HL > patients’ HL.

GP, general practitioner; HL, health literacy.

Duration of GP–patient relationship

The odds of underestimating HL are higher for patients who have been seeing their GP for a relatively short time: HL is more likely underestimated in patients consulting with their GP for less than 1 year (OR 6.7, 95% CI: 1.10 to 3.70); between 1 and 5 years (OR 4.81, 95% CI: 2.79 to 16.09); between 6 and 10 years (OR 3.70, 95% CI: 2.53 to 9.14) compared with patients who have been consulting with their GP for more than 10 years (table 4). Overestimation of HL is more likely in patients consulting with their GP between 1 and 5 years, compared with ‘more than 10 years’ (OR 1.51, 95% CI: 1.17 to 1.94) (table 4).

Gender of GPs

The odds of a male GP underestimating HL is 2.02 times the odds of a female GP underestimating patients’ HL (table 4).

Discussion

Our findings demonstrate that GPs’ estimations of patients’ HL showed a high rate of agreement with the assessment using HLS-EU-Q16 for patients with adequate literacy. However, GPs’ estimations showed a low level of agreement with the HLS-EU-Q16 for patients identified with inadequate or problematic literacy. Consequently, GPs considerably overestimated HL levels in patients identified with inadequate or problematic literacy using HLS-EU-Q16. Furthermore, GPs overestimate the proportion of patients with adequate HL and underestimate the proportion of patients with inadequate HL. GPs’ HL over-/underestimations are significantly affected by patients’ education and the duration of GP–patient relationships, as well as GPs’ gender. HL is more likely to be over-/underestimated with decreasing educational level, compared with higher education. The likelihood to over-/underestimate HL is also higher in patients who have been consulting their GP a relatively short period of time (less than 10 years), compared with patients who have been consulting their GP for over 10 years.

Comparison with previous studies

The majority of GPs perceived their patients to have adequate HL, although 10% of patients had inadequate HL.9 12 This corresponds to previous research in a hospital setting, demonstrating that these doctors’ HL overestimations outnumbered their underestimations with nearly 2 to 1.26 A similar outcome was reported, focusing on nurses.28 Only one study described primary care physicians overestimating patients’ HL.27

Education

GPs seem to seek guidance in patients’ educational level when estimating HL. We are, however, not sure patients ever explicitly disclosed their educational level to their GPs. Numerous studies have demonstrated the correlation between HL and education9 22 23: lower HL is observed in people with lower educational levels, although highly educated people may also have poor HL.12 23 Despite its correlation with HL, there’s a fundamental difference between literacy and HL, which is emphasised in several studies.9 12 23 Undoubtedly, literacy is a vital skill to function with or within the contemporary healthcare system. However, complementary, advanced skills are necessary to execute instructions, interact with healthcare professionals and critically appraise information.20 42 43 Our findings particularly indicate that for patients with primary education, there is an increased likelihood for GPs to overestimate, but also to underestimate patients' HL.

Duration of GP–patient relationship

Our findings indicate that a long-standing doctor–patient relationship helps GPs to get a better understanding of their patients, or at least of their HL. When seeing patients for a longer period of time, it is most likely GPs will have a better notion of their patients’ wishes, health beliefs and their preferences. Moreover, it enables them to get more insight into the lives of their patients: their living circumstances, meaningful relationships, social network, etc.

More specifically, HL of patients who have been consulting their GPs for more than 10 years is more likely to be correctly estimated. In contrast, chances are particularly high that HL of patients who have been consulting their GP less than 1 year is underestimated. This result could indicate a more cautious approach on the part of GPs, being less acquainted with these patients. Overestimation of HL, on the other hand, was only found in patients seeing their GPs between 1 and 5 years.

To explain these findings, we turned to studies focusing on continuity of care, as a key dimension of good primary care.44 Consequently, we are putting the duration of GP–patient relationship on an equal footing with (relational) continuity of care as described in previous studies. Findings presented in these studies support our results, as continuity of care correlates with improved adherence to physicians’ instructions45 and better communication.46 Moreover, being cared for less than 1 year by the same care provider was associated with decreasing communication excellence.47 Although (dis)satisfaction with communication may stem from being unaware of patients’ HL, we did not find studies linking their results to patients’ knowledge, their understanding of or the processing of health information.

Implications and recommendations for clinical practice

GPs in our research were often not able to estimate HL of their patients. We would recommend GPs to perform some sort of HL assessment, instead of going by intuition. Therefore, feasible alternatives for clinical practice should be considered,29 such as using single item questions,28 48 or prompt lists49 preferably tested by the target group to avoid comprehension problems,41 50 51 but, above all, by supporting patients to understand information. Besides asking patients directly about their understanding and the kind of information and/or (practical) support they might need, patients should be provided tools to ensure they understand, but can also recall what has been said.16 52 Two examples worth mentioning are the Ask Me 3 questions campaign53 or the use of the teach-back method.54 The former is designed to help patients receive appropriate information on: ‘What is my main problem?’; ‘What do I need to do (about the problem)?’; ‘Why is it important for me to do this?’. The latter refers to a method that consists of asking patients to repeat back what was just said (instructions, next steps to be taken). Based on their answers, it will be clear when there is a need for clarification. Moreover, educating (future) GPs and making them familiar with the HL concept and the implications of low HL are a prerequisite to address HL.55 GPs should get to know their patients. In particular it is important to know who is experiencing barriers to care and how to reach these patients. Being able to identify people with limited HL will help GPs to tackle health inequalities, for example by adequate information exchange. Hence, GPs should be equipped with a variety of strategies they can integrate in their day-to-day practice to communicate on a low HL level.55–57

Strengths and limitations

GP participation was potentially selective, although several attempts to contact and motivate GPs were undertaken. Selection bias may have occurred due to the relatively small research window. Some patients might not have had the chance to participate because of not visiting their GP, and some patients may have felt reluctant58 or ashamed to disclose HL information.50 59 60 Also, the voluntariness of patients to participate impacted on the study sample. This not only resulted in a small fraction of a GP's presumable patient population that took part in the research—the number of patients per GP was below 30, whereas we would expect a GP’s patient population to be around 1000 patients; it is also likely that participating in research is more easy for some patients (for instance, literate patients) than for others. Hence, the former would be more eager to participate than the latter. Nevertheless, many patients with low HL and low education participated. Survey weariness, the length or usability of a paper-based questionnaire may have discouraged patients from participating, particularly more vulnerable patients (illiterate, non-natives). Moreover, people experiencing barriers to care might have been missed. If so, their participation would have enriched data, especially if the reason for not visiting their GP or for not participating was linked to HL, for example because of low trust in GPs or poor self-related health.61 Some GPs had the impression that the profiles of the majority of participating patients did not reflect the diversity of their patient population. Patients were felt to be Dutch-speaking, literate and/or involved, empowered patients, predominantly without migration background (except for those general practices in the multicultural regions). This limitation also makes our findings more powerful. If HL overestimations imply that GPs assume these patients to function adequately in a healthcare setting, it would make those particular patients vulnerable to not accessing appropriate care. After all, if participating patients did not represent minority groups or patients with some vulnerability, GPs’ relatively high scores on patients’ HL indicate that GPs lack awareness on these ‘un-obvious’ patients to experience difficulties with health-related information. It would be interesting to examine how a more diverse sample—patients who are illiterate, with limited Dutch proficiency, with different backgrounds, with certain comorbidities, but also patients GPs visited at home14 and not solely within their practice—would impact results.

This research relies on HLS-EU-Q16 to measure comprehensive HL. As opposed to the original 47-item version, this 16-item version was developed for quicker assessment of HL. However, assessment with HLS-EU-Q16 does not allow statements on a HL sub-dimension. The output of these 16 items, covering only 11 of the 12 dimensions, is an overall HL score.3 Hence, GPs were required to give an overall HL score as well, instead of them being able to score a particular sub-dimension. Future research could aim to assess certain sub-dimensions and the (dis)agreement of GPs’ predictions with patients’ HL with respect to a particular sub-dimension. In this study, however, HLS-EU-Q16 was purposively chosen to allow easy assessment of HL in general population.

Responses on the HLS-EU-Q16 are prone to subjectivity because patients’—self-perceived—HL is assessed,62 as opposed to tools designed to examine people succeeding in specific problem-solving tasks that are regarded as more ‘objective’.62 Nonetheless, self-reported questions are considered valid and feasible methods to assess HL.63–65 Despite the differences between self-perceived and objective HL measurements, both benefit of being tailored to targeted groups. Unless it is part of the design, tools relying on vocabulary unfamiliar to the target group—consisting of terms this target group does not come across in everyday life—enhance the difficulty of a particular tool. Consequently, the output will reflect someone’s ability to understand the tool itself (its design, the questions, answer options, etc.), rather than one’s HL.50 Mindful of potential comprehension problems, feasibility of HLS-EU-Q16 was tested prior to this study.41 In correspondence with other studies, the level of abstraction or lacking experience regarding some health-related tasks in healthcare, health promotion or disease prevention, made it difficult to answer some items, but overall, HLS-EU-Q16 was considered a feasible instrument.41 50

Finally, almost 10% of participants were excluded from final analyses due to missing data on HL. However, looking into detail, 148 data were excluded solely because they lacked GPs’ HL estimations and not because patients’ HL could not be assessed. Excluded data often originated from GP surveys that were returned blank or incomplete. Possible explanations are that GPs did not receive a GP survey from their patients, or GPs on their part might have been lacking time to complete the survey.

Conclusion

Intuitively assessing patients’ HL is difficult. Patients’ education, the duration of GP–patient relationship as well as the gender of the GP impact on the estimations GPs make regarding patients’ HL. With decreasing educational levels, the likelihood for GPs to incorrectly estimate patients’ HL increases, suggesting patients’ education is not a good indicator for patients’ HL. Consequently, GPs should be aware of the HL concept and it being different from ‘literacy’ (i.e., education). It would be beneficial to facilitate and encourage GPs to get a profound understanding of their patients and their lives. A long-standing relationship between GPs and patients can contribute to GP–patient acquaintance. Based on our findings, consulting with a particular GP for more than 10 years may lead to GPs making more correct estimations on these patients’ HL. With healthcare being redesigned to be more integrated comes an opportunity to promote in-depth communication as the cornerstone for everyone to access adequate care. To be incorporated in daily practice, GPs should be allowed to invest a sufficient amount of time in getting to know their patients.

Supplementary Material

Reviewer comments
Author's manuscript

Acknowledgments

We thank GPs, their support staff and patients at all general practices, without whom this study would not have been possible. We greatly appreciate the support and the contribution of the consultants of I-BioStat, Center for Statistics, UHasselt. Last but not least, a special thanks to those helping out with the logistics.

Footnotes

Contributors: HS was responsible for study concept and design (with valuable input from BA and NC) and for data acquisition and initial analyses. Data interpretation with all authors (HS, BA, FV, NC) led to a manuscript drafted by HS. All authors substantially contributed to critically reviewing the manuscript and all read and approved the final version. All authors had full access to all of the study data and take responsibility for the integrity and the accuracy of the data. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests: None declared.

Patient consent for publication: Not required.

Ethics approval: This study was approved by the Ethical Committee of Hasselt University (CME2015/553). Prior to participation, participants received a full explanation of the purpose of the study, their rights as participants, anonymity and confidentiality of collected data.

Provenance and peer review: Not commissioned; externally peer reviewed.

Data availability statement: Data are available upon reasonable request.

References

  • 1. Epstein RM, Fiscella K, Lesser CS, et al. . Why the nation needs a policy push on patient-centered health care. Health Aff 2010;29:1489–95. 10.1377/hlthaff.2009.0888 [DOI] [PubMed] [Google Scholar]
  • 2. Epstein RM, Street RL. The values and value of patient-centered care. Ann Fam Med 2011;9:100–3. 10.1370/afm.1239 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Sørensen K, Van den Broucke S, Fullam J, et al. . Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health 2012;12:80 10.1186/1471-2458-12-80 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Stewart MA. Effective physician-patient communication and health outcomes: a review. CMAJ 1995;152:1423–33. [PMC free article] [PubMed] [Google Scholar]
  • 5. Lindau ST, Tomori C, Lyons T, et al. . The association of health literacy with cervical cancer prevention knowledge and health behaviors in a multiethnic cohort of women. Am J Obstet Gynecol 2002;186:938–43. 10.1067/mob.2002.122091 [DOI] [PubMed] [Google Scholar]
  • 6. Paasche-Orlow MK, Wolf MS. The causal pathways linking health literacy to health outcomes. Am J Health Behav 2007;31:19–26. 10.5993/AJHB.31.s1.4 [DOI] [PubMed] [Google Scholar]
  • 7. Berkman ND, Sheridan SL, Donahue KE, et al. . Low health literacy and health outcomes: an updated systematic review. Ann Intern Med 2011;155:97 10.7326/0003-4819-155-2-201107190-00005 [DOI] [PubMed] [Google Scholar]
  • 8. Vancorenland S, Avalosse H, Verniest R, et al. . Bilan des connaissances des Belges en matière de santé. MC-Informations 2014;258:48–55. [Google Scholar]
  • 9. Van den Broucke S, Renwart A. La littératie en santé en Belgique: un médiateur des inégalités sociales et des comportements de santé. Université Catholique de Louvain, 2014: 22. [Google Scholar]
  • 10. Institute of Medicine Health literacy: a prescription to end confusion. Washington, D.C: National Academies Press, 2004: 366. [PubMed] [Google Scholar]
  • 11. Kirsch IS, Jungeblut A, Jenkins L, et al. . Adult literacy in America: a first look at the results of the National adult literacy survey. office of educational research and improvement. Washington, D.C.: U.S. Dept. of Education, 1993: 178. [Google Scholar]
  • 12. Kickbush I, Pelikan JM, Apfel F, et al. . Health literacy: the solid facts. Copenhagen: World Health Organisation (WHO): Regional Office for Europe, 2013. [Google Scholar]
  • 13. Sørensen K, Pelikan JM, Röthlin F, et al. . Health literacy in Europe: comparative results of the European health literacy survey (HLS-EU). Eur J Public Health 2015;25:1053–8. 10.1093/eurpub/ckv043 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Vandenbosch J, Van den Broucke S, Vancorenland S, et al. . Health literacy and the use of healthcare services in Belgium. J Epidemiol Community Health 2016;70:1032–8. 10.1136/jech-2015-206910 [DOI] [PubMed] [Google Scholar]
  • 15. Shrank WH, Avorn J. Educating patients about their medications: the potential and limitations of written drug information. Health Aff 2007;26:731–40. 10.1377/hlthaff.26.3.731 [DOI] [PubMed] [Google Scholar]
  • 16. Brooks C, Ballinger C, Nutbeam D, et al. . Nursing and allied health professionals’ views about using health literacy screening tools and a universal precautions approach to communication with older adults: a qualitative study. Disabil Rehabil 2019;25:1–7. 10.1080/09638288.2018.1538392 [DOI] [PubMed] [Google Scholar]
  • 17. Miller DP, Brownlee CD, McCoy TP, et al. . The effect of health literacy on knowledge and receipt of colorectal cancer screening: a survey study. BMC Fam Pract 2007;8:16 10.1186/1471-2296-8-16 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Baker DW, Parker RM, Williams MV, et al. . The health care experience of patients with low literacy. Arch Fam Med 1996;5:329–34. 10.1001/archfami.5.6.329 [DOI] [PubMed] [Google Scholar]
  • 19. Sarkar U, Fisher L, Schillinger D. Is self-efficacy associated with diabetes self-management across race/ethnicity and health literacy? Diabetes Care 2006;29:823–9. 10.2337/diacare.29.04.06.dc05-1615 [DOI] [PubMed] [Google Scholar]
  • 20. Nutbeam D. Health literacy as a public health goal: a challenge for contemporary health education and communication strategies into the 21st century. Health Promot Int 2000;15:259–67. 10.1093/heapro/15.3.259 [DOI] [Google Scholar]
  • 21. Baker DW. Health literacy and mortality among elderly persons. Arch Intern Med 2007;167:1503 10.1001/archinte.167.14.1503 [DOI] [PubMed] [Google Scholar]
  • 22. van der Heide I, Rademakers J, Schipper M, et al. . Health literacy of Dutch adults: a cross sectional survey. BMC Public Health 2013;13:179 10.1186/1471-2458-13-179 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23. van der Heide I, Wang J, Droomers M, et al. . The relationship between health, education, and health literacy: results from the Dutch adult literacy and life skills survey. J Health Commun 2013;18:172–84. 10.1080/10810730.2013.825668 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. Schillinger D, Piette J, Grumbach K, et al. . Closing the loop: physician communication with diabetic patients who have low health literacy. Arch Intern Med 2003;163:83–90. [DOI] [PubMed] [Google Scholar]
  • 25. Bergeson SC, Dean JD. A systems approach to patient-centered care. JAMA 2006;296 10.1001/jama.296.23.2848 [DOI] [PubMed] [Google Scholar]
  • 26. Bass PF, Wilson JF, Griffith CH, et al. . Residentsʼ ability to identify patients with poor literacy skills. Academic Medicine 2002;77:1039–41. 10.1097/00001888-200210000-00021 [DOI] [PubMed] [Google Scholar]
  • 27. Kelly PA, Haidet P. Physician overestimation of patient literacy: a potential source of health care disparities. Patient Educ Couns 2007;66:119–22. 10.1016/j.pec.2006.10.007 [DOI] [PubMed] [Google Scholar]
  • 28. Dickens C, Lambert BL, Cromwell T, et al. . Nurse overestimation of patients' health literacy. J Health Commun 2013;18:62–9. 10.1080/10810730.2013.825670 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Weiss BD. Quick assessment of literacy in primary care: the newest vital sign. The Annals of Family Medicine 2005;3:514–22. 10.1370/afm.405 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30. Davis TC, Long SW, Jackson RH, et al. . Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam Med 1993;25:391–5. [PubMed] [Google Scholar]
  • 31. Davis TC, Gazmararian J, Kennen EM. Approaches to improving health literacy: lessons from the field. J Health Commun 2006;11:551–4. 10.1080/10810730600835517 [DOI] [PubMed] [Google Scholar]
  • 32. Bass PF, Wilson JF, Griffith CH. A shortened instrument for literacy screening. J Gen Intern Med 2003;18:1036–8. 10.1111/j.1525-1497.2003.10651.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33. Directorate-General Statistics Belgium Population by place of residence, nationality and age, 2019. [Google Scholar]
  • 34. Pelikan JM, Röthlin F, Ganahl K. Measuring comprehensive health literacy in general populations: validation of instrument, indices and scales of the HLS-EU study, 6th Annual Health Literacy Research Conference. Bethesda, Maryland, USA, 2014. 3-4, 2014 https://www.​bumc.​bu.​edu/​heal​thli​tera​cyco​nferencehealthliteracyconference/​files/​2014/​06/​Pelikan-​et-​al-​HARC-​2014-​fin.​pdf. [Google Scholar]
  • 35. Röthlin F, Pelikan J, Ganahl K. Die Gesundheitskompetenz Jugendlichen in Österreich, 2013. [Google Scholar]
  • 36. HLS-EU-Consortium HLS-EU-Q measurement of health literacy in Europe: HLS-EU-Q47; HLS-EU-Q16; and HLS-EU-Q86. Exec Agency Heal Consum 2012:1–14. [Google Scholar]
  • 37. Pander Maat H, Essink-Bot M-L, Leenaars KEF, et al. . A short assessment of health literacy (SAHL) in the Netherlands. BMC Public Health 2014;14:990 10.1186/1471-2458-14-990 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38. Fransen MP, Leenaars KEF, Rowlands G, et al. . International application of health literacy measures: adaptation and validation of the newest vital sign in the Netherlands. Patient Educ Couns 2014;97:403–9. 10.1016/j.pec.2014.08.017 [DOI] [PubMed] [Google Scholar]
  • 39. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37–46. 10.1177/001316446002000104 [DOI] [Google Scholar]
  • 40. R Development Core Team R: A Language and Environment for Statistical Computing [Internet. Vienna, Austria: R Foundation for Statistical Computing, 2008. http://www.r-project.org/ [Google Scholar]
  • 41. Storms H, Claes N, Aertgeerts B, et al. . Measuring health literacy among low literate people: an exploratory feasibility study with the HLS-EU questionnaire. BMC Public Health 2017;17:475 10.1186/s12889-017-4391-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42. Heijmans M, Waverijn G, Rademakers J, et al. . Functional, communicative and critical health literacy of chronic disease patients and their importance for self-management. Patient Educ Couns 2015;98:41–8. 10.1016/j.pec.2014.10.006 [DOI] [PubMed] [Google Scholar]
  • 43. van der Heide I, Heijmans M, Schuit AJ, et al. . Functional, interactive and critical health literacy: varying relationships with control over care and number of GP visits. Patient Educ Couns 2015;98:998–1004. 10.1016/j.pec.2015.04.006 [DOI] [PubMed] [Google Scholar]
  • 44. Starfield B, Shi L, Macinko J. Contribution of primary care to health systems and health. Milbank Q 2005;83:457–502. 10.1111/j.1468-0009.2005.00409.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45. Dietrich AJ, Marton KI. Does continuous care from a physician make a difference? J Fam Pract 1982;15:929–37. [PubMed] [Google Scholar]
  • 46. Potter SJ, McKinlay JB. From a relationship to encounter: an examination of longitudinal and lateral dimensions in the doctor–patient relationship. Soc Sci Med 2005;61:465–79. 10.1016/j.socscimed.2004.11.067 [DOI] [PubMed] [Google Scholar]
  • 47. Katz DA, McCoy K, Sarrazin MV. Does improved continuity of primary care affect Clinician–Patient communication in Va? J Gen Intern Med 2014;29:682–8. 10.1007/s11606-013-2633-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48. Wallace LS, Cassada DC, Rogers ES, et al. . Can screening items identify surgery patients at risk of limited health literacy? J Surg Res 2007;140:208–13. 10.1016/j.jss.2007.01.029 [DOI] [PubMed] [Google Scholar]
  • 49. Sansoni J, Grootemaat P, Duncan C, et al. . A systematic literature review on question prompt Lists in health care (final report), 2014. [Google Scholar]
  • 50. Domanska OM, Firnges C, Bollweg TM, et al. . Do adolescents understand the items of the European Health Literacy Survey Questionnaire (HLS-EU-Q47) – German version? Findings from cognitive interviews of the project “Measurement of Health Literacy Among Adolescents” (MOHLAA) in Germany. Arch Public Health 2018;76 10.1186/s13690-018-0276-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51. Murugesu L, Heijmans M, Fransen M, et al. . Beter omgaan met beperkte gezondheidsvaardigheden in de curatieve zorg: kennis, methoden en tools [Dealing with limited health literacy in curative care: knowledge, methods and tools. Utrecht, 2018: 67. [Google Scholar]
  • 52. Kim SH, Utz S. Association of health literacy with health information-seeking preference in older people: a correlational, descriptive study. Nurs Health Sci 2018;20:355–60. 10.1111/nhs.12413 [DOI] [PubMed] [Google Scholar]
  • 53. Partnership for Clear Health Communication Ask me 3: good questions for your health, 2007. [Google Scholar]
  • 54. Tamura-Lis W. Teach-Back for quality education and patient safety. Urol Nurs 2013;2013:267–71. 10.7257/1053-816X.2013.33.6.267 [DOI] [PubMed] [Google Scholar]
  • 55. Kripalani S, Weiss BD. Teaching about health literacy and clear communication. J Gen Intern Med 2006;21:888–90. 10.1111/j.1525-1497.2006.00543.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56. Mackert M, Ball J, Lopez N. Health literacy awareness training for healthcare workers: improving knowledge and intentions to use clear communication techniques. Patient Educ Couns 2011;85:e225–8. 10.1016/j.pec.2011.02.022 [DOI] [PubMed] [Google Scholar]
  • 57. Karuranga S, Sørensen K, Coleman C, et al. . Health literacy competencies for European health care personnel. HLRP: Health Literacy Research and Practice 2017;1:e247–56. 10.3928/24748307-20171005-01 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58. Ryan JG, Leguen F, Weiss BD, et al. . Will patients agree to have their literacy skills assessed in clinical practice? Health Educ Res 2008;23:603–11. 10.1093/her/cym051 [DOI] [PubMed] [Google Scholar]
  • 59. Parikh NS, Parker RM, Nurss JR, et al. . Shame and health literacy: the unspoken connection. Patient Educ Couns 1996;27:33–9. 10.1016/0738-3991(95)00787-3 [DOI] [PubMed] [Google Scholar]
  • 60. Weiss BD. Health literacy and patient safety: help patients understand (manual for clinicians). American Medical association Foundation and. Chicago: American Medical Association, 2007. [Google Scholar]
  • 61. Verlinde E, Poppe A, DeSmet A, et al. . Social differences in postponing a general practitioner visit in Flanders, Belgium: which low-income patients are most at risk? Health Soc Care Community 2013;21:364–72. 10.1111/hsc.12027 [DOI] [PubMed] [Google Scholar]
  • 62. Pelikan JM, Röthlin F, Ganahl K. Comparative report of health literacy in eight EU member states. The European health literacy survey HLS-EU, 2012. [Google Scholar]
  • 63. Chew LD, Bradley KA, Boyko EJ. Brief questions to identify patients with inadequate health literacy. Fam Med 2004;36:588–94. [PubMed] [Google Scholar]
  • 64. Chew LD, Griffin JM, Partin MR, et al. . Validation of screening questions for limited health literacy in a large Va outpatient population. J Gen Intern Med 2008;23:561–6. 10.1007/s11606-008-0520-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65. Powers BJ, Trinh JV, Bosworth HB. Can this patient read and understand written health information? JAMA 2010;304 10.1001/jama.2010.896 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary data

bmjopen-2019-029357supp001.pdf (451.1KB, pdf)

Reviewer comments
Author's manuscript

Articles from BMJ Open are provided here courtesy of BMJ Publishing Group

RESOURCES