Skip to main content
PMC home page
BMJ Open logoLink to BMJ Open
. 2016 Nov 9;6(11):e013166. doi: 10.1136/bmjopen-2016-013166

Effect of an evidence-based website on healthcare usage: an interrupted time-series study

Wouter A Spoelman 1, Tobias N Bonten 1, Margot W M de Waal 1, Ton Drenthen 2, Ivo J M Smeele 2, Markus M J Nielen 3, Niels H Chavannes 1
PMCID: PMC5128895  PMID: 28186945

Abstract

Objectives

Healthcare costs and usage are rising. Evidence-based online health information may reduce healthcare usage, but the evidence is scarce. The objective of this study was to determine whether the release of a nationwide evidence-based health website was associated with a reduction in healthcare usage.

Design

Interrupted time series analysis of observational primary care data of healthcare use in the Netherlands from 2009 to 2014.

Setting

General community primary care.

Population

912 000 patients who visited their general practitioners 18.1 million times during the study period.

Intervention

In March 2012, an evidence-based health information website was launched by the Dutch College of General Practitioners. It was easily accessible and understandable using plain language. At the end of the study period, the website had 2.9 million unique page views per month.

Main outcomes measures

Primary outcome was the change in consultation rate (consultations/1000 patients/month) before and after the release of the website. Additionally, a reference group was created by including consultations about topics not being viewed at the website. Subgroup analyses were performed for type of consultations, sex, age and socioeconomic status.

Results

After launch of the website, the trend in consultation rate decreased with 1.620 consultations/1000 patients/month (p<0.001). This corresponds to a 12% decline in consultations 2 years after launch of the website. The trend in consultation rate of the reference group showed no change. The subgroup analyses showed a specific decline for consultations by phone and were significant for all other subgroups, except for the youngest age group.

Conclusions

Healthcare usage decreased by 12% after providing high-quality evidence-based online health information. These findings show that e-Health can be effective to improve self-management and reduce healthcare usage in times of increasing healthcare costs.

Keywords: PRIMARY CARE, PUBLIC HEALTH, eHealth, health care utilization, interrupted time series, health communication

Strengths and limitations of this study.

  • Long-term follow-up (6 years) in a large number of healthcare consultations (18.1 million).

  • An interrupted time series analysis is the strongest research design when randomisation is not possible.

  • A change in trend of the outcome before and after the intervention and additional use of a reference group supports a causal effect of the intervention.

  • Actual healthcare usage rates based on registries were used, instead of self-reported healthcare use.

  • Other events might have affected healthcare usage during our study period, but were carefully considered and analysed in sensitivity analyses which showed similar results as the primary analysis.

Introduction

Healthcare costs and usage are rising in western countries and projected to increase even faster due to the ageing population.1–3 Healthcare workers and policymakers are seeking ways to control healthcare costs and usage while simultaneously improving healthcare quality. An important reason for patients to use healthcare is the need for information.4 Many patients look online for health information and use this information to decide whether or not to visit a doctor.5 6 So, in theory, patients using online health information are well informed and make a better assessment on the need to visit a doctor, leading to lower usage of healthcare. However, online health information is of inconsistent quality7 and evidence for the effects on healthcare usage is scarce. Previous studies showed heterogeneous results: online information did not change or even increased healthcare usage in some studies.8–17 An important limitation of these studies is that most estimated healthcare usage by patient self-reporting instead of actual healthcare usage statistics. Furthermore, the effects of online health information might be different in subgroups of sex, socioeconomic status (SES) and age. Online health information seekers are mostly young women with a high SES.5 18–21 A recent study, investigating the effect of an online support programme on healthcare usage in infants, found no difference in this youngest-age subgroup.16

The Dutch College of General Practitioners (NHG) launched a non-commercial evidence-based public website in the Netherlands in March 2012.22 The NHG is the Dutch scientific society of general practitioners (GPs), and publishes and updates nationally implemented evidence-based guidelines.23 The content on the website is based on those guidelines. The primary goal of the website was to provide reliable and understandable medical information.22 Since its launch, the website has grown to become one of the most visited healthcare websites in the Netherlands. However, its effect on healthcare usage is not yet known. We therefore conducted a nationwide study to test the hypothesis that providing evidence-based online health information is associated with a reduction of healthcare usage. Second, we investigated whether the effect on healthcare usage differed by sex, age and SES.

Methods

Design, participants and care setting

We used a natural before–after quasi-experimental design to study the number of consultations in primary care before and after the launch of the website (http://www.thuisarts.nl) in March 2012. Observational data were used from the NIVEL Primary Care Database (http://www.nivel.nl/en/dossier/nivel-primary-care-database) from 2009 to 2014. This time period provides data points from 3 years before and nearly 3 years after the website launch, giving enough data points to evaluate changes in trends.24 25

The study was situated in the primary care system of the Netherlands. It consists of 5068 primary care practices for nearly 17 million inhabitants. Dutch GP consultations are fully covered by the mandatory basic insurance for all inhabitants. GPs function as gatekeepers for specialised care and thus are important as a first-line contact for patients seeking health information and play a central role in healthcare usage.

To ensure data quality and prevent confounding by registration quality, only GPs were included who recorded more than 70% of their consultations with International Classification of Primary Care (ICPC) codes and provided data for the entire calendar year. Nationally representative data26 of on average 230 GPs were sampled in each year, representing 911 177 inhabitants (5.4% of the total Dutch population). GP patient contacts and diagnoses are coded according to the ICPC-1.27 Data were anonymised, but contained records for sex, age and zip codes for each patient. Dutch law allows the use of electronic health records for research purposes under certain conditions. According to this legislation, neither obtaining informed consent from patients nor approval by a medical ethics committee is obligatory for this type of observational studies containing no directly identifiable data (Dutch Civil Law, Article 7:458).

Data sharing

Technical appendix, statistical code and data set available from the last author (NHC).

Intervention

The website (http://www.thuisarts.nl) was launched in March 2012 by the Dutch minister of health, welfare and sports. Since then, it is a publicly available non-commercial website, providing reliable medical information and advice.22 All information and advice are based on the guidelines of the NHG and the online advice matches the most recent GP guidelines.23 Guideline updates are synchronised with patient situations on Thuisarts.nl by a team of professionals.22 The online content is thus evidence-based, is non-commercial, uses plain understandable language and covers over 600 topics. Each Thuisarts topic consists of several ‘patient situations’; for example: I need to start inhalation medication. Information on what to do and when to contact the GP is given. The situations can include illustrations, short videos, patient decision aids and e-health self-management tools like a self-test on alcohol use. The popularity of the website has grown exponentially, having currently 3 million unique page views each month.

Measurements

The primary outcome was defined as the trend in total number of consultations per 1000 patients per month before and after launch of the website. The total number of consultations was calculated in each month and divided by the number of quarterly enlisted patients, multiplied by 1000. Since the outcome was expected to have a seasonal fluctuation, a monthly interval was chosen to enable statistical control for seasonality as described below. The secondary outcome was the trend in consultation rate for a selection of the top 10 popular topics on the website. Top 10 of the topic pages was created before starting the analyses and was based on the number of unique page views, which were extracted using google analytics. A unique page view was defined as the number of times a page was loaded by a visitor's browser. If a visitor viewed a page several times during a single session, it only counted as a one page view. The following topics (ICPC codes) were selected using this criterion before conducting the analyses: herpes zoster (S70), lower back pain (L02 and L03), bladder infection in women (U71), vaginal discharge (X14), shoulder symptoms (L08 and L92), gout (T92), constipation (D12), irritable bowel syndrome (D93), diarrhoea (D11, D70 and D73) and sinusitis (R75).

Primary analysis

The primary analysis consists of a comparison of healthcare use before and after the launch of the website. So the most important control period was the period before release of the website. However, since changes in the healthcare system could have affected healthcare usage after release of the website, we additionally incorporated a reference group with health topics that were not exposed to the intervention. These topics were selected before conducting the analyses and included when no online information was available or page views of the topic were very low on the website during the study period. These topics were head trauma (N79 and N80), premenstrual syndrome (X09) and contact eczema (S88).

Subgroup and sensitivity analyses

Prespecified subgroup analyses were performed for different types of consultations: short consultations (20 min or less), long consultations (more than 20 min) and consultations by phone. Furthermore, subgroups of sex, SES and age were analysed. SES groups were created based on the available zip codes and using data of the Netherlands Institute for Social Research. These SES scores are based on postal codes (area of residence) and provided as summary scores by the Netherlands Institute for Social Research.28 Furthermore, four sensitivity analyses were performed. First, since website page views increased exponentially, we considered using a lag-time period after the release of the website. Therefore, we calculated the relative and absolute level change 12 and 24 months after the intervention. The absolute level change between 12 and 24 months can thus be considered as the effect of the website with a lag-time of 12 months after the release. In the second sensitivity analysis, we included only the GPs recording >95% of their consultations under an ICPC code, because an improvement in recording over time could have confounded the consultation rate of the secondary outcome. GPs participating over the entire study period were included in the third sensitivity analysis, because a change of participating GP characteristics over time could have confounded the consultation rate over time. Fourth, owing to changes in insurance policies, consultations performed for chronic care (asthma, chronic obstructive pulmonary disease, diabetes and cardiovascular risk management) by nurse practitioners fluctuated during the study period. To exclude any influence of these fluctuations, we performed the primary analyses separately for chronic and non-chronic care consultations.

Statistical analysis

All analyses, including subgroup analyses, were predefined in an analysis plan before the start of the analyses. The reference groups ‘premenstrual syndrome’ and ‘contact eczema’ were added post hoc to strengthen the reference group comparison. To evaluate the longitudinal impact of the introduction of the website, an interrupted time series (ITS) analysis was used. ITS is considered a strong design when randomisation is not possible and can thus be used to study causal effects using an observational ‘natural experiment’ approach.29 ITS measures the change in outcome, taking pre-existing trends in the data into account, thereby negating the effect of other factors influencing the outcome, provided that those factors remain constant.24 25 So the primary outcome's most important control period was the trend in consultation rate before release of the website. To additionally control for factors that did not remain constant after the release of the website, a reference group was created for consultations not exposed to the intervention.29

The Cochrane Collaboration guideline on ITS analyses was used, applying an autoregressive integrated moving average (ARIMA) model.24 For the primary and secondary outcomes, the trends in consultation rate before and after the launch of the website in March 2012 were compared by calculating the slope change. Level changes at 12 and 24 months were estimated by forecasting the preintervention model and comparing this with the postintervention model. For each subgroup and the reference group, a separate ARIMA model was fitted. Since an ARIMA model relies on linearity, we checked for stationarity of the mean and variance over time by differencing.30 After differencing once, the time series reached stationarity, indicating linearity and appropriateness of an ARIMA model. We evaluated autocorrelation and seasonality by inspecting the autocorrelation (ACF) and partial autocorrelation functions (PACF), finding autocorrelations at lag 1 and 12. Hence, an ARIMA 1(1,0,0) model and seasonal ARIMA 12(1,0,0) model was fitted. After applying the model, the ACF and PACF of the noise residuals were inspected and tested by using the Box-Ljung statistic, finding no relevant residual autocorrelations, which confirmed the adequacy of the model.25 26 All analyses were performed in SPSS V.20.0 (IBM Corporation, New York, USA). Values were considered to be statistically significant at a p value <0.05.

Results

The total number of consultations during the 6-year study period was 18.1 million, with a mean (SD) of 290 (21) consultations per 1000 patients per month. The mean (SD) age of the study population during the whole study period was 40 (23) years with 51% females (see online supplementary eTable 1). The total number of unique page views of the website during the study period was 56.6 million, with a monthly mean of 450 000 in 2012 and increased to 2.9 million per month at the end of 2014 (figure 1). Trends of total number of consultations before and after the launch of the website are depicted in figure 1. Before the launch of the website, the consultation rate increased by 0.826/1000/month. However, after the launch of the website, this rate decreased to −0.794/1000/month, giving a slope change of −1.620 (p<0.001; table 1). The slope for all consultation types declined (figure 2), but only the decline for telephone consultations was significant, with a slope change of −1.056 (p=0.001; see online supplementary eTable 2).Trends for the secondary outcome are depicted in figure 3. The consultations for the top 10 topics decreased (slope change: −0.169; p=0.003; table 1), while the consultation rate for the reference group did not change (slope change: −0.001; p=0.96; table 1). When analysed separately, the consultation rate decreased significantly for six of the top 10 topics. The consultation rate did not change significantly for any of the three reference topics (table 1).

Figure 1.

Figure 1

Open in a new tab

Rate of primary care consultations and page views of the website. Consultation rates with trend lines are displayed on the left y-axis. The number of page views of the website is displayed on the right y-axis and shows a steep increase in March 2012, which represents the launch of the website.

Table 1.

Change in consultation rate before and after the launch of the website in March 2012 for all consultations, the top 10 topics of the website and the reference group

Type of consultations Constant* Preintervention slope Postintervention slope Slope change p Value
All consultations 272.109 0.826 −0.794 −1.620 <0.001
Top 10 topics 28.408 0.092 −0.077 −0.169 0.003
 Constipation 2.070 0.026 −0.007 −0.033 <0.001
 Vaginal discharge 1.665 0.013 −0.016 −0.029 <0.001
 Irritable bowel syndrome 1.019 0.009 −0.003 −0.011 <0.001
 Herpes zoster 0.504 0.002 −0.001 −0.003 <0.001
 Bladder infection in women 13.664 0.090 −0.011 −0.101 0.009
 Lower back pain 6.200 0.012 −0.013 −0.025 0.03
 Gout 0.794 0.008 0.004 −0.004 0.38
 Shoulder symptoms 4.334 0.000 −0.004 −0.005 0.46
 Diarrhoea 3.311 −0.008 −0.013 −0.005 0.72
 Sinusitis 2.922 −0.007 −0.015 −0.008 0.76
Reference group 4.611 −0.005 −0.006 −0.001 0.96
 Head trauma† 0.389 0.001 0.001 0.000 0.94
 Premenstrual syndrome‡ 0.029 0.000 0.000 0.000 0.92
 Contact eczema‡ 4.195 −0.006 −0.007 −0.001 0.96
Open in a new tab

Values are presented as consultations/1000 patients/month.

*Consultation rate at t=0.

†Consultations for topics about which no online information was provided during the study period.

‡Consultations for topics that were very unpopular on the website during the study period.

Figure 2.

Figure 2

Open in a new tab

Rate of short consultations, long consultations and consultations by phone with trend lines.

Figure 3.

Figure 3

Open in a new tab

Rate of consultations in the top 10 topics of the website and reference group with trend lines.

supplementary eTables

bmjopen-2016-013166supp_tables.pdf (37.6KB, pdf)

Subgroup and sensitivity analyses

For all subgroups of sex, age and SES, the stratified analyses showed a significant decrease in consultation rate (table 2), except the youngest age group (0–16 years). In the first sensitivity analysis, calculating level changes 1 and 2 years after launch of the website, the consultation rate had decreased by 6.2% after 1 year and by 11.6% after 2 years (see online supplementary eTable 3). In our study population, this corresponds to a reduction of 62 300 consultations in the month of March 2014. The second sensitivity analysis, including only high-grade recording GPs and the third sensitivity analysis, including only GPs recording throughout the entire study period, confirmed the results of the main analysis (see online supplementary eTables 4 and 5). In the fourth sensitivity analysis, a similar decrease in consultation rate was found for chronic and non-chronic care consultations (see online supplementary eTable 6).

Table 2.

Subgroup analyses of change in consultation rate before and after the launch of the website in March 2012

Subgroups Constant* Preintervention slope Postintervention slope Slope change p Value
Male 218.036 0.558 −0.551 −1.109 0.004
Female 325.262 1.066 −1.037 −2.103 <0.001
Age <16 202.665 −0.083 −0.647 −0.563 0.37
Age 16–40 229.494 0.533 −0.713 −1.245 <0.001
Age 41–70 297.084 0.881 −0.873 −1.754 <0.001
Age >70 462.495 2.545 −1.604 −4.149 <0.001
Low SES 277.038 1.302 −0.914 −2.216 0.001
Intermediate SES 277.756 0.608 −0.620 −1.228 <0.001
High SES 259.988 0.692 −0.897 −1.589 <0.001
Open in a new tab

Values are presented as consultations/1000 patients/month.

*Consultation rate at t=0.

SES, socioeconomic status.

Discussion

Key results and interpretation

In this nationwide natural experiment, healthcare usage in primary care decreased after the launch of an evidence-based health information website. This strongly suggests that, by providing evidence-based online health information, healthcare usage can be affected: in our study, consultation numbers decreased by almost 12% after 2 years. When extrapolated to the entire Dutch population, this translates to a reduction of 675 000 consultations in the month of March 2014 only, exactly 2 years after the intervention. Interestingly, the decline was most distinct for the consultations by phone. Those consultations usually encompass non-essential care, and are therefore most likely to be affected by providing online information.31 Importantly, healthcare use in the period before release of the website was not declining and even rising. In addition, a reference group was used to detect effects of other factors influencing consultation rates which were not accounted for by the preintervention trend. After release of the website, this reference group showed no change at all, which supports a causal relation between the launch of the website and the reduction of healthcare usage. The reduction was robustly present in all the subgroups, except in the youngest age group. To the best of our knowledge, this is the first study to evaluate the effects of providing evidence-based online health information on healthcare usage rates in the general population. A previous study implemented online health information and found no change in healthcare usage.17 However, this study was only locally based, underpowered and healthcare usage was assessed by self-reporting, which underestimates actual healthcare usage.32 33 Other studies in this field mainly addressed characteristics, motivations and choices of online health information seekers. Most of these studies suggested that online health information seekers tend to make more doctor visits and that online health information is supportive during consultations but does not replace consultations.11 13–15 However, these studies are hampered by low response rates and only use self-reported healthcare usage as an outcome. In addition, patient-specific characteristics could have confounded the results of these studies. For example, patients with low health anxiety make fewer doctor visits when looking online than patients with moderate to high anxiety.12 Furthermore, the current online health information is of inconsistent quality,7 highlighting the importance of providing high-quality online information. Our findings in the young-age subgroup are in line with a recent study which found no change in healthcare usage for infants.16 This suggests that the need for assurance by face-to-face contact cannot be replaced by eHealth in this patient group. The decrease in consultation rate was also present in the oldest subgroup. This refutes concerns about accessibility of older patients to online content and supports earlier positive findings of eHealth in this age group.34 35

Strengths and limitations

The major strengths of this study are its before–after design and additional use of a reference group. The rise in consultation rates before release of the website, the decline after release of the website and the absence of change in consultation rate in the reference group supports a causal relation between the intervention and the observed effects. A limitation is that, owing to nationwide implementation of the website, no reference group could be created which was not exposed to the website at all during the study period. As an alternative, we selected topics which were not on the website (head trauma) or had very low page views during the study period (premenstrual syndrome and contact eczema). They consider health topics for which self-management with online information is feasible and, in our opinion, thus form a valid reference group. Although the absolute number of consultations for these reference topics was substantially lower than the top 10 topics, they provide insight into the stability of healthcare usage without exposure to the website. Another strength is that we investigated the impact of this eHealth intervention using actual healthcare usage rates, instead of questionnaire-based estimates. Importantly, our data are representative of the Dutch population.26 Furthermore, we were able to create a long-term study duration of 6 years, providing enough data points to account for seasonality and preintervention trends. Although ITS is considered the strongest research design when randomisation is not possible and can in theory be used to study causal effects using observational data,29 we carefully considered other events that might have affected healthcare usage during our study period. For example, one of the main incentives for healthcare usage are reimbursements. During our study period, a stepwise increase of the deductible to €360 in 2014 occurred. However, primary care consultations were always fully covered by the health insurance, while secondary care consultations were not. In addition, a stable 3% of the Dutch population avoids primary care due to financial reasons,36 which cannot explain the decrease of 12% in consultation rate that we observed in this study. Another factor possibly influencing the primary care consultation rate is fluctuations in chronic care consultations performed by nurse practitioners. We addressed this by stratifying our analysis for chronic and non-chronic care consultations and found a similar decline in consultation rate for both. Furthermore, we considered a change over time of the general practice characteristics as a confounding variable for the primary outcome and a change in registration grade as a confounding variable for the secondary outcome. After including only GPs who participated throughout the entire study period and after including only high-grade recording GPs, a similar decline in trend was found as in the main analysis. Additionally, an increase in the population's age over time could have affected our results. Our population's age increased by 0.9 years during the study period (see online supplementary eTable 1). In our opinion, this would not have a large impact on our primary results and, if any, this would lead to a larger effect of the intervention. Although the website reached 3 million page views by the end of 2014, the reduction in healthcare use did not show a dose–response relationship. However, a dose–response relationship is unlikely, because only a fixed proportion of healthcare usage can probably be affected by providing high-quality information and patient self-management. This study suggests that this proportion is at least 12%, although future studies with a longer follow-up will provide more insight. A limitation of our study is that we counted unique page views of the website, although counting unique visitors could have added valuable information. Using unique page views to quantify exposure of the website could overestimate or underestimate the number of unique visitors, but still provides important information about the uptake of the website by the public. Finally, mass media coverage about healthcare use of the website may have influenced healthcare consumption during the study period. However, the only mass media coverage about the website took place in March 2012 and no active media or marketing actions were carried out during the rest of the study period.

Generalisability and future research

The majority of literature on this topic is focused on whether eHealth can be a replacement or is merely supplemental to regular healthcare. This simplifying approach separates the role of the physician from the patients' self-management. In the Netherlands, 90% of the GPs use http://www.thuisarts.nl as a supportive tool during their consultations and patients subsequently look up the provided information online.22 This emphasises the suitability of eHealth as a tool to integrate self-management in primary care. In our opinion, the reduction of consultations for non-life-threatening conditions in primary care is beneficial and stimulates self-management of patients. Additionally, it could improve efficiency of communication during consultations, because patients are better informed. This model of blended care has also been adopted in other fields of medicine37 38 and this study shows its full potential for the first time. Since not all health topics are suitable for blended care, future studies could focus on suitability of specific topics for eHealth and improvement of information for these topics. Furthermore, our results do not imply that quality of care or health of the population is increased by an evidence-based health website. This could be a topic for future research. We expect that our results are generalisable to other countries with other healthcare systems, because patients look for online health information and make a decision to visit a doctor in other countries as well. Still, effect sizes could possibly be stronger in other countries without insurance cover for primary care or weaker when internet penetration is lower.39 Furthermore, evaluating effects of other health websites, for example, in the UK (http://www.nhs.uk/Conditions) and online patient portals (eg, patient.info, patientslikeme.com), might yield further insights into the possibilities of the internet on healthcare consumption. Finally, future studies could investigate the economic impact of this intervention.

Conclusion

The effect of providing evidence-based online health information on healthcare usage on a population level was not yet known. This study showed that, 2 years after the launch of an evidence-based health website, nationwide primary care usage decreased by 12%. This effect was most prominent for phone consultations and present in all subgroups of sex, SES and age, except for the youngest age group. This suggests that eHealth can be effective to improve self-management and reduce healthcare usage in times of increasing healthcare costs.

Acknowledgments

The authors acknowledge R Wolterbeek of the Department of Medical Statistics of the LUMC for his statistical advice and B van den Hoogenhoff of the Section of Digital media of NHG.

Footnotes

Twitter: Follow Tobias N. Bonten at @tbonten, Ton Drenthen at @Ton_Dr and Niels H Chavannes at @NHChavannes

Contributors: WAS and TNB designed the study, analysed the data, interpreted the data and drafted the paper. MWMW and NHC designed the study, interpreted the data and revised the paper. TD and IJMS revised the paper. MMJM interpreted the data and revised the paper.

Funding: The NHG and LUMC funded the analyses for this study. Funding was provided to the Netherlands Institute for Health Services to prepare and extract the data from their data set.

Competing interests: TD and IJMS are employed by the Dutch College of General Practitioners, which financially supports the website thuisarts.nl.

Provenance and peer review: Not commissioned; externally peer reviewed.

Data sharing statement: Technical appendix, statistical code, and data set available from the last author (NHC).

References

  • 1.Centers for Medicare & Medicaid Services B. National Health Expenditure Data 2014. https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/NationalHealthExpendData/Downloads/highlights.pdf (accessed 3 Nov 2015).
  • 2.Burwell SM, Frieden TR, Rothwell CJ. Health, United States, 2014: with special feature on adults aged 55–64. National Center for Health Statistics, 2015. http://www.cdc.gov/nchs/data/hus/hus14.pdf (accessed 3 Nov 2015). [PubMed] [Google Scholar]
  • 3.Europian Union statistical office. Eurostat Health care expenditure 2015. http://ec.europa.eu/eurostat/web/health/health-care (accessed 3 Nov 2015).
  • 4.van de Kar A, Knottnerus A, Meertens R et al. . Why do patients consult the general practitioner? Determinants of their decision. Br J Gen Pract 1992;42:313–16. [PMC free article] [PubMed] [Google Scholar]
  • 5.Diaz JA, Griffith RA, Ng JJ et al. . Patients’ use of the Internet for medical information. J Gen Intern Med 2002;17:180–5. 10.1046/j.1525-1497.2002.10603.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Powell J, Inglis N, Ronnie J et al. . The characteristics and motivations of online health information seekers: cross-sectional survey and qualitative interview study. J Med Internet Res 2011;13:e20 10.2196/jmir.1600 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Eysenbach G, Powell J, Kuss O et al. . Empirical studies assessing the quality of health information for consumers on the worldwide web: a systematic review. JAMA 2002;287:2691–700. 10.1001/jama.287.20.2691 [DOI] [PubMed] [Google Scholar]
  • 8.Azocar F, McCabe JF, Wetzel JC et al. . Use of a behavioral health web site and service utilization. Psychiatr Serv 2003;54:18 10.1176/appi.ps.54.1.18 [DOI] [PubMed] [Google Scholar]
  • 9.Beck F, Richard JB, Nguyen-Thanh V et al. . Use of the internet as a health information resource among French young adults: results from a nationally representative survey. J Med Internet Res 2014;16:e128 10.2196/jmir.2934 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Bouche G, Migeot V. Parental use of the Internet to seek health information and primary care usage for their child: a cross-sectional study. BMC Public Health 2008;8:300 10.1186/1471-2458-8-300 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Dwyer DS, Liu H. The impact of consumer health information on the demand for health services. Q Rev Econ Finance 2013;53:1–11. 10.1016/j.qref.2012.11.002 [DOI] [Google Scholar]
  • 12.Eastin MS, Guinsler NM. Worried and wired: effects of health anxiety on information-seeking and health care utilization behaviors. Cyberpsychol Behav 2006;9:494–8. 10.1089/cpb.2006.9.494 [DOI] [PubMed] [Google Scholar]
  • 13.Lee CJ. Does the internet displace health professionals? J Health Commun 2008;13:450–64. 10.1080/10810730802198839 [DOI] [PubMed] [Google Scholar]
  • 14.Nicholson W, Gardner B, Grason HA et al. . The association between women's health information use and health care visits. Womens Health Issues 2005;15:240–8. 10.1016/j.whi.2005.05.004 [DOI] [PubMed] [Google Scholar]
  • 15.Suziedelyte A. How does searching for health information on the Internet affect individuals’ demand for health care services? Soc Sci Med 2012;75:1828–35. 10.1016/j.socscimed.2012.07.022 [DOI] [PubMed] [Google Scholar]
  • 16.van der Gugten AC, Uiterwaal CS, Verheij TJ et al. . E-health and consultation rates for respiratory illnesses in infants: a randomised clinical trial in primary care. Br J Gen Pract 2015;65:e61–8. 10.3399/bjgp15X683485 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Wagner TH, Hibbard JH, Greenlick MR et al. . Does providing consumer health information affect self-reported medical utilization? Evidence from the Healthwise Communities Project. Med Care 2001;39:836–47. [DOI] [PubMed] [Google Scholar]
  • 18.Fox S, Duggan M. Health Online 2013. Pew Research Center, 2013. http://www.pewinternet.org/files/old-media//Files/Reports/PIP_HealthOnline.pdf (accessed 3 Nov 2015). [Google Scholar]
  • 19.Kontos E, Blake KD, Chou WY et al. . Predictors of eHealth usage: insights on the digital divide from the Health Information National Trends Survey 2012. J Med Internet Res 2014;16:e172 10.2196/jmir.3117 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Schwartz KL, Roe T, Northrup J et al. . Family medicine patients’ use of the Internet for health information: a MetroNet study. J Am Board Fam Med 2006;19:39–45. 10.3122/jabfm.19.1.39 [DOI] [PubMed] [Google Scholar]
  • 21.Wong C, Harrison C, Britt H et al. . Patient use of the internet for health information. Aust Fam Physician 2014;43:875–7. [PubMed] [Google Scholar]
  • 22.Drenthen T, Beijaert RP, Jansen PW et al. . [What do you think of Thuisarts.nl? Experiences after 3 years of www.Thuisarts.nl]. Ned Tijdschr Geneeskd 2014;158:A8282. [PubMed] [Google Scholar]
  • 23.Dutch College of General Practitioners. NHG guidelines 2015. https://www.nhg.org/nhg-standaarden. (accessed 3 Nov 2015).
  • 24.Effective Practice and Organisation of Care (EPOC). Interrupted time series (ITS) analyses. EPOC resources for review authors. Oslo: Norwegian Knowledge Centre for the Health Services, 2015. http://epoc.cochrane.org/epoc-specific-resources-review-authors. (accessed 3 Nov 2015). [Google Scholar]
  • 25.Jandoc R, Burden AM, Mamdani M et al. . Interrupted time series analysis in drug utilization research is increasing: systematic review and recommendations. J Clin Epidemiol 2015;68:950–6. 10.1016/j.jclinepi.2014.12.018 [DOI] [PubMed] [Google Scholar]
  • 26.Sitrbu-Wagner I, Dorsman SA, Visscher S et al. . Landelijk Informatienetwerk Huisartsenzorg. Feiten en cijfers over huisartsenzorg in Nederland. Utrecht/Nijmegen: NIVEL/IG, 2009. [Google Scholar]
  • 27.Lamberts H, Wood M. International classification of primary care. Oxford: Oxford University Press, 1987. [Google Scholar]
  • 28.The Netherlands Institute for Social Research. Statusscores 2015. https://www.scp.nl/Onderzoek/Lopend_onderzoek/A_Z_alle_lopende_onderzoeken/Statusscores (accessed 1 Aug 2015).
  • 29.Kontopantelis E, Doran T, Springate DA et al. . Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis. BMJ 2015;350:h2750 10.1136/bmj.h2750 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Nelson BK. Statistical methodology: V. Time series analysis using autoregressive integrated moving average (ARIMA) models. Acad Emerg Med 1998;5:739–44. [DOI] [PubMed] [Google Scholar]
  • 31.Nagle JP, McMahon K, Barbour M et al. . Evaluation of the use and usefulness of telephone consultations in one general practice. Br J Gen Pract 1992;42:190–3. [PMC free article] [PubMed] [Google Scholar]
  • 32.Bhandari A, Wagner T. Self-reported utilization of health care services: improving measurement and accuracy. Med Care Res Rev 2006;63:217–35. 10.1177/1077558705285298 [DOI] [PubMed] [Google Scholar]
  • 33.Chishti T, Harris T, Conroy R et al. . How reliable are stroke patients’ reports of their numbers of general practice consultations over 12 months? Fam Pract 2013;30:119–22. 10.1093/fampra/cms042 [DOI] [PubMed] [Google Scholar]
  • 34.Bolle S, van Weert JC, Daams JG et al. . Online health information tool effectiveness for older patients: a systematic review of the literature. J Health Commun 2015;20:1067–83. 10.1080/10810730.2015.1018637 [DOI] [PubMed] [Google Scholar]
  • 35.Crabb RM, Rafie S, Weingardt KR. Health-related internet use in older primary care patients. Gerontology 2012;58:164–70. 10.1159/000329340 [DOI] [PubMed] [Google Scholar]
  • 36.Van Esch TEM, Brabers AEM, Van Dijk C et al. . Inzicht in zorgmijden: aard, omvang, redenen en achtergrondkenmerken. NIVEL, 2015. http://www.nivel.nl/sites/default/files/bestanden/Inzicht-zorgmijden.pdf (accessed 3 Nov 2015). [Google Scholar]
  • 37.Kouwenhoven-Pasmooij TA, Djikanovic B, Robroek SJ et al. . Design and baseline characteristics of the PerfectFit study: a multicenter cluster-randomized trial of a lifestyle intervention in employees with increased cardiovascular risk. BMC Public Health 2015;15:715 10.1186/s12889-015-2059-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Mouthaan J, Sijbrandij M, de Vries GJ et al. . Internet-based early intervention to prevent posttraumatic stress disorder in injury patients: randomized controlled trial. J Med Internet Res 2013;15:e165 10.2196/jmir.2460 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.European Internet Stats. Internet Users in Europe 2015 2015. http://www.internetworldstats.com/stats4.htm (accessed 7 Oct 2015).

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

supplementary eTables

bmjopen-2016-013166supp_tables.pdf (37.6KB, pdf)

Articles from BMJ Open are provided here courtesy of BMJ Publishing Group

RESOURCES