Skip to main content
PMC home page
Wiley Open Access Collection logoLink to Wiley Open Access Collection
. 2013 Jul 31;30(4):278–293. doi: 10.1111/hir.12037

Development of a complex intervention to improve health literacy skills

Astrid Austvoll-Dahlgren *,✉, Stein Danielsen †, Elin Opheim ‡, Arild Bjørndal §,, Liv Merete Reinar **, Signe Flottorp *,††, Andrew David Oxman ‡‡, Sølvi Helseth §§
PMCID: PMC4383652  PMID: 24251890

Abstract

Background

Providing insight into the developmental processes involved in building interventions is an important way to ensure methodological transparency and inform future research efforts. The objective of this study was to describe the development of a web portal designed to improve health literacy skills among the public.

Methods

The web portal was tailored to address three key barriers to obtaining information, using the conceptual frameworks of shared decision-making and evidence-based practice and based on explicit criteria for selecting the content and form of the intervention.

Results

The web portal targeted the general public and took the form of structured sets of tools. Content included: an introduction to research methods, help on how to find evidence-based health information efficiently based on the steps of evidence-based practice, an introduction to critical appraisal, information about patient participation rights in decision-making, and a decision aid for consultations.

Conclusions

The web portal was designed in a systematic and transparent way and address key barriers to obtaining and acting upon reliable health information. The web portal provides open access to the tools and can be used independently by health care users, or during consultations with health professionals.

Keywords: consumer health information, decision support, evidence-based practice, health education, informatics, health, information seeking behaviour

Key Messages

  • Providing insight into how interventions are developed provides greater methodological transparency

  • In this study, we describe the development of an intervention – a web portal – developed to improve specific domains of health literacy

  • The intervention we developed may be a useful tool for members of the public, and a resource for health care information professionals and health educators

  • Researchers should aspire to provide insight into the developmental process of interventions in order to inform the efforts of other researchers and developers

  • Further studies are needed to develop and evaluate interventions to improve health literacy

Introduction

Basing health care on the best available evidence is an international priority.1 Evidence-based practice ‘requires that decisions about health care are based on the best available, current, valid and relevant evidence. These decisions should be made by those receiving care, informed by the tacit and explicit knowledge of those providing care, within the context of available resources’.1

The involvement of users of health care (hereafter referred to as users) in health decisions about their own health care is central to evidence-based practice and is widely acknowledged.1–3 In the shared decision-making, model users and health professionals decide together in partnership in which information, responsibility and accountability are mutually shared.4 This shift from paternalism is argued to be the result of several factors, including political forces advocating health care user's autonomy, technological advancements making health information widely available, and the acknowledgement that user involvement is important for health outcomes and sustainable health care.4–6

However, effective participation is not only dependent on the right to involvement, but also on access to reliable health information and the ability to obtain such information.5,7 Evidence suggests that much of the health information available is incomplete, biased or not evidence-based.8,9 This is particularly of concern considering that many users have been found to have inadequate health literacy skills.10,11 Such health literacy skills are described as the knowledge and skills that enable users to obtain, understand and act upon health information, and as the desired outcome of health education by the World Health Organization.3,6 In a systematic review of the relationship between health literacy and health carried out by Berkman et al.,10 the evidence evaluated showed that low health literacy levels are associated with poorer health, increased hospitalisations and health care service use, incorrect drug use, low responsiveness to health education and a low uptake of disease prevention services. Furthermore, health literacy has been found to be a stronger predictor of health status than of the relationship between health literacy and health, the evidence evaluated showed that low health literacy levels are associated with poorer health, increased hospitalisations and healthcare service use, incorrect drug use, low responsiveness to health education and a low uptake of disease prevention services (including vaccinations). Furthermore, health literacy has been found to be a stronger predictor of health status than age, income, employment status, education level and ethnicity.7 Consequently, there is a need to prioritise interventions that facilitate health literacy competencies and enable the public to obtain and evaluate such information.5,7,12

Quality improvement initiatives in health services research can be challenging and complex in nature and may include many points of intervention.13 Grol et al.13 argue that it would be unrealistic to expect that one simple improvement measure is adequate to solve all targeted problems. Instead, they reason that these challenges are often best addressed using complex interventions in which several factors are targeted and multiple intervention strategies are used.13

Significantly, the development of interventions is often poorly reported in research. Providing insight into how these steps were undertaken and which methods were used provides greater methodological transparency. It can also help to inform the efforts of other researchers and developers.14–16 In this paper, we describe the development of a complex intervention – a Norwegian web portal – intended to improve health literacy skills among the public and the ability of users to obtain reliable health information.

Methods

There is no universally optimal way of developing complex interventions, but explorative approaches are encouraged that are also systematic and transparent.13,17 Despite the methodological uncertainties associated with such interventions, some guiding rules have been advanced.17,18 A complex intervention often includes several steps or stages (which may not necessarily occur in a linear sequence), in Van Boekhoven et al.' model, which inspired our work, these steps include problem analysis, designing the intervention, piloting and feasibility testing, implementation and evaluations (see Figure1).17,18 The development of our complex intervention included all of these steps. Decisions related to the content and main focus of our intervention, as well as about which specific health literacy skills to target, were informed by qualitative interviews including members of the general public exploring decision-making and beliefs associated with obtaining health information, and a questionnaire where we explored important predictors associated with intention to search. The results of these studies have been previously published.19–21 Supplementing these studies, we also conducted explorative searches in Medline for studies describing barriers and facilitators to obtaining health information related to health literacy, using the following terms: (public or patient or consumer) and (information or Internet or mass media) and (health behaviour or search or attitude or decision or participation). We also looked through reference lists of relevant studies and searched for studies similar to these. We also scrutinised studies that had referenced these studies.

Figure 1.

Figure 1

Open in a new tab

Model describing phases of complex interventions by van Boekhoven et al.

However, the focus of this paper is on the development phase. Essential to the development phase is analysing the problem and identifying important barriers to – and facilitators of – change, so that these can be specifically targeted.13,17,18

Analysing the problem

Identifying the target population

Although the prevalence of inadequate health literacy skills has been found to be higher in certain groups such as among the elderly, people with chronic conditions and lower socioeconomic groups; healthy literacy is found to be generally low across populations.10,11 Consequently, health literacy is considered a public health issue,3,6 and the target audience of our intervention was therefore the general public.

Operationalisation of health literacy

Our overall goal when initiating this project was to develop an intervention with the potential to improve users' health literacy skills related to their ability to obtain health information within the conceptual framework of health literacy. What health literacy skills really entail has been defined in many different ways.5,12,22,23 In our study, we used the multi-dimensional model formulated by Zarcadoolas et al.,5 which contains four central domains: fundamental literacy (reading, writing, speaking and working with numbers), science literacy (understanding and using science and technology), civic literacy (skills and abilities that enables awareness, participation and involvement) and cultural literacy (skills and abilities to recognise, understand and use of (others') beliefs, customs, worldviews and social identities). This model provided us with an explicit framework for understanding and operationalising health literacy in our intervention (see Figure2).

Figure 2.

Figure 2

Open in a new tab

A multi-dimensional model of health literacy by Zarcadoolas et al.

Identifying barriers to users ability to obtaining health information

Theoretical and empirical research has suggested that interventions should be developed to address the most important barriers to – and facilitators of – change.17,18,24–26 The underlying assumption is that this will improve the effectiveness of an intervention.17,18,24–26 Identifying important barriers to change may entail reviewing the evidence base and conducting qualitative and quantitative explorative studies involving representatives of the target group.13,17,18 The results of such explorative research, in turn, help to inform decisions about intervention content and delivery in a feedback process often referred to as ‘tailoring the intervention'.18 Following the process of interviews and literature review, we identified three key barriers related to obtaining information related to specific health literacy skills and potential targets for intervention. These are briefly described below

Barrier 1: Inability to understand and critically appraise health information: We found that the inability to understand and critically appraise health information is a key barrier to obtaining information.19,20 In particular, research on decision-making about prevention and treatment strategies (including vaccination, mammography screening and smoking) indicates that users' understanding of medical and health-related research is poor and that many find it difficult to interpret concepts such as randomisation, risk, uncertainty, causality and applicability.27–35 Similarly, many users rely on information that is based on personal anecdotes and experience as opposed to information that is based on research and may overrate the trustworthiness of such information.30,36,37 Many do not check the accuracy of the health information they find, and those that do use criteria such as: the appearance, presentation of content, funding, currency and perceived reliability of the publisher.30,36–38

Barrier 2: Inability to exchange information in consultations: A second major barrier to obtaining information is the inability of users and providers to exchange information important for decision-making during consultations.19,21 Consultations should, ideally, be an arena in which appropriate health information is exchanged. But studies have shown that users often want more information than they are actually given by health care providers, and that they may not obtain information that is important for decision-making.19,21,39–42 Users may also not be made aware of their rights or of possible treatment alternatives during consultations, may not know what to ask their provider or may not be able to remember what they have been told.19,40–42

Barrier 3: Inability to find reliable and relevant information: The third major barrier to obtaining information we identified is the inability of users to obtain information independently – in other words, the inability of users to know where and how to find reliable and relevant information.19,20 Being able to obtain information independently is important for effective participation in decision-making and for users to be able to make healthy choices. Users are often overwhelmed and frustrated by the vast amount of information available to them and unsure about whom or what they can trust.43,44 Furthermore, evidence-based information is not readily available to the public,8,45–47 and the way in which users search for health information can also be haphazard. Many use online search engines, such as Google or Yahoo,37,44 or rely on family and friends, discussion forums and general news sites,48,49 which may potentially provide information of poor scientific quality.9,45–47

Our web portal was therefore tailored to address these three key barriers to obtaining information. Below, we describe the strategies we used to address them and the decisions we made.

Designing the intervention: selecting methods and strategies

Tailoring an intervention in order to address the identified barriers can be done either by choosing an evidence-based intervention delivery that addresses identified barriers and facilitators or by tailoring the characteristics of a chosen intervention.17,18,24,25 When developing our intervention, we did both. The choice of methods and strategies was also done by considering actual practical barriers and opportunities affecting implementation, including time and financial budgets and the availability of human resources.18,25

Choice of intervention delivery

Our decision to use a web portal as to deliver the intervention was informed by evidence suggesting that web-based or computer-based interventions may be particularly effective for improving knowledge and behavioural change.7,50–54 Such strategies, often referred to as patient health informatics, may include decision support, promotion of healthy behaviours and the facilitation of information exchange and self-care.52 According to Murray et al.,52 the benefits of such applications relative to more traditional methods of promoting health education include the opportunity to: contain large and accessible volumes of information, update the information centrally, tailor the information to meet the needs of the user and present information in different video, audio and graphic formats. Furthermore, web-based interventions also allow users to take greater control over the information-gathering process at their own time and pace, thus resulting in high levels of user satisfaction.7 These ways of intervention delivery appear to benefit also those with poor health literacy.53 Finally, a web-based intervention also has the potential to reach a wide public at little cost.

The web design software we chose for the web portal met recommended accessibility standards.55 When presenting the portal content, we aimed to comply with the standards recommended by Health On the Net Foundation's ‘Code of Conduct for Medical and Health Web Sites (HONcode)’.56 Central to this code for health-related websites is the following concerns: that the authority of the information provided should be made explicit, that data confidentiality and privacy should be respected, that there should be proper attribution of sources, that financial sponsorship should be stated explicitly and that advertising should be clearly separated from editorial content (if applicable).56 The presentation of the portal content was further informed by research showing that the use of mixed media, real-life examples, plain language and the provision of information in small and ‘digestible’ quantities are effective and preferred educational strategies by users.7,41,54,57,58

Tailoring the intervention: choice of content

We used the conceptual frameworks of shared decision-making and evidence-based practice to determine how the portal content was selected and presented.1,4,59 In practice, this meant that our web portal was intended to encourage users to take an active role in decision-making. We also intended to highlight the importance of that decisions should be informed by the best available, current, valid and relevant evidence, together with the users' own values and preferences.1 Furthermore, in accordance with the principles of shared decision-making and evidence-based practice, we included content that should facilitate skills and not adherence to any specific regime.

Although the purpose of the web portal was educational, a key objective was also to develop an access point to practical tools and evidence. Thus, we decided to create a web portal of tools containing educational content instead of, for example, providing online courses. Users looking for information and answers related to health decisions, we reasoned, would be unlikely to be interested in completing online courses to access such detail. Based on discussions in our group, we decided on three facilitators or – tool-sets – to address each of the main barriers to obtaining information.

Toolset 1: Improving critical appraisal skills: To address the inability of users to understand and critically appraise health information, we decided that the web portal should provide a general introduction to research methods and explain concepts central to informed decision-making.5,59–64 Moreover, we wished to include a tool that would enable users to critically appraise health information. To this end, we searched for a tool that could address key aspects related to the quality of health information, including validity, currency, attribution and disclosures of interest.65,66 We also aimed to offer a generic tool that could be applied to different types of health information (regardless of diagnosis or health issue) and sources. This would ensure that the tool could be applied to information found across different contexts such as, for example, the Internet, printed news, pharmacies, patient organisations or healthcare providers. We also sought to provide a tool that had been evaluated for construct validity and interobserver reliability. Although previous efforts aimed to improve users' understanding of research and critical appraisal skills are few, and that this evidence is limited, it is suggested that such education may be feasible, perceived as useful by the users and may improve confidence and knowledge.67–69 The use of checklists provides the user with a systematic tool, and the use of such instruments by users has been found to produce similar ratings as those of professionals and may help users in their selection of health information.70 Because the web portal was intended to be generic and target health literacy skills, we excluded diagnosis-specific tools, tools providing expert evaluations or support, or those that required the comparison of information with practice guidelines, systematic reviews or other scientific literature. We performed systematic literature searches using MEDLINE, Embase and the Cochrane Methodology Register to identify tools, such as checklists, teaching materials and software. In addition, we searched for tools online using Google (see Table1 for a list of our search strategies). Two researchers reviewed all the search results independently. All descriptions of health information evaluation tools were retrieved. The lead researcher then categorised the findings and selected the tool best-suited to the web portal. A second researcher then reviewed these judgements.

Table 1.

Search strategy for critical appraisal tools and decision aids

Critical appraisal tools
 Database: Ovid MEDLINE(R) and EMBASE <2001 to January Week 4 2010> Search Strategy
  1. Patient Participation/(13658)
  2. Consumer Participation/(11836)
  3. (Patient or patients or consumer or consumers or client or clients or user or users or citizen or citizens or lay men or lay people or lay population*).ti,ab. (3454913)
  4. 1 or 2 or 3 (3468734)
  5. Checklist/(44)
  6. ((Checklist* or tool* or guid*) adj6 (medical literature or health* literature or health care literature or medical news or health news or health care news or media or science* or research or internet or medical information or health* information or health care information or world wide web)).ti,ab. (16117)
  7. 5 or 6 (16160)
  8. ((Critical* adj2 apprais*) or assess* or evaluat*).ti,ab. (2342349)
  9. 4 and 7 and 8 (2196)
  10. Limit 9 to year = ‘2001 –Current’ (1447)
  11. Limit 10 to humans (1366)
  12. From 11 keep 1–10 (10)
  13. From 11 keep 1–10 (10)
  14. From 11 keep 1–1366 (1366)
  15. From 14 keep 1–1366 (1366)
Checklist consultation
 Database: Ovid MEDLINE(R) and EMBASE <2001 to January Week 4 2010> Search Strategy
  1. Patient Participation/(13658)
  2. Consumer Participation/(11836)
  3. (Patient or patients or consumer or consumers or client or clients or user or users or citizen or citizens or lay men or lay people or lay population*).ti,ab. (3454913)
  4. 1 or 2 or 3 (3468734)
  5. Checklist/(44)
  6. ((Checklist* or tool* or guid*) adj6 (medical literature or health* literature or health care literature or medical news or health news or health care news or media or science* or research or internet or medical information or health* information or health care information or world wide web)).ti,ab. (16117)
  7. 5 or 6 (16160)
  8. ((Critical* adj2 apprais*) or assess* or evaluat*).ti,ab. (2342349)
  9. 4 and 7 and 8 (2196)
  10. Limit 9 to year = ‘2001 –Current’ (1447)
  11. Limit 10 to humans (1366)
  12. From 11 keep 1–10 (10)
  13. From 11 keep 1–10 (10)
  14. From 11 keep 1–1366 (1366)
  15. From 14 keep 1–1366 (1366)
Checklist consultation
 Database: Ovid MEDLINE(R) and EMBASE <2001 to January Week 4 2010> Search Strategy:
  1. (Decision* or participat* or consult*).ti,ab. (402046)
  2. Decision Making/(51332)
  3. Nurse-Patient Relations/(25939)
  4. Professional-Patient Relations/(17173)
  5. Physician-Patient Relations/(49944)
  6. Patient Participation/(13658)
  7. Consumer Participation/(11836)
  8. (Patient or patients or consumer or consumers or client or clients or user or users).ti,ab. (3449075)
  9. 1 or 2 (428028)
  10. 3 or 4 or 5 or 6 or 7 or 8 (3511881)
  11. Checklist/(44)
  12. Checklist.ti,ab. (10757)
  13. 11 or 12 (10781)
  14. 9 and 10 and 13 (648)
  15. Limit 14 to (humans and year = ‘2001 –Current’) (419)
  16. From 15 keep 1–419 (419)
 Cochrane methods register
  1. ‘Consumer involvement’ or ‘Patient involvement’.kw
  2. (Patient or patients or consumer or consumers or client or clients or user or users or citizen or citizens or lay*).ti,ab.
  3. 1 or 2
  4. ((Critical* adj2 apprais*) or assess* or evaluat*).ti,ab.
  5. ‘Scales and checklists’.kw
  6. (Checklist* or tool* or guid*)4 or 5 or 6
  7. 3 and 7
Open in a new tab
*

Search strategy indication of truncation.

Toolset 2: Enabling exchange of health information: To address the second key barrier to obtaining information, namely the inability of users and providers to obtain adequate information during consultations, we decided that the web portal should include basic information about patients' rights to participate in decision-making and what decision-making about treatment or screening options entails.2,59–61,71 Furthermore, checklists and decision aids as adjuncts to verbal communication may be effective tools and may help to reduce decisional conflict and improve the ability of users to recall information, as well as improving their sense of control, knowledge retention, sense of involvement and satisfaction.7,72 We therefore developed a search strategy and conducted a systematic search to identify generic checklists and decision aids that could be used in relations with consultations to facilitate information exchange and participation in decision-making. The results of this search were reviewed using the same process as described above (see also Table1 for the search strategies used). All tools that were described as a decision aid or a checklist for consultations were retrieved. Diagnosis-specific tools were excluded. The final checklist selection was undertaken using the International Patient Decision Aid Standards (IPDAS), which specifies that decision aids should provide information about condition of the users and the options available, help to users clarify their values and help them to share their values with their health provider and others.73 Moreover, decision aids should not make actual recommendations or replace the counselling of a professional; instead, they should be used as tools to assist in the making of decisions that matches the user's values in collaboration with their health care provider.73

Toolset 3: Improved access to reliable research-based sources of health information: To address the third key barrier to obtaining information, namely the challenge that users face as a result of not knowing where to find reliable and relevant information, the web portal aimed to improve access to reliable sources of health information. It should be noted that this barrier partly overlaps with the first barrier since knowledge about research methodology necessarily impacts user assessments about whether information is reliable, valid and relevant. Rather than simply reporting conclusions or expert interpretations, we aimed to give direct access on the web portal to research evidence to a selection of medical- and health-related research databases. The following key criteria for selecting databases were used:

  • The information provided had to be based on empirical evidence (and not, e.g. on expert opinion)

  • The links provided should reflect an ‘information pyramid’ including not only different study types, but also different levels of synthesis (e.g. primary studies, systematic reviews and decision support resources such as synopses and summaries) 74

  • The sites should provide access to summarised information using explicit and systematic criteria

  • All the resources had to be freely available to the Norwegian public

In keeping with the general principle of providing ‘digestible’ quantities of information to users, we decided to limit the number of sources provided on the web portal. The final selection of databases was discussed by members of the research team using the criteria listed above. This section was built around the steps of evidence-based practice,1,59 providing the users with a path through the identification of relevant research.

Quality assurance

For quality assurance purposes, the project protocol, details related to the development process and the final completed web portal were presented to an advisory group for peer review and professional feedback. The advisory group included four researchers from different disciplines and fields of expertise, including evidence-based practice, research dissemination, general medicine, nursing, social linguistics and public health.

Results

The web portal was developed in compliance with the HONcode standards highlighted above.56 The web portal was made available online at www.sunnskepsis.no, and all content was made freely available and intended for use by the general public. We used plain language, and the provision of information was made in small and ‘digestible’ quantities. We illustrated the content using mixed media and examples mentioned in the news and in user stories. Furthermore, we presented content using an active voice and encouraging users explicitly to adopt an active role in decision-making. We also emphasised that decisions about health care should be informed by the best available, current, valid and relevant evidence.1

We presented the content in an easily accessible and structured way organised around the three sets of tools ‘improving critical skills’, ‘enabling exchange of health information’ and ‘improved access to reliable research-based sources of health information’ as points of departure (see Figure3). In this way, users have the option of quickly locating the tool or information they need, such as, for example, help to find information on how to prevent the common cold. This approach does not force users to view the information in a certain specific or prescriptive way when accessing the portal, tools or databases, and allows them to choose what they want to read and when.

Figure 3.

Figure 3

Open in a new tab

Front page of the web portal providing access to the three sets of tools

Results toolset 1: Improving critical appraisal skills

The web portal gives an introduction to medical- and health-related research methods and the basic principles of science across the three main tool-sets as small pieces of plain language text (see Figure4).

Figure 4.

Figure 4

Open in a new tab

Example of structure and presentation of content

The web portal addresses key concepts including randomisation, risk, uncertainty, causality and the difference between experience-based and research-based information. We also included information about research ethics, participation in research, and the validity and applicability of research findings using practical examples.59–63

Only one critical appraisal tool met all our inclusion criteria, namely the tool developed by the DISCERN project (run jointly by the University of Oxford Division of Public Health and Primary Health Care, the Help for Health Trust and the Buckinghamshire Health Authority in the United Kingdom). This tool was developed for both health users and health professionals and has been evaluated positively for construct validity and interobserver reliability.70 The tool helps to evaluate the trustworthiness of all types of health treatment information.70 We included a Norwegian version of the tool translated and produced by the National Information Center for Complementary and Alternative Medicine, Norway (NIFAB), and we included this both as a printable checklist and as an interactive tool.75

Results toolset 2: Enabling exchange of health information in relation to consultations

The web portal included descriptions about decision-making related to treatment and screening, and the principle of weighing benefits against harms.59–61,71 The portal also included content on user participation decision-making rights.2

We identified two decision aids that met the criteria for inclusion on the web portal.61,76 One, developed by Irwig,61 was selected because the tool does not demand a lot of effort from the user, while providing important items essential for informed decision-making about treatment or screening. This decision aid was translated into Norwegian and adapted with permission for use on the web portal. Because the decision aid did not include items related to user health conditions/diagnosis, it was further supplemented using IPDASI criteria.73

Results toolset 3: Improved access to reliable sources of health information

In order to ensure that the web portal provided improved access to reliable sources of health information, we developed a third section to facilitate direct access to research-based information via selected databases (see Table2).

Table 2.

List of databases included in the web-portal

Level Database Description
Summaries Clinical Evidence* Produces summaries of a wide range of conditions. Owned by a major global private publisher with headquarters in the UK (BMJ Publishing Group Limited)
Best Practice* Produces summaries of a wide range of conditions. Owned by a major global private publisher with headquarters in the UK (BMJ Publishing Group Limited). In 2010, the Best Practice patient information was translated into Norwegian by the Norwegian Electronic Health Library, which is organized under The Norwegian Knowledge Centre for the Health Services (NOKC)
Up To Date* Produces summaries of a wide range of conditions and is the largest clinical community in the world dedicated to synthesizing knowledge for clinicians and patients. Up To Date is independent and does not accept funding from pharmaceutical companies or other marketers
Systematic reviews Cochrane Library A database of thousands of reviews summarizing the effects of treatments but also including primary studies. The database is run by an international network consisting of volunteers and paid professional staff. Funded by national governments, international organizations, universities, hospitals, private foundations and personal donations
The Norwegian Knowledge Centre for the Health Services Organized under the Norwegian Directorate of Health but is scientifically and professionally independent. Produces systematic reviews and health economic evaluations as well as other services to support the development of quality in the health services
Primary studies PubMed PubMed is the world's largest database of medical and health sciences, and includes references to articles from approximately 4800 international journals published from 1966 onwards. PubMed is publicly funded by the United States National Library of Medicine, which in turn is funded by the National Institutes of Health
Statistics Norway (StatBank) Provides access to primarily cross-sectional data, but also other observational data about health and welfare. A Norwegian public institution with academic and editorial independence
Norwegian National Institute of Public Health (StatBank) Provides an overview of population health, prevention and research. A national centre of excellence (organized under the Ministry of Health and Care Services) in the areas of epidemiology, mental health, control of infectious diseases, environmental medicine, forensic toxicology and drug abuse
Health Talk Online Health Talk Online is a database of patient experiences obtained through systematic qualitative research. The database includes more than 40 different diseases and health conditions. The site is powered by DIPEx Charity, a charitable organization working in close collaboration with Oxford University and funded by various organizations, including the British Ministry of Health, Macmillan Cancer Relief, British Heart Foundation, Lord Ashdown Trust and Comic Relief
Open in a new tab
*

Available through Norwegian Electronic Health Library.

This section was built around the steps of evidence-based practice.1,59 Here, we provided a short description of each database, search tips and details on how to assess the applicability of information. In this section, we also described the basic designs of medical- and health-related research methods, and links were given to different types of questions as well as databases that could provide users with useful and relevant answers to each question type.59 We described different levels of synthesis of research explaining, for example, what systematic reviews and single studies are.59,74

The barriers we identified and the strategies we chose, touch upon specific domains of health literacy. Figure5 gives an overview of the components of the intervention and corresponding health literacy domains.5

Figure 5.

Figure 5

Open in a new tab

Overview of the web portals components and corresponding hypothesised health literacy domains

Discussion

Study limitations and methodological issues

The web portal we developed had limitations as well as appreciable strengths. Decisions about how to tailor an intervention depend on judgements about available resources, as well as judgements about barriers to effectiveness and the strategies needed to overcome them.17,18 A Cochrane review found that tailored interventions to improve professional practice improved processes of care and patient outcomes, but there was insufficient evidence to determine the most effective approaches to tailoring, including how to identify barriers and how to select interventions to address identified barriers.26 Thus, although tailored interventions are more likely to be effective, important barriers (and facilitators) may be missed or not addressed appropriately. These concerns may also have affected the tailoring of our web portal. But adopting an explicit conceptual and methodological approach when developing interventions 17 can provide insight into the tailoring process itself as well as the content and intended intervention mechanisms. This, in turn, is important for reproducibility and helps to inform the efforts of other researchers and developers within the field.17

Health literacy research and initiatives facilitating user involvement and informed decision-making are currently attracting great interest. Our intervention contributes to this body of research and include elements across all four domains by Zarcadoolas et al.5 To our knowledge, the web portal we developed is the first of its kind in Norway to provide insight into these domains of health literacy. Internationally, resources similar to the web portal described in this paper have been developed and evaluated. Examples include ‘Testing treatments’ (www.testingtreatments.org), which focuses on science literacy and evaluating the effects of treatments, and ‘What are your chances’ (www.whatareyourchances.com), which focuses on numeracy and how to understand risk.

Will an online resource be available to all?

Health information is popular and many actively search for such information.44,48,49 In addition to users’ next of kin and health professionals, Internet is considered an important source for such information by users.44,48,49 Studies indicate that Internet as a tool for intervention delivery has the potential to reduce inequalities and be widely available to the public at low cost.7 However, it should be noted that younger and more advantaged groups are often able to access and use new Internet information technologies more readily, and such a ‘digital divide’ may offset the advantages noted above.7 However, much suggests that this divide is diminishing. In Norway, for example, the Internet coverage is very high: 93% of people, for instance, were estimated to have used the Internet during the last 3 months of 2010.77 Similarly, Internet use for those older than 65 years was 63%, and use among those with only primary level education was 81%.77 Despite this, Internet-based interventions may not be universally appropriate or available to all. It is therefore important to emphasise that the web portal we created is not intended as an alternative to the role of health professionals in providing health information, but should instead be seen as a supplementary user tool. The portal intervention may also prove to be useful to health professionals themselves, who may choose to use or reference it in their discussions with users. As a systematic review of computer-based approaches to user education has shown, such interventions may support communication between users and providers.53 Moreover, through improved knowledge and access to the included tools, the web portal may also potentially stimulate participation in health decision-making outside the context of professional consultations.

Conclusion

In this paper, we have described the development of a web portal developed to improve specific domains of health literacy skills of the public and the consequent ability of users to obtain and act upon reliable health information. By describing the steps we took and the decisions we made, we hope to inform other researchers and developers.

Our web portal provides open access to tools included and was intended to be used independently by users as well as during consultations with health professionals. The web portal is a unique resource for patients and healthy people who are interested in health information or want to know more about medical- and health-related research. Furthermore, the web portal may be of relevance also to health professionals and other health educators working to improve health literacy skills, but also for those who develop patient information who may use the web portal to check the reliability of their own material.

Acknowledgments

We wish to thank research librarian Marit Johannesen for her much appreciated advice on search strategies, and linguist Elin Strømme Nilsen for her comments on the research protocol and portal development process.

Competing interests

The authors declare that they have no conflict of interests. The project was funded by Oslo and Akershus University College of Applied Sciences (formerly Oslo University College).

References

  1. Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, Hopayian K, Porzsolt F, Burls A. Osborne J. Sicily statement on evidence-based practice. BMC Medical Education. 2005;5:1. doi: 10.1186/1472-6920-5-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. The Norwegian Bill of Patients' Rights. 1999. Lov om pasientrettigheter (pasientrettighetsloven)
  3. Nutbeam D. 2012. pp. 1–36. Health Promotion Glossary 1998. Accessible at: http://www.who.int/hpr/NPH/docs/hp_glossary_en.pdf.
  4. Coulter A. The Autonomous Patient: Ending Paternalism in Medical Care. London: TSO; 2002. p. 128. [Google Scholar]
  5. Zarcadoolas C, Pleasant A. Greer D. Advancing Health Literacy: A Framework for Understanding and Action. 1st edn. San Francisco, CA: Jossey-Bass; 2006. p. 368. [Google Scholar]
  6. Nutbeam D. Health literacy as a public health goal: a challenge for contemporary health education and communication strategies into the 21st century. Health Promotion International. 2000;15:259–267. [Google Scholar]
  7. Coulter A. Ellins J. Patient-Focused Interventions: A Review of the Evidence. London: Picker Institute Europe; 2006. p. 277. [Google Scholar]
  8. Glenton C, Paulsen E. Oxman A. Portals to Wonderland? Health portals lead confusing information about the effects of health care. BMC Medical Informatics and Decision Making. 2005;5:7. doi: 10.1186/1472-6947-5-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Wolfe R, Sharp L. Lipsky M. Content and design attributes of antivaccination web sites. Journal of American Medical Association. 2002;287:3245–3248. doi: 10.1001/jama.287.24.3245. [DOI] [PubMed] [Google Scholar]
  10. Berkman N, Sheridan S, Donahue K, Halpern D. Crotty K. Low health literacy and health outcomes: an updated systematic review. Annals of Internal Medicine. 2011;155:97–107. doi: 10.7326/0003-4819-155-2-201107190-00005. [DOI] [PubMed] [Google Scholar]
  11. Sanders L, Federico S, Klass P, Abrams M. Dreyer B. Literacy and child health: a systematic review. Archives of Pediatric & Adolescent Medicine. 2009;163:131–140. doi: 10.1001/archpediatrics.2008.539. [DOI] [PubMed] [Google Scholar]
  12. Nutbeam D. The evolving concept of health literacy. Social Science and Medicine. 2008;67:2072–2078. doi: 10.1016/j.socscimed.2008.09.050. [DOI] [PubMed] [Google Scholar]
  13. Grol R, Baker R. Moss F. Quality improvement research: the science of change in health care. In: Grol R, Baker R, Moss F, editors; Quality Improvement Research: Understanding Change in Health Care. London: BMJ Publishing Group; 2004. pp. 1–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Bastian H. Health literacy and patient information: developing the methodology for a national evidence-based health website. Patient Education and Counseling. 2008;73:551–556. doi: 10.1016/j.pec.2008.08.020. [DOI] [PubMed] [Google Scholar]
  15. Straub C, Nebling T. Muller H. Translating research into practice: a German sickness fund supporting patient participation. Patient Education and Counseling. 2008;73:544–550. doi: 10.1016/j.pec.2008.07.019. [DOI] [PubMed] [Google Scholar]
  16. Jones JM, Nyhof-Young J, Friedman A. Catton P. More than just a pamphlet: development of an innovative computer-based education program for cancer patients. Patient Education and Counseling. 2001;44:271–281. doi: 10.1016/s0738-3991(00)00204-4. [DOI] [PubMed] [Google Scholar]
  17. Craig P, Dieppe P, Macintyre S, Mitchie S, Nazareth I. Petticrew M. Developing and Evaluating Complex Interventions: New Guidance. Medical Research Council; 2008. Accessible at: www.mrc.ac.uk/complexinterventionsguidance. [Google Scholar]
  18. Van Bokhoven M, Kok G. Van der Weijden T. Designing a quality improvement intervention: a systematic approach. In: Grol R, Baker R, Moss F, editors; Quality Improvement Research. London: BMJ Publishing Group; 2004. pp. 147–164. [Google Scholar]
  19. Austvoll-Dahlgren A. Helseth S. What informs parents' decision-making about childhood vaccinations? Journal of Advanced Nursing. 2010;66:2421–2430. doi: 10.1111/j.1365-2648.2010.05403.x. [DOI] [PubMed] [Google Scholar]
  20. Austvoll-Dahlgren A, Falk R. Helseth S. Cognitive factors predicting intentions to search for health information: an application of the Theory of Planned Behaviour. Health Information and Libraries Journal. 2012;29:296–308. doi: 10.1111/hir.12006. [DOI] [PubMed] [Google Scholar]
  21. Austvoll-Dahlgren A. Helseth S. Public health nurses' barriers and facilitators to the use of research in consultations about childhood vaccinations. Scandinavian Journal of Caring Sciences. 2012;26:271–278. doi: 10.1111/j.1471-6712.2011.00928.x. [DOI] [PubMed] [Google Scholar]
  22. Peerson A. Saunders M. Health literacy revisited: what do we mean and why does it matter? Health Promotion International. 2009;24:285–296. doi: 10.1093/heapro/dap014. [DOI] [PubMed] [Google Scholar]
  23. Freedman DA, Bess KD, Tucker HA, Boyd DL, Tuchman AM. Wallston KA. Public health literacy defined. American Journal of Preventive Medicine. 2009;36:446–451. doi: 10.1016/j.amepre.2009.02.001. [DOI] [PubMed] [Google Scholar]
  24. Oxman A. Flottorp S. An overview of strategies to promote implementation of evidence-based health care. In: Silagy C, Haines A, editors; Evidence-Based Practice in Primary Care. 2nd edn. London: BMJ Publishing Group; 2001. pp. 101–119. [Google Scholar]
  25. Grol R. Wensing M. Selection of strategies. In: Grol R, Wensing M, Eccles M, editors; Improving Patient Care the Implementation of Change in General Practice. London: Elsevier limited; 2005. pp. 122–134. [Google Scholar]
  26. Baker R, Camosso-Stefinovic J, Gillies C, Shaw E, Cheater F, Flottorp S. Roberison N. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews. 2010;3:CD005470. doi: 10.1002/14651858.CD005470.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Schwartz L, Woloshin S, Black W. Welch H. The role of numeracy in understanding the benefit of screening mammography. Annals of Internal Medicine. 1997;127:966–972. doi: 10.7326/0003-4819-127-11-199712010-00003. [DOI] [PubMed] [Google Scholar]
  28. Woloshin S, Schwartz LM, Black WC. Welch HG. Women's perceptions of breast cancer risk: how you ask matters. Medical Decision Making. 1999;19:221–229. doi: 10.1177/0272989X9901900301. [DOI] [PubMed] [Google Scholar]
  29. Peterson G, Aslani P. Williams KA. How do consumers search for and appraise information on medicines on the Internet? A qualitative study using focus groups. Journal of Medical Internet Research. 2003;5:e33. doi: 10.2196/jmir.5.4.e33. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Sillence E, Briggs P, Harris PR. Fishwick L. How do patients evaluate and make use of online health information? Social Science & Medicine. 2007;64:1853–1862. doi: 10.1016/j.socscimed.2007.01.012. [DOI] [PubMed] [Google Scholar]
  31. Lloyd A. The extent of patients' understanding of the risk of treatments. Quality in Health Care. 2001;10(Suppl 1):i14–i18. doi: 10.1136/qhc.0100014... [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Weinstein N. What does it mean to understand a risk? Evaluating risk comprehension. Journal of the National Cancer Institute. 1999;25:15–20. doi: 10.1093/oxfordjournals.jncimonographs.a024192. [DOI] [PubMed] [Google Scholar]
  33. Tickner S, Leman P. Woodcock A. Factors underlying suboptimal childhood immunisation. Vaccine. 2006;50:7030–7036. doi: 10.1016/j.vaccine.2006.06.060. [DOI] [PubMed] [Google Scholar]
  34. Robinson E, Kerr C, Stevens A, Lilford R, Braunholtz D. Edwards S, et al. Lay Public's Understanding of Equipoise and Randomisation in Randomised Controlled Trials. Research Support, Non-U.S. Gov't. NHS R&D HTA Programme, 2005 Mar. Report No.: 1366-5278 (Linking) Contract No.: 8. [DOI] [PubMed]
  35. Stead M, Eadie D, Gordon D. Angus K. “Hello, hello – it's English I speak”! A qualitative exploration of patients' understanding of the science of clinical trials. Journal of Medical Ethics. 2005;31:664–669. doi: 10.1136/jme.2004.011064. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Fox S. Jones S. The Social Life of Health Information. 2009. p. 88. Accessible at: http://www.pewinternet.org/Reports/2009/8-The-Social-Life-of-Health-Information.aspx. [Google Scholar]
  37. Schwartz K, Roe T, Northrup J, Meza J, Seifeldin R. Neale A. Family medicine patients' use of the Internet for health information: a MetroNet study. Journal of the American Board of Family Medicine. 2006;19:39–45. doi: 10.3122/jabfm.19.1.39. [DOI] [PubMed] [Google Scholar]
  38. Eysenbach G. Kohler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. The British Medical Journal. 2002;324:573–577. doi: 10.1136/bmj.324.7337.573. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Legare F, Ratte S, Gravel K. Graham I. Barriers and facilitators to implementing shared decision-making in clinical practice: update of a systematic review of health professionals' perceptions. Patient Education and Counseling. 2008;73:526–535. doi: 10.1016/j.pec.2008.07.018. [DOI] [PubMed] [Google Scholar]
  40. Bugge C, Entwistle V. Watt I. The significance for decision-making of information that is not exchanged by patients and health professionals during consultations. Social Science & Medicine. 2006;63:2065–2078. doi: 10.1016/j.socscimed.2006.05.010. [DOI] [PubMed] [Google Scholar]
  41. Detmer D, Singleton P, MacLeod S, Wait S, Taylor M. Ridgwell J. The Informed Patient: Study Report. Cambridge: University of Cambridge; 2003. p. 53. [Google Scholar]
  42. Coulter A. Jenkinson C. European patients' views on the responsiveness of health systems and healthcare providers. European Journal of Public Health. 2005;15:355–360. doi: 10.1093/eurpub/cki004. [DOI] [PubMed] [Google Scholar]
  43. Andreassen H, Sandaune A, Gammon D. Hjortdahl P. Norwegian use of Internet health services. Tidsskrift for Den norske Legeforening. 2002;122:1640–1644. [PubMed] [Google Scholar]
  44. Fox S. Online Health Search. 2006. p. 122. Accessible at: http://www.pewinternet.org/PPF/r/190/report_display.asp. [Google Scholar]
  45. Glenton C. Oxman A. The use of evidence by health care user organizations. Health Expectations. 1998;1:14–22. doi: 10.1046/j.1369-6513.1998.00002.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Johansen L, Bjørndal A, Flottorp S, Grøtting T. Oxman A. Evaluation of health information in newspapers and brochures. What can we believe? Tidsskrift for Den norske Legeforening. 1996;116:260–264. [PubMed] [Google Scholar]
  47. Hoye S. Hjortdahl P. “New wonder pill!”–what do Norwegian newspapers write. Tidsskrift for Den norske Legeforening. 2002;17:1671–1676. [PubMed] [Google Scholar]
  48. Wangberg S, Andreassen H, Kummervold P, Wynn R. Sorensen T. Use of the internet for health purposes: trends in Norway 2000–2010. Scandinavian Journal of Caring Sciences. 2009;23:691–696. doi: 10.1111/j.1471-6712.2008.00662.x. [DOI] [PubMed] [Google Scholar]
  49. Spadaro R. European Union Citizens and Sources of Information about Health: Eurobarometer 58.02003 24.01.2012. Accessible at: http://ec.europa.eu/public_opinion/archives/ebs/ebs_179_en.pdf.
  50. Wantland D, Portillo C, Holzemer W, Slaughter R. McGhee E. The effectiveness of Web-based vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes. Journal of Medical Internet Research. 2004;6:e40. doi: 10.2196/jmir.6.4.e40. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Webb T, Joseph J, Yardley L. Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. Journal of Medical Internet Research. 2010;12:e4. doi: 10.2196/jmir.1376. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Murray E, Burns J, See TS, Lai R. Nazareth I. Interactive health communication applications for people with chronic disease. Cochrane Database of Systematic Reviews. 2005;4:CD004274. doi: 10.1002/14651858.CD004274.pub4. [DOI] [PubMed] [Google Scholar]
  53. Lewis D. Computer-based approaches to patient education: a review of the literature. Journal of the American Medical Informatics Association. 1999;6:272–282. doi: 10.1136/jamia.1999.0060272. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Trevena L, Davey H, Barratt A, Butow P. Caldwell P. A systematic review on communicating with patients about evidence. Journal of Evaluation in Clinical Practice. 2006;12:13–23. doi: 10.1111/j.1365-2753.2005.00596.x. [DOI] [PubMed] [Google Scholar]
  55. Staley K. Exploring Impact: Public Involvement in NHS, Public Health and Social Care Research. Eastleigh: INVOLVE; 2009. [Google Scholar]
  56. Health on the Net Foundation (HON) 2010. Trustworthy Health and Medical Information: The Health on the Net Initiative [cited 2010 30.11.2010]. Accessible at: http://www.hon.ch/HONcode/Webmasters/
  57. Keselman A, Logan R, Smith CA, Leroy G. Zeng-Treitler Q. Developing Informatics tools and strategies for consumer-centered health communication. Journal of the American Medical Informatics Association. 2008;15:473–483. doi: 10.1197/jamia.M2744. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Epstein R, Alper B. Quill T. Communicating evidence for participatory decision making. Journal of American Medical Association. 2004;291:2359–2366. doi: 10.1001/jama.291.19.2359. [DOI] [PubMed] [Google Scholar]
  59. Bjørndal A, Flottorp S. Klovning A. Knowledge Management in Medicine and Health Sciences. Oslo: Gyldendal akademisk; 2007. p. 195. [Kunnskapshåndtering i medisin og helsefag]. [Google Scholar]
  60. Woloshin S, Schwartz L. Welch H. Know Your Chances: Understanding Health Statistics. London: University of California Press; 2008. p. 142. [PubMed] [Google Scholar]
  61. Irwig L. Smart Health Choices: Making Sense of Health Advice. London: Hammersmith; 2008. p. 242. [PubMed] [Google Scholar]
  62. Benestad H. Laake P. Research Methodology in the Medical and Biological Sciences. Oslo: Gyldendal akademisk; 2004. p. 476. [Google Scholar]
  63. Johannessen A, Tufte P. Kristoffersen L. Introduction to Social Science Methodology. Oslo: Abstrakt forlag as; 2007. p. 382. [Google Scholar]
  64. Regionale forskningskomiteer. 2012. Research Ethics [24.01.2012]. Accessible at: http://www.etikkom.no/no/Forskningsetikk/
  65. Kim P, Eng T, Deering M. Maxfield A. Published criteria for evaluating health related web sites: review. The British Medical Journal. 1999;318:647–649. doi: 10.1136/bmj.318.7184.647. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Gagliardi A. Jadad A. Examination of instruments used to rate quality of health information on the internet: chronicle of a voyage with an unclear destination. The British Medical Journal. 2002;324:569–573. doi: 10.1136/bmj.324.7337.569. [DOI] [PMC free article] [PubMed] [Google Scholar]
  67. Dickersin K, Braun L, Mead M, Millikan R, Wu AM, Pietenpol J, Troyan S, Anderson B. Visco F. Development and implementation of a science training course for breast cancer activists: project LEAD (leadership, education and advocacy development) Health Expectations. 2001;4:213–220. doi: 10.1046/j.1369-6513.2001.00153.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  68. Milne R. Oliver S. Evidence-based consumer health information: developing teaching in critical appraisal skills. International Journal for Quality in Health Care. 1996;8:439–445. doi: 10.1093/intqhc/8.5.439. [DOI] [PubMed] [Google Scholar]
  69. Berger B, Steckelberg A, Meyer G, Kasper J. Muhlhauser I. Training of patient and consumer representatives in the basic competencies of evidence-based medicine: a feasibility study. BMC Medical Education. 2010;10:1–9. doi: 10.1186/1472-6920-10-16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Charnock D, Shepperd S, Needham G. Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology and Community Health. 1999;53:105–111. doi: 10.1136/jech.53.2.105. [DOI] [PMC free article] [PubMed] [Google Scholar]
  71. Raffe A. Gray J. Screening: Evidence and Practice. Oxford: Oxford University Press; 2007. [Google Scholar]
  72. Stacey D, Bennett C, Barry M, Col N, Eden K, Holmes-Rovner M, Llewellyn-Thomas H, Lyddiatt A, Légaré F. Thomson R. Decision aids for people facing health treatment or screening decisions. Cochrane Database of Systematic Reviews. 2011;10:CD001431. doi: 10.1002/14651858.CD001431.pub3. [DOI] [PubMed] [Google Scholar]
  73. IPDASI. 2005. Criteria for Judging the Quality of Patient Decision Aids [cited 2010 18. 05]. Accessible at: http://www.ipdas.ohri.ca/what.html.
  74. DiCenso A, Bayley L. Haynes R. Accessing preappraised evidence: fine-tuning the 5S model into a 6S model. Annals of Internal Medicine. 2009;151:99–101. doi: 10.7326/0003-4819-151-6-200909150-02002. [DOI] [PubMed] [Google Scholar]
  75. Charnock D. 1998. p. 65. Håndboken DISCERN. Kvalitetskriterier for helseinformasjon til forbrukere om hvilke behandlingsalternativer som finnes. Accessible at: http://www.nifab.no/pasientveiledning/hvordan_vurdere_informasjon/discern_verktoey_for_evaluering_av_helseinformasjon.
  76. O'Connor A, Stacey D. Jacobsen M. 2007. Ottawa Personal Decision Guide [24.01.12]. Accessible at: http://decisionaid.ohri.ca/docs/das/OPDG.pdf.
  77. Statistics Norway. 2011. Use of the Internet, Frequency and Location of Use Last 3 Months. Percentage of Population, By Sex, Age, Education and Employment Situation. 2nd Quarter 2010 Statistics Norway. [cited 2011 19.05]. Accessible at: http://www.ssb.no/english/subjects/10/03/ikthus_en/tab-2010-09-24-04-en.html.

Articles from Health Information and Libraries Journal are provided here courtesy of Wiley

RESOURCES