Key concepts for informed health choices: Where’s the evidence?

Version Changes

Revised. Amendments from Version 1

In response to the Reviewer Reports, we made the following changes. We added information to the Introduction and the Methods, including references. We removed some redundant text, and we made some minor edits. Detailed responses to each of the reviewers’ comments, including the changes that we made, can be found in our responses to the Reviewer Reports. A minor spelling correction was made in Tables 1 and 4, and a footnote was added to Table 2. A duplicate file was removed from the Underlying data.

Abstract

Background

The Informed Health Choices (IHC) Key Concepts is a framework that provides a basis for developing educational resources and evaluating people’s ability to think critically about health actions. We developed the original Key Concepts framework by reviewing texts and checklists for the public, journalists, and health professionals and collecting structured feedback from an international advisory group. We revised the original 2015 framework yearly from 2016 to 2018 based on feedback and experience using the framework. The objectives of this paper are to describe the development of the framework since 2018 and summarise their basis.

Methods

For the 2019 version, we responded to feedback on the 2018 version. For the current 2022 version, in addition to responding to feedback on the 2019 version, we reviewed the evidence base for each of the concepts. Whenever possible, we referenced systematic reviews that provide a basis for a concept. We screened all Cochrane methodology reviews and searched Epistemonikos, PubMed, and Google Scholar for methodology reviews and meta-epidemiological studies.

Results

The original framework included 32 concepts in six groups. The 2019 version and the current 2022 version include 49 concepts in the same three main groups that we have used since 2016. There are now 10 subgroups or higher-level concepts. For each concept, there is an explanation including one or more examples, the basis for the concept, and implications. Over 600 references are cited that support the concepts, and over half of the references are systematic reviews.

Conclusions

There is a large body of evidence that supports the IHC key concepts and we have received few suggestions for changes since 2019.

Introduction

At the end of the last century, at least 28 frameworks were published describing the need for competencies and curriculum changes for primary and secondary schools in the 21 ^st century. ¹ Since then, many countries have moved or are moving from knowledge-based to competence- or skill-based primary and secondary school curricula. ² Critical thinking is one of the most often included competences. ^{1 – 3}

Critical thinking is not a new idea. It dates at least to Socrates over 2,400 years ago, and the need to teach critical thinking in school has been argued for over 100 years. ^{4 , 5} There are many different definitions of critical thinking, ⁵ and debate about how it should be taught. ⁶ However, generally, the focus is on dispositions and competences that help people to decide what to believe or do.

A major new challenge to deciding what to believe or do is increased access to information online and in social media, and the need to evaluate that information, much of which is misinformation. ⁴ A huge amount of health information can be found online, in addition to information that is disseminated through other channels of communication. This problem has been exacerbated by the COVID-19 pandemic, which has been accompanied by an ’infodemic’ – an overload of information including false or misleading information. ⁷

In the context of health, the skills needed to decide what to believe or do are sometimes referred to as critical health literacy. ^{8 , 9} Although both critical thinking and health are widely included in primary and secondary school curricula, critical thinking about health or critical health literacy may not be, ^{10 – 13} and many people find it difficult to make decisions about what to believe or do regarding ’treatments’ or ’health actions’ (things that they can do to care for their health or the health of others). ¹⁴

Being able to understand and apply some basic principles or concepts is essential for using reliable information appropriately and avoiding being misled by unreliable information. This includes concepts about claims, comparisons (research evidence from treatment comparisons), and choices ( Table 1). As noted by Dewey, ¹⁵ “it would be impossible to overestimate the educational importance of arriving at conceptions: that is, of meanings that are general because applicable in a great variety of different instances in spite of their difference; that are constant, uniform, or self-identical in what they refer to, and that are standardized, known points of reference by which to get our bearings when we are plunged into the strange and unknown.” ¹⁶

Table 1. Three main groups of concepts and 10 high-level concepts in the 2022 version of the Informed Health Choices (IHC) Key Concepts. ¹⁸ .

1. Claims	2. Comparisons	3. Choices
Claims about effects that are not supported by evidence from fair comparisons are not necessarily wrong, but there is an insufficient basis for believing them.	Studies should make fair comparisons, designed to minimize the risk of systematic errors (biases) and random errors (the play of chance).	What to do depends on judgements about a problem, the relevance of the available evidence, and the balance of expected benefits, harms, and costs.
1.1 Assumptions that treatments are safe or effective can be misleading. 1.2 Seemingly logical assumptions about research can be misleading. 1.3 Seemingly logical assumptions about treatments can be misleading. 1.4 Trust based on the source of a claim alone can be misleading.	2.1 Comparisons of treatments should be fair. 2.2 Reviews of the effects of treatments should be fair. 2.3 Descriptions of effects should clearly reflect the size of the effects. 2.4 Descriptions of effects should reflect the risk of being misled by the play of chance.	3.1 Evidence should be relevant. 3.2 Expected advantages should outweigh expected disadvantages.

The Informed Health Choices (IHC) Network has developed a framework that includes 49 key concepts as a starting point for deciding what to teach. ^{17 , 18} The framework provides a basis for developing learning and teaching resources and evaluating learners’ ability to think critically about health actions. Most of the concepts in the framework are relevant to other types of actions or interventions, including agricultural, educational, and environmental interventions. ¹⁹

The concepts serve as the basis for developing learning resources to help people understand and apply the concepts when claims about the effects of health actions are made, and when they make health choices. ²⁰ They are also the basis for an item bank of multiple-choice questions (the Claim Evaluation Tools item bank) that can be used for assessing people’s ability to apply the IHC Key Concepts. ²

The concepts are principles for recognising when a claim about the effects of health actions has an untrustworthy basis, recognising when evidence from comparisons of health actions is trustworthy and when it is not, and making well-informed health choices. They can help anyone, not just researchers, to think critically about whether to believe a claim and what to do. The concepts are intended for people using research, not for doing research.

The concepts and the basis for the concepts tend to focus on ways in which claims and comparisons (evidence) can be misleading, and choices can be misinformed. This is not because we underestimate the tremendous gains that have been made in health care based on appropriate use of research. Our aim is to promote healthy scepticism, not excessive scepticism, and to help people decide what to believe and what to do. How sure we can be about the effects of health actions varies, as does how sure we can be about the balance between the advantages and disadvantages of health actions. When teaching, learning, or using these concepts, it is important to bear this in mind.

Although most people prefer to make medical decisions together with a health professional, some people prefer to delegate decisions to a health professional. ²¹ Unfortunately, health professionals’ perceptions of their patients’ desire to be involved in decisions are often inaccurate. ²² They may be more likely to underestimate the extent to which patients prefer to be involved in decisions. Regardless of who decides, decisions should be consistent with a patient’s values. Decision aids can help patients to clarify their values and may help them to make choices that are more consistent with their values compared to choices made without decision aids. ²³ There is some evidence that patients choose more conservative approaches when they become better informed. ²⁴

Even people who prefer to delegate medical decisions to a health professional are confronted with an over-abundance of information about things they can do for their health, much of which is unreliable. ^{7 , 25 – 30} Although much of this information can simply be ignored, to avoid being misled by claims with an untrustworthy basis, people need to be able to recognise claims about the effects of health actions, question the basis for those claims, and recognise when a claim is relevant and important, and warrants reflection.

People have the right to participate collectively as well as individually in the planning and implementation of their health care. ³¹ In addition to being a democratic right, public participation in deliberative and decision-making processes has the potential to improve the quality of the judgements and decisions that are made, build trust, improve adherence, and help to ensure transparency and accountability. ³² However, the extent to which potential benefits of public participation are achieved (and potential harms are avoided) depends, among other things, on the ability of citizens to think critically about collective decisions as well as personal choices.

People often disagree about the effects of health actions, including experts. Who makes a claim, how likable they are, or how much experience and expertise they have do not provide a reliable basis for assessing how reliable their claim is. This does not mean that conflicting opinions should be given equal weight – or that the existence of conflicting opinions means that no conclusion can be reached. It also does not mean that all sources of information are equally reliable. How much weight to give an opinion should be based on the strength of the evidence supporting it. How much trust to place in a source of information about health actions should be based on whether it provides information about the effects of health actions based on systematic reviews. ²⁵

The objective of this paper is to describe the development of the IHC Key Concepts from 2019 to 2022 and to summarise the development of the framework over the past decade, its basis, and how it has been used.

Development of the Informed Health Choices Key Concepts 2013-2018

We have previously summarized development of the IHC Key Concepts up to the 2018 version. ¹⁷ Development of the IHC Key Concepts started in 2013 as the first step in a five-year research project funded by the Research Council of Norway (GLOBVAC project 220603/H10). ³³ The objectives of this project were to develop and evaluate resources for primary schools and mass media to improve people’s ability to assess claims about the effects of treatments. We began by extracting all the concepts addressed in Testing Treatments, ³⁴ a book that was written to promote more critical public assessment of claims about the effects of treatments. We then searched the literature for other relevant material, including books and checklists for the public, journalists, and health professionals. ³⁵ Our aim has been to include all concepts that are important for people to consider, while minimising redundancy.

Initially, we collected structured feedback from an international advisory group using four questions: ³⁵

• 1.Are concepts included that should not be?
• 2.Are there important concepts that are missing?
• 3.Are the concepts organised in a logical way?
• 4.Do you have any other comments regarding the concepts?

We published the first version of the list in 2015. ³⁵ That list included 32 concepts in six groups. Subsequently, we collected feedback at a series of workshops in 2017 and 2018, ¹⁷ and from colleagues working in other fields besides health in 2018. ^{19 , 36} We have used four criteria in deciding on changes to the list of concepts. ¹⁷ New key concepts must:

• •be within the scope of the IHC Key Concepts – standards for judgment, or principles for evaluating the trustworthiness of treatment claims and treatment comparisons (research) used to support claims, and to inform treatment choices,
• •address ways in which treatment claims and comparisons are frequently misleading or ways in which poorly informed decisions are taken,
• •be useful for people without a research background to use research, not just for researchers or for doing research, and
• •overlap as little as possible with other key concepts.

In addition, revisions have been informed by a review of related frameworks in 2018, ³⁷ and using the concepts to:

• •systematically review the effects of educational interventions to improve people’s understanding of the key concepts, ^{38 , 39}
• •create a database of educational interventions to improve people’s understanding of the key concepts, ⁴⁰
• •develop and evaluate educational resources. ^{10 – 12 , 41 – 53}
• •develop an item bank and outcome measures with multiple-choice questions that assess an individual’s understanding of and ability to apply the key concepts, ^{54 – 59}
• •develop a plain language glossary of evaluation terms, ⁶⁰
• •ensure coverage of an international core curriculum for teaching evidence-based health care to health professional students, ^{20 , 61}
• •survey the public’s ability to assess treatment claims, ¹⁴ and
• •systematically review the quality of information in news reports about the effects of health interventions. ⁶²

We published revised versions yearly from 2016 to 2018. The 2016 version included 34 concepts in three groups, and the 2017 version included 36 concepts. The 2018 version included 44 concepts reorganised within each of the three main groups, and we added three subgroups (higher-level concepts) to each of the three main groups. This helped to clarify the logic behind how the concepts were organised and helped to make the long lists of concepts less overwhelming.

Methods

The following invitation was included in the 2018 and 2019 versions: Please send any comments or suggestions to: contact@informedhealthchoices.org. In addition, we asked people to provide feedback at workshops where we presented the Key Concepts. Seven individuals provided feedback on the 2018 version and four provided feedback on the 2019 version. The suggestions came from members of the IHC Network and other researchers.

For the 2019 version, we reviewed and responded to feedback on the 2018 version. For the current, 2022 version, in addition to reviewing and responding to feedback on the 2019 version, ^{17 , 63} we reviewed the evidence base for each of the concepts. For each concept, we have provided one or more examples to illustrate the explanation, and the basis for the concept. The examples, for the most part, were taken from relevant methodological research. Due to the nature of the research, most of the examples are medical. We selected examples that clearly illustrate the concept and that can easily be understood with little explanation by a wide audience without a medical background.

Whenever possible, we have referenced systematic reviews that provide a basis for a concept. We started with reviews with which we were familiar, including some that we had co-authored. Additional systematic reviews were identified by searching and screening the following sources:

• •All Cochrane methodology reviews ⁶⁴ (n = 36, on 29 June 2021)
• •Epistemonikos ⁶⁵ using the terms “methodology review” OR “meta-epidemiological” in the title or abstract (n = 161, on 11 October 2021)
• •PubMed using the terms “methodology review” OR “meta-epidemiological” (n = 193, on 11 October 2021)
• •Google Scholar using the terms “methodology review” OR “meta-epidemiological” in the title, restricted to “Review articles” (n = 370, on 11 October 2021)

We did not restrict searches or exclude reviews based on language or the date of publication.

In addition, we searched Epistemonikos, PubMed, and Google Scholar for systematic reviews that support the explanation for each concept using key terms or phrases from the explanation or related terms. These searches were conducted and screened by one of the authors (ADO). The searches varied for each concept. For example, when our initial searches did not find a recent systematic review, we used citation searches in Google Scholar to search for a recent review or more recent research. We did not record the searches that were conducted for each concept. The purpose of these searches was to summarise the evidence supporting each concept, in addition to the logical explanations.

We included systematic reviews of methodological studies and overviews of reviews, such as Cochrane methodology reviews, methodological studies based on a systematic review of research (“meta-epidemiological” studies), and systematic reviews of treatment effects. We categorised as systematic reviews any review of research that included a methods section that described the search strategy for finding studies and selection criteria.

Ethical considerations

This research was undertaken as part of two larger projects funded in part by the Research Council of Norway (Project numbers 220603/H10 and 284683). Because the projects will not generate new knowledge about health and disease, approval by the Regional Committee for Medical Research Ethics in Norway was not required, as confirmed by the committee (reference number 30713).

The only people who participated directly in this research were people who provided feedback on earlier versions of the IHC Key Concepts framework. That feedback was given voluntarily with the understanding that it would be used to improve the framework.

Results

The 2019 version of the framework included 49 concepts. ⁶³ We added five new concepts in response to feedback:

• •Assumptions that fair comparisons are not relevant can be misleading.
• •Your own prior beliefs may be wrong.
• •Consider the baseline risk or the severity of the symptoms when estimating the size of expected effects.
• •Consider how important each advantage and disadvantage is when weighing the pros and cons.
• •Important uncertainties about the effects of treatments should be reduced by further fair comparisons.

We organised the concepts in the same way as in the 2018 version of the framework, under three higher-level concepts in each of the three main groups ( Table 2).

Table 2. Overview of revisions to the Informed Health Choices (IHC) Key Concepts.

Version	Groups	Subgroups (higher-level concepts)	Concepts	Competences	Dispositions
2015 35	1. Recognising the need for fair comparisons of treatments. 2. Judging whether a comparison of treatments is a fair comparison. 3. Understanding the role of chance 4. Considering all the relevant fair comparisons. 5. Understanding the results of fair comparisons of treatments. 6. Judging whether fair comparisons of treatments are relevant.		32
2016 66	1. Claims 2. Comparisons 3. Choices		34
2017 67			36
2018 68		1. Beware of treatment claims like these 1.1 Beware of claims that are too good to be true. 1.2 Beware of claims based on faulty logic. 1.3 Beware of claims based on trust alone. 2. Check the evidence from treatment comparisons 2.1 Don’t be misled by unfair comparisons. 2.2 Don’t be misled by unreliable summaries of treatment comparisons. 2.3 Don’t be misled by how treatment effects are described. 3. Make well-informed treatment choices 3.1 What is the problem and what are the options? 3.2 Is the evidence relevant? 3.3 Do the advantages outweigh the disadvantages?	44	10	10
2019 63			49	20 *	15 *
2022 18		( Table 1)	49	20	15

^*The competences and dispositions were reformulated and reorganised, as well as expanded.

In response to feedback, ⁶⁹ we also edited the list of concepts in the 2019 version of the framework to make their descriptions more consistent, and we edited some of the explanations. We also clarified our goal ( Table 3), increased the list of competences needed to achieve that goal from 10 to 20, and increased the list of dispositions from 10 to 15.

Table 3. Goal of the Informed Health Choices (IHC) Key Concepts.

Our goal is to enable people to make good decisions * about which claims to believe about the effects of things they can do for their health, the health of others or for other reasons, and about what to do to achieve their goals.

^*A good decision is one that makes effective use of the information available to the decision maker at the time the decision is made. A good outcome is one that the decision maker likes. The aim of thinking critically about treatments is to increase the probability of good outcomes (and true conclusions), but many other factors affect outcomes aside from critical thinking (for example genetic and environmental factors). ⁷⁰

We received little feedback on the 2019 version (10 suggestions) and decided that revisions were not needed in 2020 and 2021. ⁶⁹ The 2022 version ¹⁸ includes the same concepts as the 2019 version. We now provide examples in the explanation for each concept and the basis for each concept, as well as the implications. The 2022 version includes over 600 references, over half of which are to systematic reviews.

We incorporated eight suggestions in the explanations for concepts in this version. For example, a suggestion to include the concept that harms can be irreversible was incorporated in the explanation for the concept “do not assume that treatments are safe” by adding the following to the explanation for that concept: “The harm that is caused may be minor, but treatments also sometimes cause serious, irreversible harms, including death.” ⁶⁹ We also reorganised the concepts into four subgroups (high-level concepts) within each of the first two main groups (claims and comparisons) and into two subgroups within the third main group (choices) ( Table 1). We did this to make the organisation of the concepts more logical and the long list of concepts in some of the subgroups less overwhelming. The 10 high-level concepts also make it easier to get the gist of the concepts and makes the list for some of the subgroups less overwhelming and easier to remember. Table 4 is an overview of the 49 concepts in the three main groups and 10 subgroups.

Table 4. Overview of the Informed Health Choices (IHC) Key Concepts. ¹⁸ .

1. Claims	2. Comparisons	3. Choices
Claims about effects that are not supported by evidence from fair comparisons are not necessarily wrong, but there is an insufficient basis for believing them.	To identify treatment effects, studies should make fair comparisons, designed to minimize the risk of systematic errors (biases) and random errors (the play of chance).	What to do depends on judgements about a problem, the relevance of the available evidence, and the balance of expected benefits, harms, and costs.
1.1 Assumptions that treatments are safe or effective can be misleading. Do not assume that a)treatments are safe, b)treatments have large, dramatic effects, c)treatment effects are certain, d)it is possible to know who will benefit and who will be harmed, or e)comparisons are not needed. 1.2 Seemingly logical assumptions about research can be misleading. Do not assume that a)a plausible explanation is sufficient, b)association is the same as causation, c)more data is better data, d)a single study is sufficient, or e)fair comparisons are not applicable in practice. 1.3 Seemingly logical assumptions about treatments can be misleading. Do not assume that a)treatment is needed, b)more treatment is better, c)a treatment is helpful or safe based on how widely used it is or has been, d)a treatment is better based on how new or technologically impressive it is, or e)earlier detection of ‘disease’ is better. 1.4 Trust based on the source of a claim alone can be misleading. Do not assume that a)personal experiences alone are sufficient, b)your beliefs are correct, c)opinions alone are sufficient, d)peer review and publication is sufficient, or e)there are no competing interests.	2.1 Comparisons of treatments should be fair. Consider whether a)the people being compared were similar, b)the people being compared were cared for similarly, c)the people being compared knew which treatments they received, d)outcomes were assessed similarly in the people being compared, e)outcomes were assessed reliably, f)outcomes were assessed in all (or nearly all) the people being compared, and g)people’s outcomes were analysed in the group to which they were allocated. 2.2 Reviews of the effects of treatments should be fair. Consider whether a)systematic methods were used, b)unpublished results were considered, c)treatments were compared across studies, and d)important assumptions were tested. 2.3 Descriptions of effects should clearly reflect the size of the effects. Be cautious of a)verbal descriptions alone of the size of effects, b)relative effects of treatments alone, c)average differences between treatments, and d)lack of evidence being interpreted as evidence of “no difference”. 2.4 Descriptions of effects should reflect the risk of being misled by the play of chance. Be cautious of a)small studies, b)results for a selected group of people within a study, c)p-values, and d)results reported as “statistically significant” or “non-significant”.	3.1 Evidence should be relevant. a)Be clear about what the problem or goal is and what the options are. Consider the relevance of b)the outcomes measured in the research, c)fair comparisons in laboratories, animals, or highly selected people, d)the treatments that were compared, and e)the circumstances in which the treatments were compared. 3.2 Expected advantages should outweigh expected disadvantages. a)Weigh the benefits and savings against the harms and costs of acting or not. Consider b)the baseline risk or severity of the symptoms when estimating the size of expected effects, c)how important each advantage and disadvantage is when weighing the pros and cons, d)how certain you can be about each advantage and disadvantage, and e)the need for further fair comparisons.

Discussion

We made many changes to the IHC Key Concepts after they were first published in 2015. ³⁵ The original version included 32 concepts in six groups. There are now 49 concepts in three main groups and 10 subgroups. In addition, there are 20 competences and 15 dispositions. There have been few suggestions for changes to the 2019 version and we have made only minor changes to the explanations for some of the concepts. We therefore have decided that the 2022 version will be the last revision made by us. ¹⁸ This does not mean that this list of concepts cannot be further improved, but we will leave any further development of the IHC Key Concepts to others.

Although we have attempted to use plain language in describing the key concepts and their basis, the list of key concepts is not intended to be a learning resource for people with no relevant research background. It is a framework, or starting point, for identifying and developing learning resources and evaluation tools. It has proven to be useful for this. ^{43 , 48 , 57 , 71 – 73}

We have organised the concepts logically. Although it may sometimes make sense to organise learning resources using the same logic, the logic that is used does not reflect the difficulty of the concepts or the order in which the concepts should be learned. Moreover, the full list of concepts can be overwhelming, and it is likely to be necessary to prioritise which concepts to include in learning resources and evaluation tools. ⁵¹ For example, some concepts are too difficult for primary school children to understand and use, and it may not be possible to incorporate all the concepts in secondary and primary school curricula. ^{41 , 51}

Ideally, the concepts should be taught and learned using a spiral curriculum, that maps out what students should learn, where they should begin, and how they should progress. ^{74 – 76} However, there are many other demands on what to include in primary and secondary school (and other) curricula. ⁷⁷ This is reflected in the findings of a process evaluation of the IHC primary school intervention in Uganda. ⁴⁴ The intervention was shown to have a large effect on primary school children’s ability to think critically about health claims, ⁴³ which was sustained after one year. ⁴⁵ Teachers who used the primary school intervention in the trial thought the IHC Key Concepts were important. They were motivated to teach the concepts, and the children were enthusiastic about the lessons. Nonetheless, use of the resources has not been scaled up. A key barrier to scaling up use of the intervention was the need to incorporate the lessons in the national curriculum. The IHC lessons were viewed as an addition to what was already a packed primary school curriculum.

Conclusions

The IHC Key Concepts framework is central to critical thinking and evidence-based practice, both of which have broader scopes than this framework. ³⁷ An important weakness of frameworks with a broader focus is that they usually do not provide an adequate basis (i.e., necessary concepts) for thinking critically about claims about effects and decisions about what to do. The IHC Key Concepts are applicable to a great variety of claims about the effects of interventions, not just health interventions, ¹⁹ and they are essential points of reference for deciding which claims to believe and what to do.

There is a substantial body of evidence supporting the 49 concepts, as well as logic. Although it is possible to further improve this framework, we will leave any further development of the IHC Key Concepts to others. More importantly, there is a need to incorporate the key concepts into school curricula and to develop, evaluate, and scale up the use of effective educational interventions.

Data availability

Underlying data

Zenodo: Suggestions for changes to the IHC Key Concepts 2018-2022. https://doi.org/10.5281/zenodo.6849090. ⁶⁹

The project contains the following underlying data:

• •Suggestions for changes to the IHC Key Concepts 2018 – 2022.pdf. (Suggestions for improvements to the 2018 and 2019 versions of the Informed Health Choices Key Concepts and responses).

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Zenodo: Key Concepts for assessing claims about treatment effects and making well-informed treatment choices (Version 2022). http://doi.org/10.5281/zenodo.6611932. ¹⁸

The project contains the following underlying data:

• •Informed Health Choices Key Concepts 2022.pdf (Key Concepts for assessing claims about treatment effects and making well-informed treatment choices [Version 2022] - the 2022 version of the Informed Health Choices (IHC) Key Concepts).

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Funding Statement

This work was supported by the Research Council of Norway (Project numbers 220603/H10 and 284683). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.