A Consumer Health Informatics (CHI) Toolbox: Challenges and Implications

Abstract

Consumer health informatics (CHI) is a rapidly evolving sub-discipline of medical informatics. Such developing fields typically share common needs, such as harmonizing terms and building a common foundation of research methods and instruments. The authors describe a pilot study to conceptualize and develop a “CHI toolbox,” a repository of existing methods and instruments across relevant established fields. The challenges encountered in attempting to organize concepts in a nascent, interdisciplinary field are discussed. The authors’ experiences in creating a comprehensive CHI toolbox suggest that a larger, concerted effort to develop a similar product by members of the relevant research communities could accelerate the development of common terms, operational definitions, variables, and instruments within the CHI field.

INTRODUCTION

As the consumer health informatics (CHI) field matures, an increasing number of concepts, methods, and instruments will be derived from associated disciplines (e.g., health communication, mass communication, information science, and medical informatics). Many of these concepts, methods, and instruments will be integrated into an ongoing scholarly discourse about CHI.

While multidisciplinary cooperation is a positive development, normalizing terminology and standardizing the underlying conceptual issues derived from diverse disciplines are among the challenges of a growing hybrid field. Friedman and Wyatt explain that emerging disciplines, such as CHI, need a common ground for scholarship to grow within a linear, step-wise manner [1]. To create a common multidisciplinary framework, health informatics researchers need consistent operational definitions to describe terms as well as grounded variables and instruments. The use of consistent conceptual and methodological frameworks enables the faster creation of a body of evidence, which advances theories and working hypotheses into more grounded constructs. For example, the concept of consumer/patient self-efficacy might become a valuable construct in CHI if a consistent operationalization of the term is adopted and researchers begin to use identical scales to measure self-efficacy, either as an independent or dependent variable. However, the use of dissimilar terms (e.g., confidence) and different scales potentially delays the opportunity to advance the concept of self-efficacy as an evidence-based construct within the CHI literature. In short, the pace of growth in CHI research is partially associated with the adoption of common conceptual and methodological frameworks, which currently are in flux because of the diversity of disciplines that contribute to the field [2].

In this study, the authors attempt to help peers prepare for a common multidisciplinary terminological and conceptual convergence by exploring strategies to catalog formal tools and instruments used in consumer health-related research. This procedure was deliberately derived from a range of interdisciplinary, peer-reviewed journals. The authors’ original primary goal was to develop a “toolbox” of research and evaluation instruments to promote reuse of validated or semi-validated tools by CHI researchers. The effort was initiated to alleviate the “reinventing the wheel” phenomenon by enabling researchers to discover how conceptual frameworks, variables, and instruments are used in previous research within health communication, information science, medical informatics, mass communication, and other pertinent disciplines.

In conducting exploratory research to develop a strategy to identify, collect, and categorize the instruments (e.g., MeSH descriptors for primary topic), the authors observed conflicting nomenclature and diverse approaches for conceptualizing related ideas. The authors describe the approach used in a pilot study to develop a CHI toolbox. The challenges and high-level questions that arose during the project, which have greater implications for CHI, are discussed. Finally, we propose a comprehensive, multi-disciplinary, integrated effort to foster more consensus on the scope, foundations, and tools for the field.

BACKGROUND

While many consumer health information systems are available, there is relatively little data on how consumers seek information (consumer health information seeking or CHIS) and its effects on health outcomes. Unlike the “traditional” medical domain that is increasingly reliant on evidence-based medical research procedures, medical informatics does not yet have a well-developed system of evaluation, theory building, construct development, or a broader, among-study consistency (e.g., [3–5]).

For example, Eysenbach [6] called for registration of eHealth studies upon their inception at the International eHealth Study Registry in order to be considered for publication. Based on the International Committee of Medical Journal Editors (ICMJE) model [7], compulsory registration as a condition of publication will help create more consistent standards of reporting. While this is a welcome development, Friedman and Abbas [8] note there is a related, but more fundamental issue that should be addressed —study instrument validation (i.e., internal validity, reliability, and generalization). Based on a review of 27 medical informatics journal articles, they found that insufficient attention has been paid to validation of methodologies — what many may consider to be a hallmark of a “mature” discipline.

Inattention to measurement issues generally, and the paucity of reuse of measurement methods in particular, have likely consequences for scientific productivity in medical informatics. First and foremost, investigators performing outcome studies may have to do double work. They are required to develop—and ideally, test—their own measurement methodology before proceeding to execute the demonstration aspects of their studies. There are numerous hazards to this practice. (p. 269) [8]

While validated instruments in consumer health are becoming more common, the authors believe they are sufficiently rare to warrant a database of instruments across disciplines involved in CHI whereby investigators may share, or be informed by, available tools and the constructs they measure.

Two aspects of CHI research were explored in a pilot study:

• Conceptual issues, including devising an operational definition for consumer health research; selecting representative CHI-related disciplines; and selecting/creating common controlled terms across disciplines to represent common concepts

• Technical issues, such as identifying appropriate fields to represent any given instrument and to facilitate retrieval (again, by researchers in multiple disciplines) and extracting metadata from published journal articles, especially when the instrument is not published.

METHODS

An initial phase was conducted to develop a template to codify CHI research approaches. Friedman and Wyatt [1] provide a foundation to enumerate and define basic attributes of informatics research approaches. However, since their focus was a broader evaluation of medical informatics research, the authors adapted it for CHI research. Some basic research tools and approaches (e.g. a study’s instruments, variables, constructs, measurement scales and primary findings) were operationalized into a preliminary template, which codified how researchers approached some of the research approaches noted above.

The preliminary codification began by reviewing consumer health-related articles from the Journal of Internet Medical Research (JMIR) published since 1998.¹ The authors reviewed each JMIR article independently. After about a two-week period to code a few articles within a preliminary template, they met to discuss whether the template encompassed the diverse research approaches that emerged. During repeated meetings, the template was refined by developing new fields and guidelines.

After the authors reached a preliminary consensus on a template (Figure 1), the number of refereed journals from CHI-related disciplines was expanded to include: health communication, mass media, information science, medical informatics, psychology, and sociology. To focus on recent literature, the authors restricted the articles by publication dates (1994–2004, inclusive) and language (English). Ten databases were consulted, including the National Library of Medicine (NLM) PubMed^®, ComAbstracts², and PsycINFO³, using search strategies tailored to the domain and products.

Figure 1.

Key fields of the CHI toolbox template

To standardize the terminology of the toolbox based on the template, the NLM Medical Subjects Heading (MeSH^®) initially was selected. However, after initial extraction and coding, MeSH alone was found to be insufficient for non-medical domain CHI concepts (e.g., information seeking). Other challenges in using a controlled vocabulary to represent how scholars described key issues within diverse disciplines include: topic areas, constructs, variables, instruments, sampling issues, and even a study’s findings. As a result, the descriptive vocabulary used to codify terms was expanded beyond MeSH definitions and the authors began to use common terms from the psychology, sociology, information science, health communication, and mass communication literature. While this process did not change the template, it did present a new coding challenge: representing research within diverse fields precisely.

Once the template and range of terms within fields were stabilized, each author continued to select and review articles from different domain-specific journals. The resulting CHI toolbox represents a data set of 89 refereed research papers that either report CHI research, or research on consumer health, more broadly, using constructs, variables, or instruments applicable and/or similar to CHI research. (Four of the authors conducted the trial, template development, and implementation in a pilot study.)

Two of the four authors/reviewers had extensive research experience and two others were more experienced in development and implementation of medical controlled vocabularies. Intercoder agreement was measured by:

• Mean percentage of agreement across 15 of the empirically comparable fields used in the pilot study between the author with most collective experience in CHI, health communication and mass communication research and each of two authors with more controlled vocabulary experience and

• Percentage of agreement across the same 15 fields between the two authors with extensive research experience.

Twelve of 89 articles were randomly selected to calculate intercoder agreement. The mean intercoder agreement between authors with controlled vocabulary and extensive research experience was 64%. The intercoder agreement between the authors with extensive research experience was about 80%. Intercoder agreement, a measure of reliability, was measured by a formula proposed by Holsti [9]. The higher inter-coder agreement between judges with extensive research experience also provides modest evidence of the template’s face validity [1].

A secondary analysis compared the consistency between operational definitions of consumer health research. In addition, terms extracted from the original article were compared to MeSH citation terminology for the same manuscript. The intent was to demonstrate – visually and qualitatively –differences in definitions of consumer health research and differences between the publisher’s selection of topic terms and applicable MeSH headings. This analysis was conducted by one of the authors not involved in the pilot study.

RESULTS

The reported findings herein are selected from larger data sets. The entire data set for the pilot study/CHI toolbox as well as the secondary analysis could not be provided because of space constraints, but are available from the authors.

In the pilot study, n=89 refereed research papers were selected from 40 journals, representing 148 instruments in 8 disciplines (informatics, information science, psychology, sociology, mass media, public health, medicine, and nursing).

About 89% of the research papers reviewed in the pilot study represented demonstration studies. A demonstration study “uses the measured value of an attribute to draw conclusions about performance, perceptions, or effects of an information resource” [1]. In contrast, a measurement study “determines how well (accurately and precisely) an attribute of interest can be measured in a population of objects belonging to the same class” [1].

Among all 89 reviewed articles in the pilot study/CHI toolbox, about 10% of reported instruments used interval scales, while nearly 9% used nominal and 20% used ordinal scales respectively. About 19% of reported instruments were qualitative.

Approximately 17% of the papers reviewed in the pilot study/CHI toolbox reported significant reliability efforts, and 21% reported significant validity efforts. About 16% of the 89 papers reviewed in the pilot study/CHI toolbox reported both significant reliability and validity efforts. Finally, 173 unique theoretical constructs and 352 applied constructs were recorded.

The findings from the secondary analysis demonstrate visually some of the differences found among the articles reviewed in the pilot study/CHI toolbox (Table 1).

Table 1.

Differences in publisher keywords and MeSH terms from two articles from the secondary analysis (n=16)

Publisher Keywords	PubMed Citation (MeSH Terms)
Internet, patient education, communication, health status	Adult, Age Factors, Comparative Study, Continental Population Groups, *Delivery of Health Care, Female, *Health Status, Humans, Information Services/*statistics & numerical data, Internet/*statistics & numerical data, Male, Patient Education/statistics & numerical data/trends, Sex Factors
Internet, communication, doctor-patient relationship, decision-making, information, quality	Adolescent, Adult, *Communication, Comparative Study, Cross-Sectional Studies, Data Collection, Decision Making, Female, Humans, *Internet, Male, Middle Aged, Patient Participation, *Physician-Patient Relations, Primary Health Care/*organization & administration, Research Support, U.S. Government, P.H.S.

In the first example (Table 1), the key terms provided by the publisher omitted descriptions such as: delivery of health care; female; health status; Internet/statistics and numerical data that the secondary analysis found were both germane to the study and represented the most applicable MeSH terms. Conversely, the MeSH term descriptions omitted communication, a term that the article’s original author(s) used.

Qualitative differences in the operational definition of CHI research were found among 16 articles selected from the pilot study/CHI toolbox (Table 2). The differences among the descriptions illustrate inconsistency among operational definitions for one of the most basic questions within any CHI study: What is consumer health research?

Table 2.

Varied operational definitions of consumer health research from two articles from the secondary analysis (n=16)

	Definition
1	Use of Internet health information by two different groups of consumers: sick and healthy
2	“Consumer Health Informatics research contributes to the health care sector by attempting to systematize and codify consumer's needs, values and preferences and by trying to build and evaluate information systems that interact directly with consumers and patients.”

DISCUSSION

This pilot study suggests that across the disciplines where CHI is actively researched there are:

• Inconsistencies in existing use of an operational definition of “Consumer Health Research”

• Inconsistencies in creating a controlled vocabulary to codify the literature; MeSH and other existing terminologies are not well-developed for CHI. The field may require a “metathesaurus” approach cross-linking existing vocabularies

• Inconsistencies in validated instruments to support CHI researchers

• Needs for researchers to obtain an overview of the instruments, variables, constructs and measurement scales that are widely and idiosyncratically used.

The inconsistencies among and between the 89 articles selected for the pilot study/CHI toolbox suggest the challenges scholars face in finding common operational definitions, terms, grounded instruments, consistently named variables and instruments in research scattered across the CHI disciplines. Even well-intentioned scholars, who are motivated to comprehensively review the CHI research literature and utilize existing operational definitions, find grounded instruments, identify narrowly defined variables and utilize optimum instruments, will find the field’s status quo does not provide much consensus guidance.

Among many examples, the current dissimilarity of terms, scales, and instruments hinders the opportunity to empirically evolve single concepts, such as self-efficacy, into grounded constructs. Similarly, the content validity and, potentially, the criterion-related validity of studies are undermined by inconsistent operational definitions, such as consumer health research [1]. The resulting uncertainty confirms the flux that Napoli [2] as well as Friedman and Abbas [8] found in the development and use of common conceptual and methodological frameworks, which thwarts a linear development of the CHI field.

Conversely, the creation of a CHI toolbox and this pilot study reinforce the vitality of approaches in other disciplines where scholars investigate CHI-related issues. The CHI toolbox also reinforces the importance that CHI researchers learn from the experience of other “meta-disciplines.” Understanding how such other disciplines conceive, operationalize, and investigate how consumers use health and other sources of information on the Internet should facilitate the development of a coherent field, as long as common conceptual and methodological frameworks emerge.

CONCLUSIONS

The authors conducted a pilot study to develop a CHI toolbox, a repository of methodologies and evaluation instruments from relevant disciplines. The study’s primary goal was to support and facilitate research within the CHI community. In the process of conceptualizing and designing the toolbox, the macroscopic challenges that emerged included:

• Understanding the underlying challenges facing the multidisciplinary, yet intersecting, domains that are related to consumer health research and

• Creating a framework to encompass these differences.

Some more specific challenges that emerged included: devising an operational definition of consumer health research; selecting representative CHI-related disciplines; selecting/creating common controlled terms across disciplines to represent common concepts; identifying appropriate fields to represent any given instrument and to facilitate retrieval by researchers in multiple disciplines; and extracting metadata from published journals.

In standardizing definitions and defining appropriate fields, the lack of consistent operational definitions, constructs, instruments, and dependent and independent variables was operant within the multidisciplinary literature that examines how consumers converge on health informatics and health information Web sites.

While results of the CHI toolbox pilot study support the notion that existing instruments might be adopted (e.g., REALM [10] for consumer health literacy) or refined/adapted (e.g., questionnaires) for other studies, they also highlighted a need for formal validation of the tools (e.g., generalizability, reliability), before being applied more broadly. Further, the process of designing a CHI toolbox revealed potential opportunities to accelerate (and support) the convergence of CHI component disciplines. In particular, necessary conditions include (1) an operational definition of consumer health research; (2) standardization of key terms across disciplines (e.g., a “CHI metathesaurus”; and (3) systematic characterization and validation of new and existing tools. Since a CHI toolbox project seems to be important to the field’s development, the authors propose that the endeavor might be led by organizations such as AMIA CHI-WG with significant support from the CHI community. We believe that the issues described are not unique to CHI and may be applied broadly to any multidisciplinary field (e.g., “social informatics”).