A scoping review of qualitative research in JAMIA: past contributions and opportunities for future work

Abstract

Objective

Qualitative methods are particularly well-suited to studying the complexities and contingencies that emerge in the development, preparation, and implementation of technological interventions in real-world clinical practice, and much remains to be done to use these methods to their full advantage. We aimed to analyze how qualitative methods have been used in health informatics research, focusing on objectives, populations studied, data collection, analysis methods, and fields of analytical origin.

Methods

We conducted a scoping review of original, qualitative empirical research in JAMIA from its inception in 1994 to 2019. We queried PubMed to identify relevant articles, ultimately including and extracting data from 158 articles.

Results

The proportion of qualitative studies increased over time, constituting 4.2% of articles published in JAMIA overall. Studies overwhelmingly used interviews, observations, grounded theory, and thematic analysis. These articles used qualitative methods to analyze health informatics systems before, after, and separate from deployment. Providers have typically been the main focus of studies, but there has been an upward trend of articles focusing on healthcare consumers.

Discussion

While there has been a rich tradition of qualitative inquiry in JAMIA, its scope has been limited when compared with the range of qualitative methods used in other technology-oriented fields, such as human–computer interaction, computer-supported cooperative work, and science and technology studies.

Conclusion

We recommend increased public funding for and adoption of a broader variety of qualitative methods by scholars, practitioners, and policy makers and an expansion of the variety of participants studied. This should lead to systems that are more responsive to practical needs, improving usability, safety, and outcomes.

INTRODUCTION

“Since designers do not routinely visit work sites or talk to users, they are unlikely to come across information that would cast doubt on the generalized beliefs about work and users on which their systems are based. Lacking such data—and excluding as unscientific the informal, local information that could help them to design systems better suited to real users in particular workplaces—it is little wonder that these scientists produce systems that users do not want to use.”

-Diana E. Forsythe¹

The above epigraph was written about 2 decades ago by Diana Forsythe,¹ on the state of user research in medical informatics. At the time, qualitative methods were rarely used in health informatics research. Qualitative methods are a range of techniques for making meaning, which take written, oral, visual, and artifactual accounts of everyday practice as evidence.² Qualitative methods are well-suited for studying how people design and work with health information technologies to construct meaning and order action.^3–6

In Studying Those Who Study Us, the 2001 book excerpted in the epigraph above, Forsythe noted a historical dearth of qualitative evidence in health informatics, related to the distance between designers and field sites, generalized beliefs about the nature of clinical work, and a commitment to formal knowledge as the only valid frame for building systems.¹ Forsythe attributed low user acceptance to the dominance of the formal knowledge paradigm.¹

While qualitative methods have made significant inroads in health informatics in the intervening 2 decades,¹ the field still has a long way to go in leveraging the strengths of qualitative research to meet the promise of health information technologies to improve clinical practice and patient outcomes. In this article, we report a scoping review of 158 qualitative articles published in JAMIA from its inception in 1994 through 2019. We then offer recommendations for health informatics scholars, including those who sit on editorial boards, health information technology (IT) professionals (including vendors and payers), and policy makers (research funders and health IT policy makers) about how to expand the range of qualitative approaches in future health informatics work.

Health information technologies have been instrumental in radically reconfiguring clinical work,^7,8 the power relations between medical professionals and hospital administrations,⁹ public health practices,¹⁰ and the relationships between laypersons and their own bodies within and outside of clinical environments.¹¹ Not all health IT disruptions have been universally beneficial.^12–16

To reduce the risk of harm to health workers and consumers, public health agencies, and research, health informatics researchers and practitioners stand to gain by seeking to understand healthcare practices as they are actually performed at the front lines of care before technologically intervening in them. Qualitative methods are excellent for producing contextualized analyses, as shown by both the tradition of health informatics in JAMIA as well as other technology-oriented fields, such as human–computer interaction (HCI),¹⁷computer-supported cooperative work (CSCW),¹⁸ and science and technology studies (STS).¹⁹ In addition to generating rich accounts of the experience of working with health information technology platforms, some qualitative methods such as ethnographic fieldwork or long-term interview studies can also help health informatics researchers and practitioners, as they constantly reevaluate how technologically enabled interventions reconfigure clinical practices.²⁰

While quantitative methods are useful for quantifying and predicting problems, qualitative methods can shed light on the impressions, narratives, and discourses that underlie human behavior. For example, qualitative studies of clinicians’ “noncompliance” with process regimens or patients’ “nonadherence” to medical regimens open avenues for institutional change to adapt to the practical and often unpredictable needs of clinicians and patients.^7,21 Further, in health informatics, qualitative methods such as grounded theory^22,23 and thematic analysis²⁴ have been used to study and mitigate safety hazards created by usability issues,²⁵ to mitigate the disruptions to workflow produced in the course of software rollouts,²⁶ and to coordinate care for patients with special needs.²⁷ However, qualitative methods have not been widely used within health informatics in the past.¹ In the spirit of this special issue, we have taken the opportunity to take stock of the full scope of qualitative work in JAMIA and to identify opportunities for future work.

Objective

This scoping review investigated the following research question: how have qualitative methods been used in JAMIA? We paid particular attention to the varieties of objectives, populations studied, collection methods, cited analysis methods, and fields of analytical origin. In the discussion, we contrast the use of a relatively limited range of methods in the qualitative research in JAMIA with the wider variety of approaches used in HCI, CSCW, and STS. In these other technology-oriented fields, researchers and practitioners have used a wider variety of qualitative approaches to illuminate issues related to the design of systems, the contexts of their use, and patterns of historical development that shape technological deployment and uptake. These topics are of great interest to health informatics researchers and professionals, and the field could benefit from adopting a broader range of qualitative methods.

MATERIALS AND METHODS

We conducted a scoping review of qualitative research published in JAMIA, following the framework proposed by Arksey and O'Malley.²⁸ Scoping reviews are better suited to heterogeneous and broad topics, such as the 1 approached in this study.²⁹ The review included the following stages: identifying the research question, identifying relevant studies, study selection, charting (ie, extracting) the data, and collating, summarizing, and reporting the results. We used the PRISMA extension for scoping reviews³⁰ as a guide. We opted not to conduct a critical appraisal in this scoping review; for coherence, we focused on uncovering absences. A detailed review protocol may be requested from the corresponding author.

Data sources and search strategy

In January 2020, we queried the PubMed database to identify potentially relevant work for this study, using keyword terms associated with qualitative methods (Table 1). We included articles entered into the PubMed database between 1994, the year of the journal’s inception, and 2019. Because the study focuses on 1 journal and has a well-defined scope (qualitative empirical research), the search query was iteratively developed by the members of the study team, who are health informatics researchers.

Table 1.

Search query structure

Journal keyword	“Journal of the American Medical Informatics Association: JAMIA”
Methods Keywords	qualitative, “grounded theory”, ethnomethodology, ethnomethodological, interview, interviews, ethnographic, observational, observations, “constant comparative,” “constant comparison,” “mixed method,” “mixed methods,” “mixed-method,” “mixed-methods”
Excluding	“systematic review”
Date Entry	from “1994/01/01” to “2020/01/01”
Full Query	(“Journal of the American Medical Informatics Association: JAMIA”[Journal]) AND (qualitative OR “grounded theory” OR ethnomethodology OR ethnomethodological OR interview OR interviews OR ethnographic OR observational OR observations OR “constant comparative” OR “constant comparison” OR “mixed method” OR “mixed methods” OR “mixed-method” OR “mixed-methods”) NOT (“systematic review”) AND ((“1994/01/01”[Date—Entry] : “2020/01/01”[Date—Entry]))

Study selection

We included articles that documented original, qualitative empirical research published in JAMIA from 1994 to 2019. These included exclusively qualitative articles (eg, ethnographic works)^3,31 and some mixed-methods articles in which qualitative methods had a significant role in determining the results of the study (eg, studies in which interviews and quantitative surveys were conducted and triangulated).³² We excluded articles in which results were exclusively or almost exclusively quantitative (eg, Likert scale-based surveys) or used qualitative collection or coding methods and then analyzed the data with quantitative (eg, quantifying frequencies of medical errors), machine learning, or natural language processing methods (called system development in Figure 1). We also excluded reviews, essays, white papers, articles assessing electronic health records (EHRs) or database data quality, proposals, and articles that were not empirical or that did not provide enough information for our analysis.

Figure 1.

PRISMA flow diagram.

Screening, data charting, and synthesis of results

The literature search identified 531 potentially relevant articles for screening. We divided the screening and data extraction tasks among the research team. We then assessed titles, abstracts, and article bodies to select eligible articles following the criteria above. To ensure consistency among reviews, we first screened and assessed several articles together, discussing possible inconsistencies. Then, we divided the entire set of articles into roughly equal-sized subsets, assigned them to each member of the team, and determined eligibility within those subsets independently. We reviewed one another’s subsets and met to build consensus. We excluded 373 articles that did not fit the eligibility criteria, resulting in a total final sample of 158 articles. The process is shown in Figure 1.

In our analysis of the 158 articles in our sample, we operationalized the use of qualitative methods along these parameters: study objectives, populations of interest (ie, who was studied), the data collection methods used, the analysis methods, and the fields from which the analytical methods were drawn.

We extracted study objectives from articles’ abstracts and introductions. Populations and data collection methods were extracted from the methods section. Data analysis methods were extracted from methods sections and cross-referenced in references sections. We extracted the fields from which the analytical methods were drawn from the referenced texts—for manuscripts, this was available in the publication venue’s stated field, and for books, it was available in the Library of Congress categorization available on the copyright page. We independently charted these data in a spreadsheet and discussed the results as a team to ensure consistency.

After extraction, we drew upon thematic analysis²⁴ to identify the objectives of the studies and to synthesize the objectives into 7 main categories. Specifically, we followed the first 5 steps of Braun and Clarke’s method (data familiarization, code generation, theme identification, theme review, and theme definition), sparing the 6 and final reporting step; the purpose was to draw a scope rather than to closely examine the themes we found. For populations of interest, data collection methods, and analytical methods, we extracted, cleaned, and merged the common patterns (eg, populations were grouped into healthcare providers, healthcare consumers, and other stakeholders). At least 2 researchers analyzed and coded each dimension. In the next section, we present our results as descriptive analyses of study objectives, data collection methods, analytical methods, and fields of analytical origin. A complete list of included articles and their classifications can be found in the Supplementary Material.

RESULTS

From 1994 to 2019, a total of 3791 articles were published in JAMIA. As shown in Figure 1, of these articles, 158 mainly used qualitative analysis methods. References and data are available in the Supplementary Material.

Although qualitative articles represented only 4.2% of total JAMIA articles, the number of qualitative articles published in JAMIA has increased over the years. Figure 2 presents the proportion of qualitative articles published in JAMIA throughout its history and the number of qualitative studies published per year, normalized by the total number of articles.

Figure 2.

Proportion of qualitative articles in JAMIA, overall (A) and over time (B).

These 158 studies used different collection and analysis methods to explore different objectives and to target different populations in the health informatics domain. In the following sections, we detail these objectives, populations of interest, collection methods, analytic methods, and fields of analytical origin in order to characterize the qualitative body of work published in JAMIA.

Study objectives

From our thematic analysis, we identified 7 broad themes from the objectives of JAMIA’s body of qualitative work. The definitions for these 7 categories of objectives are summarized in Table 2. In order of frequency, the most common objective category was described technology use, followed by understand needs, learn from deployment, then initial evaluation. Assess system impact, analyze technology development, and conceptual or definitional were less common. Except for conceptual or definitional (5 articles), we found that all the others could be separated into 2 groups directly related to the system development process: studying health informatics systems before deployment (76) and after deployment (77).

Table 2.

Types of study objectives in qualitative JAMIA literature

	Objective category	N
Before deployment (N=76)	Understand needs: these studies focused on analyzing medical practices and users' needs. They often involved understanding people's processes, workflows, and mental models, to gather requirements for the development of an IT system that would align with user needs, expectations, and practices. Example: Moen and Brennan33 conducted interviews with health consumers about their experiences managing health information at home, aiming to derive implications for consumer health informatics systems.	41
	Analyze technology development: these studies focused on analyzing the processes used to develop health informatics technologies, to analyze how health informatics systems were conceptualized, negotiated, and implemented. Example: Ratwani et al34 conducted interviews with EHR vendor staff to analyze their user-centered design practices aiming to identify challenges related to system development that play a role in EHR usability.	6
	Initial evaluation: these studies described an early evaluation of a system that had not been deployed in a real environment, such as a clinical trial or usability test, to identify opportunities for improvement. Example: Wilcox et al35 conducted a pilot study of a medication-tracking tool, interviewing patients and healthcare providers (clinical pharmacists).	29
After deployment (N=77)	Learn from deployment: these studies focused on identifying best practices and potential challenges (particularly sociotechnical challenges) of deploying a system or an institutional program for use in a real-world setting, often aiming to support the success of future efforts. Example: Novak et al36 explored the work a group of nurses performed as mediators of the adoption and use of a barcode medication administration system in an inpatient setting.	26
	Assess system impact: these studies assessed the impact of a system deployed to supplement or replace established paper-based practices or software systems. They have often used methods such as a longitudinal study in 1 setting, or compared different settings, shedding light on uptake and effectiveness, and highlighting issues that need to be addressed to better support end users. Example: Richardson and Ash37 analyzed the effects of introducing hands-free communication devices in 2 hospitals.	8
	Describe technology use: these studies typically focused on understanding how people use technologies that have been used in a real setting for a significant time, often focusing on unintended consequences and work-arounds. These studies may promote reconfigurations of work practices or computer systems. Example: Winkelman et al38 analyzed how patients with chronic inflammatory bowel disease used and valued internet-based patient access to electronic patient records.	43
Separate from deployment	Conceptual or definitional: these studies defined a nomenclature, the curriculum of a course, policies, or a research space. Example: Embi and Payne39 described the field of Clinical Research Informatics and identified its main challenges and opportunities.	5

In the following, we present trends over time for the top 4 most common objective categories (Figure 3). Articles that focused on understanding needs have increased each year since 2011. We observed a similar pattern for initial evaluation articles, which increased after 2015. However, this increasing trend did not hold for either describe technology use or learn from deployment; both of these objective categories increased after 2002, with no sustained monotonic trend since that year.

Figure 3.

Number of qualitative articles from the 4 most common objectives, displayed as number of articles over time. Each quantity is divided by the number of articles published in JAMIA during that year.

Study populations

We grouped populations of interest into 3 stakeholder groups: healthcare providers, healthcare consumers, and other stakeholders. There have been studies which focused on 1, 2, or all 3 stakeholder groups at the same time.

The first group, healthcare providers, encompasses clinicians (eg, physicians, nurses, pharmacists, health professional students) and any healthcare staff, such as hospital managers and health IT staff. Studies of healthcare providers have often examined clinical systems deployed in the workspace, such as EHRs, computerized decision support (CDS), or hands-free communication device systems for use in hospitals.

The healthcare consumers group encompasses patients and caregivers, such as family members and friends. Many of the articles studying this population have analyzed systems intended for use outside of clinical settings (eg, self-management tools or social media websites), by patients themselves or by patients alongside healthcare providers (eg, patient portals). Articles focusing on this population have also explored systems used within traditional clinical settings, such as analyzing patients’ experiences concerning physicians’ use of EHRs during consultations or focusing on existing or potential systems intended to support inpatients’ needs.

Third, the other stakeholders group includes actors that are not healthcare providers or healthcare consumers, such as researchers, government agencies, health IT vendors, payers, and consumer technology companies.

Figure 4Ashows a Venn diagram of the number of articles studying each group. Articles studying only healthcare providers were in the majority (89), followed by healthcare consumers (28), and finally other stakeholders (15). Comparatively, there were fewer articles focusing on multiple stakeholder groups. Although healthcare providers and healthcare consumers can be considered the main populations for healthcare informatics, few articles (18) studied them at the same time. There were 3 articles in which all 3 groups were studied at the same time.

Figure 4.

Studied populations of qualitative work in JAMIA, overall (A) and over time (B). In (B), quantities are divided by the total number of articles published in JAMIA each year.

We also analyzed the stakeholder groups in terms of relevant articles published per year, normalized to total publication counts. In Figure 4B, the lines for healthcare providers and for healthcare consumers represent the number of articles which exclusively studied the referred population (89 and 28 articles, respectively) as well as the articles studying them alongside other stakeholders (6 and 2, respectively). The line representing healthcare providers and consumers includes the 15 articles which studied both providers and consumers at the same time, as well as the 3 articles which additionally included other stakeholders. The line representing other stakeholders includes all 26 articles that studied any “other” stakeholders (eg, researchers, health IT vendors, and payers). We found that the proportion of qualitative articles focusing on healthcare providers followed a relatively constant trend over the years. With healthcare consumers, however, there was an upward trend of qualitative articles starting in 2011.

Data collection methods

As shown in Figure 5, interviews were by far the most common collection method (111 articles), followed by observations (64), artifact analysis (31), focus groups (28), and surveys (18). In the “other” category, 7 used think-aloud protocols, 2 used participatory design, 1 each used Delphi methods, EHR timestamp data to qualitatively study clinical workflow, app store reviews, social media data, and usability inspection. Many studies used a combination of methods, such as interviews and observations.

Figure 5.

Number of articles that used each qualitative collection method. Some articles had more than 1 category (eg, observations and interviews).

Analysis methods

As shown in Figure 6, a plurality of articles (66 of 158) did not cite an analysis method, although some of these reported an analysis method by invoking its name (eg, “grounded theory”). We focused mainly on cited analysis methods, which provided us with the specific methodological lineages of each study, including the fields from which the authors drew the analytical method. The most commonly cited methods were Straussian (33 articles) and Glaserian (16) grounded theory, followed by thematic analysis (14). Other methods used included content analysis (8), mixed-methods analytical techniques (4), case study (3), and concept mapping (3) methods. There were 2 citations for each of the following analysis methods: ethnography, process tracing, and usability analysis. There was 1 citation for each of these: action research (a critical reflection of action with research), affinity diagrams (an idea-mapping approach), contextual design (a user-centered design process), discourse analysis (an approach analyzing language in social contexts), the empirical program of relativism (an approach within STS), focus groups, frame analysis (a multidisciplinary approach focused on identifying how individuals understand situations and activities), framework analysis (an applied policy approach), hierarchical task analysis (an approach used to describe activities), participatory design (an approach which engages end users with design), and template analysis (a structured approach to thematic analysis).

Figure 6.

Number of articles that cited each analysis method. Some articles had more than 1 category (eg, Straussian grounded theory and hierarchical task analysis.).

Fields of origin

Regarding the fields of origin of cited analysis methods (Figure 7): out of the 92 articles that provided at least 1 such citation, a majority (50) cited sociological sources (books or articles), 19 cited interdisciplinary sources, 17 cited medical or health informatics sources, 11 cited psychological sources, 6 cited design sources, and 5 cited education sources. There were 2 citations to information systems and management sources, 1 cited a social services source, and 1 cited an anthropological source. Some articles cited analysis methods from more than 1 field (eg, sociological and interdisciplinary sources).

Figure 7.

The fields of origin of cited analysis methods. Some articles drew analysis methods from more than 1 field (eg, sociology and psychology.) In some articles, the specific analysis method was less than clear (eg, citations to entire books on “qualitative analysis,” which is not a specific method)—so, there are more “None Cited” in this figure than there are “No Specific Method Cited” in Figure 6.

DISCUSSION

The proportion of qualitative work, particularly including consumers, has been increasing in JAMIA over time. At the same time, the scope of qualitative research reported in JAMIA appears to be relatively restricted in comparison with the venues of other technology-oriented fields, such as the ACM Conference on Human Factors in Computing Systems (CHI), whose published proceedings function as the flagship journal of the field of HCI; the ACM Conference on CSCW, the similarly structured flagship venue for computer-supported cooperative work; and Science, Technology, & Human Values, the flagship journal for STS. We were also surprised to find only 2 ethnographic works and only 1 citing an analytical method originating in STS; Forsythe was an anthropologist and STS scholar.⁴⁰

HCI, CSCW, and STS have paid attention to work practices that are complex, collaborative, and require expertise, such as healthcare, aviation. and nuclear power plant operation.^41–45 Many of the methods used in HCI, CSCW, and STS have already been successfully applied to health technologies, albeit outside of health informatics venues.^1,26,46 Health informatics will likely benefit from adopting those approaches more centrally.

In the interest of fostering productive interdisciplinary engagements, we provide specific recommendations for scholars (authors, reviewers, editorial board members), practitioners (EHR vendors, hospital administrators, insurance, and payers), and government representatives (research funders and health IT policy makers). We recommend (1) broadening the scope of qualitative approaches to include those already used in other technology-oriented fields, (2) understanding research, practice, and policy making in health informatics as a sociotechnical enterprise—as social in addition to technical—bringing social considerations into the realm of legitimate scientific investigation; and (3) supporting broader and more socially oriented studies both financially and structurally.

Broaden the scope of qualitative methods to improve research and practice

Here, we provide examples of some types of qualitative work more commonly found in other technology-oriented fields and which health informatics researchers and practitioners may find valuable. We describe how methods relatively absent in our review—such as ethnographic approaches, distributed cognition, usability inspections, think-aloud protocols, and participatory design—can be used to enhance the field of health informatics. These methods can help researchers and practitioners understand practices prior to developing technology solutions, to fix usability issues early (while they are inexpensive to fix and before they cost lives) and to serve users throughout software design and development.

Understand practices prior to planning technological interventions

Aligned with Forsythe's view,⁴⁷ studies from other technology-oriented fields⁴⁸ have emphasized the importance of understanding how social activity is structured within a community of potential users prior to the introduction of technological interventions. This approach is opposed to assuming that perceived problems can be easily fixed through a technological solution. HCI and CSCW have shown the importance of conducting thorough research in daily clinical work prior to planning technological interventions. This is key to minimizing technologically induced disruptions in clinical practice which endanger patients.¹²

Therefore, we recommend researchers, practitioners, and policy makers understand people's daily practices before developing or mandating technological solutions. There are several methods for studying everyday activity.⁴⁹ For example, ethnographic approaches have been used to study battleship navigation⁵⁰ and the relative safety of paper “flight strips” to computer software in air traffic control.^51,52 Distributed cognition, an ethnographic approach, has been used to study critical areas of healthcare, such as trauma care and cardiac surgery, because such work is collaborative, complex, uncertain, and ever-changing.^4–6,53 Such work has even been published in health informatics journals such as Journal of Biomedical Informatics and International Journal of Medical Informatics ,^5,6,53 but it was rare in JAMIA.

Evaluate systems early and often, prioritizing and fixing usability issues while they are inexpensive to fix and before they cost lives

In our scoping review, observations and interviews were common. However, other methods, such as usability inspections and think-aloud protocols, were rare. These methods have been more commonly used in HCI,^17,54 a field closely integrated with corporate software design.^46,55

In qualitative usability inspections, several trained usability experts independently inspect the interactive design of a software system for usability issues,^17,56 similar to the way peer reviewers examine a manuscript. The qualitative results are then collated to guide changes to the design, before a prototype is built. Inspections may be repeated throughout design and development.

In think-aloud protocols, the participants use a software prototype—even 1 made of paper—to accomplish a task, while thinking out loud. This often uncovers qualitative reasons for user confusion (“usability bugs”) early,¹⁷ so they may be addressed while the “system” is still a prototype which is inexpensive and easy to revise, before it becomes an enterprise software system which is expensive and difficult to revise,⁵⁷ and before any patients are harmed.⁵⁸

Usability inspections were mandated by Meaningful Use,^59,60 a US government program that incentivized the expanded use of EHRs in healthcare.⁶¹ However, according to Halamka, who served as cochair of ONC's Health IT Standards Committee ,⁶² policy makers prioritized data collection requirements, creating usability issues associated with repetitive and low-value data entry. We recommend that policy makers and practitioners prioritize usability.

Support users as they lead iterative software design and development

All too often, software systems are simply foisted upon users.^63,64 Proponents of participatory design and community-driven design⁶⁵ argue that this is both ineffective and unethical and that the actual end users must drive key decisions during every stage of software design.⁶⁶ Proponents of participatory software design, in the spirit of the Scandinavian social democratic labor movement, were originally concerned with the democratic design of computer software for the workplace;⁶⁶ as computing has expanded beyond the workplace, so has the realm of participatory software design.⁶⁷

Participatory design methods were rare in JAMIA; we found only 2 articles. To better support end user needs, health informatics researchers and practitioners should collaborate with experienced participatory designers and researchers, centralizing end users as key decision-makers. Additionally, by validating and meaningfully addressing the needs of end users, wide adoption of participatory design’s social–democratic stance⁶⁶ among researchers, practitioners, administrators, and policy makers may address some of the deeper issues in healthcare, such as clinician burnout, patient dissatisfaction, and patient safety hazards.^68,69

Understand health informatics as a sociotechnical enterprise to guide its continual reform

Diana Forsythe studied “those who study us.”¹ This reflected her commitment to feminist STS scholarship—she questioned the assumptions of power that typically define the divides between “investigators” and “subjects.”^40,70 To continue her legacy, health informatics funders and researchers can invest in carrying out the extensive and delicate work of socially analyzing health informatics across policy, research, business, and practice.

Study stakeholders other than patients and providers

Most of the qualitative studies in our review focused on providers (89), and a steady stream of consumer-focused work had emerged (28). Notably, other stakeholders—such as health IT vendors, payers, administrators, and nonclinical personnel, such as medical secretaries, social workers, and the researchers themselves—represented the smallest portion of our review (15). Health informatics is a complex sociotechnical enterprise involving many partners, regulators, institutions, and personnel; leaving out key stakeholders frequently leads to serious problems.^71–73

Because EHR developers and managers, hospital IT staff and administrators, payers, and policy makers shape the technological environments that clinicians and patients contend with, the field will benefit from more qualitative study of these influential groups. Funders should enable researchers to study these key stakeholders using methods such as those from CSCW⁷⁴ and STS.⁷⁵

Study geographically distributed social practices

Multisited ethnography⁷⁶ is a method that allows for the study of distributed practices. In the language of ethnographers in STS, the work of materializing, shaping, and sustaining the apparatus known as the EHR is distributed between vendors, healthcare organizations, payers, policy makers, and patient advocacy groups such as Leapfrog. Such study, once enabled by funders, should enable researchers and practitioners to understand the broad sociotechnical contexts that influence the development and uptake of health information systems—and perhaps even to rationalize barriers to success that have been considered too sensitive to redress.

Increase the focus on clinical encounters

We found a recent stream of qualitative work focused on healthcare consumers. This recent interest may have been motivated in part by an emerging consumer health technology market (eg, mobile health apps), as well as regulatory incentives to implement patient portals⁷⁷ and to incorporate patient-generated health data into healthcare practices.^78,79 It has become critical to analyze and understand how patients use these tools, and how their use can influence clinical practices; clinical encounters involving patient-generated health data may influence patient satisfaction, a measure to which healthcare organizations are increasingly made accountable.⁷⁹ Despite the critical importance of the topic, studies analyzing interactions between healthcare providers and consumers were uncommon. So, there are opportunities for researchers to qualitatively study technologies intended to facilitate relations between experts and laypersons.

Study how policies shape organizational objectives, which affect clinicians and patients in turn

In our review, we found that studies have tended to focus on specific implementations at the level of the healthcare organization. Since public policy increasingly shapes the objectives of healthcare organizations,⁶¹ it may be fruitful for researchers to also study how policies shape organizational actions and how those actions affect clinicians and patients in turn. Policy makers should fund such research, and researchers should be involved in the policy making process; ideally, there should be a 2-way relationship between health IT policy makers and researchers studying the front lines of care.

Expand support for impact studies

There has been a recent and rapid expansion of US healthcare technology markets, which have been tightly linked to and driven by regulatory incentives that have not foregrounded issues of usability, focusing instead on technological capacity for data capture, storage, and transmission.⁸⁰ In our analysis, learning from deployment and describing technology use were considerably more common than assessing system impact—on workers and consumers—after a system was deployed. This may be attributable in part to policy: in response to Meaningful Use, now called Promoting Interoperability, many institutions acquired such systems, creating opportunities to study technological deployment and use, instead of creating opportunities to decide whether or not it was desirable to have deployed those technologies in the first place. An expansion of multiyear and even multidecade longitudinal impact studies may warrant funding.

Support broader, socially oriented studies in health informatics

There are multiple standards of qualitative rigor, and we provide recommendations for addressing some of the challenges that the authors, reviewers, and editorial boards of health informatics venues may encounter when attempting to accommodate them.^81–84

Value novel views of established practices, not only novel technologies

As discussed in the section titled "understand practices prior to planning technological interventions," the study of ordinary, everyday, well-established clinical work is essential to minimizing disruption. Such research rarely involves presenting a marketable, novel technology or predictive model, but often reveals novel views of established practices; funders, reviewers, and editorial boards should view it as valuable, essential, prestigious, and central to health informatics—and therefore worthy of funding and publication in its flagship journal.

Cite original sources to substantiate analysis methods

Some articles referred to “conventional” or “standard” forms of qualitative analysis methods, often citing interdisciplinary sources, medical sources, and entire textbooks, but over 40% of eligible articles did not cite the analytical method used. There are many such methods, each with its own disciplinary and paradigmatic context,^23,85 and which should be interpreted in context. Authors should at least cite specific, original sources to substantiate analysis methods.

Explain and respect disciplinary norms and paradigmatic commitments

There is a wide variety of qualitative methods available, each of which is rigorous, and each of which judges rigor differently. For example, in ethnographic work, “open data” would present an unethical breach of participant confidentiality.⁸⁶

More broadly, in the past half-century, new paradigms have emerged within qualitative research; Lincoln and Guba have characterized these paradigms as interpretivist, and they have long held that it is inappropriate to evaluate such research using positivist or postpositivist standards of scientific rigor.^81–84 Specifically, instead of seeking reproducible results to uncover general mechanisms that govern the social world from an impartial perspective, many qualitative researchers reject assumptions that such general mechanisms exist, instead seeking interpretations that, while always and by necessity partial, are fair, revealing, and meet the unique needs of social movements, such as the data justice movement.^81,87

In our experience, STS has a strong base of interpretivist scholarship, followed by CSCW and HCI. When composing interpretivist articles for HCI venues, one is expected to specify and explain the analytical methods used, to delineate their disciplinary orientation and paradigmatic commitments, and to support those norms and commitments with citations. Such practice may benefit interpretivist authors aiming to publish in JAMIA.

Increase or remove specific space limits for qualitative studies

Authors will need to be provided the space necessary to explain divergent disciplinary norms and paradigmatic commitments ^81–84 and to provide the rich descriptions that are characteristic of qualitative work. Science, Technology and Human Values has a word limit of 8000.⁸⁸ The published proceedings of CSCW and CHI no longer have specific space limits; reviewers judge length based on appropriateness for methods and contributions.^89,90 So, to better accommodate qualitative research, editorial boards should increase or remove specific space limits.

Cultivate reviewer bases for evaluating multiple forms of rigor

According to Lincoln, Guba, and others, not all qualitative methods include replicability or reproducibility in their standards of rigor—this does not mean that certain standards are lower; they are instead incomparable.^2,81–84 It is often difficult to ensure replicability or reproducibility for questions of immediate interest to nonresearchers.⁹¹ There may be much of importance to learn, then, from paradigms that do not assume reproducibility and concomitant generalizability. To make space for such research,^81–84 the criteria of replicability, reproducibility, and generalizability should be applied more selectively by reviewers; editorial boards should work to cultivate reviewer bases with experience evaluating alternative orientations.^81–84

Limitations

Our review has some limitations. First, we focused on 1 journal. However, because JAMIA is widely considered the flagship journal in health informatics, there is value in analyzing the type of qualitative research it publishes to identify possible gaps and opportunities for future studies. Also, due to scope, we did not perform a comprehensive review of HCI, CSCW, and STS, so recommendations about the qualitative methods available to health informaticists from these fields are constrained by our extensive collective experience in those fields. Further, we did not include critical essays, an important and distinct method of inquiry that deserves focused study. Finally, although critical appraisal of articles is not required in scoping reviews, such analysis would be useful to identify areas to improve quality in the literature.²⁹

Future work

In the short term, further review involving quality appraisal of qualitative work—in JAMIA, in other health informatics venues, and even health informatics work published in HCI, CSCW, and STS—may be in order. These initial reviews will be critical for the course of research that follows—if history is any guide, quality assessment will be an item of interest and controversy.^81–84

As for the long term—it is clear that there is much to be done to continue the legacy of Diana Forsythe. A monumental and many-pronged task lies ahead—to invite scholars from HCI, CSCW, and STS to deeply and extensively engage with the JAMIA authorship and readership to foster long-term collaborations with those scholars, to cultivate a scholarly community that can recognize and evaluate multiple forms of rigorous qualitative work that have been largely absent from JAMIA, and which may inform health policy. Scholars, practitioners, and policy makers will need to resist, in every space and at every turn, the foreclosure of potentially beneficial avenues of inquiry.

CONCLUSION

We found a growing presence of qualitative research in JAMIA, though relatively limited in scope. We have provided specific suggestions for expanding the variety of qualitative approaches considered central to the research, practice, and policy of health IT and health informatics, especially those which are commonly used in the fields of HCI, CSCW, and STS. This will enable researchers and practitioners to produce work that is more responsive to the needs of clinicians, patients, and other users and subjects of health data, improving patient outcomes and safety.

FUNDING

This work was supported by the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, grant number TL1TR001415-05. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

AUTHOR CONTRIBUTIONS

MIH and MCF made substantial contributions to the conception and design of the work, the acquisition and analysis of data, and interpretation of results, drafting, and critical revisions for important intellectual content. BDT and ZS provided substantial contributions to data acquisition, analysis, and results interpretation. SM provided substantial contributions to the interpretation of data for the work, as well as drafting and revisions, and critical revisions for important intellectual content. EVE provided substantial contributions to the design of the work, the acquisition of data and interpretation of results for the work, and drafting and critical revisions for important intellectual content. YC made substantial contributions to the design of the work, as well as the interpretation of results for the work, and critical revisions for important intellectual content. All authors provided final approval of the version to be published and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.