Tentative Renderings: Describing Local Data Infrastructures that Support the Implementation and Evaluation of National Evaluation Initiatives

Abstract

Objective:

Data journeys are a way to describe and interrogate “the life of data” (Bates et al 2010). Thus far, they have been used to clarify the mobile nature of data by visualizing the pathways made by handling and moving data. We wanted to use the data journeys method (Eleftheriou et al. 2018) to compare different data journeys by noticing repetitions, patterns, and gaps.

Methods:

We conducted qualitative interviews with 43 evaluators, implementers and administrators associated with 21 clinical and training programs, called “Enterprise-Wide Initiatives” (EWIs) that are part of a national health system in the United States. We used inductive and deductive coding to identify narratives of data journeys, and then we used the “swim lane” (Collar et al 2012) format to make data journey maps based on those narratives.

Results:

Unlike the actors in Eleftheriou et al. (2018)’s work, who built a data infrastructure to manage clinical data, the actors in our study built data infrastructures to evaluate clinical data. We created and compared two data journey maps that helped us explore differences in data production and management. In tracing the pathways available to the data entity of interest, and the processes through which the actors interacted with it, we noticed how the same piece of information was made to work in different ways.

Conclusions:

Researchers often must build a new data infrastructures to respond to the unique needs of their evaluation work. Differing abilities lead to differences in what programs can build, and consequently what kinds of evaluation work they can support. With the goal of straightforward comparisons across different programs, a more limited focus on quantitative values, and a better description of the data journeys used by the evaluation teams, might facilitate more nuanced assessments of the evidence of complex outcomes.

1.1. Introduction

Noticing infrastructure means noticing the material evidence of a relationship; a road between two towns, a bus between two neighborhoods, a car shared among family members. For example, if you ask someone how they get to their office, they may reply that they take the bus along a particular route. Naming the relationship between their home and their office as a bus route enables asking questions like how long they walk to the bus stop, how many buses are available, how often the buses run, or how much it costs to ride the bus. Comparing the answers to these questions among groups of people gives a sense of the potential inequities in how different people get to their office; understanding these inequities can contribute to a broader understanding of some of the burdens that people experience, which often help contextualize other outcomes like missing days of work, tardiness, and desire for telework, among other things.

It is valuable to think about data, or “information-as-thing” (1), using the discourse associated with infrastructure because it provides a nuanced language for naming and describing the relationships that support the movement and management of data. Data, like cars or busses, are a reflection of policy choices, social relationships, and individual decisions. Social scientists have begun the work of articulating the relationships that comprise data infrastructure in the context of critical data studies (2–4) and infrastructure studies (2, 5–8). Some social scientists have used the language of data infrastructure as a novel approach to understanding policy decisions that result from the quantification of outcomes (e.g., educational policy decisions based on quantitative measures of learning) (2, 9). Less attention has been paid to describing data infrastructures themselves, but some examples include research on “data journeys” (10–13) and data science workers (14, 15).

Data journeys are a way to describe and interrogate “the life of data” (10). The idea of “life of data” is a general idea that implies how data are constructed and transmitted because of the choices that people make. Data journeys encourage particular attention to how the context in which data is created is often not the penultimate context in which it is used. As both a concept and a method, it helps one notice how the “form and use” of data changes in different contexts, and how paying attention to those changes, and how those changes happened, generates insight into power dynamics that impact data production (10, 13, 16). As a method for attending to processes, data journeys are akin to process mining (17), workflow analyses (18), and patient journeys (19); however, with data journeys data is the central object of focus for understanding/examining a process. Eleftheriou et al. (2018) used the concept of data journeys as a way to talk about how data travels between roles and through organizations in healthcare contexts (11, 12).

We wanted to use the data journeys method to compare different data journeys. As much as understanding differences in how someone gets to their office can contextualize outcomes like missing days of work and desire for telework, we wanted to explore how differences in data production and management could help us contextualize the implementation and evaluation outcomes reported by the rural-health focused programs that we studied within an integrated national health system in the United States. Unlike the actors in Eleftheriou et al. (2018)’s work, who built a data infrastructure to manage individual patient clinical data, the health care providers, administrators, and evaluators with whom we worked, built data infrastructures to evaluate clinical programs. As such, “data” was integrated from multiple sources and aggregated at various organizational levels (e.g., a single clinic, a regional health system, a national program). In developing a way to compare different data infrastructures built to evaluate similar kinds of programmatic data, we hoped to be able to use the data journeys model to offer evaluators a systematic way to characterize some of the inequities and challenges in data infrastructure that shaped their evaluative work.

1.1.1. Statement of Significance

Problem:

There is a need to develop qualitative methodologies for studying work with data in this new era of big data.

What is already known:

Process mining, workflow analyses, patient journeys, and data journeys are all existing methods for describing, characterizing, and interrogating the movement and use of data, with data journeys focusing on data as the central object of study.

What this paper adds:

This paper builds on the methodology of data journeys, which has been used to describe the management of individual patient clinical data. We use the method to describe the evaluation of clinical programs; we also offer a visual format that eases comparisons between data journeys.

2.1. Related Work

The work done by Eleftheriou et al. (2018) clarified the mobile nature of individual patient data by visualizing the pathways made by handling and moving data. The goal of their work was to identify barriers to the movement of patient data. They identified several barriers, including changes in media (e.g., paper to electronic), changes in context (e.g., one organization to a different organization), and changes in actors (e.g., a doctor to a lab technician). For them, the “data journeys” method was a way to generate information for an organization about barriers to the implementation of potential new data management systems from individual patient data. They were interested in visualizing the “factory work” (15) associated with data processing, and their data journey maps demonstrated the complex network of relationships that, for example, connected a blood draw from a patient (data entity of interest) to the patient’s test results from that blood draw (data entity of interest) that are then stored in the electronic health record (electronic data container) (11).

The conventions Eleftheriou et al. (2018) developed for drawing the routes of data journeys were intended to privilege how the movement of a piece of data corresponded to different locations in physical space and virtual space (e.g., General Practice (GP) desk (physical data container) to lab pathology secretary desk (physical data container) to pathology system database(electronic data container) to GP system database (electronic data container). Their model included several key components: landscape, data containers, medium, and actors. We wanted to use a visual format that would allow us to notice repetitions, patterns, and gaps in data journeys. Instead of privileging space, we used the visual format of “swim lanes,” common to project management (20), to foreground the propagation and distribution of elements (i.e., points of entry and use, mediums, containers, and forms).

Other similar methods include process mining (17), workflow analyses (18), and patient journeys (19). These methods, along with data journeys, provide ways to visualize and characterize a system, often with the goal of facilitating process improvement. Unlike the individual clinical data in Eleftheriou et al. (2018)’s work, the data in our work was integrated from multiple sources and aggregated at various organizational levels (e.g., a single clinic, a regional health system, a national program). Eleftheriou et al. (2018)’s practical work around cost and risk was rooted in a very theoretical way of thinking about data, which was the movement of data, or data journeys. They defined concepts that were helpful as we were trying to understand the socio-technical elements implicated in the aggregation of data and the quantification of outcomes like effectiveness. We used the data journey method to create a visual representation that integrated different contexts with distinct data infrastructures and variable cohesiveness (e.g., clinicians working in electronic health records and researchers working in databases some of which are created from data pull from the electronic health records). Rather than facilitate process improvement within a system, our goal was to open up the system to interrogation, to make it visible. In tracing data journeys, we hoped to ask questions about the systems that yielded particular infrastructures and journeys, and thus shaped work with data.

3.1. Methods

We used interview data from conversations with evaluators, implementers and administrators associated with clinical and training programs, called “Enterprise-Wide Initiatives” (EWIs) that were part of a national health system in the United States. In brief, each EWI had both clinical and evaluative components. The clinical components were clinician-led programs designed to improve health services and patient outcomes for rural-residing patients. The evaluative components were sometimes led by clinicians, but were more often led by professionals in allied fields (e.g., medical anthropology, health services research, public health, epidemiology). The authors of this manuscript were part of a group funded by the national health system, and our role was to review the annual evaluation reports and provided summaries to partners in the national health system, as well as feedback to each evaluation team about their evaluation report. We provide feedback on a range of topics, from how to structure the annual report, to how to interpret elements of the evaluation framework. At the beginning of the project, we spoke with evaluators, implementers, and administrators from active EWIs to better understand how EWI evaluation was currently working. During these conversations, evaluators described the difficulties they had in finding and using the pieces of data they needed to describe the implementation and effectiveness of their program.

From April to December 2020, members of our group interviewed 43 evaluators, implementers and administrators from 21 EWIs. Our procedures were approved as quality improvement work by the University of Iowa Institutional Review Board (#202001043, Jan. 6, 2020). All of the interview participants provided verbal consent prior to the start of the interview. Our participant sample has been described in detail in other published manuscripts (21, 22). This manuscript presents maps generated from interviews with clinicians, evaluators, and administrators working on two EWIs. EWI 1 became an EWI in 2017 and worked to incorporate pharmacists into primary care teams via telehealth. EWI 2 formally became an EWI in 2015-2016 and provided rehabilitation services via telehealth modalities.

We identified the narratives about work with data from these two EWIs through a process of inductive and deductive coding in the qualitative data analysis software, MAXQDA (23). Two qualitative analysts (JVT and NLJ) deductively coded the interview data; our deductive codes reflected emerging patterns around implementation science constructs (e.g., leadership buy-in) and elements of the evaluation framework (e.g., reach to target populations) that each EWI was tasked with applying. In describing their evaluation work, the interviewees often spoke about their work with data. Upon noticing this, one qualitative analyst conducted a secondary data analysis guided by the literature on data journeys, and deductively sub-coded using “data journey elements.” Text coded with “data journey elements” were segments that might be mappable with elements of data journeys, including data entities of interest (e.g., blood test results), forms of data (e.g., request card) , containers (e.g., GP system database), mediums (e.g., electronic), landscape-points of entry (e.g., GP organization), landscape-points of use (e.g., external agency), and actors (e.g., nurse).

Next, the qualitative analyst deductively assigned these data journey elements as sub-codes. The definition of these sub-codes, as well as examples from this dataset, are included in Table 1. Following these coding efforts, the qualitative analyst was left with narratives from 10 interviews from 10 EWIs. Next, the qualitative analyst used the MAX Maps functionality in the qualitative data analysis software MAXQDA to draft 10 maps. Finally, the qualitative analyst selected the maps with the most comprehensive narratives describing data journeys and used Microsoft PowerPoint to finalize two maps for the purpose of illustrating the method for this manuscript. Two additional maps are included for interested readers, in Appendix 1.

Table 1:

Examples of sub-codes for data journey elements

Sub-Codes: Data Journey Elements	Definitions from: Eleftheriou et al (2018)	Examples from: Eleftheriou et al (2018)	Examples from: Our dataset
Data Entity of Interest	“the data we want to move,” or “the data that a GP needs to decide on an action plan” (p. 117)	radiology report, patient identification details	whether patients have had their Prescription Drug Monitoring Program (PDMP) checked, outliers, comprehensive medication management visit, encounters, number of encounters by each clinician, workload
Actors	“people or IT systems that interact with containers to create, consume, or transform data” (p. 104)	people and/or systems	steering committee, “we, us” (e.g., field-based leads, evaluation team)
Forms of Data	electronic and physical ways the data entity of interest might manifest (p. 117)	x-ray request card, appointment invite, blood sample	list of patients that need their PDMP checked, month of reports showing trends, stop codes
Containers	“stable places where data can be stored” (p. 104)	electronic database, physical location	data population management tool, PDMP drug database, what we had built, electronic health record, where we track it ourselves
Mediums	“type of media” (p. 104)	electronic, physical (piece of paper)	electronic, face to face, conference phone calls
Landscape-Point of Entry	“the information infrastructure through which the data of interest moves” (p. 104)	General Practitioner desk	pharmacists checking drug databases, pharmacists seeing patients, VSSC issues telehealth data, champions tracking workload
Landscape-Point of Use		Pathology system database	not used, the actual report, announcement to email groups, monthly reports showing trends, evidence for increased revenue?

4.1. Results

We conducted a secondary data analysis of 43 interviews with individuals across three roles (field-based lead, program office lead, and evaluation lead), in 21 rural-health-focused EWIs in order to pilot and refine our data journeys method. Here, we illustrate what we learned using two focal examples. The two maps were selected because they were the maps with the most comprehensive data journeys; they also traced the same data entity of interest, “encounters.” Counting the number of encounters between providers and patients was a common measurement that field-based leads used to communicate with evaluators and program office leads about the success of an intervention. How different groups constructed that piece of data was not equivalent, and yet loosely resonant, as those two maps will show.

4.1.1. Rural Health Initiative 1: Mapping the Data Infrastructure of EWI 1

The start of this analysis was the interviewer’s question, “so thinking about the different implementation strategies, can you think of a time when you had to make an adaptation to the program or made any sort of modification?” The interviewee responded,

We started off measuring things a certain way, and then we started seeing outliers and issues that we needed to address and had to dig in and kinda adjust what we had built and what we were measuring to refine it to reflect what we actually wanted to measure. So one of the--, one of the memorable ones was for the encounters, um, we saw some pharmacists that had like hundreds, and hundreds of encounters more than their counterparts, and it wasn’t likely due to their high, high productivity, so we needed to dig into that to see, and kinda dig into why they had hundreds more encounters every month and whether we wanted to count them, or exclude them from what type of work they were doing. And in this case it was like a PDMP [Prescription Drug Monitoring Program], they were checking like drug databases and then encountering [tracking it as an encounter] which is, you can do in a few minutes… every state has a database where they track the controlled substances that are being prescribed to patients. And so we need to check that periodically, in cases of patients going to outside providers as well as [within system] providers for controlled substances. And one of the things that is measured is whether patients have had their PDMP checked or not [but that should not count as an encounter], and so when you’re going, you know there’s data population management tools where you can just get a list of patients that need it checked, and then if you’re going through one-by-one to go and check it, an then you know, put in an encounter, it goes by very quickly…And it’s not the comprehensive medication management that we were hoping for these pharmacists to do for increasing access … you’re not really seeing any patients, you’re not really increasing the access to rural [patients].

The interviewer then queried what kind of modification this realization prompted, and the interviewee explained:

There was a whole infrastructure, like the steering committee infrastructure, um, that included a lot of leadership and our program office, as well as clinical pharmacy leadership from facilities and some [regional leadership] out there. Um, and this was the type of thing that we would meet twice, (pause) twice a year at first, and this is the type of thing that we would go to that group to say ‘hey, you know, we said we were gonna measure this, but here’s what we’re finding, you know, just want to make sure that everyone’s kinda in agreement that this is something that we want to exclude…that we don’t want the [pharmacists] to be doing as part of this role for increasing access…[and depending on the decision of the steering committee] we would make the update on the actual report, and then it would kinda be an announcement to the email groups that we had set up to say, ‘there’s an update to the rules’, um, and so by the time that it was looked at by the steering committee, you know, there were probably a number of updates to make. So we would just send it out and say, ‘hey, these are the updates that we made, to the report based on what we’re looking at and measuring.’ A lot of the times, the people on the steering committee were the [regional] pharmacy executives from the facility…where those encounters were occurring, for instance. So we made sure that all of the [regional] pharmacy executives were involved in that particular steering committee. So if it affected their sites, then they knew, and they had agreed that for their sites, that this change was appropriate.

This segment of text is visualized in Map 1, and the examples of each sub-code are also detailed in Table 2.

Table 2:

Sub-codes visualized in Map 1

Sub-Codes: Data Journey Elements	Examples from Map 1
Data Entity of Interest	whether patients have had their PDMP checked, outliers, comprehensive medication management visit, encounters
Actors	steering committee, “we” (e.g., field-based leads, evaluation team)
Forms of Data	list of patients that need their PDMP checked, undetermined (i.e., not in the narrative)
Containers	data population management tool, PDMP drug database, what we had built
Medium	electronic, face to face
Landscape-Point of Entry	pharmacists checking drug databases, pharmacists seeing patients
Landscape-Point of Use	not used, the actual report, announcement to email groups

4.1.2 Rural Health Initiative 2: Mapping the Data Infrastructure of EWI 2

The coded segment for this data journey map started with the question, “you talked about a monthly data report, could you tell me a little bit more about that?” The interviewee responded,

Yeah, simply the VSSC [Veteran Support Service Center] issues telehealth data in a variety of formats, and for our purposes we basically want, by clinician, people to know how many encounters they’ve had. And that’s basically that month of reports, it shows their trends. They get every month in comparison to the previous month. It usually runs for the year also. So they can see, what their trends were. Interestingly … this year [2020], if you were to look at the numbers, there were some [telehealth] visits here or there through like February [2020] or March [2020], but once March kicked in, … people who had zeros through October-November [2019], December [2019], January [2020], February [2020], all of sudden had a hundred, or some crazy number like that. It was quite the spread really. And again, I think um, by necessity [because of the pandemic].

After a short back and forth, the exchange ends with the interviewer probing about who attended conference calls during which these reports were discussed. The interviewee described how,

[We had monthly conference calls, but] not all the providers [were] on the call, all the time. [Instead, it was typically the champion]…[and what was discussed on the calls] depends on where you are in the program. So early on, talking more about program setup, how do you code things [in the electronic health record]? Which stop codes will enable us to track your workload, if for whatever reason, you can’t code things a certain way, how do we get that communicated so we can track it ourselves? You know, it goes from that.

The importance of tracking workload had to do with the spread and sustainment of the program. The interviewee explained how,

Then like I said, by the 3rd year, when now the systems are in place, the staff is in place, and everybody’s doing stuff, how do we spread now? … When people see what works, and works well, then they want to incorporate it in their practice too … if what we have talked about … comes to fruition, then the people are generating workload, and that generates revenue for the medical center.

This segment of text is visualized in Map 2, and the examples of each sub-code are also detailed in Table 3.

Table 3:

Sub-codes visualized in Map 2

Sub-Codes: Data Journey Elements	Examples from Map 2
Data Entity of Interest	number of encounters by each clinician, workload
Actors	champions, “us” (e.g., field-based leads, evaluation team)
Forms of Data	month of reports showing trends, stop codes, undetermined (i.e., not in the narrative)
Containers	undetermined, electronic health record, where we track it ourselves
Medium	electronic, conference phone calls, undetermined (i.e., not in the narrative)
Landscape-Point of Entry	VSSC issues telehealth data, champions tracking workload
Landscape-Point of Use	monthly reports showing trends, evidence for increased revenue?

4.1.3 Comparing Map 1 and Map 2

In both maps, the data entity of interest (encounters) and one point of use (report) were the same. In addition, each interviewee held the same position in the EWI: field-based lead (FBL). Both interview segments suggested either the presence or potential of parallel data journeys, or multiple pathways for processing the same piece of data. In both maps, there were also at least two groups who were communicating about how to measure the data entity of interest – making claims about measuring this, and not that (EWI 2_FBL: “stop codes” if possible and what do to do if that is not possible, and EWI 1_FBL: “comprehensive medication management visits” versus “whether a patient had their PDMP checked”).

The questions the interviewer asked to elicit these narratives about data were very different. The EWI 2 field-based lead responded to a question about their monthly data reports, and the EWI 1 field-based lead responded to a question about implementation strategies. Also, while the data entity of interest was the same (encounters), the form was different on each map. On the EWI 2 map, the form was both stop codes and month of reports showing trends; on the EWI 1 map, the form was both list of patients that need their PDMP checked and undetermined. The EWI 1 field-based lead described identifying the data entity of interest (encounters) by identifying what to count (i.e., not counting outliers). The EWI 2 field-based lead described identifying the data entity of interest by identifying how to count (i.e., using the month of reports showing trends, using stop codes, or using a form that was undetermined). The outcomes sought by each field-based lead were also different. Where the EWI 2 field-based lead was interested in using number of encounters by each clinician to characterize workload and provide evidence for generating revenue for the medical center, the EWI 1 field-based lead wanted to use encounters related to comprehensive medication management visits to provide evidence for increasing access to rural patients. Another notable point of differentiation was how each field-based lead described the containers. Both field-based leads for EWI 1 and EWI 2 talked about building and using their own containers, likely so that they could aggregate and analyze the data for their evaluation work, which was distinct from the work of the other actors, including the champions and steering committee.

5.1. Discussion

This exploratory study used data from semi-structured qualitative interviews focused on the overall process of evaluating rural health initiatives to examine what we could learn about their related data infrastructures. Each of the data journeys were very different. Map 1 emerged from a question about adaptations to implementation strategies in the context of an initiative for medication management services. Map 2 emerged from an interviewer’s question about a monthly data report for an initiative focused on tele-rehabilitation services for patients. By creating the maps, we articulated the data infrastructure of each initiative more clearly and identified gaps, patterns, and repetitions in how they have built their infrastructure. Though we might be able to generate more complete information by interviewing folks with the map and data journey in mind, the characteristics of the maps are also partly a reflection of the interviewee’s knowledge of the data journey. We wonder how many people, and in which roles, would have a complete picture. Methodologically, we now know how to ask more sophisticated and pointed questions around sampling for qualitative interviews; the incompleteness of the interview data is data. The missing elements signal a need for a more comprehensive, multi-perspective approach to fully capture a given data journey.

Specifically, our use of the data journeys method helped surface three domains of further inquiry: quantification, signification, and structure. Each of the maps illuminated a distinct aspect of quantification: what to count (EWI 1), how to count (EWI 2), and, in both, who counts. Both maps traced shifts in signification: “revenue” was signified by “workload,” which was signified by “encounters” (EWI 1); “access” was signified by “visits,” which was also signified by “encounters” (EWI 2). Finally, both maps showed how these shifts in signification occurred across journeys in parallel infrastructures suggested by multiple points of entry and use. The import of this is that each of these domains, including quantification, signification, and structure, yield windows through which to ask questions about a data infrastructure.

The similarities between Map 1 and Map 2 (e.g., same data entity of interest – encounters) made it possible to think about the potential implications of the differences between the maps. Are the differences in outcomes (revenue versus access) a function of other differences, like the presence of a home-grown dashboard (Map 1, what we had built)? Is the form of the data entity of interest (e.g., stop code versus list of patients that need their PDMP checked), a function of the different outcomes of interest? Do these differences help explain differences in how the interviewees interpreted the evaluation framework? In theory, the data entity of interest (encounters) could be used to measure the reach of the EWI, but the outcome EWI 2 focused on (workload leading to increased revenue) and the outcome EWI 1 focused on (comprehensive medication management visits leading to increased access to rural patients) are different proxies for measuring the reach of the program, and it is not clear whether there was a shared understanding of the implications of using one proxy or the other. At the very least, in tracing the pathways available to the data entity of interest, and the processes through which the actors interacted with it, we can begin to notice how the same piece of information manifested, and was made to work in different ways, and we can begin to be curious about how the pathways and processes impacted what the data entity of interest came to mean (24).

Eleftheriou et al.’s (2018) method would be a good companion, adding detail about organizational relationships and potential barriers to the movement of data. Eleftheriou et al. (2018) maintain a sense of the real-world geographies through which data moves. Our maps are an abstract representation of the layers that exist at each point along the data journey. Making an abstract representation of the movement of data makes it easier to notice what is missing in the narratives you have been offered in an interview, which may add thick description (25) to the details about the organizational relationships and potential barriers to the movement of data that Eleftheriou et al.’s (2018) would illuminate. Moreover, these layers are visual analogs to process of data aggregation and analysis, which are unique to contexts of evaluation and implementation. For EWI 1, whether a panel of patients have had their PDMP checked is an aggregation of the work of pharmacists checking drug databases through an electronic medium, using both the PDMP drug database, as well as a data population management tool, and creating a list of patients that need their PDMP checked. This aggregation produces outliers, which are different from the encounters that are needed for the evaluation. Finally, though there is a sense of trajectory in both Eleftheriou et al.’s (2018) and our data journey maps, Eleftheriou et al. (2018) create a sense of trajectory through physical and virtual space, and our maps create a sense of a changing concept, a trajectory through meaning. Not all lists of patients who need their PDMP checked became outliers in every context, so when and how they did might matter a great deal when producing evidence for the effectiveness of the program, for example.

At the same time, making the maps, and noticing the slippage among the categories (e.g., something could be a data entity of interest on the first read, and then a form of data while mapping), reminded us about old lessons on anthropological field methods (26) and how what you notice often depends on what you have been habituated into noticing. It is possible (even likely) that the field-based lead for either EWI 1 or EWI 2 would make a data journey map differently than we did; or, if given the opportunity to comment on our data journey map, would question some of our representations. Noticing those differences would be instructive in ways that we can anticipate (e.g., we might have missed how the EHR is part of the journey) and in ways that we cannot imagine. Also, paying attention to the practice of mapping itself, as a geographer might do (27), would likely reveal cultural assumptions and power dynamics that would be compelling to examine. Before we can respond to the call to participate in critiques of big data (14, 16) and interrogate the role of data infrastructures in silently shaping our understanding of big ideas (e.g., educational attainment gaps among all students in the UK) (2, 5, 9), we have to first find a way to invite more people, who have been trained to notice different things, into the conversation (28). We have to do the work of encouraging multiple partners (not just statisticians or data managers) to think about, and become able to talk about, data journeys.

5.1.2. Limitations & Future Directions

This analysis has several limitations. First, we did not conduct the interviews with these ideas about data journeys in mind, so the analysis we generated around data journeys might be limited to contexts we did ask about, namely evaluation and implementation. Second, all of the EWI programs were based in a national health system in the United States, which in itself was a unique context, though lessons learned through research and evaluation in this national health system are often broadly applied to healthcare contexts as a whole. In addition to these contextual limitations, another limitation is that our interview questions did not include any informatics – related questions; at the same time, the fact that interviewees talked about it anyhow is illuminating, and perhaps reveals what is top of mind about data-related topics when not prompted to think specifically about them.

Though these maps were first created in a specific context, they can be applied more broadly and used to draw comparisons and contrasts between contexts. Building visual representations of the social world is a familiar anthropological technique, akin to an anthropologist making a kinship map. As much as kinship maps made relationships, histories, and beliefs available to non-natives, these data journey maps do the same. Moreover, kinship maps helped anthropologists theorize about a place and a people; similarly, data journey maps allow us to ask questions that an informaticist or data analyst might not ask, and thus illuminate elements of context and work that might otherwise remain under-theorized. For example, temporality is not well-represented in a kinship map; it is implied by generations, but sometimes an uncle is younger than a nephew, and generations often co-exist in time and space. The data journey maps, using the format of swim lanes, create a similar kind of ambiguity. Eleftheriou et al.’s (2018) maps implied linear time, and used temporal relationships as a way to characterize the relationships between the data elements (e.g., what happens first, second, third, etc.). The swim late format de-emphasized temporality and helped us notice conceptual relationships between the data elements (e.g., how does one start with encounters and get to access, or start with encounters and get to revenue).

5.1.4. Theoretical Implications

A data element is not information; information is data in context. Data journeys could be theorized as a way to visualize the transformation of data into information, or putting data in context, by noticing how the same data element is different information depending on how it is used, what role it plays, and how other people interact with it. In ongoing conversations about measurement, and quantifying outcomes, we tend to talk about information, not data. Work with data journeys shows us that what we measure (e.g., checking the PDMP or having a comprehensive medication management visit), and how we measure (e.g., with stop codes or with months of reports showing trends), are reflections of policy choices, structural constraints, social relationships, and individual decisions. When measuring outcomes to, for example, make decisions about how best to standardize an element of care, it may be useful to know what values, choices, and capacities are baked in to the information being used to make that decision. Surfacing those details may reveal latent assumptions that impact the meaning of the information.

5.1.5. Practical Implications

One potential practical use case for this kind of data journeys method might be as part of a needs assessment prior to an implementation and evaluation effort that relies on work with data. Using Proctor et al (2011) (29) to guide attention about how work with data impacts implementation and service outcomes, as well as Powell et al (2015) (30) to guide attention about which implementation strategies rely on work with data, researchers could use the data journeys method to answer some of the key questions posed around naming, defining, and specifying how to measure implementation strategies, as encouraged by Proctor et al (2013) (31).

When funding is secured for the implementation of an intervention, researchers often must build a data infrastructure to support implementation and evaluation. In an ideal world, additional resources are available to support the infrastructure building. In reality, these resources are often not available, and infrastructures are patched together. Differing abilities to patch things together likely lead to inequities in what different programs are able to build, and consequently what kinds of evaluation work they are able to support. Perhaps in efforts to support and sustain strong data infrastructures for evaluation, our partners in the national health system could develop shared resources across EWIs to leverage rural data and infrastructure expertise. Short of that, with the goal of straightforward comparisons across different programs, a more limited focus on the final “number” and a better description of the data journeys used by the evaluation teams might facilitate assessment of the evidence of complex outcomes (e.g., effectiveness).

Figure 1:

Map 1

Figure 2:

Map 2

Data Availability Statement

The datasets presented in this article are not readily available in accordance with federal requirements and standards and guidelines for the protection of participants’ privacy and to maintain confidentiality. Requests to access the datasets should be directed to Dr. Heather Schacht Reisinger (heather.reisinger@va.gov)