Developing a model for understanding patient collection of observations of daily living: A qualitative meta-synthesis of the Project HealthDesign Program

Abstract

We conducted a meta-synthesis of five different studies that developed, tested, and implemented new technologies for the purpose of collecting Observations of Daily Living (ODL). From this synthesis, we developed a model to explain user motivation as it relates to ODL collection. We describe this model that includes six factors that motivate patients’ collection of ODL data: usability, illness experience, relevance of ODLs, information technology infrastructure, degree of burden, and emotional activation. We show how these factors can act as barriers or facilitators to the collection of ODL data and how interacting with care professionals and sharing ODL data may also influence ODL collection, health-related awareness, and behavior change. The model we developed and used to explain ODL collection can be helpful to researchers and designers who study and develop new, personal health technologies to empower people to improve their health.

1. Introduction

The process of observing and collecting data about one’s health-related behavior can increase awareness [1] and engage and empower an individual to take a more active role in his/her own health [2, 3]. The concept of collecting data on health behaviors to improve health and wellbeing has been a subject of study since 1980 when Lois Verbrugge found that the use of diaries could be expanded from data collection to reflection and self-improvement [1].

Observations of Daily Living (ODLs) are defined as “the patterns and realities of daily life…such as diet, physical activity, quality and quantity of sleep, pain episodes and mood” [4] which can inform patients and health care professionals about a patient’s health. The vision for ODLs is for them to be patient-generated but also patient-informed: rather than just collecting prescribed data, patients have the opportunity to work with health care professionals to choose the type of cues, behaviors, and experiences to record. In this way, a balance can be reached in terms of collecting the evidence-based, clinically relevant data and details most important to the people experiencing them [5]. ODLs can include measurable health activities such as steps walked, and calories burned that are collected through sensors, as well as self-reported information on eating habits, medication adherence, and illness symptoms (e.g. stomach pain, headache).

Collecting ODL data requires time and dedication, as the information needs to be recorded consistently, and as close to in real time as possible to avoid recall bias [6]. Historically, the most common method for collecting ODL-type data has been the paper diary, which is cumbersome, time-consuming and tedious to complete [7]. Recently, however, new mobile technologies, such as smartphones, have created opportunities to expand and expedite the way people record and share health data [8]. The accessibility and prevalence of mobile technology [9] have paved new ways for how people record and share information, with active data collection tools, such as tracking applications [10–13], and passive data collection tools, such as pedometers and other health tracking technology available to the general population [14–16]. For instance, people can use sensor-based tools to monitor exertion, duration and calories burned during a workout and use these data to inform their eating and subsequent exercise behaviors.

Collecting individualized health information can help to improve individual knowledge and self-management of health issues [17], including self-monitoring of diabetes [18, 19], physical activity [20, 21] and dietary monitoring and weight loss [7]. Health applications can also be used to monitor behavioral issues such as smoking behaviors [22], as well as mental health issues such as variations in a person’s feelings to help identify patterns and triggers for depressions and anxiety, as two examples [23, 24]. While opportunities to use new technologies to collect ODLs and engage patients in their health are expanding, little is known about what deters or motivates people to use such tools.

We conducted a meta-synthesis of data collected by five studies funded to develop, test, and implement applications using new technologies to support ODL data collection. Using these data, we developed a model for understanding factors related to patient collection of ODLs.

1.1. Project HealthDesign

Project HealthDesign (PHD) was a national program of the Robert Wood Johnson Foundation (RWJF). In this program, there were two rounds of funding. In Round 1, grantees focused on the issue of enabling patients to create a personal health record. It was out of this work that the PHD program identified ODLs as an important type of patient-generated information for personal health records. This important concept informed Round 2 of the program. In Round 2, PHD grantees developed and tested innovative health information technology (health IT) tools to help people capture, sort and use ODLs with the aim of using the information to foster patient engagement and inform personal health decision-making and clinical care. This report focuses on Round 2 of the PHD program (April 2010 – July 2012).

The RWJF created a national program office (NPO) to lead and administer this program. PHD applicants were researchers at US-based academic centers who were either health IT experts or were collaborating with health IT experts. Five teams were selected for funding. On average, grant awards were $480,000, and the study period for each grantee team was approximately two and a half years.

2. Methods

We conducted a qualitative meta-synthesis [25–31] of the five projects that comprise the PHD program – Round 2. This involved interpretation and synthesis of documents and data we collected from each of the PHD projects and examining these data to identify patterns and summarize key findings [25].

The aim of the meta-synthesis was to understand (1) the role of technology in developing, implementing and using applications and devices to collect ODL; (2) health care professionals’ attitudes towards ODL data, as well as their implementation and use of these data with patients; and (3) patients’ experience with and use of ODL data. This paper focuses on patients’ experiences with and use of ODL data. In particular, we were interested in unpacking what variations in motivation and ODL data collection can be observed across the PHD projects, what factors appear to motivate or deter people (patients and caregivers) from collecting ODLs, and in what ways these factors assert themselves differently in these highly varied contexts.

2.1. Data collection

The data for this study included documents pertaining to the PHD program and its grantees (e.g. grant applications, publications, consultant reports, interim and final grantee reports, blog posts, meeting minutes and videos; a dissertation) and qualitative data from the PHD grantees. These were data each grantee team collected as part of their study evaluation and agreed to share with us. While the documents we examined were similar, we report the details of the qualitative data we received from each team in Table 1 (see below) since there was variation in the types of data we received.

Table 1.

Data Collected from the PHD Grantees

PROJECT	DATA SOURCES
	Study Phases	Data Collection Method	Type of Data Shared
1	Design phase, Post intervention	Patient focus groups	Notes from 2 Design phase and 3 Post intervention focus groups
2	Not applicable	Not applicable	Not applicable
3	Design phase, Midpoint, and Post intervention	Advisory board, User focus groups, and User Interviews	Transcripts from 2 Design phase and 2 Midpoint focus groups, and 16 Post intervention interviews
4	Design phase, Post intervention	Patient focus groups, Provider focus groups	Notes from 4 focus groups
5	Design phase, Midline, and Post intervention	Focus group transcripts, Patient Interviews, Provider Interviews, and Control Interviews	Focus group transcripts, transcripts from 2 providers and 14 patients (Midline and Post intervention transcripts)

2.2. Data management

Focus group and interview notes and transcripts were de-identified by PHD grantees before sharing these data with us. Documents (e.g. reports, blog posts) collected were not de-identified since these are publicly available. Data were put into ATLAS.ti (version 7) for data management and analysis. All study data were saved on a password-protected networked drive maintained by OHSU.

2.3. Analysis

We used an immersion-crystallization and cross-case comparison approach to analyze data. [32–35] In each step in the data collection process we read data to engage in and understand what was happening in each project (immersion) and then discussed these data and our insights to identify key themes (crystallization) within a single project, (i.e. case). We engaged in this immersion-crystallization process twice: first, to analyze the documents we collected and then to conduct a secondary analysis of the data PHD grantees shared with us. The first phase culminated with the creation of a case summary for each project. In the second phase, we expanded and added to each case summary the new information identified through the secondary data analysis. After each phase we shared the summaries with the PHD grantees for feedback. Additionally, we debriefed with our co-authors (Dorr, Hayes, Ash, Sittig) as well as with leaders from the RWJF, the NPO and a PHD grantee representative. Using the summaries created we conducted a cross-case comparative analysis to look for patterns and variations in motivation and ODL data collection across the PHD projects and to identify factors that influence the collection of ODLs by patients and caregivers. Through this process we identified a range of factors related to patients’ motivation to collect ODLs. Then, we developed our model by discussing findings and iterating versions of a prototype until we developed a model to adequately explain how motivational factors might relate to ODL collection, sharing, and behavior change.

3. Results

The five projects participating in PHD (Round 2) are described below (see Table 2). In all of the studies, the people asked to collect ODLs were living with or caring for someone with substantial health concerns. The PHD grantees developed tools that targeted people with five different types of health experiences; people living with asthma (Project 1) or Crohn’s disease (Project 4), elders nearing a time when issues of cognitive decline might emerge (Project 2), youths (13–24 years old) who were overweight or obese and may have had depressive symptoms (Project 3), and caregivers for premature infants (Project 5). In three of these projects the use of the ODL application and device was finite (Projects 1, 3, and 5). For example, the device might be used to help patients gain better asthma control and, once this was achieved, the tool would no longer be needed. In contrast, Project 4 created tools that could be used indefinitely by patients living with Crohn’s disease. In Project 2, the tool created for use with elders was initially conceived as a time-limited, diagnostic tool. However, during testing the team saw potential for using sensor technology indefinitely to assist elders with, for instance, medication adherence.

Table 2.

Description of PHD Projects

Project	Focal health-related experience	Device used	Primary Users
1	People living with moderate to severe asthma. Patients used the device to collect ODLs such as asthma triggers, medication usage, peak flow (team provided peak device) and physical activity. ODLs shared with clinic nurse who, following a protocol, took appropriate action to improve asthma control. Physicians are notified when necessary.	Smartphone	Patients and clinicians (physician/nurse pairs)
2	Elders who may be experiencing or nearing a time when they could experience cognitive decline. Passive sensors were placed in elders’ homes to collect ODL data. Data shared with elders, caregivers, and healthcare professionals to identify likelihood of decline. Sensor data can also be used to keep elders on track, particularly with medications.	Sensors placed in coffeemaker, pillbox and telephone	Patients and caregivers
3	Overweight or obese youths 13–24 years old who may simultaneously be struggling with anxiety or depression. Collection of ODLs included food consumption, physical activity and mood. Collection of ODLs done by goal setting in conjunction with a health coach to foster patient awareness raising and behavior change.	iPod Touch	Patients, health coaches, and physicians
4	Patients with Crohn’s disease. The ODLs patients collected included weight, physical activity, mood, and symptoms relevant to their illness. Patients shared this information with their clinician to gain greater awareness of the factors that trigger symptom exacerbation and to help patients and clinicians manage their medical treatment as well as psychological and social wellbeing.	iPad, biometric devices (scale, activity monitor)	Patients, study gastroenterologists
5	Parents/guardians of significantly premature infants. The application developed helped caregivers keep track of medical appointments and collect ODLs including infant’s weight (team provided scale), eating and elimination patterns, and fussiness. This tool helped caregivers to provide needed information for communicating with healthcare professionals.	Smartphone	Caregivers, High Risk Infant Follow-up Case Managers

3.1. A model to explain user motivation and ODL collection

Based on our analysis, we created a model to explain six motivating factors we found in the data, the relationships between these factors, the role of ODL data collection and the potential pathway to behavior change (see Figure 1). On the left of this figure we depict the six factors that can motivate ODL data collection by showing them as overlapping circles to illustrate the interdependencies among these factors. Although these circles are the same size, one factor might play a more or less important role than others with regard to ODL data collection. This depended on the situation, and we observed this when the projects were considered collectively.

Figure 1.

A model to explain user motivation and ODL collection

3.2. Factors influencing collection of ODLs

Across the PHD projects we identified six factors reported by participants to influence the collection of ODL data: the usability of the ODL application and device; the information technology infrastructure; the degree of burden of collecting ODL data; the perceived relevance of the ODL data collected to the participant; the patients’ level of emotional activation or concern about the health problem; and variations in the health or illness state for which ODL data collection is relevant. We describe each of these factors below.

3.2.1. Usability

Usability is a highly overloaded term with numerous definitions, however, for the purposes of this paper, we define usability as ease of use including the ability to learn and understand the functioning of an application or device [36], such as those developed and tested in the PHD program. The design and layout of an application influenced users’ experiences. PHD grantees employed user-centered design approaches [37] involving a series of activities (interviews, focus groups, cooperative design sessions, and beta testing) with potential users to identify users’ needs and iterate the design of the data collection tools. Participants preferred simple and easy to use tools with a small learning curve to collect ODL data because participants were not necessarily “technically savvy” (Project 1, Debriefing Focus Group Notes).

Participants described issues that detracted from usability such as difficulty with ODL search tools (e.g. difficulty finding the food name to be entered), a lack of functionality to protect against loss of entered ODLs, and not having the flexibility to add ODLs for a previous day. As an example, one participant reported that medication tracking “got lost” in the application. Although we do not know what “got lost” means, we know that this made collection of this information difficult, “Med tracking is not regular–this gets lost in the app. I don’t look forward to tracking because the app does crash and it’s buggy” (Project 4, Patient Post Intervention Focus Group).

Participants also said that slow performance and applications that crash deter ODL collection:

Participant: …Like for the weight, I want to go back sometimes and look at the weight but it just shows the dots and I want to know exactly how much it is.

Interviewer: Right.

Participant: And then, it's a little frustrating that it takes a while [to load] (Project 5, Interview).

3.2.2. Information technology infrastructure

Infrastructure is the “persistent socio-technical system over which services are delivered” [38]. Information technology infrastructure is the foundation or system we have for moving and storing information. In the PHD projects, infrastructure emerged as a factor that influenced ODL collection. For instance, as Kim et al, describes in this issue in greater detail, lack of access to Wi-Fi can deter users from collecting ODLs. When asked by the interviewer how long the participant used the ODL tool, one user from Project 3 responded, “I didn’t use it because I never had Wi-Fi” (Participant Exit Interview).

Information technology infrastructure issues also emerged when study teams were placing sensors in users’ homes, as was the case with Project 2:

After countless visits to apartments to find that the computers having trouble connecting to a Wi-Fi signal that teetered between "weak" and "no signal," we realized that we needed to boost the signal with the Wi-Fi repeater. Finally, after installing two additional Wi-Fi repeaters, we were able to get everyone back on Wi-Fi for the final month of the study. This took nearly a week and a half of trial and error with debugging…This Wi-Fi-induced nightmare should be a lesson that evaluating systems in the field can be difficult, particularly if the evaluation lasts for a long time. The difficulties are twofold: 1) the unpredictability of people's lives (e.g. people had to move apartments) and 2) debugging IT infrastructure that is mostly invisible [39].

3.2.3. Perceived relevance of the ODLs

Participants in the PHD studies reported that collecting ODL data they perceived as relevant to them and meaningful to their illness/health experience was a factor that affected ODL data collection. During the design process grantees found that some flexibility was needed in the ODLs collected to ensure they could be individualized and relevant. As one participant noted:

Provider: It really depends on the patient. It varies for each one. My experience is that the stage of the disease, the amount of time the patient has had the disease, what the patient’s “new normal” is determines what they want to track and why (Project 4, Provider Post intervention Focus Group Notes).

Users reported ODLs must be relevant to the health experience and be flexible enough to accommodate individuals’ tracking needs. In addition, the perceived relevance of ODLs was also important to healthcare professionals and this could, in turn, influence patients’ ODL tracking behavior. The example below shows how people were disappointed when their doctor did not review ODL data with them:

Participant: Yeah, I showed her the graph.

Interviewer: And how did she-- did she ask you for that information?

Participant: She doesn't ask me because I think, you know why too--it's so embarrassing, because she spends a lot of time with us so I think by the time--I'm too embarrassed to bring something up to her like give her more something to look at or do. So like by the time she is done with the kids, I'm like uuuggghh, or she will, when she did look at it, she was like oh that's nice" or whatever and then she just left. I think she was already done with us (laughter) (Project 5, Final Interview).

Additionally, we observed that patients’ willingness to collect these data is reduced if healthcare professionals do not reinforce the value of collecting ODLs by reviewing with them.

3.2.4. Degree of burden involved in collecting ODLs

Across the five PHD projects, four projects required participants to engage in active ODL data collection by manually entering ODL data into an application via a user interface. A few projects attempted to use sensors to collect ODL data but, due to technological problems, these passive approaches failed. For example, weight sensors in a crib were untenable in practice for Project 5 because infants did not weigh enough for accurate measurement with available pad-based weight sensors. An important contrast emerged between the passive and active ODL collection approaches regarding the extent to which participants used the ODL data collection tools. As the case below shows, some of the youth advisors who gave feedback on the initial design for this study found that manually entering data was too great a burden [40]:

Participant 4: Yeah, I think taking pictures is definitely better than writing it down because, I mean, I’ve had to write stuff down in school and I can’t stand doing that. It’s just like writing down what you eat is just awful…

Participant 2: It’s like keeping a diary. People say they’re going to do it but they try and after about a week or two it just falls through because you just can’t keep up with it… (Project 3, Youth Advisory Board Focus Group).

Active data collection requires manual data entry, often several times a day. The information to be entered can be quite detailed, placing a greater degree of burden on a user, particularly when that person is busy doing other tasks. Across the PHD projects we observed that some participants were more likely to enter ODL data at a later time, when they were not busy with other tasks. However, delaying ODL data entry can lead to recall bias.

In contrast, once people agreed to use sensing devices (e.g. coffeemaker, pillbox, phone, pedometer, scale), and provided permission to have sensors collect and share information with others (e.g. research team, health care professionals), there was, in theory, little effort required. In practice, however, the contrast is not as stark in passive and active data collection primarily due to the limitations of existing—and even prototype—sensing systems. Configuration of systems requires substantial human effort. As seen in Project 2, a researcher visited homes to install the sensors at the beginning of the deployment, monitored and maintained them throughout (including replacing batteries at regular intervals) and removed the sensors at the study’s end.

In those situations in which equipment replaces an existing device, any new tools introduced must fit the needs and preferences of the users who may be accustomed to very specific interactions with existing consumer products. For instance, in the case of the coffeemaker in Project 2, participants often prioritized the quality of coffee made over any therapeutic or diagnostic benefit of the tool. Likewise, in Project 4, the use of a Bluetooth enabled scale allowed participants to report these data automatically to the system. However, the tool still required the active engagement of participants stepping on the scale. This task could be frustrating when participants’ own scales were perceived as simpler to use and more accurate, or simply less time-consuming, given the need to wait for synchronization with the system on the new scale. As one participant reported, “Withings almost never synched for me; the latest build doesn’t work, cannot launch it on my iPad” (Project 4, Post Intervention Focus Group).

When currently used options are replaced with new tools for the purposes of data collection, the new tools must be similar to those already in use in order to reduce users’ learning curves. Teams using smartphones had variable success, in part tied to whether the participant already used a smartphone and, if so, whether the operating system (OS) (i.e. Android, Blackberry, iOS, Windows) was the same. In previous work, researchers have found that having a variety of phone options—both in terms of OS and form factor—can address this issue to some degree [41]. Likewise, in Project 2, the research team redesigned the sensor-based pillbox so that it would be simpler and similar to the ones elders tended to use.

3.2.5. Emotional activation and variations in health or illness state

Emotional activation, a person’s feelings and level of concern about his/her health problem or the illness of a person to whom they are providing care, is also an important factor influencing ODL data collection. Feelings about an illness vary across individuals and one’s own level of concern may heighten and decline over the course of a health or illness experience as well. For example, the relevance of ODLs may wax and wane as a patient gains control over his/her asthma or as a premature baby gets stronger, and people may feel that collecting this ODL is no longer warranted. Some participants said they might collect ODLs more consistently when they are not feeling well and less consistently when they are feeling better, “When I am not feeling well, I track every day; but if I am feeling well, maybe once a week or once every two weeks” (Project 4, Post Intervention Focus Group Notes). Interestingly, variations in and barriers to active ODL collection did not manifest in the one PHD project that used a passive, sensor-based approach to collecting ODLs.

3.2.6. Interdependencies among motivating factors

It should come as no great surprise that these factors were deeply interrelated in all of these projects. Technological difficulties and usability issues can influence the degree of burden of collecting ODLs and the relevance of an ODL can ebb and flow as the urgency and seriousness of a health problem or illness changes. Additionally, the relative importance of the factors we describe herein depend, in part, on the use case for the ODL tool. Infrastructure challenges, such as the lack of Wi-Fi, may be major deterrents to ODL collection when target users are low-income individuals. Ease of use and perceived relevance of the ODL may be more significant factors to consider when the target users are collecting data to manage a chronic disease. Thus, there is interplay between the factors influencing ODL data collection and these factors may vary in their relative importance depending on the target user and contextual factors driving their use.

3.3. The path from motivation to ODL collection and behavior change

Motivation can influence the collection of ODL data, which in turn can increase information and awareness for both those collecting the data and those viewing it, and, possibly, lead to behavior change, ideally leading to improved health outcomes. While the PHD projects were small pilot studies and of short duration, participants across the five projects stated in interviews and focus groups that collecting ODL data helped them to be more proactive in their healthcare management, reduced stress and worry, and led to improvement in treatment and/or outcomes i.e. weight loss resulting from changes in health behaviors associated with collecting ODL data. When this was reported, participants indicated these changes were stimulated by an increased level of knowledge and awareness regarding relationships between behaviors, aspects of their daily activities and their health. A participant from Project 3 reported:

Interviewer: …So how has it changed things for you?

Participant: I noticed that like I’m starting to notice stuff like when I’m bored I eat more. And when I’m happy I like eat but I eat like portioned. I don’t eat like a bunch of snacks that I um would eat when I’m bored. So, I ‘m starting to watch that (Project 3, Midpoint Focus Group).

In this case, the collection of ODL data helped this person recognize behaviors, such as calorie consumption, and the relationship between cues and behaviors, such as mood and eating behavior, that might otherwise go unnoticed. Once noticed, as this participant reports, the behavioral pattern could be changed.

In addition, collecting ODL data could enhance the communication between patients and healthcare professionals, as a Project 4 participant reported, “The app helped me stay fresh as to what actually transpired. Having it written down was golden. I took the list of ODLs to my provider and we talked about what was going on with me” (Project 4, Post Intervention Focus Group).

In Project 1, health care professionals reported the asthma-tracking tool increased their knowledge of patients’ symptoms as well as how patients were taking medications. This, they reported, led to re-educating patients in how to take their medications, to changes in medication, and to referrals for patients who were discovered to have Chronic Obstructive Pulmonary Disease:

…Upon reviewing one patient’s ODL on rescue and maintenance medication use, the nurse discovered the patient was using the rescue medication daily and the maintenance medication episodically due to the patient’s confusion about which medication was for which purpose. The nurse provided appropriate education to reinforce how to use the medications appropriately (Project 1, Final Report).

As these instances show, ODL tracking can increase awareness and lead to behavior change that enables people to better manage their health.

The model provides an overall view of how this behavior change process can—and often did—occur. However, there are of course variations, which can be seen when examining all five of these projects together. In the case of improving asthma control, once changes are made and asthma control improves, the collection of ODL data can be discontinued. This is shown in the bottom right corner of the model by the dotted line that breaks the path between behavior change and a subsequent, more informed iteration of ODL collection (see Figure 1, #1–3). In contrast, ODL data collection might lead to behavior changes that inform a next iteration of ODL collection. Examples of using ODL data as feedback for subsequent behavior change include: 1) collecting ODLs about food consumption and physical activity, and 2) collecting ODLs for managing a chronic illness such as Crohn’s.

3.4. Breakdowns in the path to behavior change

There are a number of places where people may either discontinue the collection of ODLs or the path to behavior change may breakdown. In the model in Figure 1 we depict these points where users may stop collecting ODLs with a dotted oval and a number (see Figure 1, #1–3).

3.5. Collecting ODL data

One of the primary challenges to even beginning behavior change through data collection is having the desire and ability to collect the data in the first place. Lack of motivation for data collection can result from a combination of the six factors we identified earlier. While PHD grantees designed tools generally meeting users’ specifications, there were, nonetheless, a small number of instances in which the perceived relevance of the ODL emerged as a problem allowing us to see the interplay between perceived relevance and users’ motivation to collect ODLs. Additionally, the perceived relevance of an ODL may change over the course of an illness episode. As the caregiver below describes, the mood ODLs, while difficult to enter, were valuable for a while but the value of the ODL waned as the baby’s mood leveled out and the caregivers stopped entering these data:

Interviewer: Have you found any of the, any of the features to be more difficult or the things you are supposed to enter, are any of them more difficult to enter than others?

Participant: The mood one is always a difficult one. [My husband] always enters that. I stopped doing the baby moods… [My daughter] got to a point where she's just, she plateaued. She's a really happy baby most of the time. So it would just be putting in the kind of same thing and then with an occasional spike. So, I did stop doing that, but I did find it useful for a time (Project 5, Participant Interview).

Additionally, the burden of collecting ODL data acted as a barrier to use of these tools. In the case of Project 3, participants felt that the amount of detail they had to input was too much; instead of selecting a bowl of cereal, they needed to select all aspects of what made up the bowl of cereal (see Kim et al. in this issue pages XXX - XXX) for more detail on this barrier).

3.6. Increasing information and awareness

The ability to view ODL data over both the short- and the long-term is important for increasing user awareness. Short-term visualization tools provided individuals with immediate data such as taking medication or feedings. In Project 2, elders reported wanting “short-term views of the data, particularly for giving them a sense of extra security for the memory-intensive task of taking medications.” [42]^p.38 Long-term visualizations gave patients the ability to see patterns over time and this changed users’ awareness about specific behaviors. A participant in Project 3 related, “when I would go into the internet and see TheCarrot.com, I would go over the months of what I ate and I thought it was too much of this or that. So it helped me notice to cut that off” (Project 3, Exit Interview).

It is possible, however, that users collect ODLs but never use the visualization tools to look at patterns over time and they may not, therefore, gain new information or a change in awareness from this data collection experience. This is represented in the model by the dotted oval (see Figure 1, #2).

3.7. Changing outcomes and behaviors

Even when patients and caregivers are willing and able to collect ODL data, and they learn something new from doing this, behavior change is hard in the short-term and incredibly difficult to maintain [43, 44]. This is shown by the third dotted oval in the model (Figure 1, #3). Importantly, health care professionals can play a significant role in motivating ODL data collection, changes in user information and awareness, and behavior change. In Projects 1, 3, and 4, healthcare professionals reviewed ODL data with patients and, in some cases, these interactions led patients to see information in new ways, stimulating behavior change that might not have otherwise occurred. In Project 3 health coaches used adolescent patients’ ODL summaries to discuss their behaviors in an attempt to stimulate motivation and set behavior change goals:

The health coach drew out the participant’s individual attitudes through conversation and reinforced the values of health, well-being, and empowerment to make change. She also facilitated self-reflection about the ODLs and identification of trends and patterns in the combinations of ODLs [45].

In contrast, in projects 3 and 5, since caregivers and healthcare professionals were not formally part of the study, it was up to participants whether or not they shared ODL data with them. In some cases, participants said they wanted their healthcare professional to look at ODL data with them and to help them make sense of emerging patterns. In our model, this lack of communication with healthcare professionals about ODL data is represented as a dotted line in the arrow between oval #3 and the box labeled “Interaction with healthcare professionals” (See Figure 1).

4. Discussion

Our meta-synthesis of the five PHD projects led to the development of a model that can be used to explain peoples’ collection and use of ODL data. ODL data collection can have implications for patient engagement and communication with their health care providers. This is evident as behavior changes (i.e. medication adherence or food consumption) take place. Additionally, the findings from this analysis highlight a number of important tensions that manifest themselves when designing and implementing ODL data collection tools.

First, ODL collection is heavily influenced by the degree of user burden and usability of the process and technologies in place. For example, in Project 4, in which sensor-based technology was the sole method for collecting ODLs and, once technological challenges were addressed in the field, the ease of data collection for participants led to consistent collection of ODLs. Yet, these approaches leave open substantial product design work to ensure that, in the clinical and commercial space and absent researcher intervention, appropriate customization and adaptation are possible. However, as sensing systems improve, both in terms of their ease of use and configuration and in terms of their robustness under the stresses of the home environment, less and less burden will be required, allowing them to ostensibly become invisible in use [46]. One possible downside of the increasing ‘invisibility’ of sensing technologies may be the loss of user awareness and reflection; users do not get the inherent benefit of thinking about their data as they are manually recording it [47] and may require additional intervention to gain these benefits.

Where collection of ODL data has a greater degree of burden, various factors enhance or deter people from putting in the effort required to collect ODL data. This finding suggests future research and development efforts in this area may best be focused on developing sensor devices that reduce or eliminate the exertion required for ODL collection [48]. Additionally, emerging research in this area suggests that using a “snippet technique,” of having people engage in the low burden task of capturing small bits of text or data throughout the day and then using these “jottings” to prompt a full entry of ODL data later, when there is more time, might be productive, as this approach is perceived by users to be less time-consuming and results in longer entries [49].

Second, we observed that the perceived relevance of ODL data is critical, including collecting data at the right level of detail for the cue, behavior, or feeling of interest. This can, however, get complicated when patients lack the desire to enter information at the level of detail required for the health issue of focus. While details may be required to increase the awareness and elicit behavior change, individuals who are not motivated toward that change will also likely remain unmotivated to collect ODL data. This suggests that it may be most worthwhile to put active data collection tools in the hands of those motivated to use them and to further explore whether or not passive technologies (e.g. a scale that automatically tracks portion size) to increase awareness and motivation in the unmotivated may be a worthy area for future study.

Third, the temporal quality to ODL data collection, often related to changes in the illness experience, can lead to an ebb and flow in ODL collection or ODL collection ending entirely. Researchers, software developers, and others who may implement ODL tools in clinical practice need to be aware of the temporal quality of ODL collection and work to understand peoples’ experiences with collecting ODL data, specifically, patterns of ebb and flow and how to manage them so that useful information is collected. If a patient with Crohn’s disease only collects ODLs during times when they are feeling sick, the ODLs collected may not allow the person or the clinician to have the information necessary to compare times of heightened symptoms and healthier periods. This limit could in turn constrain the potential for awareness raising, learning, and the treatment improvement these tools are designed to foster.

There are a variety of opportunities and challenges to ODL data collection for both the end users and healthcare professionals. While many patients and caregivers may not be explicitly making cost-benefit analyses about these tools when considering whether or not to use them, the degree of burden of using a tool to collect ODL data must not dramatically outweigh the potential or perceived benefits, including both short- and long-term burdens and healthcare outcomes. Use of ODL tools requiring less effort from users, such as those using sensor-based technologies, might offer opportunities to rebalance this cost/benefit equation. However, even sensor-based tools may be too burdensome (e.g. require putting on and sleeping with an uncomfortable sensor), and there are significant potential downsides to developing methods for low-burden ODL collection tools. There is the risk that data will be collected beyond a point at which they are useful, either because the situation has changed (e.g. information on baby weight beyond the point when the baby is growing successfully) or the data are simply too plentiful to parse (e.g. caloric intake and metabolic output on minute by minute basis for weeks or years on end). This kind of massive long-term data collection requires effort to synthesize, summarize, and make it meaningful for users. If users lose interest in the information, the cost of collecting and managing data may exceed its perceived or potential utility. Thus, the collection of ODLs, regardless of where it falls on the spectrum of burden, and passive and active data collection, requires careful consideration of these cost/benefit equations. While many of these tools are widely available in the marketplace, use of these tools in health care settings requires additional investigation to determine the circumstance and timing of use of such tools.

4.1. Limitations

There was variation across the PHD grantees projects and this breadth, both in the types of tools developed and in the types of users and health care experiences these tools were designed to target, led to the development of a robust model for understanding peoples’ behaviors regarding collecting and using ODLs. Nonetheless, there are important study limitations worth noting. First, the people who used the tools that PHD grantees developed were study participants. As such, their use of these tools is influenced by the study context. Had they not been a part of these studies these people might have never used applications to collect ODL data. Additionally, participants were compensated for study participation and, as a result, may have continued to use these tools beyond the point they would have if they were using these tools outside of a study. Second, we did not have data regarding the personality characteristics of users. This is a factor missing from the model and personality characteristics, such as being compulsive about tasks, highly detail-oriented, or particularly anxious might influence ODL collection in ways that we cannot fully understand from these studies. Third, our findings are based on a secondary analysis of projects funded by the same overall program. As such, our findings are limited by the data that PHD grantees collected, and shaped by the structure, processes and ideas of the program, inasmuch as these shaped the work of grantees. This limitation was mitigated by the variety of PHD projects funded, the overall richness of the data grantees collected and shared with us, and by the opportunity to draw out lessons across five studies that might not otherwise be observed if conducting a single study. These limitations highlight important opportunities for future researchers, developers and designers. This area of investigation would benefit from research exploring the individual characteristics that drive ODL collection and examining the percent of the population likely to use these tools, given that only a small fraction of smartphone users have downloaded a health-centric application.[9] To answer both of these questions will require larger studies with different designs from those employed by PHD grantees. By answering these questions, one can examine, on a larger scale, the value and impact of ODL collection.

4.2. Implications

Collection of ODL data is a fertile area for exploring the role patient-centered health informatics tools may play in informing healthcare decisions and improving health. Our findings indicate that the prospective challenges, degree of burden, and costs must be balanced against the likely opportunities and benefits the use of these tools affords. Additionally, the timescale of these costs and benefits must be considered in both their design and use.

The need to balance risks and rewards, costs and benefits, and challenges and opportunities has been noted before in considering how best to design data capture technologies in terms of privacy [50, 51]. Likewise, these concerns have been raised in association with telemedicine [52, 53], arguably a precursor to ODL data collection and sharing. As patients and caregivers increasingly take on the roles of both healthcare workers and IT managers, this balance will likely become more salient to end-users and more impactful on healthcare quality. Beyond improving the usability and robustness of the technologies themselves, we can also work to balance this equation through improved alignment between effort (e.g. Who is placing effort to collect the data? Review the data?) and incentives (e.g. Who gains from collecting the data? How are people compensated for time spent reviewing patient-collected data?).

In healthcare, particularly when dealing with chronic conditions, as all of the PHD projects were, these tensions must be balanced across a variety of timescales. Thus, ODL data collection tools may well be designed with the ability to adapt as situations change. Alternatively, designers may wish to target particular contexts, while recognizing that other tools may be used before or after any given tool. Data collection tools may be designed with only finite use intended, enabling patients and caregivers to move through an acute incident or change in health status. Even in those cases, however, once the data are collected, they may be archived and analyzed long into the future. Thus, questions arise about not only when to end ODL collection in such a scenario but also when—or if—to destroy the data. These are important considerations when implementing these tools, as individuals who may not be interested in collecting data infinitely might be convinced of the importance of collecting ODL data for a short time. On the other hand, even those who choose to collect data indefinitely may not want older health information archived and available.