Abstract
Purpose
To overcome literacy-related barriers in the collection of electronic family health histories, we developed VICKY, an animated “virtual counselor”. This study examines the acceptability and accuracy of using VICKY to collect family histories, among underserved patients, compared to My Family Health Portrait (MFHP).
Methods
Participants were recruited from a patient registry at a safety net hospital and randomized to use either VICKY or MFHP. Accuracy was determined by comparing tool-collected histories to those obtained by a genetic counselor.
Results
A total of 70 participants completed this study. Participants rated VICKY easy to use (91%) and easy to follow (92%), would recommend VICKY to others (83%) and were highly satisfied (77%). VICKY identified 86% of first degree relatives, and 42% of second degree relatives; combined accuracy was 55%. Compared to MFHP, VICKY identified a greater number of health conditions overall (49% VICKY vs 31% MFHP, IRR: 1.59, 95% CI 1.13, 2.25, p=.008); in particular, hypertension (47% vs 15%; IRR = 3.18; 95% CI: 1.66, 6.10; p=.001) and type 2 diabetes (54% vs 22%; IRR = 2.47; 95% CI: 1.33, 4.60; p=.004).
Conclusion
These results demonstrate that technological support for documenting family history risks can be highly accepted, feasible, and effective.
Keywords: family health history, genetic literacy, health literacy, validation, sensitivity, My Family Health Portrait, relational agent
Introduction
The family health history is one of the most important risk factors for many chronic conditions including cardiovascular disease, diabetes, and cancer and represents an integration of disease risk stemming from genetic, environmental, and behavioral factors.1-3 When compared with genotypic information, family history remains a strong, independent risk factor for disease.4,5 As such, family history assessment remains the current gold standard for clinical risk assessment, 2,6,7 and is considered a genomic tool and proxy to genetic predisposition that can serve as a means to better guide and personalize medical care and disease prevention.1,5,8,9
Although the importance of family health history is evident, the collection of family history information by patients and the integration of family history assessment into clinical practice has been surprisingly poor in frequency and quality.10 Numerous barriers preclude the systematic documentation of family history in primary care settings.11 The most commonly documented barriers include lack of time, lack of physician compensation for the efforts, physician lack of knowledge and skills, and other logical barriers such as lack of standardization in family history collection methods.9,10,12,13 Even when family history is collected in primary care, it is often lacking in quality or detail that would yield useful information about disease risk.10,12
Due to the importance of family history assessment and its lack of systematic documentation, several national efforts have been undertaken to improve the documentation and use of family history, particularly in primary care settings.1,3 Yet, in spite of these national efforts to promote family history tools, concerns have been raised about the appropriateness of these tools for low literate, underserved populations.14 Health literacy has been defined as “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.”15 Approximately half of U.S. adults have limited health literacy, which disproportionally affects those who are less educated, elderly, poor, minorities, or have limited English proficiency.16-18 Although computer-based family history tools have been developed with the goal of increasing genetic literacy,1 there is evidence to suggest that existing tools may be challenging to use by a large portion of the U.S. population.14
In efforts to overcome the aforementioned barriers, we developed a Relational Agent or “virtual counselor” to collect family health history information. Relational Agents are computer-animated characters that use speech, gaze, hand gesture, prosody, and other nonverbal modalities to emulate the experience of human face-to-face conversation. They can be programmed and used for automated health education and behavioral counseling interventions, and have been demonstrated to establish and maintain therapeutic relationships through these and other interactions.19 These agents have been successfully used to facilitate medication adherence,20 health document explanation,21,22 breastfeeding promotion,23 and exercise and weight loss education and motivation.24-27 Use of the relational agent system requires very minimal reading skills. We have previously shown that the interface can be designed in a manner that is usable for people with limited health literacy, limited reading capacity, and no prior computer experience, 19,22,28,29 which makes it a potentially useful platform to collecting detailed family history information in an electronic format.
We developed a prototype virtual counselor that we named VICKY (VIrtual Counselor for Knowing Your Family History, see Figure 1). VICKY is an animated computer character, designed to collect family health history information by asking a series of questions about the user's family health history, targeting common chronic conditions including heart disease, diabetes, hypertension, stroke, and various cancers. Users respond to VICKY's verbal questions by selecting a pre-formulated simple response on a touch screen, with the choices updated at each turn in the conversation. Response options are kept short and easy to read. Minimal reading and typing are required, thus reducing the literacy burden. Moreover, additional opportunities are interwoven throughout the program to let respondents tell VICKY when they are uncertain about the meaning of a response option. For example, as shown in Figure 1, there is a response option indicating “not sure what these problems are.” Subsequent screens have VICKY asking whether they would like information about a health problem. Selected options would then be verbally explained to the participant, rather than presenting an explanation or definition in text format on the screen. VICKY was deployed on a touch-screen tablet computer, and interleaves her interview with displays of the family history pedigree chart as it is incrementally constructed.
Figure 1. VICKY: “Have you had any of these health problems?”.
The relational agent used within the VICKY program was developed and evaluated on several prior automated health counseling interventions.19,22 The dialogue content written specifically for VICKY was developed by experts in computer science, health communication, health literacy and genetic counseling, and extensively tested by developers and research assistants using family test cases to check for errors on flow, logic, and completeness. In addition, user testing interviews were conducted on 10 patients to identify further problematic areas for fixing prior to the study. During user testing interviews, participants were instructed to use the tool and asked to “think aloud” as they were using the VICKY program. Research assistants observed participants as they were using the tool and documented areas that caused confusion and errors in data entry. Participants were also asked about their general experience with VICKY via a series of both open- and closed-ended questions. Results from the user testing were used to make updates to our prototype, which we then subsequently evaluated in a pilot study.
This paper reports on the results from the pilot study. Specifically, it examines the feasibility of using VICKY to collect family health history information, within an underserved patient population. The acceptability of the program was evaluated during an interview process. Accuracy of the information collected was determined by comparing family health histories collected by VICKY to those generated independently by a certified genetic counselor (gold standard). In addition, as part of our study, we randomized patients to use either VICKY or another computer-based tool that has been widely promoted (Surgeon General's My Family Health Portrait - https://familyhistory.hhs.gov/) to compare the acceptability and accuracy of VICKY to an existing tool.
Materials and Methods
Participants
Patients were recruited from the ReSPECT Registry, a recruitment services program that provides support to research investigators at Boston Medical Center (BMC). BMC is the largest safety net hospital in New England. Approximately 73% of BMC patients come from underserved populations, including low-income families, elders, people with disabilities, and immigrants. Individuals are recruited to the ReSPECT registry from various BMC and community venues, as well as online advertisements that link directly to the registry website.
Participants were eligible for this study if they were 18 years or older, could read and write in English, and were currently or had ever been a patient at BMC. The registry staff contacted eligible participants via email, letter or telephone and provided a brief summary of the study and asked if they would be interested in participating. A list of registry members who were interested in participating was provided to the study research assistant who called to confirm study eligibility and extend an invitation to participate in the study. An appointment was subsequently scheduled for those agreeing to participate. The CONSORT diagram for this study is presented in Figure 2.
Figure 2. Study CONSORT diagram.
Procedures
All participant interviews took place at BMC with a trained research assistant (RA). Following the consent process, participants were randomized to use either VICKY or MFHP and instructed to use the tool to enter their family health history information. Because the study evaluated the usability of the standalone tools, participants were not provided with additional assistance or guidance to understand the tool instructions or complete their histories. Participants were provided with as much time as needed to complete this process, with most participants completing this step within 15-30 minutes. Following the interaction with VICKY or MFHP, participants were interviewed in-person by the RA to obtain detailed feedback about their experiences with the tool. Participants were then interviewed by a genetic counselor over the telephone to obtain a detailed family health history. The genetic counselor was blind to the study arm and followed a general script for the interview, which emphasized the collection of information for common chronic conditions. A single genetic counselor conducted all the interviews and created a “gold standard” pedigree for each participant, generated in Progeny (http://www.progenygenetics.com/). All pedigree information was based solely on participant self-report. Participants received a $20 gift card for their time. In addition, participants were offered a copy of their family history pedigree, which they could receive in the mail within 1-2 weeks of participation.
Study Measures
Demographics
Standard demographic information collected for all participants included age, gender, education, race/ethnicity, and income.
Computer Experience
Participants were asked “How much experience do you have with computers?” and responded by selecting either “I've never used one,” “I've tried one a few times,” I use one regularly,” or “I'm an expert.” They also rated their computer skills on a scale from 1-Poor to 5-Excellent.
Tool Evaluation
Likert scale questions were used to obtain general feedback about the family health history tools. Questions were on a 5-point scale (not at all – very) and included, “How easy was it to use the tool,” “How easy was it to follow the flow of the tool,” “How easy was it to understand the information being asked,” “How much do you like this tool,” “How likely are you to use this tool on your own,” “Would you recommend this tool to others,” and “Overall, how satisfied are you with this tool?” Percent endorsement or agreement for each item was derived from those responding either a 4 or 5 on the 5-point scale. In addition, a single item also asked “Overall, how would you rate the quality of the family history tool,” (5-point scale, poor/fair/good/very good/excellent).
Health Literacy
The Rapid Estimate of Adult Literacy in Medicine (REALM) tool was used to assess health literacy.30,31 The REALM includes a list of 66 medical words that participants are instructed to read. A reading grade equivalent is determined based on the number of words pronounced correctly.
Accuracy: Family Members Identified
Using the pedigree generated by the genetic counselor as the gold standard, the number and relationship of family members identified by a computer tool was compared to family members identified by the genetic counselor. Accuracy rates were calculated by dividing the number of tool identified relatives by the number identified by the genetic counselor and were derived for 1st degree relatives, 2nd degree relatives, and combined total 1st and 2nd degree relatives.
Accuracy: Health Conditions Identified
The accuracy of health conditions identified was derived by calculating sensitivity estimates for first and second degree (and total) relatives for each health condition. Sensitivity, or true positive rate, was defined as the disease cases reported in a tool that were also identified by the genetic counselor (true positive) divided by the disease cases not captured by the tool but captured by the genetic counselor (false negatives) plus the true positives. The health conditions assessed included heart disease, type 2 diabetes, stroke, hypertension, and cancers of the breast and colon.
Analytic Plan
Descriptive statistics were used to compute means and standard deviations for continuous variables and counts with percentages for categorical variables. A series of bivariate analyses were used to examine the effectiveness of the randomization by comparing and testing distributions of baseline variables by intervention arm using cross-tabulation with chi-square or Fisher's exact tests as appropriate. Any lack of balance on these variables between the two groups was addressed by the inclusion of these variables as covariates in multivariable analyses. Study arms were compared on percent endorsement of the tool evaluation items using multivariable logistic regression. Multivariable Poisson regression models were used to examine the effect of study arm on identification of relatives and health conditions. Rate ratios illustrating relative differences in accuracy and their 95% confidence intervals were computed from these models to compare the study arms. All analyses were conducted using SAS version 9.3. Analyses were conducted using 2-sided tests, and a p-value < 0.05 was considered statistically significant.
Results
Participant Demographics
A total of 74 individuals were enrolled in the study with 70 individuals completing the protocol (Figure 2). Four individuals were dropped from the study analyses due to technical issues wherein pedigrees generated from the tools (n=3 for VICKY and n=1 for MFHP) were not saved. Among the 70 individuals who completed the protocol, the majority were age 45 or older (74%), 60% were female, and 63% were African American (Table 1). Over half of the study population (51%) had the equivalent of high school education or less and 60% had a household income of $25,000 or less per year. According to the REALM health literacy assessment, 38% of participants had a reading equivalent of 8th grade or lower. Approximately 30% of participants had limited computer experience. Mean ratings for computer skills was 3.31 (SD=1.18). Chi-square and Fisher's exact tests were conducted to examine demographic differences at baseline between the two study arms. Results showed a borderline difference between groups on gender composition, with significantly more females in the VICKY arm (71%) compared to the MFHP arm (49%, p=.051). No other differences were noted. Because of this borderline difference, all subsequent analyses were repeated adjusting for sex to determine its impact on study outcomes.
Table 1. Participant Demographics (N=70).
| VICKY (N=35) N (%) |
MFHP (N=35) N (%) |
TOTAL (N=70) N (%) |
p-value | ||
|---|---|---|---|---|---|
| Gender | |||||
|
| |||||
| Female | 25 (71%) | 17 (49%) | 42 (60%) | 0.0510 | |
| Male | 10 (29%) | 18 (51%) | 28 (40%) | ||
|
| |||||
| Age (Years) | |||||
|
| |||||
| 18-24 | -- | 1 (3%) | 1 (1%) | 0.6970 | |
| 25-34 | 4 (11%) | 2 (6%) | 6 (9%) | ||
| 35-44 | 6 (17%) | 5 (14%) | 11 (16%) | ||
| 45-54 | 10 (29%) | 15 (43%) | 25 (36%) | ||
| 55-64 | 10 (29%) | 7 (20%) | 17 (24%) | ||
| 65 and over | 5 (14%) | 5 (14%) | 10 (14%) | ||
|
| |||||
| Ethnicity/Race | |||||
|
| |||||
| Hispanic/Latino | 3 (9%) | 1 (3%) | 4 (6%) | 0.6139 | |
| Caucasian | 8 (23%) | 10 (29%) | 18 (26%) | 0.8177 | |
| African American | 23 (66%) | 21 (60%) | 44 (63%) | ||
| Asian | 1 (3%) | -- | 1 (1%) | ||
| Other or multiple | 3 (8%) | 4 (11%) | 7 (10%) | ||
|
| |||||
| Education | |||||
|
| |||||
| Less than 9th grade | -- | 1 (3%) | 1 (1%) | 0.6557 | |
| HS no diploma | 8 (23%) | 7 (20%) | 15 (21%) | ||
| HS diploma or GED | 10 (29%) | 10 (29%) | 20 (29%) | ||
| Some college | 7 (20%) | 11 (31%) | 18 (26%) | ||
| College degree | 7 (20%) | 3 (8%) | 10 (14%) | ||
| Graduate or doctoral degree | 3 (8%) | 3 (8%) | 6 (9%) | ||
|
| |||||
| Household Income | |||||
|
| |||||
| Less than $25,000 | 24 (69%) | 18 (51%) | 42 (60%) | 0.2305 | |
| $25,001-35,000 | 2 (6%) | 7 (20%) | 9 (13%) | ||
| $35,001-50,000 | 4 (11%) | 3 (8%) | 7 (10%) | ||
| $50,001-75,000 | 2 (6%) | 2 (6%) | 4 (6%) | ||
| More than $75,000 | 3 (8%) | 2 (6%) | 5 (7%) | ||
| Prefer not to answer | -- | 3 (8%) | 3 (4%) | ||
|
| |||||
| REALM grade | |||||
|
| |||||
| 4th-6th grade | 3 (9%) | 3 (8%) | 6 (9%) | 0.9307 | |
| 7th-8th grade | 11 (31%) | 9 (26%) | 20 (29%) | ||
| HS | 21 (60%) | 23 (66%) | 44 (63%) | ||
|
| |||||
| Computer Experience | |||||
|
| |||||
| Never used one | 1 (3%) | 4 (11%) | 5 (7%) | 0.4937 | |
| Tried one a few times | 9 (26%) | 7 (20%) | 16 (23%) | ||
| Used one regularly | 21 (60%) | 22 (63%) | 43 (61%) | ||
| Expert | 4 (11%) | 2 (6%) | 6 (9%) | ||
Note: All p-values reflect results from Fisher's Exact Test, with the exception of gender, which reflects results from Chi-square test.
Tool Evaluation
Each family history tool was evaluated for user acceptability (Table 2). The majority of participants rated VICKY as easy to use (91%), easy to follow (92%), and understood the questions being asked (97%). A majority (83%) also indicated they would recommend VICKY to others and 77% were highly satisfied. Only 57% indicated they would be likely to use VICKY on their own. VICKY was rated as very good or excellent quality by 62% of participants.
Table 2. Evaluation of tools, % endorsement*.
| Evaluation Question | VICKY | MFHP | OR (95% CI) | p-value |
|---|---|---|---|---|
| 1. How easy was it to use the tool? | 91% | 59% | 7.47 (1.90, 29.28) | 0.0017 |
| 2. How easy was it to follow the flow of the tool? | 91% | 51% | 10.07 (2.60, 39.11) | 0.0002 |
| 3. How easy was it to understand the information being asked? | 97% | 66% | 17.74 (2.16, 145.95) | 0.0007 |
| 4. How much do you like this tool? | 83% | 66% | 2.52 (0.82, 7.75) | 0.1008 |
| 5. How likely are you to use this tool on your own? | 57% | 46% | 1.58 (0.62, 4.07) | 0.3388 |
| 6. Would you recommend this tool to others? | 83% | 49% | 5.12 (1.70, 15.39) | 0.0025 |
| 7. Overall, how satisfied are you with this tool? | 77% | 54% | 2.84 (1.01, 7.98) | 0.0440** |
| 8. Overall, how would you rate the quality of the family history tool? (very good/excellent) | 62% | 46% | 1.92 (0.74, 5.01) | 0.1813 |
Percentages reflect those who rated the item either 4 or 5 on a 5-point scale, with the exception of the last item, which reflects those participants selecting either very good or excellent on a 5-point scale.
No longer significantly different after adjusting for sex (OR=2.60; 95% CI 0.90-7.47, p=.0763).
Table 2 also presents the percentage endorsement of evaluation items for the My Family Health Portrait (MFHP) tool. Notably, for five of eight items, evaluation scores for VICKY were significantly greater than for MFHP among study participants. Upon adjusting for sex, however, one of the items pertaining to overall satisfaction with the tool was no longer significantly different between VICKY and MFHP (adjusted OR = 2.60; 95% CI: 0.90-7.47, p=.0763).
Accuracy of Family Members Identified
Compared to family histories obtained by a Genetic Counselor, VICKY identified 86% (227/263) of first degree relatives and 42% (265/632) of second degree relatives for a combined accuracy rate of 55% (492/895) for both first and second degree relatives. MFHP identified 84% (231/274) of first degree relatives and 43% (300/699) of second degree relatives, for a combined accuracy rate of 55% (531/973). No significant differences were noted between the two computerized tools for identifying family members (all ps>.05). Analyses adjusting for sex were consistent with unadjusted analyses.
Accuracy of Health Conditions Identified
Table 3 presents the sensitivities for the six conditions, stratified by tool and type of relative. Overall sensitivity for the six health conditions was 49% for VICKY and 31% for MFHP (p=.008). The sensitivity or true positive rate for identifying these conditions was greater for first degree (60% VICKY, 37% MFHP) than second degree (33% VICKY, 24% MFHP) relatives, regardless of family history tool. Compared to MFHP, VICKY was more accurate in identifying cases of hypertension (p=.001) and type 2 diabetes (p=.004), the most prevalent conditions within the study sample. Results comparing VICKY to MFHP did not differ when models adjusted for sex.
Table 3. Sensitivities (%) of Identified Health Conditions (Tool versus Genetic Counselor).
| Condition | VICKY (n=35) % (Tool/GC) |
MFHP (n=35) % (Tool/GC) |
IRR* (95% CI) | p-value | |
|---|---|---|---|---|---|
| Total Conditions (6) | 1st degree | 60% (99/166) | 37% (43/116) | 1.61 (1.14, 2.26) | 0.006 |
| 2nd degree | 33% (38/114) | 24% (25/105) | 1.40 (0.73, 2.68) | 0.308 | |
| Total | 49% (137/280) | 31% (68/221) | 1.59 (1.13, 2.25) | 0.008 | |
|
| |||||
| Heart Disease | 1st degree | 65% (17/26) | 70% (14/20) | 0.93 (0.55, 1.58) | 0.800 |
| 2nd degree | 39% (7/18) | 36% (9/25) | 1.08 (0.39, 2.97) | 0.881 | |
| Total | 55% (24/44) | 51% (23/45) | 1.07 (0.65, 1.75) | 0.797 | |
|
| |||||
| Hypertension | 1st degree | 56% (38/68) | 20% (8/40) | 2.79 (1.45, 5.37) | 0.002 |
| 2nd degree | 27% (8/30) | 5% (1/21) | 5.60 (0.72, 43.42) | 0.073 | |
| Total | 47% (46/98) | 15% (9/61) | 3.18 (1.66, 6.10) | 0.001 | |
|
| |||||
| Type 2 Diabetes | 1st degree | 68% (34/50) | 34% (12/35) | 1.98 (1.06, 3.70) | 0.032 |
| 2nd degree | 33% (11/33) | 11% (4/38) | 3.17 (0.93, 10.78) | 0.065 | |
| Total | 54% (45/83) | 22% (16/73) | 2.47 (1.33, 4.60) | 0.004 | |
|
| |||||
| Stroke | 1st degree | 40% (6/15) | 40% (4/10) | 1.00 (0.32, 3.14) | 1.000 |
| 2nd degree | 43% (9/21) | 60% (6/10) | 0.71 (0.27, 1.86) | 0.492 | |
| Total | 42% (15/36) | 50% (10/20) | 0.83 (0.41, 1.68) | 0.612 | |
|
| |||||
| Breast Cancer | 1st degree | 50% (2/4) | 43% (3/7) | 1.17 (0.20, 6.87) | 0.865 |
| 2nd degree | 20% (1/5) | 25% (2/8) | 0.80 (0.08, 7.57) | 0.846 | |
| Total | 33% (3/9) | 33% (5/15) | 1.00 (0.25, 3.96) | 1.000 | |
|
| |||||
| Colon Cancer | 1st degree | 67% (2/3) | 50% (2/4) | 1.33 (0.24, 7.30) | 0.740 |
| 2nd degree | 29% (2/7) | 100% (3/3) | 0.29 (0.09, 0.95) | 0.040 | |
| Total | 40% (4/10) | 71% (5/7) | 0.56 (0.24, 1.33) | 0.190 | |
IRR (incidence rate ratio) and 95% confidence interval (CI) comparing the sensitivity of VICKY/sensitivity of MFHP based on Poisson regression model.
Discussion
Computerized tools that can facilitate the systematic documentation of the family health history have been developed in recent years, yet concerns have been raised about their usability, particularly among those with limited health literacy.14 The present study set out to examine the acceptability and feasibility of using a virtual counselor to electronically document family health history from an underserved patient population. Study participants were willing to enter family history information into the system, and found the virtual counselor easy to use, understood the questions being asked, and would recommend VICKY to others. These results demonstrate the acceptability of a virtual counselor, as well as the feasibility of using this platform to collect family health history in an electronic format, while overcoming some of the previously identified barriers for collecting this information among underserved patient populations using existing tools like My Family Health Portrait (MFHP). 32-35
VICKY and MFHP were comparable in terms of identifying the number and relationship of relatives, but performed differently with the identification of health conditions, particularly for conditions with a higher prevalence. The questions asked in VICKY and MFHP were relatively similar in regards to the identification of family members, starting with immediate, first degree relatives and then branching out to allow respondents to include other family members. This similarity in structure likely contributed to the similarities in outcomes observed. Future research should explore different options for soliciting second degree family member information since the accuracy of documenting those family members was much worse, regardless of tool.36 In the case of the VICKY prototype tested, participants were not asked about nieces or nephews, grandchildren, or half siblings, thus contributing to the lower accuracy levels. These family members will be included in the next version of the VICKY program and we will ascertain the extent to which their inclusion improves the accuracy of second degree relative documentation.
The differences between MFHP and VICKY related for disease conditions suggests that a challenge with the MFHP tool may relate to issues in the entry of disease data. MFHP is capable of collecting information on a greater number of health conditions compared to VICKY. In addition, MFHP asks about the health conditions using more advanced language and medical terminology (e.g., ‘hypertension’ instead of ‘high blood pressure’) presented within detailed drop-down menus which participants had to review to select the corresponding condition. As such, issues in tool content including the number of diseases collected and the manner in which the disease information is asked may contribute the accuracy of disease collection by any family history tool and warrants attention in future research.
The average amount of time it took study participants to complete their family history on a computer was 15-30 minutes. However, the duration was at times shorter for participants using the MFHP tool because some patients (∼11%, 4/35) gave up early after trying unsuccessfully to use the tool. This typically occurred when there were frustrations with navigation and uncertainty about where to go next, and the inability to save information that was entered, which was a necessary step in order to advance in the program. Findings from other studies using MFHP have also reported similar challenges32,35 suggesting that issues in navigation may also contribute to the disease accuracy outcomes observed for the tool. Notably, all but one of the participants in the VICKY arm was able to complete the program, which may reflect differences in navigation burden between the tools since the virtual counselor directs navigation to subsequent screens as part of the conversation. We believe this is a key strength of the relational agent system.
This pilot study is the first study to obtain validation data of MFHP web platform among an underserved patient population. Sensitivity results observed in this study were significantly lower than results from a previously published validation study on MFHP, which was conducted on a very highly educated, white sample of patients participating in genetics research within the ClinSeq cohort.36 Others have examined MFHP within different platforms (telephone) with an underserved patient population and reported lower sensitivity rates.37 Altogether, these studies along with other qualitative research reporting challenges to using MFHP as a standalone system,32-35 further highlight the need to conduct validation studies of family history tools with a diverse patient population.
There were several limitations to the present study. First, the low prevalence of certain health conditions such as cancer, along with the small sample size of the pilot, greatly limited the comparisons we could make between tools for these conditions. Second, we did not attempt to clarify “heart disease” accuracy, as was conducted in the prior validation study using MFHP,36 however, this method was used for both tools examined in this study so no bias was introduced by this approach. As such, heart disease information collected could have included a wide range of heart-related conditions. In addition, because this was pilot study to demonstrate acceptability and feasibility of the VICKY program, we were underpowered to test for interactions by health literacy or computer experience. We did find, however, that accuracy outcomes did vary by education, health literacy, and computer experience (see supplementary Tables 1 & 2). Testing for interactions is a primary aim for the larger trial currently underway. Finally, participants in our study were recruited from a volunteer patient registry, which may not be representative of the underserved patient population at our institution and has implications for the generalizability of the findings.
This study also raises questions about methods for using a genetic counselor as the gold standard for comparison. The few family history tool validation studies published to date differ in whether the genetic counselor adapts and supplements pedigree data collected by the automated tool,36 or collects the health history information independently.38 The latter approach, which we also undertook in this pilot study, raises different issues that are challenging to resolve. We observed circumstances in which patients provide family health history to the automated tool that they choose not to reveal to the genetic counselor and/or are missed by the counselor. For example, in our prototype, we also included other conditions to test feasibility, including family history of alcoholism. Notably, there was a high rate of “false positives” (45%), wherein the tool reported family members with alcoholism but they were not identified on the gold standard genetic counselor pedigree. In this study, we did not re-contact participants to get further clarification of their family histories, but it is unlikely that all such reports represent false positive histories. This is something we will pursue in our next trial in efforts to gain a better understanding of this phenomenon.
As a result of this pilot study, specific opportunities to refine VICKY were identified. These improvements will be undertaken in the next development phase include the collection of more complex family trees (e.g., half siblings, twins, nieces/nephews, adopted relatives – which contributed to lower accuracy rates), additional health conditions, the inclusion of certain navigation features to increase accuracy (e.g., incorporating a ‘back button’ to allow patients to go back a screen and correct errors in data entry), and the expansion of data elements collected in efforts to obtain the minimum core data set for family history information set forth by American Health Information Community (AHIC).39. Finally, in order to increase reach and access among underserved minorities, VICKY is also needed in other languages. As such, future plans include programming and testing a Spanish language version of VICKY.
In sum, efforts to facilitate the electronic documentation of family health history should reflect the diverse needs of the population and ensure that barriers such as health or computer literacy do not limit who will be able to access and use such systems effectively. Ultimately, our goal is to improve the systematic documentation and use of family history in primary care to identify those at greatest risk for chronic diseases who would benefit from preventive intervention efforts.
Supplementary Material
Supplementary Table 1: Accuracy of family members identified (1st/2nd degree relatives) by education, health literacy or computer experience, N=70
Supplementary Table 2: Accuracy of health conditions identified (total conditions, 1st/2nd degree relatives) by education, health literacy or comput
Acknowledgments
Disclosure of Funding: This study was funded by grants from the Boston University School of Public Health and the Boston University Clinical and Translational Science Institute (BU-CTSI)(NCRR UL1RR025771). Additionally, Dr. Wang was supported by the National Cancer Institute (K07 CA131103) and a Peter T. Paul career development professorship from Boston University.
Footnotes
Conflict of Interest Notification: The authors declare no conflicts of interest.
References
- 1.Guttmacher AE, Collins FS, Carmona RH. The family history - More important than ever. New England Journal of Medicine. 2004;351(22):2333–2336. doi: 10.1056/NEJMsb042979. [DOI] [PubMed] [Google Scholar]
- 2.Pyeritz RE. The family history: the first genetic test, and still useful after all those years? Genetics in Medicine. 2012;14(1):3–9. doi: 10.1038/gim.0b013e3182310bcf. [DOI] [PubMed] [Google Scholar]
- 3.Valdez R, Yoon PW, Qureshi N, Green RF, Khoury MJ. Family history in public health practice: a genomic tool for disease prevention and health promotion. Annu Rev Public Health. 2010;31:69–87. doi: 10.1146/annurev.publhealth.012809.103621. 61 p following 87. [DOI] [PubMed] [Google Scholar]
- 4.O'Neill SM, Rubinstein WS, Wang C, et al. Familial Risk for Common Diseases in Primary Care: The Family Healthware™ Impact Trial (FHITr) American Journal of Preventive Medicine. 2009;36(6):506–514. doi: 10.1016/j.amepre.2009.03.002. [DOI] [PubMed] [Google Scholar]
- 5.Yoon PW, Scheuner MT, Peterson-Oehike K, Gwinn M, Faucett A, Khoury MJ. Can family history be used as a tool for public health and preventive medicine. Genetics in Medicine. 2002;4(4):304–310. doi: 10.1097/00125817-200207000-00009. [DOI] [PubMed] [Google Scholar]
- 6.Doerr M, Eng C. Personalised care and the genome. BMJ. 2012;344:e3174. doi: 10.1136/bmj.e3174. [DOI] [PubMed] [Google Scholar]
- 7.Heald B, Edelman E, Eng C. Prospective comparison of family medical history with personal genome screening for risk assessment of common cancers. Eur J Hum Genet. 2012;20(5):547–551. doi: 10.1038/ejhg.2011.224. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wilson BJ, Qureshi N, Santaguida P, et al. Systematic review: family history in risk assessment for common diseases. Ann Intern Med. 2009;151(12):878–885. doi: 10.7326/0003-4819-151-12-200912150-00177. [DOI] [PubMed] [Google Scholar]
- 9.Giovanni MA, Murray MF. The application of computer-based tools in obtaining the genetic family history. Curr Protoc Hum Genet. 2010;Chapter 9 doi: 10.1002/0471142905.hg0921s66. Unit 9 21. [DOI] [PubMed] [Google Scholar]
- 10.Rich EC, Burke W, Heaton CJ, et al. Reconsidering the family history in primary care. Journal of General Internal Medicine. 2004;19:273–280. doi: 10.1111/j.1525-1497.2004.30401.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Berg AO, Baird MA, Botkin JR, et al. National Institutes of Health State-of-the-Science Conference Statement: Family History and Improving Health. Ann Intern Med. 2009;151(12):872–877. doi: 10.7326/0003-4819-151-12-200912150-00165. [DOI] [PubMed] [Google Scholar]
- 12.Kanetzke EE, Lynch J, Prows CA, Siegel RM, Myers MF. Perceived utility of parent-generated family health history as a health promotion tool in pediatric practice. Clin Pediatr (Phila) 2011;50(8):720–728. doi: 10.1177/0009922811403301. [DOI] [PubMed] [Google Scholar]
- 13.Wood ME, Stockdale A, Flynn BS. Interviews with primary care physicians regarding taking and interpreting the cancer family history. Fam Pract. 2008;25(5):334–340. doi: 10.1093/fampra/cmn053. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Wang C, Gallo RE, Fleisher L, Miller SM. Literacy assessment of family health history tools for public health prevention. Public Health Genomics. 2011;14:222–237. doi: 10.1159/000273689. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Institute of Medicine. Health Literacy: A prescription to end confusion. Washington: National Academy Press; 2004. [Google Scholar]
- 16.Kirsch I, Jungeblut A, Jenkins L, Kolstad A. Adult Literacy in America: A First Look at the Results of the National Adult Literacy Survey. Washington, DC: National Center for Educational Statistics, US Department of Education; Sep, 1993. [Google Scholar]
- 17.Paasche-Orlow MK, Parker RM, Gazmararian JA, Nielsen-Bohlman LT, Rudd RR. The prevalence of limited health literacy. J Gen Intern Med. 2005;20(2):175–184. doi: 10.1111/j.1525-1497.2005.40245.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Kutner M, Greenberg E, Jin Y, Paulsen C. The Health Literacy of America's Adults: Results from the 2003 National Assessment of Adult Literacy (NCES 2006-483) Washington, DC: National Center for Educational Statistics, US Department of Education; 2006. [Google Scholar]
- 19.Bickmore TW, Pfeifer LM, Byron D, et al. Usability of conversational agents by patients with inadequate health literacy: evidence from two clinical trials. J Health Commun. 2010;15(Suppl 2):197–210. doi: 10.1080/10810730.2010.499991. [DOI] [PubMed] [Google Scholar]
- 20.Bickmore T, Puskar K, Schlenk E, Pfeifer L, Sereika S. Maintaining Reality: Relational Agents for Antipsychotic Medication Adherence. Interacting with Computers. 2010;22:276–288. [Google Scholar]
- 21.Bickmore TW, Pfeifer LM, Paasche-Orlow MK. Health document explanation by virtual agents. Intelligent Virtual Agents, Proceedings. 2007;4722:183–196. [Google Scholar]
- 22.Bickmore TW, Pfeifer LM, Paasche-Orlow MK. Using computer agents to explain medical documents to patients with low health literacy. Patient Education and Counseling. 2009;75(3):315–320. doi: 10.1016/j.pec.2009.02.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Edwards R, Bickmore T, Jenkins L, Foley M. Use of an Interactive Computer Agent to Support Breastfeeding. Maternal and Child Health. doi: 10.1007/s10995-013-1222-0. to appear. [DOI] [PubMed] [Google Scholar]
- 24.Bickmore T, Cange A, Kulshreshtha A, Kvedar J. An Internet-Based Virtual Coach to Promote Physical Activity Adherence in Overweight Adults: Randomized Controlled Trial. Journal of Medical Internet Research. 2012;14(1):5–5. doi: 10.2196/jmir.1629. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Bickmore TW, Mauer D, Brown T. Context Awareness in a Handheld Exercise Agent. Pervasive Mob Comput. 2009;5(3):226–235. doi: 10.1016/j.pmcj.2008.05.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Bickmore TW, Silliman RA, Nelson K, et al. A randomized controlled trial of an automated exercise coach for older adults. J Am Geriatr Soc. 2013;61(10):1676–1683. doi: 10.1111/jgs.12449. [DOI] [PubMed] [Google Scholar]
- 27.Ellis T, Latham NK, DeAngelis TR, Thomas CA, Saint-Hilaire M, Bickmore TW. Feasibility of a virtual exercise coach to promote walking in community-dwelling persons with Parkinson disease. Am J Phys Med Rehabil. 2013;92(6):472–481. doi: 10.1097/PHM.0b013e31828cd466. quiz 482-475. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.King AC, Bickmore TW, Campero MI, Pruitt LA, Yin JL. Employing Virtual Advisors in Preventive Care for Underserved Communities: Results From the COMPASS Study. J Health Commun. 2013 doi: 10.1080/10810730.2013.798374. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Sequeira SS, Eggermont LH, Silliman RA, et al. Limited health literacy and decline in executive function in older adults. J Health Commun. 2013;18(Suppl 1):143–157. doi: 10.1080/10810730.2013.825673. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Arozullah AM, Yarnold PR, Bennett CL, et al. Development and validation of a short-form, rapid estimate of adult literacy in medicine. Med Care. 2007;45(11):1026–1033. doi: 10.1097/MLR.0b013e3180616c1b. [DOI] [PubMed] [Google Scholar]
- 31.Davis TC, Long SW, Jackson RH, et al. Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam Med. 1993;25(6):391–395. [PubMed] [Google Scholar]
- 32.Berger KA, Lynch J, Prows CA, Siegel RM, Myers MF. Mothers' perceptions of family health history and an online, parent-generated family health history tool. Clin Pediatr (Phila) 2013;52(1):74–81. doi: 10.1177/0009922812467531. [DOI] [PubMed] [Google Scholar]
- 33.Wallace JP, Baugh C, Cornett S, et al. A family history demonstration project among women in an urban Appalachian community. Prog Community Health Partnersh. 2009;3(2):155–163. doi: 10.1353/cpr.0.0061. [DOI] [PubMed] [Google Scholar]
- 34.Arar N, Delgado E, Lee S, Abboud HE. Improving learning about familial risks using a multicomponent approach: the GRACE program. Per Med. 2013;10(1):35–44. doi: 10.2217/pme.12.107. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Arar N, Seo J, Abboud HE, Parchman M, Noel P. Veterans' experience in using the online Surgeon General's family health history tool. Per Med. 2011;8(5):523–532. doi: 10.2217/pme.11.53. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Facio FM, Feero WG, Linn A, Oden N, Manickam K, Biesecker LG. Validation of My Family Health Portrait for six common heritable conditions. Genet Med. 2010;12(6):370–375. doi: 10.1097/GIM.0b013e3181e15bd5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Murray MF, Giovanni MA, Klinger E, et al. Comparing Electronic Health Record Portals to Obtain Patient-Entered Family Health History in Primary Care. J Gen Intern Med. 2013 doi: 10.1007/s11606-013-2442-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Cohn WF, Ropka ME, Pelletier SL, et al. Health Heritage(c) a web-based tool for the collection and assessment of family health history: initial user experience and analytic validity. Public Health Genomics. 2010;13(7-8):477–491. doi: 10.1159/000294415. [DOI] [PubMed] [Google Scholar]
- 39.Feero WG, Bigley MB, Brinner KM. New standards and enhanced utility for family health history information in the electronic health record: an update from the American Health Information Community's Family Health History Multi-Stakeholder Workgroup. J Am Med Inform Assoc. 2008;15(6):723–728. doi: 10.1197/jamia.M2793. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplementary Table 1: Accuracy of family members identified (1st/2nd degree relatives) by education, health literacy or computer experience, N=70
Supplementary Table 2: Accuracy of health conditions identified (total conditions, 1st/2nd degree relatives) by education, health literacy or comput