Abstract
Objectives
This study is a randomized controlled trial comparing the efficacy of a virtual counselor (VICKY) to the My Family Health Portrait (MFHP) tool for collecting family health history (FHx).
Methods
A total of 279 participants were recruited from a large safety-net hospital and block randomized by health literacy to use one of the digital FHx tools, followed by a genetic counselor interview. A final sample of 273 participants were included for analyses of primary study aims pertaining to tool concordance, which assessed agreement between tool and genetic counselor.
Results
Tool completion differed significantly between tools (VICKY=97%, MFHP=51%; p<.0001). Concordance between tool and genetic counselor was significantly greater for participants randomized to VICKY compared to MFHP for ascertaining first and second-degree relatives (ps<.0001), and most health conditions examined. There was significant interaction by health literacy, with greater differences in concordance observed between tools among those with limited literacy.
Conclusions
A virtual counselor overcomes many of the literacy-related barriers to using traditional digital tools and highlights an approach that may be important to consider when collecting health histories from vulnerable populations.
Practice Implications
The usability of digital health history tools will have important implications for the quality of the data collected and its downstream clinical utility.
Clinicaltrials.gov:
Keywords: family health history, virtual counselor, health literacy, ehealth, health disparities, vulnerable populations
1. INTRODUCTION
Family health history represents one of the most important risk factors for chronic disease, remains the “first genomic test in precision medicine,” and is a clinical standard of care for the identification of at-risk individuals [1–3]. In spite of efforts undertaken over the past decade to improve the collection of family health history information, approximately two-thirds of the U.S. population have never actively collected their family health history [4]. Clinically, the detailed assessment of family health history is underutilized [2, 5]. Numerous barriers preclude the systematic documentation of family history in primary care settings including the lack of provider time to collect the data, lack of compensation for the efforts, physician lack of knowledge and skill, and lack of access to digital systems to collect such data in a manner that can be easily used to generate risk assessments [2, 6–9]. As such, better digital tools are needed [10].
Although several digital family history tools exist [11], few have undergone validation efforts to determine their utility in assessing accurate family histories [12–15]. Moreover, concerns have been raised that many may be inaccessible for diverse and vulnerable patient populations, due to health literacy barriers. Our prior research has shown that the reading grade level of publicly available family history tools averaged between 12 to 13, beyond the reading capacity of at least half of the adults in the U.S. [16]. Other studies have demonstrated that challenges observed in the usability of digital family history tools among some populations may preclude their effective use [14, 15, 17–20].
Our work has previously demonstrated the potential value of using a virtual counselor to collect family health histories [14]. Our virtual counselor “VICKY” (Virtual Counselor for Knowing Your Family History) is an animated computer character designed to collect a detailed family health history by asking the user a series of questions. In a pilot study, VICKY was demonstrated to be acceptable and feasible as an approach to collect family history information from a diverse, vulnerable patient population. Over 90% of participants rated VICKY as easy to use and follow and 77% were highly satisfied with the tool. Moreover, VICKY identified 86% of first-degree relatives [14]. We subsequently developed VICKY 2.0, which incorporated additional diseases to be ascertained by the virtual counselor and developed a Spanish-language version of VICKY [21].
We conducted a randomized trial of VICKY versus My Family Health Portrait (a widely available web-based tool for collecting family health histories), comparing each to a professional standard of histories collected by a certified genetic counselor [12, 13]. Both English and Spanish language versions of each tool were included. The purpose of this paper is to present our trial results and to determine whether tool concordance varies as a function of health literacy.
2. METHODS
2.1. Study Design and Participants
A randomized controlled trial was conducted to compare the efficacy of using a virtual counselor (VICKY) to the existing My Family Health Portrait (MFHP) tool for collecting family health history among an underserved primary care patient population (Clinicaltrials.gov: NCT02344433). Study participants were recruited from Boston Medical Center (BMC), the largest safety-net hospital in New England, where over 70% of the patient population are racial/ethnic minorities. Patients with upcoming appointments were mailed a study invitation letter with the option to return an opt-out card if they were not interested. Those who did not op-out were contacted via telephone to determine if they were eligible for the study. Those who were over the age of 21, able to speak English and/or Spanish, and a current BMC patient with an upcoming scheduled appointment within 3 months were deemed eligible to participate and scheduled for a baseline visit. Baseline visits occurred separate from any medical appointments for clinical services at BMC.
2.2. Study Procedures
Upon arrival for the baseline visit, participants were consented for the study by a study research assistant and subsequently asked baseline survey questions, which included sociodemographic and psychosocial survey items in addition to one [22] of the two health literacy assessments asked of all participants [22, 23]. All survey responses were recorded in RedCap (Research Electronic Data Capture, https://www.project-redcap.org/).
Participants were then block randomized by health literacy [22] and asked to use one of the two tools to enter their family health history information. Research assistants were instructed to encourage participants to enter all their known family history to the best of their ability and not assist participants in the use of either of the digital tools. Any challenges to tool use, including participant requests to discontinue using a tool, were noted by research assistants in RedCap. Following completion of this activity, research assistants connected participants to a genetic counselor over video conferencing and left the room. Participants then spoke with a genetic counselor to collect a detailed, three-generation family health history, which served as the professional standard comparison for the digital tools. Interviews with the genetic counselor were recorded to enable audit checks for quality, data cleaning and study implementation. Genetic counselors were blinded to the tool used by participants. All counselors were provided with a general script to follow for collecting family history information, which detailed the family members and health conditions of interest for the study. Following the genetic counselor interview, research assistants re-entered the room and asked immediate follow-up survey questions, including the Newest Vital Signs (NVS), which is a formal test of health literacy (vs self-report)[23]. Participants were then provided with a copy of their pedigree generated from the tool (if available) at the end of their session.
2.3. Intervention and Control Arm Tools
2.3.1. Intervention: VICKY
VICKY (Virtual Counselor for Knowing Your Family History – see Figure 1) has been described in detail elsewhere [11, 14, 21]. Using a conversational agent platform, users respond to VICKY’s verbal questions by selecting from pre-formulated simple responses on a touchscreen, simulating face-to-face conversations between a patient and health care provider. The initial VICKY 1.0 prototype was previously evaluated by our team for acceptability and feasibility of use [14]. We subsequently developed VICKY 2.0, which included a more comprehensive list of health conditions (e.g., heart disease, stroke, cancer, diabetes, hypertension, high cholesterol, mental health issues, schizophrenia, addiction, and alcoholism), and also created a Spanish language version of VICKY [21].
Figure 1.
VICKY: Virtual Counselor for Knowing Your Family History (selected screenshots)
2.3.2. Control: MFHP
My Family Health Portrait (MFHP; https://phgkb.cdc.gov/FHH), developed by the Office of the Surgeon General and the National Human Genome Research Institute, was first launched in November 2004 in both English and Spanish [24]. It has gone through several updates over the years and remains one of the few publicly available online tools to document family health history [11]. MFHP uses a traditional interface wherein pop-up windows and drop-down menus are used to facilitate data entry. Conditions listed in the drop-down menus include heart disease, stroke, cancer, diabetes, hypertension, high cholesterol, and psychological disorders. Other conditions not included in drop-down menus can be added under the “other- add new” option.
2.4. Outcome Measures
2.4.1. Tool Completion
Tool completion was defined as when participants could enter all known family history data into the system. Study research staff observed participants as they used one of the family history tools and documented tool completion and reasons for not completing the tool within RedCap using designated, non-mutually exclusive options: a) technical issue – slow network, b) could not use computer, c) chose not to continue (i.e., quit), and d) could not navigate tool.
2.4.2. Tool Concordance
Tool concordance was defined as the agreement between family histories derived by each tool compared to family histories derived from a genetic counselor (GC). Concordance for family members identified was calculated by dividing the number of tool-identified relatives by the number of relatives identified by the GC. Separate estimates were derived for first and second-degree relatives. Concordance for health conditions identified was calculated by dividing the number of disease cases reported in a tool by the number of disease cases identified by the GC, and derived separately for first and second-degree relatives. The conditions assessed included: heart problem (including heart attack), stroke, diabetes, cancer (breast, colon, ovarian, prostate, lung, skin), hypertension, high cholesterol, alcoholism, addiction, mental health issues (anxiety/panic disorders, depression, bipolar, schizophrenia), and Alzheimer’s disease.
2.5. Data Collection
Data collection for the trial occurred between November 2016 and October 2019. The study CONSORT is presented in Figure 2. A total of 2694 patients were invited to participate in the study via mailed letters, 2508 were deemed eligible for telephone contact by the study team, and 1943 were reached via phone and screened for study eligibility. Out of those reached, 643 (33%) declined to participate, 828 (43%) did not meet study inclusion criteria, and 472 (19%) were scheduled for the baseline appointment. A total of 279 participants kept their appointment, completed informed consent, and were enrolled and block randomized by health literacy [22] in the study. Based on receipt of allocated study intervention (i.e., family history tool) and follow-up with a GC, a final sample of 273 participants (138 controls, 135 intervention) were included for analyses of primary study aims.
Figure 2.
Participant flow CONSORT
2.6. Statistical Analysis
Descriptive statistics, including means and standard deviations for continuous variables and counts and percentages for categorical variables, were used to describe baseline characteristics of the randomization arms. Student’s t-test (continuous variables) and chi-square or Fisher’s exact tests (categorical variables) were used to compare the randomization arms on baseline characteristics. There were a few respondents who reported very large numbers of second-degree relatives. To address this skewness in the outcome analyses, if the summed numbers of second-degree relatives (overall and by type) for a given pedigree were above the 95th percentile, we recoded the value to the 95th percentile. In addition, there were some circumstances where the GC pedigree did not specify an exact number of a given relative type who had a health condition (i.e., notes of a condition without indicating number of affected people). In these situations, we took a conservative approach and coded the number of relatives with that health condition as 1. Poisson regression models were used to examine the effect of study arm on identification of relatives and health conditions, with the number of relatives identified (or number of relatives with a given health condition identified) through the tool as the dependent variable, randomization arm as the independent variable, and the log of the number of relatives identified by the GC as the offset. Incidence rate ratios (IRRs) and 95% confidence intervals describing relative differences in concordance were obtained from these models to compare the study arms. We further assessed the possibility of effect modification by health literacy level by performing stratified analyses and formally testing for interactions in the above noted statistical models. We used an alpha level of 0.05 for all statistical testing. We performed all analyses using SAS version 9.4.
2.7. VICKY Audit Checks for Quality and Implementation
Qualitative audit checks of the VICKY tool were performed during data cleaning and analyses. In particular, we focused on addressing findings from our prior feasibility study that showed a high degree of variability in concordance between the VICKY tool and the GC across health conditions [14]. Approximately 20% (n=28) of the GC interview recordings were audited (by TNC and MT-T) to gain insight into the potential sources of error between the tool and GC pedigrees, selecting cases where larger discrepancies were observed in either direction (i.e., difference of 2 or more cases in first degree relatives identified by either the GC or VICKY).
3. RESULTS
3.1. Participant Demographics
Participant demographics are presented in Table 1. Approximately two-thirds of participants were female, and almost three-quarters were African American or Hispanic/Latino, and had a possible or high likelihood of limited literacy (as measured using Newest Vital Signs, [23]). Over half of the participants indicated they use computers regularly. There was a borderline difference between study arms by gender. However, because concordance did not vary overall as a function of gender, models did not adjust for this covariate.
Table 1.
Participant Demographics (N=273)
| VICKY (n=135), n (%) |
MFHP (n=138), n (%) |
Total (N=273), n(%) |
p value | |
|---|---|---|---|---|
| Gender | 0.0536 | |||
| Female | 83 (61.5%) | 100 (72.5%) | 183 (67.0%) | |
| Male | 52 (38.5%) | 38 (27.5%) | 90 (33.0%) | |
| Age (years) | 0.5615 | |||
| 18–24 | 0 (0.0%) | 3 (2.2%) | 3 (1.1%) | |
| 25–34 | 14 (10.4%) | 12 (8.7%) | 26 (9.5%) | |
| 35–44 | 16 (11.9%) | 14 (10.1%) | 30 (11.0%) | |
| 45–54 | 38 (28.1%) | 37 (26.8%) | 75 (27.5%) | |
| 55–64 | 58 (43.0%) | 65 (47.1%) | 123 (45.1%) | |
| ≥65 | 10 (7.1%) | 7 (5.1%) | 16 (5.9%) | |
| Ethnicity/Race | 0.2985 | |||
| Hispanic/Latino | 21 (15.7%) | 27 (19.6%) | 48 (17.6%) | |
| Caucasian | 19 (14.2%) | 20 (14.5%) | 39 (14.3%) | |
| African American | 83 (61.9%) | 71 (51.4%) | 154 (56.6%) | |
| Asian | 3 (2.2%) | 3 (2.2%) | 6 (2.2%) | |
| Other or multiple | 8 (6.0%) | 17 (12.3%) | 21 (7.7%) | |
| Education | 0.4783 | |||
| Less than 9th grade | 7 (5.2%) | 9 (6.5%) | 16 (5.9%) | |
| HS, no diploma | 14 (10.4%) | 20 (14.5%) | 34 (12.5%) | |
| HS graduate/GED | 44 (32.6%) | 35 (25.4%) | 79 (28.9%) | |
| Some college | 29 (21.5%) | 33 (23.9%) | 62 (22.7%) | |
| College degree | 24 (17.8%) | 30 (21.7%) | 54 (19.8%) | |
| Graduate or doctoral degree | 17 (12.6%) | 11 (8.0%) | 28 (10.3%) | |
| Household income | 0.6959 | |||
| ≤$25,000 | 68 (51.9%) | 73 (53.7%) | 141 (52.8%) | |
| $25,001–35,000 | 12 (9.2%) | 12 (8.8%) | 24 (9.0%) | |
| $35,001–50,000 | 19 (14.5%) | 15 (11.0%) | 34 (12.7%) | |
| $50,001–75,000 | 5 (3.8%) | 11 (8.1%) | 16 (6.0%) | |
| >$75,000 | 3 (2.3%) | 2 (1.5%) | 5 (1.9%) | |
| Prefer not to answer | 24 (18.3%) | 23 (16.9%) | 47 (17.6%) | |
| Health literacy1 | 0.3803 | |||
| High likelihood of limited literacy | 42 (31.1%) | 54 (39.1%) | 96 (35.2%) | |
| Possibility of limited literacy | 56 (41.5%) | 50 (36.2%) | 106 (38.8%) | |
| Almost always adequate literacy | 37 (27.4%) | 34 (24.6%) | 71 (26.0%) | |
| Computer experience | 0.8347 | |||
| Never used one | 13 (9.6%) | 11 (8.0%) | 24 (8.8%) | |
| Tried one a few times | 35 (25.9%) | 40 (29.0%) | 75 (27.5%) | |
| Use one regularly | 72 (53.3%) | 75 (54.3%) | 147 (53.8%) | |
| Expert | 15 (11.1%) | 12 (8.7%) | 27 (9.9%) |
As assessed by the Newest Vital Signs health literacy measure [23]
3.2. Tool Completion
Tool completion rates differed significantly (p< .0001) between the two study arms and was 97% (131/135) for participants in the VICKY arm compared to 51% (71/138) in the control arm for participants who used My Family Health Portrait (MFHP). For VICKY, reasons for non-completion among the 4 non-completers included: 1) could not use computer (n= 2/4, 50%), 2) could not navigate tool (n= 1/4, 25%) and 3) chose not to continue (n= 1/4, 25%). For MFHP, the reasons non-completion among the 67 non-completers were as follows: 1) could not navigate tool (n=54/67, 81%), 2) could not use computer (n= 23/67, 34%), 3) chose not to continue (n= 15/67, 22%), and 4) technical issue – slow network (n= 2/67, 3%).
3.3. Primary and Secondary Outcomes
3.3.1. Primary Outcome: Tool Concordance
Tool concordance outcomes are presented in Table 2. Concordance was significantly greater for participants randomized to VICKY compared to MFHP (control arm) for ascertaining first and second degree relatives (IRRs: 1.53 and 2.50 respectively, ps<.0001). When examining disease conditions for first-degree relatives, the concordance was also significantly higher for the majority of the conditions examined for those in the VICKY arm compared to the MFHP control arm, particularly for heart conditions, diabetes, hypertension, alcoholism, addiction, and some mental health conditions.
Table 2.
Concordance between family history tool versus genetic counselor for identifying relatives and health conditions (includes both English and Spanish, N=273)
| VICKY (n=135) % (tool/GC) |
MFHP (n=138) % (tool/GC) |
Incidence Rate Ratio (95% CI) | p-value | |
|---|---|---|---|---|
| Relatives identified | ||||
| First Degree Relative (any)1 | 100.5% (919/914) | 65.8% (651/990) | 1.53 (1.27, 1.85) | <.0001 |
| Mother or Father | 98.9% (266/269) | 60.1% (166/276) | 1.64 (1.41, 1.92) | <.0001 |
| Brother or Sister | 102.7% (424/413) | 73.0% (313/429) | 1.41 (0.87, 2.28) | 0.1652 |
| Child | 98.7% (229/232) | 60.4% (172/285) | 1.64 (1.34, 2.00) | <.0001 |
| Second Degree Relative (any)2 | 82.7% (2237/2705) | 33.1% (1022/3089) | 2.50 (2.07, 3.01) | <.0001 |
| Aunt or Uncle | 76.0% (717/944) | 55.6% (640/1152) | 1.37 (1.10, 1.70) | 0.0047 |
| Niece or Nephew | 81.1% (576/710) | 0.0% (0/812) | n/a | -- |
| Grandparent | 96.3% (520/540) | 60.4% (332/550) | 1.60 (1.36, 1.88) | <.0001 |
| Grandchild | 74.0% (151/204) | 0.0% (0/245) | n/a | -- |
| Half-Sibling | 86.0% (209/243) | 0.9% (2/222) | 95.40 (12.02, 756.95) | <.0001 |
| Conditions identified among first degree relatives (FDR) | ||||
| Heart Problem (including heart attack) | 108.5% (89/82) | 49.5% (46/93) | 2.19 (1.30, 3.69) | 0.0031 |
| Stroke | 83.3% (25/30) | 44.7% (17/38) | 1.86 (0.84, 4.15) | 0.1279 |
| Diabetes (any) | 87.9% (109/124) | 37.1% (43/116) | 2.37 (1.51, 3.73) | 0.0002 |
| Type 1 | 733.3% (22/3) | 500.0% (5/1) | 1.47 (0.42, 5.19) | 0.5516 |
| Type 2 | 49.2% (59/120) | 20.2% (22/109) | 2.44 (1.49, 3.99) | 0.0004 |
| Type not specified | 2900% (29/1) | 266.7% (16/6) | 10.86 (3.69, 31.97) | <.0001 |
| Cancer (any) | 100.0% (50/50) | 80.9% (38/47) | 1.24 (0.75, 2.05) | 0.4108 |
| Breast Cancer | 84.6% (11/13) | 50.0% (7/14) | 1.69 (0.94, 3.06) | 0.0812 |
| Colon Cancer | 128.6% (9/7) | 83.3% (5/6) | 1.54 (0.76, 3.12) | 0.2275 |
| Ovarian Cancer | 400.0% (4/1) | 133.3% (4/3) | 3.00 (1.16, 7.75) | 0.0236 |
| Prostate Cancer | 86.7% (13/15) | 77.8% (7/9) | 1.11 (0.69, 1.80) | 0.6602 |
| Lung Cancer | 121.4% (17/14) | 76.5% (13/17) | 1.59 (0.98, 2.58) | 0.0626 |
| Skin Cancer | (2/0) | (2/0) | n/a | -- |
| Hypertension | 127.6% (217/170) | 43.1% (75/174) | 2.96 (1.84, 4.76) | <.0001 |
| High cholesterol | 228.6% (80/35) | 85.7% (48/56) | 2.67 (1.25, 5.68) | 0.0109 |
| Alcoholism | 122.4% (60/49) | 9.1% (5/55) | 13.47 (4.69, 38.66) | <.0001 |
| Addiction (all)3 | 151.9% (41/27) | 22.7% (5/22) | 6.68 (2.05, 21.77) | 0.0016 |
| Mental Health Issue (any) | 56.5% (39/69) | 37.7% (29/77) | 1.50 (0.90, 2.51) | 0.1204 |
| Anxiety/Panic Disorders | 214.3% (15/7) | 43.8% (7/16) | 4.90 (1.61, 14.94) | 0.0052 |
| Depression | 83.3% (30/36) | 35.5% (11/31) | 2.35 (1.22, 4.50) | 0.0102 |
| Bipolar | 116.7% (14/12) | 11.1% (2/18) | 10.49 (2.39, 46.14) | 0.0019 |
| Schizophrenia | 83.3% (5/6) | 28.6% (2/7) | 2.92 (1.18, 7.20) | 0.0204 |
| Other | 66.7% (14/21) | 35.0% (7/20) | 1.90 (0.85, 4.29) | 0.1200 |
| Alzheimer’s | 160.0% (16/10) | 111.1% (10/9) | 1.44 (0.66, 3.15) | 0.3613 |
Includes parents, siblings, children.
Includes aunts/uncles, nieces/nephews, grandparents, grandchildren, half-siblings.
Addiction represents all drug addiction, including prescription. It does not include alcoholism or nicotine addiction.
3.3.2. Secondary Outcome: Impact of Health Literacy
Health literacy (using the Newest Vital Signs - NVS) showed adequate participant distribution across health literacy categories (see Table 1). Table 3 presents concordance rates stratified by health literacy. Interactions between family history tool and health literacy were significant for first-degree (p<.05) and second-degree (p<.001) relatives, as well as for hypertension (p<.005), suggesting that differences in agreement between study arms was greater among those with limited literacy. Among the subgroup with the highest health literacy levels, concordance was significantly higher for VICKY compared to MFHP for second-degree relatives (p<.0001), alcoholism (p<.005) and addiction (p<.05).
Table 3.
Concordance between family history tool vs. genetic counselor stratified by health literacy
| High likelihood of limited literacy (n=96) | ||||
|---|---|---|---|---|
| Relatives and Health Conditions Identified | VICKY n=42 % (tool/GC) |
MFHP n=54 % (tool/GC) |
Incidence Rate Ratio (95% CI) | p-value |
| First Degree Relative (any)1 | 94.4% (323/342) | 45.6% (201/441) | 2.07 (1.41, 3.05) | 0.0002 |
| Second Degree Relative (any)2 | 75.0% (716/955) | 20.9% (266/1275) | 3.59 (2.39, 5.39) | <.0001 |
| Heart Problem (including heart attack) | 103.2% (32/31) | 22.7% (5/22) | 4.54 (0.92, 22.47) | 0.0636 |
| Stroke | 90.0% (9/10) | 16.7% (2/12) | 5.40 (0.91, 32.03) | 0.0634 |
| Diabetes (any) | 58.2% (32/55) | 29.8% (17/57) | 1.95 (1.11, 3.44) | 0.0210 |
| Cancer (any) | 100.0% (18/18) | 31.3% (5/16) | 3.20 (1.13, 9.03) | 0.0281 |
| Hypertension3 | 126.2% (82/65) | 17.2% (11/64) | 7.34 (2.57, 20.94) | 0.0002 |
| High cholesterol | 188.9% (34/18) | 81.8% (9/11) | 2.31 (0.44, 12.21) | 0.3249 |
| Alcoholism | 107.1% (15/14) | 0.0% (0/19) | n/a | -- |
| Addiction (all) | 170.0% (17/10) | 0.0% (0/8) | n/a | -- |
| Mental Health Issue (any) | 32.3% (10/31) | 3.3% (1/30) | 9.67 (2.18, 42.90) | 0.0028 |
| Alzheimer’s | 233.3% (7/3) | 66.7% (2/3) | 3.50 (0.64, 19.22) | 0.1494 |
| Possibly limited literacy (n=106) | ||||
| Relatives and Health Conditions Identified | VICKY n=56 % (tool/GC) |
MFHP n-50 % (tool/GC) |
Incidence Rate Ratio (95% CI) | p-value |
| First Degree Relative (any) | 104.1% (407/391) | 73.3% (255/348) | 1.42 (1.06, 1.90) | 0.0185 |
| Second Degree Relative (any) | 81.8% (924/1129) | 32.7% (378/1155) | 2.50 (1.86, 3.36) | <.0001 |
| Heart Problem (including heart attack) | 119.4% (37/31) | 41.5% (17/41) | 2.88 (1.40, 5.90) | 0.0039 |
| Stroke | 73.3% (11/15) | 41.7% (5/12) | 1.76 (0.40, 7.69) | 0.4527 |
| Diabetes (any) | 104.3% (49/47) | 27.5% (11/40) | 3.79 (1.56, 9.19) | 0.0032 |
| Cancer (any) | 95.0% (19/20) | 69.2% (9/13) | 1.37 (0.61, 3.07) | 0.4403 |
| Hypertension | 134.3% (94/70) | 25.4% (17/67) | 5.29 (1.99, 14.10) | 0.0009 |
| High Cholesterol | 377.8% (34/9) | 43.8% (7/16) | 8.63 (1.64, 45.56) | 0.0111 |
| Alcoholism | 153.3% (23/15) | 0.0% (0/19) | n/a | -- |
| Addiction (all) | 175.0% (14/8) | 100.0% (2/2) | 1.75 (0.13, 24.49) | 0.6777 |
| Mental Health Issue (any) | 76.2% (16/21) | 66.7% (16/24) | 1.14 (0.46, 2.85) | 0.7746 |
| Alzheimer’s | 75.0% (3/4) | 33.3% (1/3) | 2.25 (0.73, 6.91) | 0.1567 |
| Almost always adequate literacy (n=71) | ||||
| Relatives and Health Conditions Identified | VICKY n=37 % (tool/GC) |
MFHP n=34 % (tool/GC) |
Incidence Rate Ratio (95% CI) | p-value |
| First Degree Relative (any) | 104.4% (189/181) | 97.0% (195/201) | 1.08 (0.87, 1.34) | 0.5060 |
| Second Degree Relative (any) | 96.1% (597/621) | 57.4% (378/659) | 1.68 (1.40, 2.01) | <.0001 |
| Heart Problem (including heart attack) | 100.0% (20/20) | 80.0% (24/30) | 1.25 (0.54, 2.89) | 0.6014 |
| Stroke | 100.0% (5/5) | 71.4% (10/14) | 1.40 (0.33, 5.88) | 0.6458 |
| Diabetes (any) | 127.3% (28/22) | 78.9% (15/19) | 1.61 (0.64, 4.05) | 0.3099 |
| Cancer (any) | 108.3% (13/12) | 133.3% (24/18) | 0.81 (0.28, 2.34) | 0.7000 |
| Hypertension | 117.1% (41/35) | 109.3% (47/43) | 1.07 (0.52, 2.19) | 0.8496 |
| High Cholesterol | 150.0% (12/8) | 110.3% (32/29) | 1.36 (0.36, 5.08) | 0.6480 |
| Alcoholism | 110.0% (22/20) | 29.4% (5/17) | 3.74 (1.54, 9.10) | 0.0036 |
| Addiction (all) | 111.1% (10/9) | 25.0% (3/12) | 4.44 (1.15, 17.23) | 0.0310 |
| Mental Health Issue (any) | 76.5% (13/17) | 52.2% (12/23) | 1.47 (0.75, 2.86) | 0.2629 |
| Alzheimer’s | 200.0% (6/3) | 233.3% (7/3) | 0.86 (0.20, 3.63) | 0.8341 |
Significant tool × health literacy interactions:
First degree relatives: p < .05
Second degree relatives: p < .001
Hypertension/high blood pressure: p < .005
3.4. Audit Checks for Quality and Implementation
Supplemental Table 1 highlights the pattern of discrepancies between VICKY and the GC for each of the conditions assessed in the tool. Certain health conditions were more frequently documented by VICKY, such as addiction and alcoholism. In contrast, Type 2 diabetes and mental health conditions were more frequently documented by the genetic counselor.
Among the audited recordings, several patterns emerged across health conditions that appeared to reflect distinct sources of error: 1) participant nondisclosure (participant did not disclose condition when asked by the GC); 2) genetic counselor omission (GC did not ask about the condition); and 3) VICKY logical errors (e.g., skip patterns in virtual counselor script). For conditions such as addiction and alcoholism, the source of error was primarily participant nondisclosure to the GC. For other conditions such as heart disease, hypertension and high cholesterol, the source of error was often GC omission. For Type 2 diabetes and mental health, the sources of error were participant nondisclosure.
4. DISCUSSION AND CONCLUSION
4.1. Discussion
This randomized controlled trial is the first to obtain validation data on the efficacy of both VICKY and MFHP for collecting family history data among an underserved patient population, in two languages. Study findings suggest that using a virtual counselor to collect family health histories overcomes many of the literacy-related barriers to using traditional digital tools in this context and highlights an approach that may be important to consider when collecting family history information. Recent work by other investigators have also similarly reported the advantages of using an interactive conversational interface compared to a more traditional graphical user interface for capturing family health history [25, 26].
In the present study, only half of participants randomized to My Family Health Portrait were able to use the tool to enter their family history. Inability to use the tool had large implications for the concordance rates noted. Not surprising, within the MFHP arm, tool completers had higher health literacy than non-completers, with approximately 88% (30/34) of those with almost always adequate literacy completing MFHP versus 44% (22/50) of those with possible limited literacy and 35% (19/64) of those with high likelihood of limited literacy completing MFHP. In contrast, completion rates for VICKY were 100% (37/37), 96% (54/56) and 95% (40/42) for participants with almost always adequate, possible limited literacy and high likelihood of limited literacy, respectively. These findings further demonstrate how limited health literacy serves as an important barrier to effectively using some tools.
Additional analyses among the subset of tool completers showed fewer differences in concordance rates across the two study arms, yet significant differences remained for data collected of first- and second-degree relatives, as well as alcoholism, addiction, and schizophrenia (see Supplemental Table 2). Notably, the large difference between the tools in the collection of half-siblings suggests that half-siblings are grossly underreported in the MFHP tool, and likely erroneously captured as full siblings, thus explaining the higher rates of first-degree relatives observed within MFHP among tool completers. Moreover, MHFP does not include alcoholism or addiction on its list of conditions in the drop-down menus and required participant to use the “other-add new” option to add them, which may explain the differences noted between the tools for these conditions in particular.
We also conducted additional analyses focused on the subset of pedigrees where the number of family members (i.e., first-degree relatives) matched between tool and genetic counselor. Results of these analyses revealed that the majority of the differences between VICKY and MFHP regarding conditions identified were no longer significant when FDRs were matched. Notably, only 36 out of 138 participants in the MFHP arm had matching FDRs between tool and genetic counselor pedigrees. As such, concordance outcomes between the tools are likely mostly attributed to participant ability to enter family members. Future efforts to design effective family history tools may need to prioritize user-friendly methods to ascertain family members to ensure greater data quality.
Usability issues with My Family Health Portrait have previously been reported by our team as well as others [18, 20, 27–29]. Numerous navigation challenges were observed when participants used this tool, including entering the tool from the home screen, entering date of birth, and adding/saving conditions either for self or relatives. In particular, the error message that was generated when date of birth was not entered as requested (mm/dd/yyyy) was “invalid date of birth,” which caused further confusion among participants who would indicate aloud that it was their birthday. When participants chose not to continue the program (n=15), their top reason for not continuing was frustration with the tool (93%, 14/15), followed by being upset with it (27%, 4/15).
Consistent with prior research, the present study found that the accuracy of family history tools is greater for first-degree relatives (FDRs) compared to second-degree relatives (SDRs) [13, 14]. Concordance rates across various diseases for SDRs are available in Supplemental Table 3. Additional analyses within each arm indicated no differences in concordance by gender, with the exception of type 2 diabetes, wherein females were more concordant with the genetic counselor compared to males (data not shown).
Our team set out to develop a virtual counselor that could accurately collect family health histories among vulnerable patient populations. Findings from both in our prior work [14] and current study suggest a high degree of variability in concordance across health conditions. Our audit revealed several patterns across health conditions that reflect distinct sources of error. Certain health conditions were more frequently documented by VICKY, such as addiction and alcoholism, a finding we previously observed for alcoholism in a pilot study [14]. For these conditions, the source of error was primarily participant nondisclosure to the GC. In contrast, GC omission was often the source of error for other conditions such as heart disease, hypertension and high cholesterol. Together, these findings highlight the potential value of using virtual counselors to collect family history information that may be potentially stigmatizing and embarrassing to disclose by participants or inconsistently collected by a genetic counselor (i.e., not asked). Moreover, these results also highlight situations wherein digital tools may outperform genetic counselors that often serve as the professional standard for collecting family histories.
In contrast, Type 2 diabetes and mental health conditions were more frequently documented by the genetic counselor. Based on interview audits, several programming issues were identified that likely contributed to the errors in documentation. First, participants in VICKY reported Type 1 diabetes at a very high frequency level. It became clear from the audits that often, these were instances of Type 2 diabetes, noted accordingly in the GC pedigree. The genetic counselor often asked follow-up questions about the age of disease onset to get clarification on the type of diabetes in the family member. Although VICKY had a glossary of terms that participants could use to get clarification of diabetes type, it became apparent that many respondents may simply have selected Type 1 diabetes without understanding the differences between the two types. VICKY was not programmed to provide proactive clarification on the distinction between the two types of diabetes when asking about diabetes in the family, which likely resulted in lower concordance. Second, the genetic counselor frequently provided examples of mental health conditions (e.g., anxiety and depression) when asking about mental health, whereas VICKY simply asked about “mental illness,” which may not have been as well understood by participants. Future efforts to collect health histories for both diabetes and mental illness may need to consider clarification prompts at the outset to increase accuracy in their documentation within family history tools.
In spite of some challenges with how VICKY ascertained certain diseases, it its notable that VICKY 2.0 improved upon the concordance rates observed with VICKY 1.0, detailed in our pilot study [14]. In particular, concordance rates were higher in the current study than the VICKY pilot for family members ascertained (e.g., 100% for first-degree relatives and 83% for second-degree relatives for VICKY 2.0 vs 60% and 33% respectively for VICKY 1.0) and for some disease conditions among first-degree relatives including heart disease (109% vs 65%), stroke (83% vs 40%), breast cancer (85% vs 50%) and colon cancer (129% vs 67%).
There were several limitations to the present study that should be noted. First, although we set out to test the efficacy of VICKY versus MFPH in two languages, the Spanish language version of MFHP became unavailable shortly after we began enrollment of Spanish speakers to the trial, resulting in a small number of Spanish speakers recruited overall to the study. We have reported on the evaluation of Spanish VICKY elsewhere [21]. Even with the limited number of Spanish speakers in our trial, we noted some differences in the population (e.g., larger family size among Latinos) that may have important implications in the tool concordance [12]. Our future work will examine potential differences in outcomes by language administration, which remains an understudied area in this field. Second, as reported, health literacy moderated significant differences in concordance between VICKY and MFHP. The interaction term between tool and health literacy was significant for first and second-degree relatives, as well as for hypertension and Type 2 diabetes (data not shown). However, we were unable to analyze tool × health literacy interactions for less prevalent health conditions (approximately half of conditions examined) due to insufficient power. Third, although the GCs involved in the study were provided with a script to follow when ascertaining a three-generation family health history, which included all the conditions of interest to be captured, it became apparent that GCs would not always ask about all the conditions during the interviews. Our team did not intervene to enforce adherence by GCs to the script, however, this source of error was not considered a differential effect by study arm as GCs were blinded to study arm. Finally, our study focused on comparing the concordance related to counts of relatives and disease conditions, which are the inputs often needed for calculating disease risk using standardized risk algorithms. We thus cannot conclude from this work whether any enumerated family members from a digital tool are the same as those members enumerated by a GC. As such, it is unknown as to the extent to which the tool pedigrees would be useful in ascertaining inheritance patterns within a family.
4.2. Conclusion
Using a virtual counselor to collect family health histories overcomes many of the literacy-related barriers to using traditional digital tools in this context and highlights an approach that may be important to consider when collecting family history information from underserved patient populations. Results from this study can also inform the development and refinement of digital tools that may serve as a critical supplement in the collection of family health histories in clinical practice.
4.3. Practice Implications
The usability of any digital family health history tool will have important implications for the quality of the data collected and its downstream clinical utility. Using a virtual counselor to collect family health history may address literacy-related barriers that often preclude the effective use of digital tools for documenting family health histories among those who are most vulnerable in the population.
Supplementary Material
ACKNOWLEDGMENTS
We would like to acknowledge the following individuals for their contributions to the development of the VICKY program and the execution of the study: Ha Trinh, Juan Fernandez, Emily Armstrong, Adriana Black, Candice Bangham, Maria Cerda Diez, Noreen Jesani, Julie Whyte, Sarah Cochrane, Sarah Ollis, Courtney Manning, and Kathleen Berensten Swenson.
FUNDING
This work was supported by the National Human Genome Research Institute (NIH grant number R01HG007746 and 3R01HG007746–03S1). Access to VICKY is currently available to researchers via a compiled system.
Footnotes
DECLARATION OF COMPETING INTEREST
The authors declare no competing interests to report.
REFERENCES
- [1].Bennett RL, Family Health History: The First Genetic Test in Precision Medicine, Med Clin North Am 103(6) (2019) 957–966. [DOI] [PubMed] [Google Scholar]
- [2].Ginsburg GS, Wu RR, Orlando LA, Family health history: underused for actionable risk assessment, Lancet 394(10198) (2019) 596–603. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [3].Pyeritz RE, The family history: the first genetic test, and still useful after all those years?, Genet Med 14(1) (2012) 3–9. [DOI] [PubMed] [Google Scholar]
- [4].Welch BM, O’Connell N, Schiffman JD, 10 years later: assessing the impact of public health efforts on the collection of family health history, Am J Med Genet A 167A(9) (2015) 2026–33. [DOI] [PubMed] [Google Scholar]
- [5].Nathan PA, Johnson O, Clamp S, Wyatt JC, Time to rethink the capture and use of family history in primary care, Br J Gen Pract 66(653) (2016) 627–628. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [6].Kanetzke EE, Lynch J, Prows CA, Siegel RM, Myers MF, Perceived utility of parent-generated family health history as a health promotion tool in pediatric practice, Clin Pediatr (Phila) 50(8) (2011) 720–8. [DOI] [PubMed] [Google Scholar]
- [7].Giovanni MA, Murray MF, The application of computer-based tools in obtaining the genetic family history, Curr Protoc Hum Genet Chapter 9 (2010) Unit 9 21. [DOI] [PubMed] [Google Scholar]
- [8].Rich EC, Burke W, Heaton CJ, Haga S, Pinsky LE, Short P, Acheson L, Reconsidering the family history in primary care, J Gen Intern Med 19 (2004) 273–280. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [9].Wood ME, Stockdale A, Flynn BS, Interviews with primary care physicians regarding taking and interpreting the cancer family history, Fam Pract 25(5) (2008) 334–40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [10].Welch BM, Dere W, Schiffman JD, Family health history: the case for better tools, JAMA 313(17) (2015) 1711–2. [DOI] [PubMed] [Google Scholar]
- [11].Welch BM, Wiley K, Pflieger L, Achiangia R, Baker K, Hughes-Halbert C, Morrison H, Schiffman J, Doerr M, Review and Comparison of Electronic Patient-Facing Family Health History Tools, J Genet Couns 27(2) (2018) 381–391. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [12].Cohn WF, Ropka ME, Pelletier SL, Barrett JR, Kinzie MB, Harrison MB, Liu Z, Miesfeldt S, Tucker AL, Worrall BB, Gibson J, Mullins IM, Elward KS, Franko J, Guterbock TM, Knaus WA, Health Heritage(c) a web-based tool for the collection and assessment of family health history: initial user experience and analytic validity, Public Health Genomics 13(7–8) (2010) 477–91. [DOI] [PubMed] [Google Scholar]
- [13].Facio FM, Feero WG, Linn A, Oden N, Manickam K, Biesecker LG, Validation of My Family Health Portrait for six common heritable conditions, Genet Med 12(6) (2010) 370–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [14].Wang C, Bickmore T, Bowen DJ, Norkunas T, Campion M, Cabral H, Winter M, Paasche-Orlow M, Acceptability and feasibility of a virtual counselor (VICKY) to collect family health histories, Genet Med 17(10) (2015) 822–30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [15].Wu RR, Myers RA, Sperber N, Voils CI, Neuner J, McCarty CA, Haller IV, Harry M, Fulda KG, Cross D, Dimmock D, Rakhra-Burris T, Buchanan AH, Ginsburg GS, Orlando LA, Implementation, adoption, and utility of family health history risk assessment in diverse care settings: evaluating implementation processes and impact with an implementation framework, Genet Med 21(2) (2019) 331–338. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [16].Wang C, Gallo RE, Fleisher L, Miller SM, Literacy assessment of family health history tools for public health prevention, Public Health Genomics 14 (2011) 222–237. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [17].Wu RR, Myers RA, Buchanan AH, Dimmock D, Fulda KG, Haller IV, Haga SB, Harry ML, McCarty C, Neuner J, Rakhra-Burris T, Sperber N, Voils CI, Ginsburg GS, Orlando LA, Effect of Sociodemographic Factors on Uptake of a Patient-Facing Information Technology Family Health History Risk Assessment Platform, Appl Clin Inform 10(2) (2019) 180–188. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [18].Ponathil A, Ozkan NF, Bertrand J, Welch B, An investigation of the usability issues of a family health history compiling application, Human Factors and Ergonomics Society, Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting, 2018, pp. 1699–1703. [Google Scholar]
- [19].Murray MF, Giovanni MA, Klinger E, George E, Marinacci L, Getty G, Brawarsky P, Rocha B, Orav EJ, Bates DW, Haas JS, Comparing Electronic Health Record Portals to Obtain Patient-Entered Family Health History in Primary Care, J Gen Intern Med (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [20].Arar N, Seo J, Abboud HE, Parchman M, Noel P, Veterans’ experience in using the online Surgeon General’s family health history tool, Per Med 8(5) (2011) 523–532. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [21].Cerda Diez M, D EC, Trevino-Talbot M, Bangham C, Winter MR, Cabral H, Norkunas Cunningham T, D MT, D JB, M KP-O, Bickmore T, Wang C, Designing and Evaluating a Digital Family Health History Tool for Spanish Speakers, Int J Environ Res Public Health 16(24) (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [22].Chew LD, Bradley KA, Boyko EJ, Brief questions to identify patients with inadequate health literacy, Fam Med 36(8) (2004) 588–94. [PubMed] [Google Scholar]
- [23].Weiss BD, Mays MZ, Martz W, Castro KM, DeWalt DA, Pignone MP, Mockbee J, Hale FA, Quick assessment of literacy in primary care: The newest vital sign, Ann Fam Med 3(6) (2005) 514–522. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [24].Guttmacher AE, Collins FS, Carmona RH, The family history - More important than ever, New England Journal of Medicine 351(22) (2004) 2333–2336. [DOI] [PubMed] [Google Scholar]
- [25].Ponathil A, Ozkan F, Welch B, Bertrand J, Chalil Madathil K, Family health history collected by virtual conversational agents: An empirical study to investigate the efficacy of this approach, J Genet Couns (2020). [DOI] [PubMed] [Google Scholar]
- [26].Ponathil A, Ozkan NF, Bertrand J, Welch B, Madathil KC, Comparing Usability of User Interfaces to Collect Family Health History, Stud Health Technol Inform 264 (2019) 2005–2006. [DOI] [PubMed] [Google Scholar]
- [27].Wang C, Gallo RE, Breen-Lopez CJ, Smith GA, Cousler Emig J, Beene-Harris R, Fleisher L, Miller SM, Identifying Barriers to Genetic Literacy among African Americans and Latinos in the Community, American Public Health Association annual meeting, Philadelphia, PA, 2009. [Google Scholar]
- [28].Wang C, Gallo RE, Smith GA, Cousler Emig J, Beene-Harris R, Fleisher L, Miller SM, Family health history tools for public health prevention: Preferences among African Americans and Latinos in the community, American Public Health Association annual meeting, Denver, CO, 2010. [Google Scholar]
- [29].Wang C, Bickmore T, Paasche-Orlow M, Norkunas Cunningham T, Cabral H, Winter M, Campion M, Bowen D, Usability of family health history tools among underserved patients. Poster (549.F) American Society of Human Genetics, https://www.ashg.org/wp-content/uploads/2019/10/2017-Poster-Abstracts.pdf, Orlando, FL, 2017, p. 89. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.