Structured abstract
Background:
Patient Reported Outcomes Measurement Information System (PROMIS) measures can monitor patients with chronic illnesses outside of healthcare settings. Unfortunately, few applications that collect electronic PROMIS measures are designed using inclusive design principles, which ensure wide accessibility and usability, thus limiting use by older adults with chronic illnesses.
Objectives:
To establish the feasibility of using an inclusively designed mobile application tailored to older adults to report PROMIS measures by examining (1) PROMIS scores collected with the application, (2) patient-reported usability of the application, and (3) differences in usability by age.
Design:
Cross-sectional feasibility study.
Setting:
Inpatient and outpatient cardiac units at an urban academic medical center.
Participants:
168 English- and Spanish-speaking older adults with heart failure.
Intervention:
Participants used an inclusively designed mobile application to self-report PROMIS measures.
Measurements:
Eleven PROMIS short-form questionnaires (Anxiety, Ability to Participate in Social Roles and Activities, Applied Cognition-Abilities, Depression, Emotional Distress-Anger, Fatigue, Global Mental Health, Global Physical Health; Pain Interference, Physical Function, Sleep Disturbance), and a validated health technology usability survey measuring Perceived Ease-of-Use and Usefulness of the application.
Results:
Twenty-eight percent of participants were between 60–69 years of age, 20% were 70 years or older, 63% were male, 32% were White, and 96% had two or more medical conditions. There was no missing PROMIS data and mean PROMIS scores showed the greatest burden of pain, fatigue, and physical function in the sample. Usability scores were high and not associated with age (Perceived Ease-of-Use p=0.28; Perceived Usefulness p=0.44).
Conclusion:
It is feasible for older adults to use an inclusively designed application to report complete PROMIS data with high perceived usability. To ensure data completeness and the opportunity to study multiple domains of physical, mental, and social health, future work should use inclusive design principles for applications collecting PROMIS measures among older adults.
Keywords: Patient Reported Outcome Measures, Heart Failure, Symptoms, Health Status, Mobile Health
Introduction
Hundreds of healthcare organizations seek a reliable way to periodically collect symptoms and functioning on patients with multiple comorbid conditions in order to enhance clinical care.1 In recognition of this, the National Institutes of Health (NIH) invested in the Patient Reported Outcomes Measurement Information System (PROMIS), a system that standardizes the capture and format of patient-reported outcome (PRO) data about symptoms and functioning in the domains of physical, mental, and social health.2,3These measures are freely available and have been translated and linguistically validated in multiple languages. PROMIS measures may be especially useful among older adults with multiple chronic conditions with overlapping symptoms because the instruments are designed to assess global domains of health and are disease-agnostic.
PROMIS measures can be collected electronically. This versatility increases clinical utility because they can be collected more frequently and easily, aggregated across time points or across populations, and integrated into the electronic health record (EHR).1 Although adoption of mobile technologies such as smartphones (42%) and tablets (32%) has risen over the last 5 years among older adults,4 PROMIS uses generic survey software which is not optimized for older adults who may have unique sensory or motor capabilities or low technology-related self-efficacy which create different design requirements.5 It is critical that applications for collecting PROMIS are created using inclusive design principles, which ensure applications “are accessible to, and usable by, people with the widest range of abilities within the widest range of situations.”6 The objective of this study was to design a mobile application using inclusive design principles to collect PROMIS measures among older adults with heart failure (HF).
HF patients frequently experience a range of symptoms and decreased functioning that reduce health-related quality of life. An estimated 75% of HF patients have multiple comorbid conditions,7 which increase symptom burden, blunt patients’ ability to recognize and respond to symptoms, and complicate attribution to a specific condition.8 Together these factors make self-reporting of symptoms and functioning challenging for HF patients, and yet critically important because early symptom changes can signal decompensation.9 Earlier detection of decompensation can reduce risk of hospitalizations and death.10 As such, symptom monitoring is a core aspect of HF self-care.
We conducted a cross-sectional study to determine the feasibility of older adults with HF using an inclusively designed mobile application to report symptoms and functioning using PROMIS measures. To determine feasibility, we examined PROMIS scores collected through the mobile application, usability of the mobile application, and whether usability differed by patient age. Usability is a critical endpoint because it indicates how easily an intended user can adopt and sustain use of a given application.11 High usability regardless of patient age confirms that inclusive design principles have been met.
Methods
Participants and Recruitment
Patients with HF were recruited from a cardiac inpatient unit and an ambulatory cardiac clinic at a large, urban academic medical center from October 2016 through January 2017. Eligible patients were (1) diagnosed with HF as confirmed by clinical exam, echocardiographic evidence, or cardiologist, (2) able to speak English or Spanish, and (3) willing and able to provide informed consent. Participants were excluded if they had a diagnosis of dementia or active psychosis, or were on isolation precautions for infectious disease. Participants provided written informed consent in English or Spanish and were offered $35 as compensation for the time spent in study participation. We compared sample characteristics to other patients on the units during active study recruitment using the EHR to confirm generalizability. The Columbia University Medical Center Institutional Review Board approved the protocol prior to recruitment.
Developing a tailored mobile application using inclusive design
We undertook several steps to inclusively design the application; these steps have been reported in greater detail elsewhere.12–15 Briefly, we identified design features that researchers have reported improve usability for older adults, including simple navigation, large touch-target regions, large readable fonts, consistent interaction patterns, verbose error messages, and encouragement messages.16,17 We then systematically searched and evaluated existing commercially-available HF applications to identify key areas for improvement in terms of PRO reporting.14 Finally, we interviewed patients and healthcare providers to identify additional design requirements not described in the literature, and which symptoms they considered most important to monitor.13
The resulting application, called mi.Symptoms, included several features intended to enhance usability for a broader set of patients, such as progress bars, large font, colorful icons, high contrast, graphical aids for answer choices, and a Spanish version for Spanish speakers. mi.Symptoms included 11 PROMIS short-form questionnaires (47 questions total) which together measure all aspects of the PROMIS adult self-reported health framework.18 These surveys measured physical (Fatigue v1.0, Pain Interference v1.0, Sleep Disturbance v1.0, Physical Function v2.0), mental (Emotional Distress-Anger v1.1, Anxiety v1.0, Depression v1.0, Applied Cognition-Abilities v2.0), and social (Ability to Participate in Social Roles and Activities v2.0) health domains. Upon signing into mi.Symptoms, demographic information was pulled from the EHR so participants could verify their information such as age, and gender, racial and ethnic identity. After completing all PROMIS measures, participants received a summary of their most burdensome PROMIS domains and could select three to discuss with their healthcare provider.
Measures and Analysis
PROMIS scores were converted to standardized T-scores, which have a mean of 50 and a standard deviation of 10 in the general population in the U.S.18 A higher T-score indicates worse symptoms and functioning for six measures: anxiety, depression, emotional distress-anger, fatigue, pain interference, and sleep disturbance. A lower score is worse for five measures: physical functioning, applied cognition-abilities, ability to participant in social roles and activities, and global physical and mental health measures. A research assistant also collected sociodemographic characteristics, and clinical information was retrieved directly from the EHR. Clinical characteristics included presence of multimorbidity (2 or more comorbid conditions),19 polypharmacy (5–9 medications),20 and hyperpolypharmacy (10 or more medications).20
At the end of the session, participants assessed the usability of mi.Symptoms with an instrument validated for mobile health applications, the Health Information Technology Usability Evaluation Scale (Health-ITUES).11 Health-ITUES is based on theories of user acceptability and includes 12 items that are scored into two subscales: Perceived Usefulness (the belief that using an application would enhance ability to perform a specific task) and Perceived Ease-of-Use (the belief that using an application would require minimal effort). Scores range from one (lowest usability) to five (highest usability). Average scores of approximately four and above are typically interpreted as high usability.11,21,22
Standard descriptive statistics of frequency, central tendency, and dispersion were used to describe the sample and summarize patient characteristics, Health-ITUES usability, and PROMIS scores. Analysis of variance (ANOVA) was used to determine whether Health-ITUES usability scores differed by age group (<65 years, 65–74 years, ≥75 years).
Results
Participant characteristics
A total of 168 participants completed the study, of which 27% (n=46) were between 65–74 years and 10% (n=17) were 75 years of age or older (Table 1). More than half of the sample were men (n=106, 63%) and racial minorities (n=114, 68%). Nearly half of participants reported not having enough financial resources to make ends meet, high school or less education, and inadequate health literacy. Almost a third of the sample did not have access to a computer at home and a quarter did not have access to the Internet. Multimorbidity was prevalent in nearly all participants (n=161, 96%), and polypharmacy (n=46, 27%) and hyperpolypharmacy (n=108, 64%) were also common in the sample. Compared to all patients on the units from which we recruited, the participants were slightly younger, similar with respect to gender, and more balanced by race and ethnicity (Supplemental Table 1).
Table 1.
Study Participant Characteristics (n=168)
Characteristic | n (%) |
---|---|
Age (years) | |
< 65 | 105 (62.5) |
65–74 | 46 (27.4) |
≥ 75 | 17 (10.1) |
Gender (Men) | 106 (63.0) |
Race | |
White | 54 (31.7) |
Black | 61 (36.0) |
Asian and Other | 53 (32.3) |
Ethnicity (Not Hispanic or Latino) | 107 (63.6) |
Preferred Interview Language (English) | 135 (80.4) |
Marital Status (Single) | 94 (56.0) |
Financial status | |
Having more than enough to make ends meet | 24 (14.3) |
Having enough to make ends meet | 70 (42.0) |
Not having enough income to make ends meet | 74 (43.7) |
Highest level of education | |
Graduate school | 16 (9.5) |
College | 79 (47.0) |
High school | 73 (43.5) |
Technology Experience | |
Computer at home | 117 (69.6) |
Internet access | 124 (73.8) |
Use of Internet for health materials | 107 (64.1) |
New York Heart Association (NYHA) Class | |
II | 39 (23.2) |
III | 81 (48.2) |
IV | 48 (28.6) |
Prevalence of multimorbidity (2+ comorbid conditions) | 161 (95.8) |
Prevalence of polypharmacy | |
Polypharmacy (5–9 medications) | 46 (27.3) |
Hyperpolypharmacy (≥ 10 medications) | 108 (64.0) |
Distribution of PROMIS scores
We collected complete PROMIS data using the mi.Symptoms application with no missingness. Means and standard deviations of each PROMIS domain are shown in Figure 1. Based on difference between mean domain score and the population mean (T score=50), the most burdensome areas of self-reported health in the sample were physical function (36.6 ± 8.0; reverse-scored), pain interference (58.3 ± 11.3), and fatigue (57.5 ± 11.2).
Figure 1.
Mean and standard deviation (SD) of PROMIS scores in the sample (n = 168).
Usability of mi.Symptoms
Participants reported mi.Symptoms to be highly usable overall according to Health-ITUES scores (Table 2); the mean score was 4.02 (±0.84) for Perceived Usefulness and 3.91 (±0.89) for Perceived Ease-of-Use. There were no significant differences in Perceived Ease-of-Use (p=0.28) or Perceived Usefulness scores (p=0.44) by age group in ANOVA analyses.
Table 2.
mi.Symptoms Usability (Health Information Technology Usability Evaluation Scale11)
Mean Scores Overall | Mean Scores by Age | ||||
---|---|---|---|---|---|
<65 | 65–74 | ≥75 | p | ||
Total Perceived Usefulness | 4.02 (± 0.84) | 3.97 (±0.75) | 3.95 (±0.89) | 4.05 (±0.90) | 0.91 |
General Usefulness | 4.07 (±0.89) | ||||
General Satisfaction | 4.06 (±0.86) | ||||
Productiveness | 4.03 (±0.85) | ||||
Performance Speed | 3.90 (±0.92) | ||||
Information Needs | 3.96 (±0.91) | ||||
Total Perceived Ease-of-Use | 3.91 (±0.89) | 3.80 (±0.79) | 3.72 (±0.98) | 3.89 (±1.06) | 0.77 |
Competency | 3.98 (±0.88) | ||||
Learnability | 3.92 (±0.98) | ||||
Ease-of-Use | 3.94 (±0.96) | ||||
Memorability | 3.70 (±0.99) | ||||
User Control | 3.67 (±0.95) | ||||
Error Prevention | 3.60 (±0.98) | ||||
System Impact | 4.10 (±0.81) |
Discussion
In this study patients with HF, a significant proportion of whom were older adults, were able to use the inclusively designed mi.Symptoms application to report a range of PROMIS measures. The complete PROMIS data, high usability scores, and lack of differences in usability by age support the feasibility of mi.Symptoms. Overall, this study justifies the use of thoughtful design approaches, including inclusive design and gerontological design.
As one measure of feasibility, cross-sectional PROMIS data were collected in this study with high levels of data completeness. Mean PROMIS scores align with those reported in the literature with smaller samples of HF patients.23–26 As an area for future work, longitudinal data collection likely will be more informative for management of chronic illnesses, especially when monitored before and after specific interventions or therapies.
Additionally, as another measure of feasibility, participants reported high usability of mi.Symptoms. We conducted inclusive user-centered design with the explicit goals of improving Perceived Ease-of-Use and Perceived Usefulness of mi.Symptoms, the core aspects of usability measured in Health-ITUES. For instance, we identified optimal application features for older adults and tested these features with our target population to improve Perceived Ease-of-Use. Similarly, we conducted extensive interviews with healthcare providers and HF patients to identify the most salient symptoms to monitor and gaps in current monitoring strategies to improve Perceived Usefulness. Importantly, the burden of completing PROMIS measures has the potential to lower perceived usability. For example, an 86-item HF-specific PROMIS profile was recently published.27 Although comprehensive, this profile may be unfeasible for individuals with the exact physical and cognitive limitations that PROMIS measures. The benefit of mi.Symptoms was the use of multiple short-form questionnaires, which can be selectively removed to reduce response burden.
As a final measure of feasibility, we found no significant differences in usability by age. Our goal with mi.Symptoms was to create an inclusively designed application in recognition of the wide age ranges of patients with HF, thus motivating our recruitment of both younger and older adults. Conversely, gerontechnology is technology design that is intended to make technology more usable specifically for older adults; it integrates research on the biological, psychological, social, and medical aspects of aging towards the goals of supporting independent living, social participation, and ultimately improving quality of life and well-being.15 Those developing applications exclusively for older adults should apply gerontological design principles. As healthcare providers increasingly use mobile applications to manage and monitor patient health, inclusive and gerontological design will be critical to ensure that older adults are not excluded from the health benefits these platforms may offer.
Strengths of this study were the inclusive design procedures that we undertook to ensure mi.Symptoms was usable for older and younger HF patients alike, and the large and diverse sample size with which we confirmed feasibility of the application. Many mobile applications are developed with tech-savvy users and later retrofitted to meet target end users’ needs. We developed this application with the explicit goal of inclusivity for older adults, which we found supported high usability in our study sample. Additionally, the use of PROMIS ensures the application is disease-agnostic and can be easily deployed in a variety of contexts. This represents an important step in making PROMIS measures, which are a priority of the NIH and numerous national health systems, more easily collected among older adults with comorbid conditions who could benefit the most.
A major limitation of this study is that several characteristics of our sample limit the generalizability of these findings to all older adults with multimorbidity. These limitations include that participants were recruited from one urban, academic medical center, were willing to participate in research involving the use of technology, and had a small proportion over age 75 (10%). Additionally, there were likely other unmeasured differences between participants and other patients from the same inpatient and outpatient units who were not included in the study, such as experience with technology. The sample characteristics, particularly slightly younger age, are concordant with our prior research involving technology with older adults.28 Importantly, in many prior studies even the older adults who were willing to participate in research which involved technology struggled when the technology was not designed with input from older adults.15,17,29,30 Therefore, our findings represent a step towards greater inclusion of older adults in research and ultimately in clinical care. Another limitation is that the sample was skewed towards more severe HF based on NYHA class and other parameters, and thus our mean PROMIS scores may not be reflective of other populations with HF.
In conclusion, this study demonstrating feasibility of an inclusively designed mobile application for older adults represents a first step towards multiple inquiries into the electronic collection and use of PROMIS for the monitoring and management of older adults in clinical practice.
Supplementary Material
Impact Statement:
We certify that this work is novel. This study contributes to the literature an evaluation of the feasibility of using a new method (a novel, inclusively designed mobile application) for older adults to self-report measures of physical, mental, and social domains of health.
Acknowledgements
Drs. Reading Turchioe and Masterson Creber would like to acknowledge Drs. Suzanne Bakken and Kathleen Hickey for their mentorship on the K99NR016275 that funded this work.
Sponsor’s Role: Research reported in this publication was primarily supported by the National Institute of Nursing Research of the National Institutes of Health under Award Number R00NR016275 (mHealth for Heart Failure Symptom Monitoring; PI: Masterson Creber).
Footnotes
Conflict of Interest: Meghan Reading Turchioe is affiliated with Iris OB Health Inc., a startup company focused on postpartum depression and maternal health.
References
- 1.Lavallee DC, Chenok KE, Love RM, et al. Incorporating Patient-Reported Outcomes Into Health Care To Engage Patients And Enhance Care. Health affairs (Project Hope). 2016;35(4):575–582. [DOI] [PubMed] [Google Scholar]
- 2.Bevans M, Ross A, Cella D. Patient-Reported Outcomes Measurement Information System (PROMIS): efficient, standardized tools to measure self-reported health and quality of life. Nurs Outlook. 2014;62(5):339–345. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Lapin B, Thompson NR, Schuster A, Katzan IL. Clinical Utility of Patient-Reported Outcome Measurement Information System Domain Scales. Circ Cardiovasc Qual Outcomes. 2019;12(1):e004753. [DOI] [PubMed] [Google Scholar]
- 4.Anderson M, Perrin A. Tech Adoption Climbs Among Older Adults. 2017; https://www.pewresearch.org/internet/2017/05/17/tech-adoption-climbs-among-older-adults/.
- 5.Mitzner TL, Savla J, Boot WR, et al. Technology Adoption by Older Adults: Findings From the PRISM Trial. The Gerontologist. 2019;59(1):34–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Keates S BS 7000–6: 2005 Design management systems. Managing inclusive design. Guide. 2005. [Google Scholar]
- 7.Manemann SM, Chamberlain AM, Boyd CM, et al. Multimorbidity in Heart Failure: Effect on Outcomes. Journal of the American Geriatrics Society. 2016;64(7):1469–1474. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Lee CS, Gelow JM, Bidwell JT, et al. Blunted responses to heart failure symptoms in adults with mild cognitive dysfunction. J Cardiovasc Nurs. 2013;28(6):534–540. [DOI] [PubMed] [Google Scholar]
- 9.Lee CS, Bidwell JT, Paturzo M, et al. Patterns of self-care and clinical events in a cohort of adults with heart failure: 1 year follow-up. Heart & lung : the journal of critical care. 2018;47(1):40–46. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Lee CS, Moser DK, Lennie TA, Riegel B. Event-free survival in adults with heart failure who engage in self-care management. Heart & lung : the journal of critical care. 2011;40(1):12–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Schnall R, Cho H, Liu J. Health Information Technology Usability Evaluation Scale (Health-ITUES) for Usability Assessment of Mobile Health Technology: Validation Study. JMIR mHealth and uHealth. 2018;6(1):e4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Baik D, Reading M, Jia H, Grossman LV, Masterson Creber R. Measuring health status and symptom burden using a web-based mHealth application in patients with heart failure. European Journal of Cardiovascular Nursing. 2019;0(0):1474515119825704. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Grossman LV, Feiner SK, Mitchell EG, R MC. Leveraging Patient-Reported Outcomes Using Data Visualization. Applied clinical informatics. 2018;9(3):565–575. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Masterson Creber R, Maurer MS, Reading M, Hiraldo G, Hickey KT, Iribarren S. Review and analysis of existing mobile phone applications to support heart failure symptom monitoring and self-care using the Mobile Application Rating Scale. JMIR mHealth and uHealth. 2016;4(2). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Masterson Creber RM, Hickey KT, Maurer MS. Gerontechnologies for Older Patients with Heart Failure: What is the Role of Smartphones, Tablets, and Remote Monitoring Devices in Improving Symptom Monitoring and Self-Care Management? Current Cardiovascular Risk Reports. 2016;10(10):30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Tao D, Yuan J, Qu X. Presenting self-monitoring test results for consumers: the effects of graphical formats and age. Journal of the American Medical Informatics Association : JAMIA. 2018;25(8):1036–1046. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Isakovic M, Sedlar U, Volk M, Bester J. Usability Pitfalls of Diabetes mHealth Apps for the Elderly. Journal of diabetes research. 2016;2016:1604609. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.HealthMeasures. Interpret Scores: PROMIS®. 2018; http://www.healthmeasures.net/score-and-interpret/interpret-scores/promis.
- 19.Johnston MC, Crilly M, Black C, Prescott GJ, Mercer SW. Defining and measuring multimorbidity: a systematic review of systematic reviews. European Journal of Public Health. 2018;29(1):182–189. [DOI] [PubMed] [Google Scholar]
- 20.Slater N, White S, Venables R, Frisher M. Factors associated with polypharmacy in primary care: a cross-sectional analysis of data from The English Longitudinal Study of Ageing (ELSA). BMJ open. 2018;8(3):e020270. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Khasnabish S, Burns Z, Couch M, Mullin M, Newmark R, Dykes PC. Best practices for data visualization: creating and evaluating a report for an evidence-based fall prevention program. Journal of the American Medical Informatics Association : JAMIA. 2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Stonbraker S, Cho H, Hermosi G, Pichon A, Schnall R. Usability Testing of a mHealth App to Support Self-Management of HIV-Associated Non-AIDS Related Symptoms. Stud Health Technol Inform. 2018;250:106–110. [PMC free article] [PubMed] [Google Scholar]
- 23.Cella D, Lai JS, Jensen SE, et al. PROMIS Fatigue Item Bank had Clinical Validity across Diverse Chronic Conditions. J Clin Epidemiol. 2016;73:128–134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Hahn EA, Beaumont JL, Pilkonis PA, et al. The PROMIS satisfaction with social participation measures demonstrated responsiveness in diverse clinical populations. J Clin Epidemiol. 2016;73:135–141. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Schalet BD, Hays RD, Jensen SE, Beaumont JL, Fries JF, Cella D. Validity of PROMIS physical function measured in diverse clinical samples. J Clin Epidemiol. 2016;73:112–118. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Schalet BD, Pilkonis PA, Yu L, et al. Clinical validity of PROMIS Depression, Anxiety, and Anger across diverse clinical samples. J Clin Epidemiol. 2016;73:119–127. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Ahmad FS, Kallen MA, Schifferdecker KE, et al. Development and Initial Validation of the PROMIS(R)-Plus-HF Profile Measure. Circ Heart Fail. 2019;12(6):e005751. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Masterson Creber RM, Grossman LV, Ryan B, et al. Engaging hospitalized patients with personalized health information: a randomized trial of an inpatient portal. Journal of the American Medical Informatics Association : JAMIA. 2019;26(2):115–123. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Zikmund-Fisher BJ, Scherer AM, Witteman HO, et al. Graphics help patients distinguish between urgent and non-urgent deviations in laboratory test results. Journal of the American Medical Informatics Association : JAMIA. 2017;24(3):520–528. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Lyles CR, Sarkar U, Ralston JD, et al. Patient-provider communication and trust in relation to use of an online patient portal among diabetes patients: The Diabetes and Aging Study. Journal of the American Medical Informatics Association : JAMIA. 2013;20(6):1128–1131. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.