Summary
Objectives
Various computerized health risk appraisals (HRAs) are available, but few of them assess health-related quality of life (HRQoL) in a goal-directed framework. This study describes the user-centered development and usability testing of an innovative HRQoL module that extends a validated HRA tool in primary care settings.
Methods
Systematic user-centered design, usability testing, and qualitative methods were used to develop the HRQoL module in primary care practices. Twenty two patients and 5 clinicians participated in two rounds of interactive technology think-out-loud sessions (TOLs) and semi-structured interviews (SSIs) to iteratively develop a four-step, computerized process that collects information on patient goals for meaningful life activities and current level of disability and presents a personalized and prioritized list of preventive recommendations linked to online resources.
Results
Analysis of TOLs and SSIs generated 5 categories and 11 sub-categories related to facilitators and barriers to usability and human-technology interaction. The categories included: Understanding the Purpose, Usability, Perceived Value, Literacy, and Participant Motivation. Some categories were inter-connected. The technology was continually and iteratively improved between sessions until saturation of positive feedback was achieved in 4 categories (addressing motivation will require more research). Usability of all screen units of the module was improved substantially. Clinician feedback emphasized the importance of the module’s ability to translate the patient-centered HRQoL Report into actionable items for clinicians to facilitate shared decision-making. Complete integration of the HRQoL module into the existing HRA will require further development and testing.
Conclusions
Systematic application of user-centered design and human factors principles in technology development and testing may significantly improve the usability and clinical value of health information systems. This more sophisticated approach helped us translate complex clinical concepts, goal-setting steps, and decision-support processes into an accepted and value-added technology.
Keywords: Health risk appraisal, user-centered design, quality of life, goal-directed care, prioritization
1. Background
The Patient Protection and Affordable Care Act (ACA) was introduced in 2010 and it aimed at improving the quality and affordability of healthcare for Americans. It also established new benefits for annual wellness visits (AWVs) for Medicare beneficiaries. In response, the Centers for Medicare and Medicaid Services (CMS) instituted billing codes for a “Welcome to Medicare” and annual follow-up visits to create and maintain a personalized wellness plan [1]. The regulation stipulated that as part of the AWV, an annual health risk assessment (HRA) must be performed. CMS also prescribed an HRA minimum dataset that must cover demographics, biometrics, self-assessed health status, psychosocial health, behavioral health, and functioning that include memory, activities of daily living (ADLs) and instrumental activities of daily living (IADLs) [2].
Several systematic reviews indicate that HRAs may be effective in worksite and community-based wellness programs [3] [4]. However, few studies have determined the optimal characteristics and the effectiveness of HRAs in primary care settings [5]. Before the HRA mandate was introduced by CMS, professional organizations, including the American Academy of Family Physicians (AAFP) voiced their concern that evidence is insufficient for the institution of routine HRAs in primary care. Clinicians have been concerned about the lack of guidance pertaining to the types of HRAs that could be optimally implemented and additional resources and care process adjustments needed to use HRAs in routine care. AAFP attempted to bridge some of these gaps by generating recommendations [6], but significantly more research is needed to produce sufficient evidence.
A seminal HRA pilot study conducted by our study team systematically designed, implemented, and tested the effectiveness of an innovative, AWV-compliant HRA tool in four primary care practices [5]. One of the distinctive characteristics of our HRA is the ability to personalize and prioritize care recommendations in a goal-directed framework. Instead of addressing health problems in the context of individual diseases, the HRA considers patient goals (e.g., living longer and living better) by assessing a wide array of risk factors and suggesting preventive recommendations in the order of their estimated impact on reaching these goals. This approach is fundamentally different from the usual, problem-focused thinking that tends to break down in primary care when multimorbidities result in a long and unmanageable tasklist, including items that have variable impact on future outcomes. The study demonstrated that a comprehensive HRA, when implemented carefully, can improve patient and clinician acceptance of the HRA approach, the level of patient focus (patient-centeredness), preventive services uptake and estimated life expectancy in ambulatory settings.
Previous studies [5, 7] suggest that patient-centered (goal-directed) prioritization of care recommendations is a potential approach to solving the “long list” challenge. Initial versions of goal-directed HRAs prioritized tailored care recommendations based on one of the fundamental patient goals: to live a longer life. However, we are not aware of any study that attempted to develop a technology framework for improving health related quality of life (HRQoL) through a goal-directed care prioritization approach. A number of tools have been developed to measure and track HRQoL over time (e.g., SF-36, QWBS, EQ-5D, SIP, etc), but these tools are not designed to support a strategic prioritization of interventions that may improve current or future HRQoL.
2. Objective
The main goal of this study was the development and usability testing of a goal-directed HRQoL module that extends an existing, comprehensive HRA in primary care settings. The secondary goal was to demonstrate the suitability and effectiveness of user-centered design techniques in the development of primary care health information technology.
3. Methods
Development of the HRQoL Module
The HRQoL module was developed over two rounds of experiments conducted during the summer of 2013 and 2014, based on several years of background work with HRAs to operationalize a goal-directed approach to improving HRQoL. The module’s design was based on the clinically relevant and patient-centered conceptualization of HRQoL as the sustained ability to perform meaningful life activities (MLAs). After a brief instruction, the module displayed a matrix of 86 MLAs organized in eight categories following an adapted version of the International Classification of Functioning [8] to help individuals prioritize activities that can help them achieve their personal goals. A second form then collected data on current, self-reported functional limitations and disabilities to refine the module’s ranking process. Finally, the module presented a prioritized list of preventive strategies and online resources (HRQoL Report). The recommendations were based on specific MLA selections which were linked to an array of corresponding functions, which in turn were connected to evidence-based approaches to improve or preserve these functions. The development of the pilot HRQoL module and its ranking algorithm incorporated substantial guidance from one of our team members, who is a doctoral-level occupational therapist. An example of the HRQoL module’s decision support path is provided in ▶Table 1.
Table 1.
An example of the HRQoL module’s decision support path demonstrating the three main steps that include the selection of meaningful life activities (MLAs), followed by the identification of functions associated with MLAs, and finally, generation of prioritized recommendations for improving or preserving functions. Participants could select a maximum of 10 MLAs from a table. Each MLA was linked to an array of functions in nine categories (lower body strength, upper body strength, balance/flexibility, endurance, cognition, social interaction, positive affect, vision, and hearing) that were chosen from matrix of associated functions and prioritized based on the relative importance of each function. On a separate screen, participants could also note if they had current limitations in each function category which was used to further tailor the order of recommendations. In the last step, function categories were linked to evidence-based recommendations for improving or preserving these functions.
| MLA Selections (maximum 10 selections) |
Top Associated Functions (prioritized from a matrix) |
Top Recommendations (without current limitations) |
|
|---|---|---|---|
| Drink liquids; Toilet self; Take own medicine; Shopping (groceries, clothes); Games, cards; Play with kids/pets; Volunteer in church Use the phone; TV watching; Emotional intimacy; |
Cognition (4x); Vision (2x); Upper body strength (2x); Hearing; Positive affect; |
Thinking & memory:
|
Vision:
|
Think-Out-Loud Sessions
Two trained medical student research assistants (RAs) completed approximately 20-minute, computerized think-out-loud sessions (TOLs) [9] with a convenience sample of 10 non-clinical, clerical staff in the University of Oklahoma Health Sciences Center (OUHSC) Family Medicine Center in two rounds conducted in July of 2013 and July of 2014. The mean age of the participants was 43 years, ranging between 34 and 57 years. Sessions were digitally audio- and video -recorded with the high resolution camera of a regular iPhone device mounted below the computer monitor stand and positioned to face the screen.
The RAs took field notes during each session to document meta-communication and other observations. Each round of recordings and field notes were collected, uploaded into separate instances of NVivo 10 (QSR International Inc, Burlington, MA) as sharable projects and systematically analyzed by two reviewers independently. Together, three reviewers participated in the entire project.
Reviewers analyzed each recording and generated a list of preliminary observations, including verbal content and meta-communication, concurrent cursor movements, number and position of clicks and navigation patterns, and background information on the personal context of participants gained through the enrollment process. The reviewers then compared their notes on each session, formulated emerging categories iteratively from observations and argued them to consensus. Horizontal coding was followed by axial coding to generate a hierarchy of categories. The process followed a constant-comparative approach via cycles of media review and clarification of individual categories. TOLs were conducted and analyzed until saturation was achieved in each category over two rounds of interviews in 2013 and 2014. The face validity of the category structure was affirmed by immersing individual observations into the structure and examining how strongly they could be linked to individual categories. Barriers or problems with the technology were acknowledged, collaboratively suggested changes to the HRQoL module were implemented in an iterative fashion and the usability of the improved design was tested in subsequent sessions.
Semi-Structured Interviews
The RAs obtained regular patient visit schedules for two periods of three weeks in July of 2013 and 2014 from the Geriatric Clinic in the OUHSC Family Medicine Center. Prospective participants were selected from the list based on their age, reason for visit, cognitive competence and willingness to participate in order to obtain a purposeful sample. Our goal was to focus on older individuals with comorbidities and potentially function-limiting health challenges when quality of life may gradually become more important relative to longevity. The status of patients and their suitability for the study were verified through the electronic medical record and by their primary care clinicians who reviewed the patient list. Thirteen patients and five clinicians participated in approximately 20-minute semi-structured interviews (SSIs) [10]. The interview guide was developed and all interviewers were trained by the primary investigator (ZJN) who has significant experience in qualitative and user-centered design methodology. Four of the participants were male, three were African American and 10 were non-Hispanic Caucasian. Participants had a wide range of socioeconomic status, education, and computer literacy levels. Targeted questions in SSIs helped us differentiate participants into five tiers (0–4) based on their computer literacy using a primary care-specific scale that we have developed in a previous study [7]. Most participants were in Tier 2 (use computers frequently for general tasks including searching for information on the web and basic communication).
The mean age was 68 years ranging from 54 to 77 years. The RAs contacted prospective participant by phone using a standardized protocol, one to three days before their scheduled appointments. They then briefly explained the study and determined whether patients would be willing to participate. Cognitive competence was assessed during the phone call. Participants were asked to sign an informed consent form following their regular office visit and complete the HRQoL module using a regular desktop computer in a private office located in the Family Medicine Center. Interviews explored participant opinions and perceptions of the module’s usability, utility, acceptability, and value. The RAs also solicited feedback about ideas to improve the module. Interviews were digitally audio recorded using an iPhone device and were analyzed separately by two reviewers in each round of the study. Together, three reviewers participated in the evaluation of SSIs. Analyses followed the approach described above. After all data were collected and summarized, the reviewers created a prioritized list of themes from the entire dataset separately and then argued them to consensus collaboratively. Finally, a category structure diagram was created to visually represent the results, establish its face validity and resolve all remaining questions and uncertainties.
4. Results
TOL and SSI participants interacted with a functional, iteratively improved version of the HRQoL module. ▶Figure 1 shows the category structure diagram. Five categories emerged from TOLs and SSIs, including (in this rank order): Understanding the Purpose of the Module, Usability, Participant Motivation, Literacy, and Perceived Value. Each of these categories was associated with two or more sub-categories and there were multiple potential interactions between categories.
Fig. 1.
Diagram of categories that emerged from qualitative analyses of HRQoL module usability testing. Five main and 11 sub-categories were identified from think-out-loud sessions and semi-structured interviews. A team of three reviewers analyzed a rich set of audio, video, and field records and established consensus in an iterative, content analytic process.
Understanding the Purpose of the Module
Prioritization of MLAs requires the understanding of somewhat complex concepts and processes that may be unfamiliar to most individuals. In early versions of the HRQoL module, participants experienced some difficulty understanding the difference between a regular, broad patient survey (“…this was a survey about me”) and a cognitively more demanding life activity prioritization process where there may be competition between personal goals. This problem was exacerbated by an ineffective design that elicited all MLA selections in a single step. The one-step forced-choice approach typically resulted in user frustration with limiting the number of MLA choices that might not represent everything the individual wanted to convey (“…it doesn’t give you a broad picture of what their life is like”). Usability testing helped us implement three successful improvements. First, we determined the optimal number of MLA choices in consecutive experiments (5, 8, and finally 10) that created a balance between a sense of restriction and the ability to discover clear priorities. Then, we introduced an orientation screen at the beginning of the module that included a simple description of the exercise and its goals, in order to help users understand the purpose of the module (framing). The introduction also emphasized that both current and future quality of life should be considered. Finally, we separated activities of daily living (ADLs) from instrumental activities of daily living (IADLs) and created a two-step approach to select MLAs. The module allowed users to spend up to three choices on ADLs in the first step and up to seven choices on IADLs in a second step. It then allowed users to revise and finalize their selections by re-evaluating them and distribute all 10 choices among all MLAs, as they desired. This approach resulted in a substantially improved task flow and smoother navigation. After these adjustments, no design-related user struggle was observed during MLA prioritization. A moderate cognitive load was still detectable due to the nature of the exercise, but no significant barriers were reported by the final group of four participants. User satisfaction with the resulting experience validated the success of design improvements (“…[the module] is interesting and makes you think about what you really want in life, what you are looking for in your life, how you wanna stay healthy…and active”). ▶Figure 2 shows the top segment of the finalized MLA matrix.
Fig. 2.
Meaningful life activity (MLA) prioritization screen after usability testing. A matrix of 86 MLAs was created based on a patient-centered adaptation of the International Classification of Functioning (WHO). Activities were separated into ADLs (half-toned first column) and IADLs (all other columns) and were displayed in a two-step exercise.
Usability of the HRQoL Module
The majority of participants expressed a high level of satisfaction with the overall design of the module pertaining to the ease of navigation and interaction with its features (“…it was very easy to follow and select my choices, I had no problem”). However, some participants exhibited uncertainty about how to transition between sections of the module (“…at the end of the screen, I did not know what to do next”) and had difficulty keeping track of how many MLA choices they already selected. Sequential usability testing resulted in a smooth navigation pattern after clarifying the “submit” buttons’ dual function: “Save AND Go to Next”. To overcome MLA selection tracking issues, we designed a JavaScript-based unobtrusive popup-counter (three seconds display with fade-in and fade-out) that indicated remaining number of selections after each click. As mentioned above, the design of the module was further improved by the separation of ADLs and IADLs which helped participants distinguish essential daily activities from activities that give meaning to their lives, greatly improving the task flow. We also experimented with various ways of keeping user instructions visible for a continuing reference while, at the same time, we attempted to optimize the use of the screen real-estate. The HRQoL Report presented at the end of the module escaped the attention of several participants, who skipped to the library of online resources. To reinforce the importance and relevance of the Report, a semi-transparent progress indicator was introduced during the development of the Report to generate an expectation and the graphical design of the Report was also improved to make it stand out on the screen (3-D shading, enhanced color scheme and tabulation). The “Print Report” button was moved to the bottom of the screen to streamline the visual review of the Report before printing. Finally, the font-resizer widget that we created to maintain Section 508-compliance was magnified to help those with visual impairment recognize this feature.
Participant Literacy
Limited participant literacy (reading and computer literacy) has been a recurring challenge. Users who were less familiar with computers experienced some difficulty interacting with the technology. Scrolling, using the standard 2-button mouse and moving from screen to screen were among the barriers. To identify those with a higher level of challenge, a tailored support approach can be implemented based on a brief computer literacy tool we developed before. Using a touch screen or a one-button mouse could also be explored to eliminate issues with accidental right-clicking. Complicated or confusing wording represented another barrier for participants (“I can read my pill bottles easily with or without glasses”). It has been generally difficult to simplify some of the HRQoL concepts in order to achieve a 7th grade reading level. The alpha version of the module was at about 9th grade reading level which then improved in the course of project. The study did not specifically measure health literacy and numeracy, however, TOLs and SSIs did not indicate significant challenges in these areas, most likely because the language of the module, in the most part, followed existing instruments used in occupational and physical therapy.
Perceived Value of the Module
The value of the module was associated with its relevance to the individual’s needs, learning value and perceived usefulness (“What will take care of it [my problem] is having somebody you can go to, to help you with these problems you’re having. I liked the idea of getting referrals.”). The module’s value was clearly affected by the relevancy of the report card, the ability of the report to provide new information, and how useful recommendations were to the user (▶Figure 3).
Fig. 3.
A typical health-related quality of life (HRQoL) report. Participants reflected upon the prioritized list of recommendations and had the opportunity to explore various existing websites containing high-quality, relevant information providing further guidance on each recommendation (“More Info”). Resources included more detailed descriptions of exercises, their beneficial effects, and further suggestions about how to apply them or find venues to obtain them. Some resources incorporated online tools, games, and mobile apps.
Several improvements were introduced that aimed at making the presentation of recommendations more appealing and relevant to users. Some participants had issues with specific recommendations. For example, “Spiritual Support”, although multi-dimensional and can be recommended for various reasons, were felt to be irrelevant to some who had not selected spiritual activities as MLAs. The recommendation algorithm was revised to respond to this concern. Recommendations were paired with online resources for more drill-down information. Initially, “More Info” links opened a new window focusing on pertinent resources, but with all other resources also available for review. This full-length information page became distracting because users could see the entire library of materials and would start browsing them. The links were changed to point only to pertinent sections of resources, helping the users focus on their personalized recommendations. SSIs indicated that discussing the HRQoL Report with the participant’s clinician could also significantly improve the perceived value of the report. A future study will address the integration of the HRQoL Report into the existing HRA that currently produces a report for life extension as a goal.
Participant Motivation
Participants exhibited a varying level of motivation based on their willingness to accept and respond to recommendations (“…I’ve come to terms with the first two [recommendations] – if you get 1 and 2 solved, there would be no third. But they’re not going to get solved”). Individuals brought a “baseline” motivation and enthusiasm about participating and contributing. The majority of SSI participants contributed (voluntarily) a personal story about themselves or someone close to them, who experienced significant functional limitations or disability. Enthusiasm about the module seemed to relate to the individual’s personal experience with limitations in life activities. Those who strongly related the exercise to a personal experience also said that the module helped them think of ways to improve their health and it would likely help them or others. A short Patient Activation Measure (PAM-13) and a “Readiness for Change” scale may be included in the next version of the module to measure and leverage participants’ inner drive and capacity for following preventive recommendations. Personal experiences can also be exploited by clinicians or health coaches to link health recommendations to personal goals and make them more relevant to the individual.
Clinician Feedback
Five clinicians (four physicians and a physician assistant) participated in 20-minute SSIs after a brief, typical case-based demonstration of the HRQoL module. Clinician feedback indicated a general interest in the innovative approach to operationalizing HRQoL and the potential utility of the Report that could be used for shared decision making as part of a comprehensive wellness visit. A clinician noted: “If my patient has taken the time to do it, and they’ve identified it as important to them, it’s one of the tenets of patient-centered care that we would actually respond to.“ Another clinician added: “I think it has value in the clinical setting. It can generate a discussion with the patient about the things that are most important to them, so you can set goals that are appropriate to their personal choices.” However, clinicians insisted that for the module to be useful for them in a clinical setting, it has to deliver actionable guidance for addressing patient recommendations. For example, memory training or weight exercise might be recommended, but the clinician will need more information about specific steps for delivering care or methods for referring their patients. Some of this information is available via the resource links, but it is not listed on the Report. A future version of the Report will include a separate list of suggested actions for the clinician. In addition to observations that overlapped with those of other participants, medical professionals helped us improve and clarify the wording of instructions and module items, aligning the sequence of MLA selections with existing clinical approaches, and initiate the development of a plan for the implementation of the technology in primary care settings.
5. Discussion
A recent paper published by the National Coordinator for Health Information Technology (ONC) indicated that about 80% of office-based clinicians now use electronic medical or heath records (EMRs/EHRs) in the United States [11]. About half of these implementations include EHRs with “more advanced” functionality. On the other hand, neither the architectural design of EHRs, nor their limited ability to interact with other professionals and patients supports readily the type of healthcare we need to provide in the 21st Century. Throughout the nation and through various venues, practicing clinicians are very vocal about the shortcomings of current health information systems [12-14]. There are many reasons for the substantial gaps between needs and available solutions, but perhaps most importantly, clinicians feel that EHRs, in general, have been designed without their input and became burdensome documentation and billing systems.
This sentiment is felt nowhere more than in primary care which is fundamentally different from all sub-specialties, in that, it is defined not by its content (specific diseases), but by its processes (accessibility, continuity, comprehensiveness, patient-centeredness, coordination, partnership, integration and accountability) [15]. Current EHRs typically offer insufficient functionality (at best) to support care defined by the above attributes. Primary care health informatics professionals argue that in order to achieve the goals of 21st Century medicine, it would be critical to redesign ambulatory information systems with the active participation of end-users (clinicians, office staff, administrators, and patients). Other successful industries (including software developers) have already created and implemented methods to achieve user-driven and user-centered design, some of which are demonstrated in this paper. Our study shows how essential user involvement is from the earliest phase of technology development and how well-established approaches can be used to ensure that tools are optimally designed, accepted, and valued.
Few analysts would argue that the course of the current healthcare system is unsustainable financially and administratively. But the prevailing, problem-oriented clinical paradigm is also unsustainable, which receives limited attention. Primary care doctors are running out of their EHRs’ maximum number of entries on their problem lists (some geriatricians have lists with over 100 entries) and the same tendency can be observed in medication lists and laboratory findings. In a recent paper, we described an alternative, HRA-driven care approach that introduces the prioritization of evidence-based interventions based on their impact on achieving personal health goals [5]. This innovative concept needed an extension that may allow us to incorporate the domain of HRQoL into the HRA process, in addition to improving survival. This pilot study is the first step in that direction. Other patient goals are also being conceptualized and incorporated into the HRA care process, including: reaching developmental mile-stones (children and young adults), planning and supporting a “good death” (end-of-life care), and developing longitudinal, nurturing relationships between healthcare professionals and their patients (partnership for health).
Analysis of the HRQoL module’s usability yielded observational categories that show considerable congruence with the widely known Technology Acceptance Model (TAM) [16]. Since TAM was not invoked specifically as a conceptual framework in our analytical approach, agreement between our findings and TAM constructs strengthen the face validity of the HRQoL module’s design.
6. Limitations
The participant sample was limited to a large academic Family Medicine Center due to the limited availability of funding.. However, the Center provides care for a diverse patient population, 60% of which is Medicaid-insured and over 25% represent ethnic minorities. Patients’ socioeconomic status, health status, and computer literacy also vary considerably. Center staff shows somewhat less diversity, but over 15% of them are African American, Hispanic/Latino, or Native American. Although the HRQoL module will be integrated into the existing HRA tool that addresses length of life, it had to be tested separately in this study for technical and feasibility reasons. This meant that participant profiles had to be “primed” with background information (e.g., diagnoses and other known risk factors) in order to enable the module’s engine to personalize recommendations. This workaround will not be necessary after integration of the module into the HRA that will feed data forward to the module. The pilot study was based on a convenience sample of participants. Although they had varied computer experience, the sample size did not allow addressing usability for a stratum of participants with low computer literacy. A larger study should examine the suitability of the technology in a low computer literacy population. The International Classification of Functioning has been used widely as a broad framework for addressing functional limitations that may relate to quality of life. However, it is not specific to cultural, ethnic, or patient groups. Such specificity goes beyond the scope of our study, but it could be considered in future research. The HRQoL module is only one element of a more comprehensive, goal-directed care model that our team has been developing. Other parts of the model include facilitation of shared decision-making and goal-setting (e.g. via motivational interviewing), longitudinal tracking of goal-attainment (e.g. via health coaching), and periodic re-evaluation (e.g. via annual wellness visits). These extend beyond the scope of the HRQoL module pilot study.
7. Conclusions
Our study demonstrates that user-centered design and usability testing can be optimally employed to develop and refine value-added eHealth technologies in ambulatory settings. The results also suggest that computer-assisted assessment of health-related quality of life can facilitate the tailoring and prioritization of preventive recommendations that may help patients achieve their personal goals. More research is needed to strategically leverage participant motivation through the HRQoL module’s design and to integrate the module into the existing HRA process.
Acknowledgments
The authors appreciate the support of the University of Oklahoma Health Sciences Center Department of Family and Preventive Medicine that provided stipends to two medical students (MS-2) in 2013 and 2014 to participate in the study as research assistants.
Footnotes
Clinical Relevance
The Patient Protection and Affordable Care Act of 2010 mandates Medicare payment for annual wellness visits (AWVs) that include a required health risk appraisal (HRA) component. However, few studies examined the optimal composition and implementation of HRAs in primary care settings, especially those that can effectively assess health-related quality of life. This study provides a detailed description of the development and usability testing of an innovative, computerized HRQoL module through user-centered design that can be implemented as part of an HRA in primary care AWVs.
Human Subject Research Approval
The study has been approved and monitored by the University of Oklahoma Health Sciences Center Institutional Review Board (IRB# 4436).
Conflict of Interest Statement
The authors report no conflicts of interest in the presented research.
References
- 1.Annual Wellness Visit (AWV), Including Personalized Prevention Plan Services (PPPS). MLN Matters – Centers for Medicare and Medicaid Services. Accessed at: http://www.cms.gov/Outreach-and-Education/Medicare-Learning-Network-MLN/MLNMattersArticles/downloads/MM7079.pdf Last Updated: March 27, 2013. Access date: 8/26/2014. 8/26/2014]. [Google Scholar]
- 2.Interim Guidance for Health Risk Assessments and their Modes of Provision for Medicare Beneficiaries. Centers for Medicare and Medicaid Services. Accessed at: https://www.cms.gov/Medicare/Coverage/CoverageGenInfo/downloads/healthriskassessmentsCDCfinal.pdf Publication date: March 23, 2011. Access date: 8/26/2014. [Google Scholar]
- 3.Rubenstein L, et al. Health Risk Appraisals and Medicare, 2003, US Department of Health and Human Services, Health Care Financing Administration: Baltimore. [Google Scholar]
- 4.Cyrus Baghelai MS, et al. Health Risk Appraisals in Primary Care: Current Knowledge and Potential Applications To Improve Preventive Services and Chronic Care, 2010, AHRQ Center for Primary Care, Prevention, and Clinical Partnerships: Rockville, MD. [Google Scholar]
- 5.Nagykaldi ZJ, et al. Novel computerized health risk appraisal may improve longitudinal health and wellness in primary care: a pilot study. Appl Clin Inform 2013; 4(1):75–87. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Academy Remedies CMS’ Lack of Guidance on Health Risk Assessment Requirement. American Academy of Family Physicians. Accessed at: http://www.aafp.org/news/practice-professional-issues/20120215healthriskassessment.html Publication date: February 15, 2012. Access date: 8/26/2014. [Google Scholar]
- 7.Nagykaldi Z, et al. Impact of a Wellness Portal on the delivery of patient-centered preventive care. J Am Board Fam Med 2012; 25(2):158–167. [DOI] [PubMed] [Google Scholar]
- 8.International Classification of Functioning, Disability and Health (ICF). World Health Organization. Accessed at: http://www.who.int/classifications/icf/en/ Last Updated: January 10, 2014. Access Date: 8/26/2014. [Google Scholar]
- 9.Lewis CH.Using the „Thinking Aloud“ Method In Cognitive Interface Design (Technical report), in IBM. RC-92651982. [Google Scholar]
- 10.Creswell JW.Qualitative Inquiry and Research Design – Choosing Among Five Approaches. Third ed, 2012: SAGE Publications Inc. [Google Scholar]
- 11.Furukawa MF, et al. Despite Substantial Progress In EHR Adoption, Health Information Exchange And Patient Engagement Remain Low In Office Settings. Health Aff (Millwood), 2014. [DOI] [PubMed] [Google Scholar]
- 12.O’Malley AS, et al. Are electronic medical records helpful for care coordination? Experiences of physician practices. J Gen Intern Med 2010; 25(3):177–185. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Lau F, et al. Impact of electronic medical record on physician practice in office settings: a systematic review. BMC Med Inform Decis Mak 2012; 12: 10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Krist AH, et al. Electronic health record functionality needed to better support primary care. J Am Med Inform Assoc 2014; 21(5):764–771. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.National Research Council. Defining Primary Care: An Interim Report: Washington, DC, 1994. [PubMed] [Google Scholar]
- 16.Davis FD.Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 1989; 13(3):319–340. [Google Scholar]