Evaluating a mobile application for improving clinical laboratory test ordering and diagnosis

Abstract

Objective

Mobile applications for improving diagnostic decision making often lack clinical evaluation. We evaluated if a mobile application improves generalist physicians’ appropriate laboratory test ordering and diagnosis decisions and assessed if physicians perceive it as useful for learning.

Methods

In an experimental, vignette study, physicians diagnosed 8 patient vignettes with normal prothrombin times (PT) and abnormal partial thromboplastin times (PTT). Physicians made test ordering and diagnosis decisions for 4 vignettes using each resource: a mobile app, PTT Advisor, developed by the Centers for Disease Control and Prevention (CDC)’s Clinical Laboratory Integration into Healthcare Collaborative (CLIHC); and usual clinical decision support. Then, physicians answered questions regarding their perceptions of the app’s usefulness for diagnostic decision making and learning using a modified Kirkpatrick Training Evaluation Framework.

Results

Data from 368 vignettes solved by 46 physicians at 7 US health care institutions show advantages for using PTT Advisor over usual clinical decision support on test ordering and diagnostic decision accuracy (82.6 vs 70.2% correct; P < .001), confidence in decisions (7.5 vs 6.3 out of 10; P < .001), and vignette completion time (3:02 vs 3:53 min.; P = .06). Physicians reported positive perceptions of the app’s potential for improved clinical decision making, and recommended it be used to address broader diagnostic challenges.

Conclusions

A mobile app, PTT Advisor, may contribute to better test ordering and diagnosis, serve as a learning tool for diagnostic evaluation of certain clinical disorders, and improve patient outcomes. Similar methods could be useful for evaluating apps aimed at improving testing and diagnosis for other conditions.

Introduction

Appropriately selecting and ordering laboratory tests is essential for accurate diagnosis.^1,2 However, test selection and ordering accounts for more testing-related errors than test interpretation does,^3,4 and primary care physicians are uncertain about what tests to order in nearly 1 in 7 patients.⁵ This uncertainty is of special concern when laboratory tests require rapid assessment and follow-up actions to prevent adverse patient outcomes. For example, for patients presenting with bleeding or thrombosis requiring expedited diagnostic work-up, consultations from hematologists may not be universally available to generalist physicians who often see such patients initially (there were fewer than 8000 practicing hematologists and hematologists/oncologists in 2011 in the United States⁶). Health information technology has the potential to improve diagnosis by facilitating timely and easy access to information, such as identifying which tests to order.^7,8 Therefore, reliable clinical decision support for generalists’ diagnostic investigations, including interpretation of laboratory test result abnormalities, should be widely available.

To facilitate accurate test ordering and diagnosis of bleeding and coagulation disorders, the Centers for Disease Control and Prevention (CDC)’s Clinical Laboratory Integration into Healthcare Collaborative (CLIHC) developed PTT Advisor,⁹ a mobile application (app) available in the iTunes store, that assists physicians with interactive step-by-step test ordering and diagnostic decision making related to certain types of coagulation and bleeding disorders. Specifically, the app guides physicians through test selection and diagnostic evaluation for patients with normal prothrombin times (PTs) and abnormal partial thromboplastin times (PTTs) (see app screenshots in Figure 1). All app options and recommendations are based on diagnostic testing algorithms previously developed by CLIHC’s team of experts in diagnosis of coagulation disorders.¹⁰

Figure 1.

Selected screenshots of the PTT Advisor app.

Despite rapid increase in both the availability of medical mobile applications and adoption of mobile devices by physicians for clinical decision support, there has been minimal evaluation of diagnostic decision-making applications.^11–14 Indeed, most physician-facing applications are not evaluated in terms of their usefulness as clinical decision support or educational tools.^11–13

Using previously piloted methods for evaluating effects of mobile applications, we aimed to (1) evaluate whether PTT Advisor improves diagnostic and test ordering decisions of general medicine physicians and (2) assess if they find the app to be useful for learning.

Methods

Development of Procedures for Evaluating the App

A multidisciplinary team composed of internal medicine physicians, laboratory testing experts, an evaluation expert, and diagnosis researchers created the materials used to evaluate the app, including 8 paper-based patient vignettes and a questionnaire, and then pilot tested and refined all materials prior to evaluating the PTT Advisor app.

Vignettes

The team created vignettes of challenging clinical cases, which would prompt physicians to make laboratory test ordering and diagnosis decisions, that could be tested while physicians sought help from one of 2 resources: PTT Advisor or the usual clinical decision support (including any internet, text, or personnel resources the physicians wanted). We based the vignettes on real, published cases (case summaries in Table 1),^15–18 but modified them to improve clarity. We incorporated questions regarding what tests to order throughout each vignette, the ultimate diagnosis for each, and physician’s self-reported confidence in decisions made after diagnosing the vignette (on a scale of 0–10, with 0 being the lowest level of confidence and 10 being the highest level of confidence; cases in Supplementary Appendix). Each of the 8 vignettes addressed 1 of the 8 diagnoses or endpoints of the algorithm.

Table 1.

Vignette Summaries

Summary	Diagnosis or Algorithm Endpoint
A 32-year-old female with acute onset of easy bruising and “rash”15	The bleeding is most likely unrelated to the prolonged PTT. Consider causes such as thrombocytopenia, platelet function defect, scurvy, Ehlers-Danlos syndrome, etc.
A 32-year-old man with a swollen, tender left knee after injury16	The prolonged PTT is most likely explained by intrinsic factors VIII, IX, or XI deficiency
A 42-year-old woman with “abnormal” coagulation result from preoperative evaluation16	The PTT is most likely explained by factor XII deficiency
A 62-year-old black man with an external anal hemorrhoid16	The prolonged PTT might be explained by rare factor deficiency—check activity levels of prekallikrein or high molecular weight kininogen
A 33-year-old man with prolonged PTT found during a routine outpatient evaluation16	PTT is most likely explained by intrinsic factor VIII, IX, or XI deficiency
A 74-year-old white man with sudden non-traumatic bleeding17	Determine the Bethesda titer of the factor VIII inhibitor to establish a semi-quantitative anti-factor VIII concentration
A 62-year-old man with right upper quadrant pain16	The prolonged PTT is most likely due to a lupus anticoagulant
A 30-year-old woman with a 24-hour history of headache and altered mental status18	Causes of false negative tests for LA include a weak antibody, high factor VIII, or platelet count >10 000/μL in a frozen plasma specimen because platelet phospholipids may neutralize LA. If there is strong clinical suspicion, repeat LA testing at a later date

Questionnaire

The questionnaire consisted of 12 Likert-type questions (all on a scale of 1-Strongly Agree to 5-Strongly Disagree) to assess physicians’ perceptions about the application’s usefulness (or potential usefulness) for clinical decision making and learning using a modified Kirkpatrick Training Evaluation Framework.¹⁹ The multidisciplinary team reviewed the items to assess content validity and suitability for inclusion. The resulting questions allowed us to assess 4 levels of outcomes:

• Level 1: learners’ reactions (to understand how physicians reacted to the app in terms of satisfaction and usability),

• Level 2: modification of attitudes and perceptions (to determine if users modified their perceptions about ordering tests or diagnosing patients with PTT abnormalities using the app),

• Level 3: change in behavior (to assess whether users think their performance on test-ordering and diagnosing PTT abnormalities will improve after using the app),

• Level 4a: change in organizational practice (to examine users’ thoughts related to whether their organizations/institutions would support or encourage use of PTT Advisor in clinical practice or as an educational tool), and

• Level 4b: benefits to patients (establish whether users think the use of the PTT Advisor app would positively impact real patients if used in real-world clinical practice).

Demographic questions and open-ended questions were also included in the questionnaire. Specifically, demographic questions inquired about participants’ age, sex, years of experience in their clinical specialty, experience with mobile applications, and what types of information sources they typically used for diagnosing patients. Open-ended questions inquired about what could be improved about the app and what additional sources participants used while seeking help from the usual clinical decision support.

Pilot testing

We pilot tested the materials for evaluating the app with 6 physicians: 3 general internal medicine physicians and 3 hematologists/oncologists. Each physician solved 4 random vignettes using each resource (the app vs the usual decision support). Based on results from the pilot test, we refined the vignette content and conducted power analyses to determine needed sample size for obtaining a power = 0.8 for the full study. Power analyses showed that we needed 24 participants for our main variable of interest, proportion of test ordering and diagnosis decisions made correctly. Additionally, 2 clinicians on the team rated the 8 vignettes for difficulty on a scale of 1–10 (with 1 being the least difficult). Mean difficulty ratings were used to counterbalance the vignettes for the full study.

Evaluating the App

Sample/participants

We recruited internal medicine physicians, including hospitalists, via emails at 7 different health care institutions to diagnose 8 paper-based patient vignettes and answer the questionnaire. The study protocol was approved by our local institutional review board (and IRBs of participating facilities when needed). No participation incentives were provided.

Study design

Each physician solved 4 vignettes while seeking help from 1 of 2 resources (the mobile app or usual clinical decision support) in a counterbalanced, “blocked” fashion. Specifically, the resources were “blocked” such that each physician solved 4 vignettes using one resource, then switched and solved 4 vignettes using the other resource. Half of the physicians started with the mobile app, and the other half with the usual clinical decision support. The vignettes each physician solved with each resource were counterbalanced, such that mean vignette difficulty for a physician was the same for each resource, but vignette order varied between physicians. This ensured that all vignettes were solved using both types of resources, but the overall effects of seeing particular cases in a particular order would be minimized.

Procedure

Participation took place in physicians’ offices or in available conference rooms where physicians had access to internet or text resources they would normally use to diagnose patients. After obtaining verbal consent, physicians were given instructions as well as the 8 vignettes to solve in a paper packet. Prior to using the mobile app, they were given instructions on how to use it. Right before using usual clinical decision support, they were told they could use any resource they would typically use to guide test ordering and diagnostic decision-making; such as the internet, text resources, or even other physicians. For all cases, participating physicians were asked to circle all answers in the packet, while a member of the research team recorded their times to assess the cases. Participation took between 30 and 60 minutes.

Measures

Participating physician characteristics. Demographic information collected in the questionnaire included age, sex, years of experience in their clinical specialty, experience with mobile applications, and mobile application types they typically used for diagnosing patients.

Impact on diagnostic decision making. We evaluated the impact on diagnostic decision making using 3 measures: percent of test ordering and diagnosis decisions made correctly on the vignettes (because many questions involved elements of both testing and diagnosis, making these concepts difficult to disentangle in measurement, these were combined), self-reported confidence on the vignettes on a scale of 0–10, and time to assess vignettes.

Perceptions about the application’s usefulness for clinical decision making and learning. The questionnaire evaluated 4 levels of physicians’ perceptions about the app’s usefulness for clinical decision making and learning using a modified Kirkpatrick Training Evaluation Framework (1—learners’ reactions; 2—modification of attitudes and perceptions; 3—change in behavior; 4a—change in organizational practice; and 4b—benefits to patients).

Questionnaire reliability. Reliability metrics (Cronbach’s α) were also calculated to assess the internal consistency of the individual levels of the questionnaire as well as the overall questionnaire.

Statistical Analysis

Test ordering/diagnostic accuracy, self-reported confidence in decisions made, and vignette assessment time were compared when using the mobile app versus using usual clinical decision support using repeated-measures analyses of variance. Questionnaire responses (both Likert-type responses and demographic information) were assessed using descriptive statistics.

Results

Participating Physician Characteristics

Forty-six general internal medicine physicians (including 26 hospitalists, 57%), from 7 different health care institutions participated and attempted to solve 368 patient vignettes. The sample of physicians represented a range of age and experience, and over half were male (61%). The majority (76%) of participants self-reported being highly experienced and comfortable with mobile applications and a majority (80%) reported consulting electronic sources of information when evaluating patients in everyday practice (see Table 2).

Table 2.

Characteristics of Physician Participants

Demographic	Mean (min–max)
Age (years)	38.0 (27–54)
Length of practice in current specialty (years)	8.4 (0.5–30)
	n (%)
Sex
Male	28 (61)
Female	18 (39)
Level of experience with mobile/smartphone/smart pad applications
Low (Somewhat comfortable using them)	2 (4)
Moderate (Comfortable using them)	9 (20)
High (Very comfortable using them)	35 (76)
Information sources they primarily consult when evaluating patients (could select multiple)
Smartphones/smartpads/mobile applications	25 (54)
Electronic reference(s): professional articles,  journals, newsletters	37 (80)
Paper reference(s): professional articles,  journals, newsletter	7 (15)
Other	5 (11)

Impact on Clinical Decision Making

Use of PTT Advisor demonstrated a significant advantage over the usual clinical decision support for the following outcome variables: mean accuracy in testing/diagnostic decisions (82.6% vs 70.2% correct; P < .001) and mean confidence in testing/diagnostic decisions made (7.5 vs 6.3 out of 10; P < .001). Additionally, the advantage of using the app over usual clinical decision support on mean vignette assessment time neared statistical significance and likely represents clinical significance, with a time savings of 21.9% (3:02 min vs 3:53 min; P = .06).

When using the usual clinical decision support, 35 of 46 people (76%) reported using UpToDate, 16 (35%) reported using Google and/or the internet, 3 (7%) reported using Pocket Medicine, 2 (4%) reported using an algorithm from Washington State, 2 (4%) reported using Medline/PubMed, and 1 (2%) reported using each of Wikipedia, Medscape, and a residency manual (sources were not mutually exclusive).

Perceptions About the Application’s Usefulness for Clinical Decision Making and Learning

In the questionnaire, physicians reported positive perceptions of the app’s potential for improved clinical decision making and learning according to the 4 levels of the modified hierarchy.

• 1. Learners’ reactions. The physicians found the app provided the right amount of information to evaluate the patients with PTT abnormalities (70% reported strongly agreeing or agreeing with this), and they found the app to be both easy to use (83% reported strongly agreeing or agreeing about the app’s ease of use) and satisfying (61% reported strongly agreeing or agreeing about being satisfied with the app).

• 2. Modification of attitudes and perceptions. The participating physicians thought they would be more confident in making testing and diagnostic decisions for their patients when using the app (72% of physicians strongly agreed or agreed to each).

• 3. Change in behavior. They also thought they would improve their testing and diagnostic decisions using the app (with 61% and 67% of physicians strongly agreeing or agreeing, respectively).

• 4a. Changes in organizational practice. The physicians did not anticipate many organizational barriers to utilizing the tool (only 5% and 13% of physicians reported strongly agreeing or agreeing that they anticipate barriers to using the tool in their organization or barriers for applying the knowledge gained from the tool in their organization, respectively).

• 4b. Benefits to patients. They also reported they would be able to improve care for their patients in real practice (67% strongly agreed or agreed with this idea; see Table 3 for a summary of all Likert-type question responses).

Table 3.

Frequencies (and Percentages) of Evaluation Responses by Modified Kirkpatrick’s Hierarchy Level (n = 46)

Modified Kirkpatrick’s hierarchy level	Evaluation question	1 or 2- Strongly agree OR Agree, n (%)	1- Strongly agree, n (%)	2- Agree, n (%)	3- Neither agree nor disagree, n (%)	4- Disagree, n (%)	5- Strongly disagree, n (%)
Level 1: Learners’ Reactions (To understand users’ satisfaction with PTT Advisor [including things such as satisfaction with content and usability].)	This educational tool, PTT Advisor, provides me with the right amount of information necessary to evaluate a patient with a normal PT and an abnormal PTT.	32	10	22	8	5	1
		(70)	(22)	(48)	(17)	(11)	(2)
	This tool was easy to use.	38	22	16	0	6	2
		(83)	(48)	(35)	(0)	(13)	(4)
	Overall, I am satisfied with this app.	28	12	16	8	6	3
		(61)	(26)	(35)	(17)	(13)	(7)
Level 2a: Modification of Attitudes/Perceptions (To determine if users modify their perceptions about assessing/test-ordering/diagnosing PTT when using the compared to when not using the PTT Advisor)	I am more confident determining which tests need to be ordered for patients with normal PTs and abnormal PTTs after using this app.	33	9	24	9	4	0
		(72)	(20)	(52)	(20)	(9)	(0)
	I am more confident diagnosing patients with normal PTs and abnormal PTTs after using this tool.	33	10	23	8	5	0
		(72)	(22)	(50)	(17)	(11)	(0)
Level 3: Change in Behavior (To evaluate whether users think their performance on assessing/test-ordering/diagnosing PTT will improve after using the PTT Advisor [looking at it as an educational tool].)	I will apply the knowledge gained from this tool to my practice.	32	9	23	12	2	0
		(70)	(20)	(50)	(26)	(4)	(0)
	As a result of using this tool, I will improve my test ordering practices for patients with normal PT and abnormal PTT.	28	9	19	14	4	0
		(61)	(20)	(41)	(30)	(9)	(0)
	As a result of using this tool, I will improve how I diagnose patients with normal PT and abnormal PTT.	31	8	23	11	4	0
		(67)	(17)	(50)	(24)	(9)	(0)
	Applying the knowledge I gained from this tool would improve clinical care in my organization.	30	11	19	13	3	0
		(65)	(24)	(41)	(28)	(7)	(0)
Level 4a: Change in Organizational Practice (To examine users’ thoughts related to whether their organizations/institutions would support or encourage use of PTT Advisor in clinical practice or as an educational tool.)	I anticipate barriers to using this tool for clinical practice from my organization.	16	4	12	4	18	8
		(35)	(9)	(26)	(9)	(39)	(17)
	I anticipate barriers to applying knowledge gained from this tool from my organization.	6	1	5	4	23	13
		(13)	(2)	(11)	(9)	(50)	(28)
Level 4b: Benefits to Patients (To establish whether users think the use of the PTT Advisor would positively impact real patients if used in real-world clinical practice.)	I believe using this tool would improve the safety of patients with normal PT and abnormal PTT in real clinical practice.	31	8	23	8	7	0
		(67)	(17)	(50)	(17)	(15)	(0)

In the open-ended responses, many of the participants commented on the app’s simple, easy-to-use interface. However, despite many positive reactions to the app, the participating physicians reported suggestions to make the app better in their free-text responses (all respondents offered free-text comments). Several participants noted a desire for the app to address broader diagnostic challenges. They also thought the step-wise approach used in the app might not be relevant to clinical practice where many physicians order multiple tests at once. Regarding the user interface of the app, they sometimes could not find the footnotes (which were located at the top of the screen) or did not understand them and desired more information. They wanted to see the entire algorithm all at once (rather than only pieces at a time), and they wished for the app to be available on non-iOS devices.

Questionnaire Reliability

Assessing the questionnaire’s reliability (both as individual scales and as one wholistic scale) revealed Cronbach’s α levels for individual levels of the hierarchy at 0.87, 0.91, 0.92, and 0.70 for levels 1, 2a, 3, and 4a, respectively (level 4b includes a single item). The questionnaire in its entirety obtained an alpha of 0.81, a reasonably high degree of internal consistency.

Discussion

Our results indicate that PTT Advisor may contribute to better test ordering and diagnosis and serve as a learning tool for diagnostic evaluation of certain coagulation and bleeding disorders. Specifically, generalist physicians achieved 12.4% higher accuracy in test ordering and diagnosis decisions when using PTT Advisor compared to other resources they chose. They also finished cases 51 seconds faster (a reduction of 21.9%) and this savings of almost a minute is potentially clinically meaningful. Physicians also thought the app could prove useful in their real clinical practice and could benefit patients.

This study has many strengths. It is the first examination of an app designed to assist physicians throughout the diagnostic process. Despite the support such apps could provide for busy clinicians, most are not evaluated. Many of the apps that have been evaluated either focus on physician- or patient-facing symptom checkers that provide a list of likely diagnoses^20,21 or on diagnosis of dermatological issues,^22,23 which represent a very different problem than general medical diagnosis. Use of these tools without a formal evaluation, such as this one, could have adverse impacts on patient care, if the tools provide unsound medical advice, or if they interfere with clinical work. Because these tools are now being developed in large numbers, it is likely that some ineffective or unsafe tools that enter clinical practice can lead clinicians to mistrust all of them, despite the potential benefits from effective ones. Another strength is the experimental design, which allowed us to compare decision making using an app vs other resources on the same cases, enabling attribution of improved test ordering and diagnosis to use of the app. Additionally, participating physicians came from 7 different health care institutions from 3 states, and represented a wide range of clinical experience.

Despite these strengths, this study has some limitations. Although we attempted to sample a variety of physicians in regards to experience and comfort with mobile apps, the participating sample was mostly at a high level of experience and comfort. This may have altered our results, such that they could easily use and benefit from the app more than physicians with low comfort levels. Since only 2 physicians reported low experience and comfort levels, we were not able to conduct a sub-analysis to determine if this were true. However, an increasing number of physicians are reporting using mobile applications,¹¹ so this sample may reflect the growing comfort with using such technology. An additional limitation is that we did not assess the effect of using the app on real clinical cases, which may have altered how physicians used other resources. For example, physicians using the usual clinical decision support were given flexibility to use any resource of their choice, including other physicians, but they did not do so. In real practice, physicians would likely consult colleagues or refer patients when unsure about cases. However, the experimental method to evaluate performance allowed us to compare physicians on the same cases using different resources (as noted above). Nevertheless, this app likely has its largest benefit for physicians who do not have access to specialists for quick consultations or referrals.

Cases unfolded piece by piece, requiring physicians to order one or a few tests at a time. Algorithm developers were seeking to discourage overtesting and overutilization of resources, which is more likely to occur when physicians order multiple tests at once. Tools such as the algorithm contained in the PTT Advisor app and the app itself may decrease both underutilization and overutilization of testing. However, this would warrant testing in real clinical settings when clinicians are trying to fit the use of the app into real clinical workflows.

Our results underscore several important policy implications to consider as health care apps are increasingly adopted. A multitude of mobile applications have been created to support physicians in their diagnostic decision-making, but few have been evaluated in terms of how they improve test ordering and diagnostic decision-making performance.^11–13 While our methods were able to test clinical decision making and potential impact on patient care, most apps do not undergo similar testing. Physicians thus currently have no way of knowing which apps are useful and which are not trustworthy. All of these factors might limit app usefulness in the real world.²⁴ The use of extensive physician expertise in the development of PTT Advisor may have contributed to its perceived usefulness in this study. It is unknown to what extent app developers seek continuous expert physician input. While guidance on decision support software is still being developed,²⁵ some researchers, physicians, and policy experts suggest that health care apps should be regulated,²⁶ others strongly argue for self-regulation,²⁷ and others recommend the creation of guidelines for developing these apps.²⁸ Regardless of which route is taken, physicians may need help selecting helpful versus unhelpful apps. Evaluating them is one important way to do this.

Conclusion

In conclusion, PTT Advisor may contribute to better test ordering and diagnosis and serve as a learning tool for diagnostic evaluation of certain hematologic disorders. Also, users’ perceptions show its potential to improve care for patients in real practice. Methods used in this study could serve as a model for evaluating potential benefits of other mobile applications on clinical decision making. Future efforts focused on integrating rigorously tested decision-support tools into physicians’ clinical work flow may promote better patient care.

Funding

This work was supported by the Centers for Disease Control and Prevention, Division of Laboratory Systems (IAA 15FED1513732), and the Houston VA Center for Innovations in Quality, Effectiveness, and Safety (CIN 13-413). Dr Singh is also supported by the VA Health Services Research and Development Service (CRE12-033; Presidential Early Career Award for Scientists and Engineers USA 14-274), the VA National Center for Patient Safety, the Agency for Healthcare Research and Quality (R01HS022087 and R21HS023602). The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Department of Veterans Affairs nor the Centers for Disease Control and Prevention.

Competing interests

None.

Contributors

All authors made substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; and drafting the work or revising it critically for important intellectual content; and final approval of the version to be published; and agreeing to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.