Skip to main content
Journal of the American Medical Informatics Association : JAMIA logoLink to Journal of the American Medical Informatics Association : JAMIA
. 2018 Jun 7;25(8):1074–1079. doi: 10.1093/jamia/ocy060

Trends in user ratings and reviews of a popular yet inaccurate blood pressure-measuring smartphone app

Timothy B Plante 1,, Anna C O’Kelly 2, Zane T Macfarlane 3, Bruno Urrea 4, Lawrence J Appel 5, Edgar R Miller III 5, Roger S Blumenthal 6, Seth S Martin 5,6
PMCID: PMC6664848  PMID: 29878236

Abstract

Objective

To understand whether user reviews of Instant Blood Pressure (IBP), an inaccurate, unregulated BP-measuring app reflected IBP’s inaccuracy, to understand drivers for high and low ratings, and to understand if disclaimers prevented medical use.

Materials and Methods

All iTunes app reviews for IBP v1.2.3 were downloaded and assessed for themes by two reviewers. Summary statistics for themes were tabulated with their associated star ratings.

Results

Common themes included perceived accuracy (42% of all reviews, star rating mean 4.8, median 5), inaccuracy (10%, 2.0, 1), and convenience (34%, 4.7, 5). Nine percent documented IBP use in medical conditions (4.6, 5), and 2% mentioned IBP’s disclaimer (2.7, 3).

Discussion

User reviews and ratings of a popular, inaccurate BP-measuring app were positive and uncommonly commented on its inaccuracy. Disclaimers attempting to prevent medical use of the app were ineffective. These findings support the need for more rigorous regulatory review of apps prior to their release.

Keywords: mobile health, user reviews, smartphone, blood pressure, user impressions

BACKGROUND AND SIGNIFICANCE

The smartphone revolution bolstered a rapid proliferation of mobile apps of various functionalities, including mobile health (“mHealth”) apps that deliver health and wellness technologies. As of 2015, consumer app stores had >165 000 mHealth apps available for download, doubling the number from just 2 years prior.1 The Food and Drug Administration (FDA) has largely avoided applying its regulatory oversight to this emerging marketplace.2,3 Consumer advocacy groups and professional organizations are largely absent or ineffective in providing independent reviews of the quality of these apps given the sheer number available.

Smartphone owners consider user app ratings and reviews when deciding whether to download a given app.4 These ratings are typically scored out of 5 stars with higher ratings indicating a more favorable user experience. Narrative reviews typically document a user’s reasoning for his/her rating. In absence of assurance of quality through FDA oversight or independent review by advocacy or professional groups, consumers deciding whether to download mHealth apps are left considering user reviews and ratings. These ratings and reviews may foster crowd-sourced self-regulation of apps, as critical public commentary of poorly performing apps can reduce downloads, thereby driving app developers to release high-quality apps. To what extent these ratings and reviews reflect the quality of apps and promote self-regulation of the app market is unknown.

Instant Blood Pressure (IBP) is an inaccurate $4.99 mHealth app claiming to measure blood pressure (BP) with just an iPhone and no cuff. In a previous validation study,5 we found IBP to achieve the lowest possible British Hypertension Society accuracy grade and found it to misclassify 78% of hypertensive measurements as non-hypertensive.

The method by which it produces its result is not disclosed, but it involves entry of user demographics predictive of BP (date of birth, sex, height, and weight), placement of the finger over the rear camera and illuminated light, placement of the microphone over the chest, and holding still for 45 seconds.6,7 We previously found IBP to inappropriately rely on demographics in producing a BP estimate, as these factors explained 66% of systolic and 82% of diastolic BP variability. In contrast, these factors explained only 12% of systolic and diastolic BP variability using a standard automated BP cuff.8

Before its unexplained removal from the Apple app store in the summer of 2015, it was one of the most popular for-sale apps available, earning >$600 000 in revenue.9 Despite replicating the functionality of Class II non-invasive BP monitors,10 it never underwent review by the FDA. IBP prominently displayed several disclaimers including that it was for recreational (ie, entertainment) use only and that it was not a medical device. It is unclear if these warnings prevented medical use of IBP. It is also unclear if users considered IBP to be accurate. Understanding the content of IBP app reviews and quantifying the correlating ratings can help elucidate whether these can serve as effective guides for patients in determining the appropriateness of apps in medical care. Such findings would support the ability of the consumer mHealth market to self-regulate.

OBJECTIVE

The objective of this study was to analyze and describe IBP app store user ratings and reviews to examine app usage, drivers for favorable and unfavorable impressions, the frequency of perceived quality of the BP measurement, and acknowledgement of IBP’s disclaimers.

MATERIALS AND METHODS

All iTunes ratings and reviews for IBP were downloaded from an aggregating website.11 User reviews were excluded if they were not for the most recent version of the app. Duplicate reviews were also excluded.

Coding scheme

Using an iterative process, two reviewers (TBP and ACO) independently assessed user reviews for common content themes and developed a unified coding scheme. Additional uncommon themes related to disclaimers were included. Each reviewer then independently re-reviewed each user review and applied the unified coding scheme. The reviewers resolved coding discrepancies through discussion and consensus.

Analysis

Frequencies of narrative themes and corresponding mean star ratings were tabulated. Despite a lack of normality in the distribution in star ratings, we decided to present means and standardized deviations (SD) of star ratings because the “average” star rating is commonly presented in app stores. We also presented median star ratings and interquartile ranges (IQR). Coding scheme reviewer agreement was assessed with the kappa statistic. We compared the star ratings of thematically similar reviews using the Mann-Whitney U test.12 All analyses were performed with Stata MP 14.2 (StataCorp, College Station, TX).

RESULTS

Between June 5, 2014, and July 29, 2015, 995 ratings and reviews were written for IBP. After excluding ratings and reviews for older versions of the app (n = 730) and duplicate reviews (n = 4), there were 261 remaining ratings and reviews.

Overall ratings

The overall mean (SD) star rating was 4.0 (1.5) stars, and the median (IQR) star rating was 5 (4-5) stars. The most frequent star rating was 5 of 5 stars, comprising 59% of all reviews. Less frequent ratings include 4 stars (17%), 1 star (16%), 3 stars (5%), and 2 stars (2%). Figure 1.

Figure 1.

Figure 1.

Distribution of star ratings for IBP.

Review themes

Rater agreement was high, with the kappa statistic ranging from 0.89 to 1.00. As Table 1 displays, themes that were associated with a higher star rating were IBP being accurate, BP log functionality, IBP being convenient, IBP successfully producing a measurement, and use of IBP for medical purposes. Frequent themes associated with a lower star rating commented on IBP’s inability to successfully produce a BP measurement, IBP’s inaccuracy, and the user wanting a refund. Reviews representative of the themes are in Table 2.

Table 1.

Frequency of review themes and star ratings

Theme N (%) of reviews Star rating
Kappa
Mean (SD) Median (IQR)
IBP accuracy and functionality
 IBP is accurate 109 (42) 4.8 (0.5) 5 (5 - 5) 0.89
  Compared IBP to another BP-measuring device 50 (46*) 4.9 (0.3) 5 (5 - 5) 0.91
 IBP is inaccurate 27 (10) 2.0 (1.2) 1 (1 - 3) 0.96
  Compared IBP to another BP-measuring device 16 (59*) 1.9 (1.2) 1 (1 - 3) 0.94
 Unsure if IBP is accurate 9 (3) 4.1 (1.4) 5 (4 - 5) 0.99
  Compared IBP to another BP-measuring device 0 (0*) 0.98
 IBP successfully produces a BP measurement 27 (10) 4.8 (0.5) 5 (5 - 5) 0.94
 IBP does not successfully produce a BP measurement 31 (12) 1.3 (1.0) 1 (1 - 1) 0.94
Disclaimer and medical use of IBP
 Mentions IBP’s disclaimer** 6 (2) 2.7 (1.6) 3 (1 - 4) 0.98
 Uses IBP for medical purposes 24 (9) 4.6 (0.9) 5 (5 - 5) 0.97
 IBP provides insight into overall health 12 (5) 4.8 (0.4) 5 (5 - 5) 0.95
Medical professionals
 User claims to be a medical professional 6 (2) 4.2 (1.6) 5 (4 - 5) 1.00
 User claims that a medical professional known to him/her approves of the use of IBP 11 (4) 4.9 (0.3) 5 (5 - 5) 0.99
 User claims that a medical professional known to him/her disapproves of use of IBP 2 (1) 1.0 (0.0) 1 (1 - 1) 0.99
Added functionality wanted by the user
 Wants Apple iWatch compatibility 3 (1) 3.3 (1.5) 3 (2 - 5) 1.00
 Wants Apple HealthKit integration 24 (9) 3.8 (0.9) 4 (3 - 4) 0.96
 Wants a log of BP results 41 (16) 4.2 (0.8) 4 (4 - 5) 0.94
Other themes
 Recommends IBP to others 20 (8) 4.8 (0.4) 5 (5 - 5) 0.92
 Wants a refund 13 (5) 1.2 (0.6) 1 (1 - 1) 0.98
 IBP is convenient 90 (34) 4.7 (0.6) 5 (5 - 5) 0.89
*

Percentage of the parent group.

**

Mentioning the disclaimer includes describing that it is for “entertainment purposes only” (n = 5) or that it is “not a medical device” (n = 1).

Table 2.

Select examples of narrative themes

Theme Representative themes with star rating, review title, and review narrative
IBP is accurate 5 stars – “Excellent and practical!!!!”: Easy to use, can take pressure anytime without arm or wrist gadgets and purely with the phone! To cap it all, this app is much more accurate and reliable than the clunky Omron machine I recently bought (and now returned!).
IBP is inaccurate 1 star – “Not Even Close “: I read all of the great reviews prior to making this purchase. I really wanted to like this app! I took several readings with both the app and my cuff and the app is way off. Average 110/72 with the app, and the cuff average is 135/84. I’d say that’s a pretty big difference! So disappointed that I actually spent money on an app that doesn’t work as advertised.
Unsure if IBP is accurate 4 stars – “Great app”: It’s easy to use. I don’t know how accurate the readings are but it gives me an general idea of my vitals. I’d give it a five but there’s no ability to save/track measurements.
IBP successfully produces a BP measurement 5 stars – “So so helpful”: I wish this app came out earlier - this would save me $$I spent on unreliable gadgets. Very good job, guys. Works like magic. Thank you!
IBP does not successfully produce a BP measurement 1 star – “Junk”: Have tried multiple times and never had success getting a reading do not buy.
Mentions IBP’s disclaimer 4 stars – “It’s pretty amazing, but…”: …at five bucks a pop, it doesn’t share information with the Health app? Does Apple question its accuracy enough that it’s “for entertainment purposes only”?
Uses IBP for medical purposes 5 stars – “Very accurate”: Been using this along side a pressure cuff due to extreme “white coat” syndrome associates with taking blood pressure. Very accurate and with no cuff pressure during the reading helps mitigate the effects of the syndrome. Hope this finds a way into an accessory on a wrist so you wouldn’t even be aware when the reading was taking place.
IBP provides insight into overall health 5 stars – “Heart data”: Love how I can watch my heart in real time. It’s like a window into the heart. Wish I could store my data on the app and chart It out for me. Until then I guess taking occasional screen grabs will do. Really cool to see what certain foods, drinks and supplements do to your blood pressure in real time.
User claims to be a medical professional 4 stars – “Great app”: I like this app a lot and have tested it against monitoring devices at work (I’m an OR nurse) and it’s pretty accurate. I just wish it had a tracker for my daily measurements.
User claims that a medical professional known to him/her approves of the use of IBP 5 stars – “Excellent App…”: My Cardiologist is pleased with way of tracking my BP and Pulse!!
User claims that a medical professional known to him/her disapproves of use of IBP 1 star – “Should only be used for fun”: I have normal blood pressure usually around 120/80. So at first it seemed that the readings were legit because they always hovered around those numbers. I was in a car accident a week ago, I wasn’t hurt bad but I also have anxiety and started to panic. I started doing my breathing exercises and used the app to track my blood pressure and heart rate. My heart rate was a little high for me and my BP was in the normal range. Then when I went to the ER to get checked out and they took my blood pressure it was 150/100. They said this happens a lot when people are in accidents b/c of the stress. I told them I took my BP 10 minutes before I came in and it was 123/82. They said basically the app is Bullcrap and that it will give everyone a reading w/i normal to high/normal range therefore the majority of people think it’s accurate. I thought about it and the next time I went to the gym I took my BP it’s not rocket science to know that when you exercise your blood pressure elevates. I took it at the gym and sure enough 125/79 then I used the blood pressure cuff at the gym and the reading was 145/90 I took it not even a minute later. Yes the app is fun and it does detect your pulse fairly accurately but as far as your BP its a joke.
Wants Apple iWatch compatibility 3 stars – “Works good once you figure it out”: I had a rough time getting the app to register a reading for me at first. Now that I’ve been doing it for a week, it’s a little bit better. I still have some issues with the finger on the camera but usually I can get the reading on the first or second try. Hoping the can use apple watch for the heart rate part soon and then you would just have to use the pulse for users with the watch. Little pricey at $4.99 but a good app to learn where you are at.
Wants Apple HealthKit integration 4 stars – “Time to go prime time…”: Love the app and it’s ability to deliver consistent and reliable BP measurements once you learn how to properly place the phone and your finger. PLEASE…TIME TO LINK TO APPLE HEALTH KIT!!!
Wants a log of BP results 4 stars – “Just one more thing!”: This app is great! I just wish it stored my blood pressure readings so I would have them to share with my doctor. I was quite shocked that it didn’t already do this. Please add this soon….then it would be even more awesome!
Recommends IBP to others 5 stars – “BP”: Wonderful app! Very consistent was pleasantly surprised! Would recommend this to anyone who wants/needs to keep an eye on their BP
Wants a refund 1 star – “Don’t waste your money”: I couldn’t get it to work. Would like to know how to get my money back.
IBP is convenient 5 stars - “Convenient and Easy”: Works well easy to use. Glad I can get a BP with ease of use.

Review themes pertaining to accuracy

Commentary on perceived IBP accuracy, based on anecdotal experience, was the most frequent narrative theme, comprising 42% of all reviews. This was associated with mean and median star ratings of 4.8 (0.5) and 5 (5-5). Of these narratives, 46% reported comparing it to another BP-measuring device (mean rating 4.9 [0.3], median 5 [5-5]). Commentary on IBP being inaccurate comprised 10% of narrative themes (mean rating 2.0 [1.2], median 1 [1-3]), with 59% of these reviews reporting comparing IBP to another device (mean rating 1.9 [1.2], median 1 [1-3]). Of the 66 reviews reporting a comparison against another BP-measuring device, only 16 (24%) reported IBP being inaccurate.

Star ratings were higher for reviews documenting IBP to be accurate than those documenting IBP to be inaccurate (P < .001) or to be of an unsure level of accuracy (P = .02). Star ratings were lower for those documenting inaccuracy versus an unsure level of accuracy (P < .001).

Review themes pertaining to IBP’s disclaimer and medical use of IBP

Of the 6 reviews (2%) mentioning a component of IBP’s disclaimer, 1 commented on IBP not being a medical device and 5 commented on IBP being for recreational or entertainment purposes only. These reviews were associated with a mean rating of 2.7 (1.6) and median rating of 3 (1-4). Star ratings were lower for reviews mentioning the disclaimer than those that did not mention the disclaimer (P = .02).

Twenty-four (9%) reviews documented the use of IBP for medical purposes (mean 4.6 [0.9], median 5 [5-5]). Of the 11 reviews documenting use of IBP in managing hypertension, 2 reviews documented that IBP helps diagnose “white coat” hypertension, and 1 review documented that it was useful in following BP levels when forgetting to take antihypertensive medications. Other reviews documented the use of IBP in managing end-stage renal disease (ESRD, n = 1 review), postural orthostatic tachycardia syndrome (POTS, n = 1 review), and post-heart transplant care (n = 1 review). There was a non-statistical trend towards higher star ratings among reviews mentioning medical use than those that did not mention medical use (P = .052).

Review themes pertaining to medical professionals

Of the 6 reviews claiming that the reviewer was a medical professional, 4 reported being nurses, 1 reported being a certified residential medication aide, 1 reported being a doctor, and 1 reported being a healthcare professional, not otherwise specified. These reviews were associated with a mean and median star rating of 4.2 (1.6) and 5 (4-5), respectively. Star ratings for reviews claiming to be from a medical professional were similar to those that did not claim to be from a medical professional (P = .70).

Of 11 reviews claiming that a medical professional known to the user approved of IBP, 7 were reported to be nurses and 4 were reported to be doctors. One doctor was reportedly so impressed with IBP that he/she began to recommend the app to other patients. Mean and median star ratings for these reviews were 4.9 (0.3) and 5 (5-5), respectively. Of 2 reviews claiming that a medical professional known to the user disapproved of IBP, 1 was reported as an unspecified emergency department staff member and 1 as a nurse. Both reviews had 1 star ratings. Star ratings were higher for those with medical professionals approving of IBP use than those disapproving of IBP use (P < .001).

DISCUSSION

We assessed for themes among iTunes app store reviews of a popular, inaccurate, BP-measuring app and tabulated their related ratings. We found IBP to have overall high ratings, which were driven by users documenting the app’s accuracy and convenience. We also found that approximately 10% of reviews were related to medical use of this app. Interestingly, health professionals who left a review tended to have favorable impressions of the app.

There is a need for improved education of consumers, patients, and health care professionals to promote responsible use of mHealth technologies and the need for tighter regulation of the mHealth marketplace, as the case study of IBP does not support self-regulation of this market. Validation studies confirming accuracy must be performed before non-invasive BP measuring devices are sold to consumers.10 Among IBP users leaving ratings and reviews, most who anecdotally compared IBP to a reference device perceived IBP to be accurate, despite the opposite finding after rigorous clinical assessment.5 While it is possible that users perceiving IBP to be accurate were more likely to leave a review than those finding IBP to be inaccurate, consumers deciding whether to download and use IBP were exposed to reviews that disproportionately emphasized app accuracy.

The app store description of IBP included disclaimers stating that the app is not a medical device and is for recreational use only. We are unaware of any recreational uses of a non-invasive BP monitor. While 6 reviews acknowledged these disclaimers, 4 times as many reviews documented IBP’s use in medical conditions. Some of these were serious BP-related conditions, including hypertension, POTS, ESRD, and post-heart transplant care. We are particularly concerned by the users with “white coat hypertension” feeling reassured by this app. In our validation study, IBP missed hypertensive-range BP 78% of the time. For these individuals, they may be self-diagnosing white coat hypertension when in fact they could have uncontrolled hypertension. These findings suggest that disclaimers are ineffective in preventing medical use of this technology. An optimal strategy would ensure safety and accuracy of mHealth apps by requiring clinical validation prior to their release.

Literature assessing the role of user ratings in determining the quality of mHealth apps is very limited. Among smoking cessation apps, one study found that higher user ratings were associated by higher app adherence to the 5As of smoking cessation (ask, advise, assess, assist, and arrange follow-up) that are included in clinical practice guidelines.13 Other surveys of women’s health-specific, hepatitis C-specific, and medication reminder apps identified that many mHealth apps lack user reviews in general. One survey of 137 apps for various chronic diseases identified poor correlation between mean user rating and perceived clinical utility of the app by a reviewing medical professional.14 These studies are limited assessments of the correlation between user ratings and reviews and objective app quality. Given the impact of user reviews in driving app uptake, we think that there is an urgent need to expand literature behind mHealth app ratings and reviews.

There is also an urgent need to improve safety of the mHealth marketplace as it currently stands. Numerous apps with equivalent functionality to IBP are still available to consumers. One such app, iCare Health Monitor, measures BP, lung capacity, and oxygen saturation with just a phone and no external devices.15 It has a mean rating of 4.5 stars and has been downloaded >1 000 000 times, dwarfing the success of IBP. It has not undergone independent validation and, like IBP, has not been reviewed by the FDA. We are concerned that the iCare Health Monitor cuffless BP readings likely suffer from similar inaccuracies to those we reported previously for IBP. More recently, the flagship Samsung Galaxy S9 Android phone has shipped with native optical BP-measuring functionality similar to IBP.16 We are unaware of any validation data. As the Galaxy S8 sold 20 million devices,17 it is likely that the Galaxy S9 will place this technology into the hands of a large number of adults.

In contrast to Google’s mostly automated app release process, Apple has attempted to increase the stringency of its approval process.18,19 We are encouraged by this policy change and the interest of some regulatory bodies to improve the safety of the mHealth marketplace. For example, after publication of our initial validation results, the Federal Trade Commission pursued litigation against the manufacturers of IBP, eventually settling for approximately $600 000, with payment suspended for inability to pay.9 Hopefully, regulatory bodies will continue to engage mHealth app developers.

Limitations

This study has several limitations. We did not include ratings and reviews from the Android version of IBP. We made this decision because the Android version of the app was downloaded by only 1000 to 5000 users when it was available.20 Second, as these reviews constitute a convenience sample of vocal users, they may not represent the experiences of the typical user of IBP. However, this convenience sample is what is available to individuals deciding whether or not to purchase the app. Finally, we are unable to verify reported disease statuses, professional affiliations, or medical use of IBP.

CONCLUSION

We found user reviews of an inaccurate BP-measuring mHealth app to document high ratings, largely driven by perceived accuracy and convenience. User ratings and reviews, therefore, do not provide fair assessment of the quality of the functionality of this app. As 9% of these reviews document app use for medical purposes, disclaimers appear to be ineffective in preventing medical use.

FUNDING

This work was supported by a PJ Schafer Cardiovascular Research Grant. Dr Plante was funded by Institutional National Research Service Awards from the Health Research Services Administration (T32HP10025B0) and a National Institutes of Health National Heart, Lung, and Blood Institute (2T32HL007180-41A1).

CONTRIBUTORS

TBP designed the study, obtained data, performed the analysis, and wrote the manuscript. AOK performed the analysis and wrote the manuscript. ZTM obtained the data and critically revised the manuscript. BU obtained the data and critically revised the manuscript. LJA designed the study, assisted in the analysis, and critically revised the manuscript. ERM designed the study, assisted in the analysis, and critically revised the manuscript. RSB obtained funding, designed the trial, and critically revised the manuscript. SSM obtained funding, designed the trial, assisted in the analysis, and wrote the manuscript. All authors have given final approval for publication and agree for accountability for all aspects of the work.

Conflict of interest statement. Dr. Martin reports research support from Google, Apple, Nokia, iHealth, and the Aetna Foundation. There are no other relevant conflicts of interest.

Acknowledgments

We would like to thank Geetanjali Chander, MD MPH, for feedback on study design and on feedback of an earlier version of this manuscript. A version of this research was presented at the 2016 Society of General Internal Medicine conference.

REFERENCES


Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of Oxford University Press

RESOURCES