Assessment of content, quality and compliance of the STaR mobile application for smoking cessation

Abstract

INTRODUCTION

Smoking cessation support with smartphone applications is widely practised, however, quality and novel content is yet to be established. This study examined content, quality and compliance of the STaR (Sihat Tanpa Rokok - Healthy Without Smoking) smartphone application to smoking cessation clinical practice guidelines.

METHODS

Mobile Application Rating Scale (MARS) was applied to assess the quality of the mobile application. A total of 33 compliance assessment criteria were developed, based on the smoking cessation clinical practice guidelines adopted by the Brunei Ministry of Health and used to rate the compliance.

RESULTS

The STaR app has an average quality and compliance score. The mean MARS score of 3.31 and a subjective quality score of 2.50 was measured on a 5-point scale (from 1=inadequate to 5=excellent). The application contents that adhere to smoking cessation clinical practice guidelines suggest adequate adherence to the 5A (Ask, Advice, Assess, Assist, Arrange) and 5R motivational interventions (Relevance, Risks, Rewards, Roadblocks, Repetition) approach for smoking cessation.

CONCLUSIONS

The STaR mobile app with the integration of all components of smoking cessation clinical practice guidelines (such as 5A approach and pharmacotherapy intervention) may improve quality scores.

INTRODUCTION

Mobile applications and the computing ability of smartphones have changed mobile health (mHealth). More than 3 billion mHealth apps are estimated to have been downloaded in 2015¹. Mobile applications can be easily downloaded, and many users can receive tailored text messages and information at low cost. The use of smartphone technology is increasing rapidly globally. Between 2014 and 2015, the global use of smartphones grew by 566 million. There is rapid adoption of the smartphone as demonstrated by its estimated broadband penetration of 47% and that 3G coverage has nearly reached 70% of the world population². In the same period, the Asia-Pacific region has experienced an 83% increase in internet mobile data traffic².

The national Information Communication and Technology (ICT) data show that the mobile penetration rate in Brunei has kept above 100% since 2008³, and the mobile internet penetration as of 2016 has reached 100 per cent, leading to increased use of mobile applications and social media^{4, 5}. Public App Stores such as iTunes or Google Play serve many consumer needs that include government mobile apps and limited private or enterprise apps. Open standards such as HTML5 for app development is widely used by the Brunei ICT industry. One-third of the mobile applications in Brunei have started using cross-platforms, thus extending local and international links⁶.

The use of mHealth interventions for smoking cessation represent the WHO (World Health Oganisation) recommended best buys to curb the global public health threat by the tobacco epidemic⁷. Additionally, a study suggests that a high prevalence of smokers in the US and UK use a smartphone, send and receive texts, download, and use apps, use Facebook and browse health-related internet sites⁸. Most importantly, smoking cessation by 40 to 60 year-olds reduces the risk of premature death by 90% to 40%, respectively⁹. Based on this evidence, the Tobacco Free for Life Program of the Health Promotion Center, Ministry of Health , launched the smoking cessation mobile application STaR (Sihat Tanpa Rokok - Healthy Without Smoking) in conjunction with the World No-Tobacco day, 3 June 2015¹⁰.

The health apps can improve with user-friendly app content, design and interactive features. Content analysis from the sample of smoking cessation application suggests quality and content are interrelated^{11, 12}. Therefore, this study was conducted to outline features and design of the STaR app, and estimate its quality based on the Mobile Application Rating Scale (MARS). Furthermore, we evaluated the compliance of the app content with smoking cessation clinical practice guidelines developed by the Ministry of Health.

METHODS

App development, design and technical features

The smartphone application was developed using in-house expertise within the Ministry of Health, including an ICT expert, a behaviour-change psychologist, and a public health and health promotion expert. Table 1 summarises the information on the app classification design and technical features. The STaR mobile application uses the Malay language. The app is free in Brunei’s app store and published by the Ministry of Health. The primary purpose of the application is to encourage smoking cessation. STaR uses HTML5 with adaptive cross-platform design. The technical features include sharing of Facebook content, possessing an app community of users, and space for dedicated profiles. The theoretical basis of the content was derived from cognitive behaviour change theories, assessment and motivation. The app targets adolescent, young adults and current adult smokers. The app contains 7 sections and 9 sub-sections.

Table 1.

App classification, design and technical features

Characteristics	Description
App name	STaR (Sihat Tanpa Rokok - Healthy Without Smoking)
Language	Malay
Available platform	Apple Store and Play Store
Developer	In-house development (smoking cessation counsellor and ICT trained health personnel within MoH)
Version	1.6
Last update date	17 May 2015
Development cost	2937 USD*
Type	Free, Native, HTML 5, adaptive design
Security policy	Yes ( compliance with Google and Apple security policy)
User rating	3.8**
Affiliation	Government
Purpose	App that gives smokers in Brunei a supportive group and a channel to find help to stop smoking
Technical features	Allows sharing with Facebook, has an app community, requires login, does not require web access to function, no password protection features
Strategies applied	Assessment, information and awareness, cognitive changes (thought challenging ), advice/ tips/ motivation, cognitive behaviour change (positive events), behaviour change, motivational theories and health promotion
Targeted population	Adolescent age 13-17, young adult 18-25, adults using smartphone
Total section in the app and their type	Users profile , latest updates, about smoking cessation, smoking cessation clinic , contact us, users, chat Total : 7 sections and 9 sub-sections

ICT-Information Communication and Technology

MoH-Ministry of Health Brunei

^*Estimate

^**based on Google Play Store rating

Study design and duration

We report on records of the mobile app activity over the app development website, Android and Apple smartphones. The study was conducted from July to August 2016.

Tools and technique

Data extraction was conducted using the pre-validated tools (MARS scale) and a checklist of criteria with themes derived from the smoking cessation clinical practice guidelines¹³ and past studies^{11, 14}. The compliance of content with clinical practice guidelines was accessed to extract and classify: the app content as a basic intervention; the ABC approach (Ask, Brief Advice, Cessation Referral) as an intensive intervention; and the 5A approach (Ask, Advice, Assess, Assist, Arrange) and 5R motivational interventions (Relevance, Risks, Rewards, Roadblocks, Repetition) for people motivated to quit, as described in clinical practice guidelines¹³. The mobile application quality rating scale (MARS) was applied to rate the quality of the content used in the application. The Ms-Excel proforma was prepared as suggested by Stoyanov et al¹⁴. The leader of the app development team was consulted to access the background information during the quality and compliance assessment process. The MARS quality scale measured app quality on five domains; engagement, functionality, aesthetic, informative and subjective. Quality was measured using 19 quality indicators each on a 5-point scale (from 1=inadequate to 5=excellent). The compliance of the app information with the national smoking cessation guidelines was assessed by scanning the content of the app. The authors individually rated the level of conformity for each section with a yes, no or not enough information to determine the criteria. The rating exercise was performed independently and the discrepancies were settled with discussions among experts. Inter-rater reliability of 85% was achieved using Ms-Excel proforma.

Ethical consideration

The Ethical approval was obtained from the research ethics committee of the Institute of Health Science, University Brunei Darussalam. The privacy and confidentiality of the application users were preserved with security features such as password protection, as outlined in the application terms of use.

RESULTS

App utilisation

There were 145 downloads from the Google Play Store and 52 from the Apple Store. Apple Store downloads represent almost one-fourth (26%) of the total. An app rating of 3.8 for the Google Play Store was observed. There were 39 current users. The number of people accessing the app on Facebook was 155. In the past six months, the majority (56%) of the messages uploaded were pictures. More than three quarters (78%) of the messages provide information on the benefits of Smoking Cessation (Table 2).

Table 2.

App utilization and quality

Characteristics		N (% )
Downloads	Apple Store	52 (26)
	Google Play Store	145 (74)
Current users		39
Numbers of likes to the social media page of the app		155
Type of information uploaded to app in past 6 month
	Text messages	5(28)
	Pictorial messages	10(56)
	Audio visuals materials	3(16)
Nature of the information provided in past 6 month
	Smoking cessation clinic	2 (11)
	Benefits of smoking cessation	14 (78)
	Promotion of organizational events	2 (11)
Mean Mobile App Rating Scale (MARS Score) (from 1= Inadequate to 5 = Excellent) (N=19)		3.31
	Engagement	2.8
	Functionality	3.5
	Aesthetics	2.29
	Information	3.57
	Subjective quality score	2.5

Quality analysis

The mean MARS score was 3.31. The average score on each of the four components of quality ranged from 2.50 to 3.57. The functionality feature had the lowest mean score of 2.50 and the information component had the highest mean score of 3.57. The app mean subjective quality was 2.50.

Compliance with clinical practice guidelines

The content of the app compliance with the national smoking cessation guidelines shows low compliance to the Level 1 intervention, i.e. the ABC approach. The intensive Level 2 interventions, i.e. the 5A approach and the motivational intervention, i.e. the 5R approach, exhibit high to moderate level compliance, respectively. For the Level 3, i.e. the pharmacy-based intervention, the app contains minimal information to judge the criteria. Overall, the app content showed compliance with 50% of the criteria of the Smoking Cessation guidelines. Detail criteria and their assessment are presented in Table 3.

Table 3.

Compliance with clinical practice guidelines on smoking cessation

Sections	Component	Criteria Y = criteria met, N = criteria not met, NC = not enough information to judge the criteria, Yes = 18(50%), No = 10 (27.8%.2), NC = 8(22.2%)	Observation
Level 1 Intervention ABC Approach	Ask	Status of tobacco use for every app user, based on do you smoke or use any form of tobacco	NC	40%
		Classify the users as current, former, never, or passive smokers	NC
	Brief advice	Stop-tobacco-use advice to smokers regardless of intention to quit	Y
		Triggering message that links to smokers existing tobacco related medical condition	NC
	Cessation referral	Do app features contain or provide information on cessation referral (smoking cessation counselling, pharmacotherapy or smoking cessation clinic or all)	Y
Level 2 Intensive Intervention 5A approach	Ask	Smoking cessation status/classify smoking status	N	53%
		Numbers of cigarettes smoked per day	Y
		Duration of smoking	Y
		Previous quit attempts (method, duration, what helped and what did not help)	N
		Reasons for relapse	N
		Medical conditions such as high blood pressure, heart attack, stroke, DM, hypercholesterolemia, chronic lungs disease	N
		Family history of high blood pressure, heart attack, stroke, diabetes, lung cancer, other cancer	N
		Contains positive reinforcement for non-smokers	Y
	Assess	The smokers degree of nicotine addiction (Fagerstrom score )	Y
	Advice	Information on harmful effects of tobacco	Y
		Information on relapse	Y
		Common fears and concerns about quitting	Y
		Tailored messages according to users medical conditions	N
		Patient readiness to stop smoking	NC
	Assist	Contains features to set quit date	N
		Sharing or getting help from friends, family, app community	Y
		Message and technique on problem solving	NC
	Arrange	Have features to set follow up session or hotline or online help	Y
		Have information on behavioural technique (delay /escape/ avoid/distract technique)	Y
Motivational Intervention The 5R approach	Relevance	Does the app have features or information that encourage the users to indicate why quitting is personally relevant	Y	57%
	Risk	Does the app information demonstrate short term/long term or environmental risk of smoking?	Y
		Enables or provides potential negative consequences of tobacco	Y
	Reward	Does the app information demonstrate potential benefits of quitting?	Y
		Is the information tailored?	N
	Roadblocks	Enables smokers to identify barriers to quitting and suggests solution to those barriers?	NC
	Repetition	App has features of reminder of quit date	NC
Pharmacotherapy	NRT/Varenicline	Availability	Y	40%
		Remind patient to stop smoking with the use	Y
		Strategies to address withdrawal and relapses	NC
		Information on duration of use, dose, adverse reaction and contraindication for use	N
		Mechanism of action, dose, side effects and warning and contraindication	NC
		Compliance Inter-Rater reliability	50% 85%

NRT- Nicotine Replacement Therapy

DISCUSSION

The study presents content and quality analysis of the STaR mobile app for smoking cessation. The goal was to compare the content of the STaR mobile application with predefined standards, outlined in clinical practice guidelines, and to estimate its quality on the Mobile App Rating Scale (MARS). The results indicate that the STaR app has average quality and compliance scores followed by low app use.

Given that a large proportion of the population uses smartphone applications¹⁵, the country has the highest mobile internet coverage among South-East Asian (ASEAN) countries¹⁶, and that it has a high prevalence of smoking¹⁷, the STaR mobile application should adopt the highest spectrum of technological features in order to engage with the smokers and track their cessation process, as outlined in the smoking cessation clinical practice guidelines. Accordingly, a cross-sectional mHealth study among Brunei and Indonesian populations found that promoting support mechanisms, quality of service, and quality of information are prominent factors identified by health-care staff for an effective mHealth program¹⁸. The high prevalence of mHealth opportunity justifies the development of STaR apps that target smokers.

Despite the high prevalence of cigarette smoking, the app utilisation statistics were low based on downloads, current users and persons assessing social media content. There are supply and demand side factors at play. Especially for the government health system, mobile apps have low inter-operability¹⁹. A patchwork of legal and liability issues, such as privacy and security, and human resources to control, monitor and update mobile applications in the health sector are yet to be addressed globally^{20, 21}. In addition, our study app was largely developed by health care professionals with minimal expertise in ICT. Few interventional studies on Smoking Cessation apps suggest that targeting social groups, such as working women, university students, and industrial workers, with tailored messages based on push notification increases app utilisation²². In our study app, push notification on tailored targeted populations was rare.

Quality scores from the MARS scale are widely used to rate mHealth apps involving physical activity, smoking cessation and disease management^{11, 14, 23}. Our study analyses on quality using the MARS score are concurrent with analyses of popular smartphone mobile apps for smoking cessation²³. Quality scores are heavily decreased by feedback features, tailored user content and client preference, while our app is in its infancy to regulate and monitor these features¹². This app is a non-intended informative app without those aforementioned tailoring and tracking features. Moreover, a recent analysis of smartphone apps suggests low availability and usability of informative apps¹².

The compliance of the app content was measured with 33-point criteria derived from the verbatim outlined in the smoking cessation guidelines. The majority of these criteria are in accordance with US public health clinical practices to treat tobacco dependence. The mean adherence index of 12.9 (SD=6.8) on a 42-point scale was found in an analysis of 98 smartphone smoking cessation applications¹². In our study, the compliance was low for the ABC approach, 5R approach and pharmacotherapy component, but the best adherence was found with the 5A approach.

This study builds on progress and provides foresight on the use of the mHealth application for smoking cessation. Limitations include failure to document the user’s perspective on app utilisation and the quality assessment process. Perspectives on a smoker’s ease of use and need would be future research priorities. Additionally, the compliance criteria of the app are built around the assumption that the recommendations in the Clinical Practice Guidelines adopted by health care personnel of Brunei in a clinical setting are applicable to a mobile app setting. It has been shown that further studies are needed to test this argument^{11, 24}. Finally, the personnel rating the app are not experts and receive no training, despite their time and effort in learning from online videos. This may have generated additional quality and adherence scores. Other sources of systematic bias that is accounted for, but not very likely, include observer bias and reference standard bias, as we found no evidence on critical appraisal of smoking cessation clinical practice guidelines. Unlike other similar studies^{11,12, 24-26}, our study is unique in its analyses of a single mobile application for smoking cessation.

CONCLUSIONS

The results of our study indicate that the STaR app could be improved through the better adoption of every component and strategy outlined in the smoking cessation clinical practice guidelines. Improving aesthetic and functional features to integrate smokers by continuous engagement and follow-up activities with the application is recommended.

CONFLICT OF INTERESTS

The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

FUNDING

There was no source of funding for this research.

PROVENANCE AND PEER REVIEW

Not commissioned;

externally peer reviewed