Usability testing of a wireless individual indicator system application: Monitoring exposure to outdoor air pollution among Malaysian Traffic Police

Abstract

Objectives

Designs for low-cost air monitors and associated performance data appear in many peer-reviewed articles; however, few manuscripts provide feedback from end user's experiences or comprehensive evaluation. The present study addresses the usability of the wireless outdoor individual exposure indicator system from the viewpoint of the Malaysian Traffic Police (end users). This study is one of the first to chronicle end user experiences for low-cost pollution sensing.

Method

The evaluation involved 12 target end users to assess the usability of a prototype for Malaysian Traffic Police to manage their exposure to outdoor air pollution. The test evaluation includes a pre-test, post-task and post-test questionnaire (Post-Study System Usability Questionnaire). The main components in this Post-Study System Usability Questionnaire are Overall satisfaction, System Usefulness, Information Quality and Interface Quality.

Findings

The results of the Post-Study System Usability Questionnaire indicated the mean score of the Overall satisfaction item (2.33), System Usefulness (2.25), Information Quality (2.36) and Interface Quality (2.17) on a scale of 1–10. Prototype users were satisfied with the system because the score is close to 1 on the Post-Study System Usability Questionnaire.

Conclusions

A user-friendly wireless outdoor individual exposure indicator system is now available for Malaysian Traffic Police. Users have stated that they are happy to use the system at work. However, in addition to more technological advances, practical implementation requires evidence supporting its efficacy, viability and effectiveness.

Background

Low-cost sensors built into low-power sensor systems are used in wireless sensor networks (WSNs) to monitor the urban air quality of hazardous chemicals and particulate matter with high spatial and temporal resolution at the outdoor level. ¹ Low-cost sensor technology has the potential to transform air pollution monitoring by providing high-density spatiotemporal pollution data. Such information can be used to supplement traditional emissions monitoring, improve exposure predictions, and raise community awareness of air pollution. ² The introduction of sensors capable of measuring the gases and particles that make up air pollution, especially in cities and industrial areas, has prompted several academic studies that evaluate sensors and compare performance against reference techniques. ³

Construction companies have been striving for a dust monitoring tool that does not require an external power source, and various initiatives are currently using low-cost sensors to accomplish this goal. ⁴ Emissions monitoring in road tunnels and related ventilation ducts are examples of civil engineering applications. Many countries, including Scotland, Ireland and Greece, have conducted studies to better understand the relationship between traffic levels, mitigation measures and air quality. ⁵ Local air quality data, when combined with wind speed and direction information, can provide a powerful attribution of the source of pollutants, which can be applied to fence line monitoring of industrial facilities, airports and other locations. ⁶ These emerging sensor systems can provide a new stream of data that complements and calibrates the data provided by the fixed traditional air monitoring stations as the reference stations. Furthermore, because they are self-sufficient in terms of electricity and communications, they allow for greater mobility and new possibilities.

Recent studies have introduced these innovative technologies equipped with low-cost sensors that can monitor air pollution directly and use the WSN to provide online and real-time data. ⁷ According to the recommendations of the US Environmental Protection Agency (EPA) and the Directive of the European Union (EU), multiple researches on the use of real-time air monitoring using low-cost sensors have been performed.^8–10 Low cost, mobile, relatively small size, low power consumption, large coverage area and most significantly, online data access on websites and smartphone apps are the advantages of this new technology. ¹¹

Although the fixed traditional air monitoring stations provide many benefits, the need for air monitoring devices closest to the working location of the traffic police is significant because they are mobile at all times. ¹² The downside is often its immobilization, only in fortified locations, costly, restricted spatial resolution and inefficient data communication. ¹¹ Significantly, research by Kim et al. ¹³ reports that because of the low spatial and temporal resolutions, air quality data communication via government websites is insufficient and inconvenient. Anyhow, a low spatial-temporal resolution is adequate and sufficient for environmental monitoring, but not for the community, especially outside workers, to understand their exposure to air pollution and not to address individual health risks. ¹⁴

The need for a practical system that can effectively help detect air pollutants was illustrated in several previous studies by the Malaysian Traffic Police.^15,16 A system using real-time wireless air monitoring was designed to cater to the needs of the Malaysian Traffic Police. Detailed methods for the design are discussed in another article by the same author. The system uses WSN to provide online and real-time data which is communicated through website and smartphone applications. Despite all the studies on low-cost air monitoring and its performance in data measurement, to the extent of the author's knowledge, a few have been produced with feedback from end users or comprehensive evaluation. This study was conducted to assess, from the Malaysian Traffic Police viewpoint, the usability of the wireless outdoor individual exposure indicator system prototype for its website and apps.

Methods

Design

This study used heuristic evaluation which is a comprehensive assessment of the website, or app's user experience. This process of broad rules of thumb allows for flaws identifications in the user experience and comparing the website or apps to usability best practises. ¹⁷ The evaluation was used to assess the air pollution website (https://airpolutionmonitor.web.app/) and smartphone application for the wireless outdoor individual exposure indicator system prototype as the user interface (Figures 1 and 2).

Figure 1.

Overview of the website.

Figure 2.

Smartphone application.

Procedure

The first step in the evaluation is to define the objective which is to analyse the usability testing score for both the website and the smartphone application of the prototype. Next, a usability testing plan is developed to be used during the evaluation. In this study, a usability test plan comprised of a few test sessions according to the order namely;

Screener test

The test was administered prior to the collection of data in order to classify respondents who met the study criteria. The respondents who fulfilled the criteria were asked to proceed with the next test. The criteria for the users were working age at 18–56 years old, ¹⁸ have Internet access and regular users of the Internet. ¹⁹ In this study, having Internet connectivity and being a regular Internet user is critical to ensure that the prototype's website and applications are accessible for the usability testing.

Pre-test questionnaire

In usability testing, the pre-test is essential to define the profile of a user who will be performing the testing. A quick pre-test questionnaire is frequently undertaken before the user begins to experience the product. It is designed to assist the developer to better understand the user's views, goals and behaviour in relation to the product under test.^20–22 At this point, all respondents completed a questionnaire primarily regarding self-information, occupational data, knowledge of air quality, health effects of bad air exposure and measures taken to prevent bad air exposure. This part of the questionnaire, adapted from Karki et al., ²³ was translated into Bahasa Malaysia.

Prototype testing in real-world application

By utilizing the developed prototype (discussed in another article by the same author) in a field study, the recruited subject was asked to utilize the prototype and track their exposure at work. The prototype showed users real-time air pollution measurements while they were exposed by working outside regulating traffic flow.

Post-task questionnaire (task scenarios)

Before using the prototype, the respondents were requested to install the application of this method on their smartphones and use them at work. Then, the respondents were asked to follow the instructions given by the examiner based on the scenarios. The task scenarios in the usability test plan were adopted from Barnum. ²⁴ The specific goals of this task were to:

• Identify the current level of exposure.

• Search for previous exposure information by date and time.

• Save and download the information searched.

In this test, the assessment was based on the time taken to complete the task by the respondent, the number of respondents able to complete the task and the number of attempts needed to accomplish the task. The investigators for this test were another researcher who was not involved in any part of the study to control the bias.

Post-test questionnaire (Parts F, G and H)

The final stage of the usability plan was the revised version three of standardized Post-Study System Usability Questionnaire (PSSUQ) adopted from Lewis. ²⁵ The PSSUQ was widely used for user experience study on any interface, hardware, software, mobile applications or websites. ²⁶ Moreover, the questionnaire is proven to help identify the problems in website and mobile application developments.^27–30 The questionnaire has also been used in other recent user-centred application research proving the usefulness in problem recognition of their product.^31–33

PSSUQ follows a 7-point Likert scale where the overall result is calculated by averaging the scores given to statements. The PSSUQ score ranges from 1 (strongly agree) to 7 (strongly disagree). The better the performance and satisfaction, the lower the score. In PSSUQ, it has three subscales, namely System Usefulness (SYSUSE), Information Quality (INFOQUAL) and Interface Quality (INTERQUAL). ²⁴ The subscales give a more in-depth look at the various aspects that influence the website, software, system, or product.^34,35 The scores of the PSSUQ are categorized into subscales as such; ³⁵

• ▪Overall: the average scores of questions 1–16
• ▪SYSUSE: the average scores of questions 1–6.
• ▪INFOQUAL: the average scores of questions 7–12.
• ▪INTERQUAL: the average scores of questions 13–15.

Since the PSSUQ could be broken down into four sections: overall, SYSUSE, INFOQUAL and INTERQUAL, a careful review of the scores for each of the 16 questions is required. ²⁴ By averaging the scores of the questions within the subscales generated the scores for each subscale. This is required in order to provide depth reviews of which subscales should be improved. The standardized questionnaire (PSSUQ) was translated into Bahasa Malaysia so that the respondents can easily understand the question given. The questionnaire that is adopted had undergone constructive testing and reliability testing. After conducting the pre-test, the PSSUQ had acceptable reliability (Cronbach's α = 0.95) which is also similar to Schnall et al. ³⁵ (Kappa index = 0.70–0.95). This result indicates that the questionnaire is a reliable and valid test for usability evaluation of the end users of a wireless outdoor individual exposure indicator system prototype.

Data analysis

Once the usability testing is conducted, the results is analysed and summarized to produce a usability score where the score close to 1 indicates good usability of the website or app. ²⁵ Overall, the heuristic evaluation is a clear list of usability problems and with this information, a quick and informed change could be done to improve the experience. ¹⁷ The summary of heuristic evaluation is shown in Figure 3. For estimating the sample size required in the testing of a web interface, the rule of thumbs of 4 ± 1 or 10 ± 2 model is widely used to obtain a proportion of at least 80% of the problem discovery. ³⁶ A smaller sample size is crucial in comprehending the underlying problem in a user experience study and attaining respectable levels of problem recognition, according to experts in the field. ^{37 , 27 , 28 , 29 , 30} Hence, the total respondents for this study are 12 end users.

Figure 3.

Summary of heuristic evaluation.

Results and discussions

The study involves 12 respondents recruited among the Point Duty Unit of Malaysian Traffic Police from Kuala Lumpur Traffic Police Station. There were six male respondents and six female respondents with a mean age of 28.8 years (range: 24–37 years old). Screener test results suggest that all respondents possessed a device with Internet access and were online multiple times a day. This requirement is important so that the respondents involved in this study were familiar with the devices and the Internet. As noted in Table 1, the majority of the respondents were Malay (83.3%), married (58.3%) and acquired secondary education (75%). Other characteristics were 66.7% of them were a smoker and the ranks were distributed in a range from a Constable to a Sergeant. Respondents served at least 2 years for as many as 10 years as traffic police.

Table 1.

Characteristics of respondents.

Characteristics	n (%)
Gender
Male	6 (50)
Female	6 (50)
Ethnicity
Malay	10 (83.3)
Chinese	0
Indians	0
Others	2 (16.7)
Marital Status
Single	5 (41.7)
Married	7 (58.3)
Divorcee/Widow	0
Education
Primary	0
Secondary	9 (75)
Certificate/Diploma	3 (25)
Degree/Master/PhD/Professionals	0
Smoking
Yes	8 (66.7)
No	4 (33.3)
Rank
Constable	3 (25)
Lance Corporal	4 (33.3)
Corporal	3 (25)
Sergeant	2 (16.7)
Sergeant Major	0
Sub Inspector	0
Years in service as traffic police
2–5 years	8 (66.7)
6–10 years	4 (33.3)
More than 10 years	0
Total	12 (100)

To further understand their knowledge, the questionnaire inquired for data on awareness and understanding of air pollution. The findings were descriptively illustrated in Table 2. The findings revealed that most of them had knowledge of air pollution (91.7%) and all of them care about air pollution around their workplace (100%). The study also wanted to know if the respondents think that the air quality level at their workplace is satisfactory. It was noted that 58.3% were satisfied with the air quality level of 41.7% were not satisfied. The respondents also reported that the source of air pollution were traffic emissions (83.3%). When asked about the air quality index, however, half of the respondents (50%) show no knowledge of the air quality index proving the data communication on air quality is lacking. ¹⁵

Table 2.

Awareness and understanding of air pollution and health.

Question	Yes, %	No, %
Do you know about air pollution?	91.7	8.3
Do you care about air pollution around your workplace?	100	0
In your opinion, is the air quality at your workplace now is satisfactory?	58.3	41.7
Source of pollution in your workplace
Traffic emissions	83.3	16.7
Industrial emissions	66.7	33.3
Agriculture	25	75
Domestic emissions	75	25
Open burning	41.7	58.3
Incinerators	25	75
Shipping activity	25	75
Airport	33.3	66.7
Do you know about air quality index?	50	50
Source of information on air quality index
Department of Environment website	50	50
Newspaper	58.3	41.7
Radio	75	25
Smartphone application	75	25
Others	0	100
In your opinion, the right prevention of air pollution?
Proper waste management	91.7	8.3
Wear a mask or any PPE	100	0
Regular medical surveillance	83.3	16.7
Exposure monitoring via an easily accessed portable device	75	25
Are you provided with any protective equipment in the workplace?	66.7	33.3
Do you wear them while working?	58.3	41.7

Generally, the respondents obtained the information on air quality from the government (Department of Environment) website. For prevention purposes, all of the respondents agreed on wearing a mask is the best way to curb air pollution exposure. However, only 66.7% reported they were provided with masks at their workplace and only 58. 3% wore them at work. The respondents’ remark on the reason they decided not to wear the mask is due to the difficulty of wearing the mask when working, the constraint of interacting with the public while wearing the mask as they need to issue orders during their job. This is also reported by Jamil et al. ¹⁵

For the post-task questionnaire, the results were shown in Table 3. The mean score for the time taken to complete the task was below 2 min (range: 0.33–1.67 min) which is very good. The estimated time was 5 min and no respondents exceeded the estimated time. Most of the comments received from the respondents were optimistic and the tasks were graded as easy to perform.

Table 3.

Scenarios for the post-task questionnaire.

Tasks	Mean in minutes (SD)
Review home page	0.33 (0.52)
Search for PM2.5, NO2, O3, CO and temperature	1.17 (0.41)
Search for a date today and find 7 a.m. exposure	1.67 (0.82)
Search for a date today and look for PM2.5 reading, set by highest	1.67 (0.82)
Add a start date and end date, search for all parameters, save and download	1.50 (0.84)
Open the file and read	1.50 (0.55)

Eventually, after the tasks were completed by the respondents, questions about their view of the system were posed. As noted in Table 4, all of the respondents agreed that their knowledge of air pollution levels in their surroundings is increased. After recognizing their level of exposure, a majority (83.3%) of the respondents know the action to take. Relocating work areas frequently with colleagues to reduce exposure, switching to low-pollution routes to work, planning exercise concentrating on respiratory health and utilizing N95 facemasks while at work, under suitable circumstances, are just a few of the efforts that can be implemented.^38,39 Nevertheless, all of the respondents agreed they have more knowledge of the air quality around them and they believe this system can help to protect their respiratory health. The respondents also reported that the task was very easy to perform (100%).

Table 4.

Respondents’ perspective towards the system.

Question	Yes, %	No, %
Do you know about your level of exposure to air pollution after using this system?	100	0
After recognizing your level of air pollution exposure, do you know what action to take?	83.3	16.7
After using this tool, do you think you have more knowledge of the air quality around you?	100	0
Does it help you to protect your respiratory health as you work after using this system?	100	0

Table 5 shows the scores on PSSUQ which is ranged from 1 to 7 with 1 being most agreed and 7 being most disagreed. Score 4 is considered to be neutral. Based on Table 5, the mean scores of all the items in the questionnaire was in the range of 1.83–2.67 indicating that the respondents mostly agreed on the usability of the prototype. In Table 6, the scores based on Table 5 were further analysed according to the category as explained in the methods above. The results of the PSSUQ showed that the respondents were satisfied with the usability of the website and the apps (Table 6). The mean score of all the items according to category; (Overall: 2.33; SYSUSE: 2.25; INFOQUAL: 2.36; INTERQUAL: 2.172) showed a lower range which indicates better usability. The findings were supported by another study on the smartphone application of a decision support app for nurses to facilitate aging in place of people with dementia by Thoma-Lurken et al. ⁴⁰ which reported similar results. Nevertheless, to date, there is still a lack of usability reporting regarding user-centred application of real-time wireless outdoor exposure monitors. The usability of the website and apps is assumed to be successful as it involves the users in the final stage of its evaluation. A strength of this research is that the principles of user-centred design were upheld in the development phase.^40,41

Table 5.

Post-Study System Usability Questionnaire (PSSUQ) score among the respondents (website and apps).

No.	Variables	Mean scores a	Maximum	Minimum
	Website
1	Overall, I am satisfied with how easy it is to use this system b .	2.50	4	1
2	It is easy to use this system b .	2.17	3	1
3	I can complete tasks and scenarios quickly using this system b .	2.33	4	1
4	I feel comfortable using this system b .	2.17	3	1
5	Easy to learn using this system b .	2.00	4	1
6	I believe I can be productive quickly using this system b .	1.83	3	1
7	The system gave me an error message telling me exactly how to solve the problem c .	3.00	4	1
8	When I make a mistake using the system, I can recover easily and quickly c .	2.33	4	1
9	The information (such as notifications, on-screen messages and other documentation) provided by this system is clear c .	2.33	3	1
10	Easy to find the information I need c .	2.33	3	1
11	This information is effective in helping me complete tasks and scenarios c .	2.55	3	1
12	The organization of information about screen systems is clear c .	2.33	3	1
13	The interface of this system is pleasant d .	2.17	3	1
14	I like to use the interface of this system.	2.00	3	1
15	This system has all the functions and capabilities I expected it to have d .	2.50	4	1
16	Overall, I am satisfied with this system d .	2.00	3	1
Smartphone Application
1	Overall, I am satisfied with how easy it is to use this system b .	2.17	3	1
2	It is easy to use this system b .	2.33	3	1
3	I can complete tasks and scenarios quickly using this system b .	2.33	3	1
4	I feel comfortable using this system b .	2.50	3	1
5	Easy to learn using this system b .	2.33	3	1
6	I believe I can be productive quickly using this system b .	2.50	4	1
7	The system gave me an error message telling me exactly how to solve the problem b .	2.67	4	1
8	When I make a mistake using the system, I can recover easily and quickly b .	2.50	4	1
9	The information (such as notifications, on-screen messages and other documentation) provided by this system is clear b .	2.50	3	1
10	Easy to find the information I need b .	2.17	4	1
11	This information is effective in helping me complete tasks and scenarios b .	2.67	4	1
12	The organization of information about screen systems is clear b .	2.33	4	1
13	The interface of this system is pleasant d .	2.33	3	1
14	I like to use the interface of this system d .	2.17	3	1
15	This system has all the functions and capabilities I expected it to have d .	2.17	3	1
16	Overall, I am satisfied with this system d .	2.33	3	1

^a Rating (range: 1–7); 1 (good)–7 (bad).

^b System Usefulness (SYSUSE): the average scores of questions 1–6.

^c Information Quality (INFOQUAL): the average scores of questions 7–12.

^d Interface Quality (INTERQUAL): the average scores of questions 13–15.

Table 6.

Results of Post-Study System Usability Questionnaires (PSSUQ).

Scores on PSSUQ	Mean score; Range 1–7 Lower scores indicate better usability
Overall satisfaction	2.33
System Usefulness (SYSUSE)	2.25
Information Quality (INFOQUAL)	2.36
Interface Quality (INTERQUAL)	2.17

Conclusion

The study suggests that the end user is satisfied with the prototype (wireless outdoor individual exposure indicator system). The PSSUQ results showed that the mean score of the Overall satisfaction item is 2.33, which is close to 1 as a successful usability evaluation score, indicating that the prototype users are satisfied with the system as a whole. As a result, a usable website and smartphone application for Malaysian Traffic Police working in Point Duty Unit, which helps to effectively monitor their exposure to air pollution are available. However, additional development and research are needed before the actual implementation in practice should be considered. This is to strengthen the implementation of the prototype (wireless outdoor individual exposure indicator system) while traffic police are working in terms of mobility and efficacy.