From luxury to necessity: Progress of touchless interaction technology

Abstract

Touchless Technology is facilitating the move to Zero User Interface(UI) propelled by the COVID-19 pandemic which has accelerated the use of this technology due to hygiene requirements. Zero UI can be defined as a controlled interface that enables user interaction with technology through voice, gestures, hand interaction, eye tracking, and biometrics such as facial recognition and contactless fingerprints. Smart devices, IoT sensors, smart appliances, smart TVs, smart assistants and consumer robotics are predominant examples of devices in which Zero UI is becoming increasingly adopted. These control interfaces include natural interaction modes such as voice or gestures.

Touchscreens and shared devices such as kiosks, self-service counters and interactive displays are present in our everyday lives. Each of these interactions however is a concern for consumers in a post-COVID-19 world where hygiene is of utmost importance.

The one-stop solution to hygienic interactions includes touchless technology such as voice control, remote mobile screen take over, biometric, and gesture control as Zero User interfaces. With the breakthroughs in image recognition and natural language processing, powered by advanced computer vision and machine learning, “Zero UI” is becoming a new normal. This paper is focusing on the progress of the touchless interaction technology during the COVID-19 pandemic, which actually accelerated development in this concept and moved it from being a luxury to a life necessity.

1. Introduction

When we talk about “interactive”, the use of touch screens automatically comes to mind but this concept has evolved in the last few years and over the pandemic to interaction with gestures, motion sensors, hand interaction and screen take over with personal devices. These types of touch-free interactions were already with us but considered as a luxury.

What is Zero User Interface (UI) Zero UI means interaction with machines without interacting with a screen or physical buttons. It allows a user to be free of traditional Graphics User Interfaces (GUI).

Zero UI technology is emerging very fast in every field of life. From automatic doors, toilets, touchless sanitizing dispenser [1], faucets, and hand dryers to Apple's Face ID [2] and Visa's Tap to Pay, companies have been pursuing innovative ways to make public places more convenient and more hygienic for years. In last two decades, touchscreens dominated the world but the COVID-19 crisis has significantly dented the public's demand and trust in shared touchscreens. As a precautionary measure, the need to avoid touching devices & surfaces at public and shared places has become necessary to stop the spreading of the disease. Touchless technology gained an expanded recognition during this time and it became very easy to understand the need of this technology in digital interactions and self-service places from vending machines [3] to identity checks at airports [4]. Touchless thermometers are the most widely used examples of touchless technology in the COVID-19 pandemic to stop the entry of affected people at access points.

From gesture control to virtual keyboards [5], these individual solutions have helped retailers, museums and other businesses to adapt to the demands during COVID-19 and will continue to dominate in future years.

These solutions have created the foundation of growing demand for touchless environments in the modern world but actually these solutions weren't designed for health and safety. For example, RFID was actually designed for asset tracking, identification of animals, and proximity cards such as IDs [6]. However, these touchless solutions need to be chosen correctly for their implementation to be useful, as RFID may be touchless but is only accurate to few metres. Precise movements require technology such as a gesture recognition and motion detection which are more appropriate examples as automatic touchless soap & sanitizing dispensers becomes more popular, convenient, and easier to use when compared to their manual counterparts.

Section 2 of the paper is presenting different types of software solutions to enable touchless interaction. Section 3 presents the touchless technology progress during COVID-19 using the software and hardware solutions provided in section 2. Section 4 is discussing the potential future opportunities and prospective use of touchless interaction in different domains of human life.

2. Types of touchless (contactless) interfaces

Touchless technology is defined as a type of interaction which does not need physical touch to operate. Different back to work strategies are being developed using IOT devices which allow for avoiding touch to conform to new pandemic related hygiene policies. There are different types of touchless interfaces that have been commonly employed to avoid touching digital surfaces;

• ●Screen Takeover, enabling users to control public screens with their mobile phones
• ●Live AR, enabling virtual interaction with Augmented Reality in the real world [7].
• ●Voice Control, interaction with virtual assistants using voice interaction
• ●Reactive Display, systems using motion sensors to enable interaction
• ●Eye Tracking, enabling interaction with screens using eye tracking technology

Fig. 2 explains the types of touchless technologies with potential use in different industries.

Fig. 2.

Types of Touchless technologies and their potential uses in different industries. This diagram is created based on the existing software and hardware solutions to enable touchless interaction.

As shown in Fig. 1 , at the very beginning of the pandemic, the touchless sensing market was projected to reach USD 15.3 billion in 2025, from USD 6.8 billion in 2020 at a CAGR(Compound annual growth rate) of 17.4% while the gesture recognition market will grow from $9.8 billion in 2020 to $32.3 billion in 2025 [8]. Since the COVID-19 pandemic is still prominent in several countries across the globe, the need for touchless technology is more important than ever and even in the future such adoption will provide hygienic digital interfaces going forward.

Fig. 1.

Projected 5 year increase in the Touchless sensing and gesture recognition market analysis [8].

From different use cases of contact free interfaces in healthcare and the travel industry, this adoption has started to gain traction. Keeping aside the basic requirement to track humans in some manner, touchless gesture technology concept can vary quite widely: starting from simple gesture tracking to advanced real-time hand interaction. Commonly, it is considered hand interaction technology is the same as gesture recognition technology but technically these are totally different approaches. Touchless technology varies depending on the input source. The range of different touchless technologies interaction is explained in Table 1 .

Table 1.

Types of Technology/Devices, working range and use cases.

Technology/Device	Detection Range	Use Cases
Infrared	up to 10 m (30 ft)	Doors, elevators, lavatories, health
NFC	10 cm	Payments
Personal Device	Depends on Wifi Range	Public Screens, smart homes
Leapmotion	25–600 mm (1 inch–2 ft)	Education, Health
Microsoft Kinect	1.2–3.5 m (3.9–11.5 ft)	Education, Health
QR Code	Depends on size of QR code	Kiosks, Payments, Screen takeover

2.1. Personal devices for touchless interaction

Touchless interaction is possible by using personal smartphone devices to allow for scanning QR Codes [9] for taking over a screen, connecting by Bluetooth and to pay using NFC.

Using mobile devices for proxemic interactions can help to stop the spread of infections [10]. This touch-free experience is actually very simple and intuitive to get control over screens without the need of downloading any applications. Many businesses are considering replacing interactive touchscreens with a touch free interface to provide a better customer experience.

2.2. Gesture recognition & hand interaction

Hand gesture is a well adopted touchless interaction allowing humans to interact with machines and it is the next level in the evolution of motion-sensors. Devices like Leapmotion, kinect, Azure kinect and deep learning based developments like Google Mediapipe and Manomotion provide the ability of gesture based touchless interfaces or more enhanced real-time hand interaction.

2.3. Motion sensing

Use of motion sensors as touchless interaction is the oldest touchless technology which is the most common and indeed it is the most affordable touchless technology. These sensors detect if a person is occupying a precise space. Motion sensing allows touchless interaction with different types of infra-red sensors where sensors detect humans to activate the system operation like trigger automatic doors, hand dryers, taps, and lights. Table 1 explains the range of these sensors.

2.4. Eye/gaze tracking

Eye tracking is the process of measuring the motion of eyes with eye trackers. Eye tracking technology is not new, it has a long history and now it is an important tool in many domains. Use of eye tracking as touchless technology adopted in Tobii Rex using a pair of infrared sensors to track the user's eyes to allow interaction with computers like with a mouse cursor [11].

2.5. Touchless typing

Touchless typing allows typing without touching the keyboard or keypad. It is mostly known as a special assistive technology to enable operation of standard electronic equipment or typical computers. Just like Airwriting technology that allows writing text messages or composing emails by writing in the air [12].

2.6. Contactless payments

Contactless card payments using Radio Frequency Identification (RFID) technology and use of Near Field Communication (NFC) technology allows contactless payments with the kiosks, exchanging digital contents and shopping centers [13]. As a better security option, it works within a 10 cm radius which is makes it more ideal for secure data exchange. Like a teacher logging on using NFC card or making payment with Google Pay which creates a contactless interaction without touching a shared or public device.

2.7. Voice control/virtual assistant

Vocal instructions for technologies like Amazon Alexa, Google Home or Apple's Siri are getting popular very rapidly. Voice recognition is also a perceptual user interface, based on NLP, which enables a machine or program to recognize spoken language and, as a result, understand and carry out voice commands. The driving force behind customized voice assistants is creating a more accessible, branded experience for consumers in healthcare and the auto industry. Voice and gesture control are applicable mostly in different fields in response to specific tasks requiring their use.

3. Touchless interfaces progress during COVID-19 in different industries

In every field of life where public interaction is required such as in health, education, workplaces, travel and social life, touchless technology has gained attention as a potential future solution during the COVID-19 pandemic.

3.1. Touchless technology in workplaces

Digital transformation of workplaces has a long history which rests upon whether there is an adoption or resistance of new technologies [14]. COVID-19 has changed the work environment and even when the pandemic is over it will never be the same as before. Touchless technologies are taking a front and central role in the new normal in the workplace across the globe. In the past, these technologies were considered as a way of embracing the future in the workplace but now with COVID-19 they are a necessity for the creation of safe shared work spaces. The contactless or touchless technology in the working place reduces the risks of spreading germs by touching contaminated surfaces to reduce sickness and increase productivity.

Touchless technology will be an important component of digital ecosystems of tomorrow. At the workplace there are several situations where avoiding touch for employees can effectively control the spread of germs.

Using shared devices as revaan access control solution has been an important part of the security industry for years in revaworkplaces. Biometric access was widely adopted before COVID-19 but now it has been proven as a major way of transmitting infectious diseases through touch-enabled biometrics system [17] which has created a new demand for contactless biometric [18]. Touchless Fingerprints [19,20] in Fig. 3 and touchless fingerprint recognition system for smartphones [21] are the new innovations in biometric authentication to avoid touch in workplaces at access points.

Fig. 3.

(i)Touchless biometric finger scan [15] (ii) Hand scanning as access control to workplaces [16].

Water dispensing units at workplaces and schools are one of those points which can easily transmit germs via contact. Fig. 4 shows the touchless technology used to innovate the water filling stations for eliminating contact possibilities with the machines. It is using infrared technology to completely remove the need of touching the machine, by activating it with a simple hand wave.

Fig. 4.

(i) Mechanical way of avoiding hand touch in water dispensing [22] (ii) Sensor based fully touchless water dispenser [23].

Elevators are commonly used in the workplaces to move between floors. According to research the number of bacteria on an elevator is 40 times more than it on a toilet's seat [24]. This research explains clearly the need for touchless technology for elevators for a more hygienic building.

Fig. 5 shows the touchless interaction mechanism installed in the elevators to avoid touch and enable button presses with pointing fingers without physically touching the buttons. This concept is equally important for the lift operations in shopping centers, educational institutes, hospitals and other places where people use the elevators. Another study tested the design of an Infrared light field solution to provide a touchless interface for elevator [27]. At a workplaces, the first place people touch is an entry door. Singapore based stuck design's ‘kinetic touchless 2.0’, the door uses motion as feedback, and as the user's hand moves towards it, the system is activated and responds with a slider interface.

Fig. 5.

(1) Concept of touchless elevator button (2) STUCK Labs Kinetic Touchless Elevator push button [25] (3) Concept of touchless door sliding using sensor technology in the workplace [26].

A touchless mask dispensing machine introduced by Cypress Vending (shown in Fig. 6 ) which can be mounted on a wall or stand, and can be branded to fit the needs of the space [28]. This mask vending machine prevents the spread of COVID-19 by being completely touchless. It provides contactless payment as well using Tap Pay, Debit/Credit Cards, Apple and Google Pay [29]. This is also a good case study to provide masks at work place entrance for employees, similar to touchless sanitizing stations (see Fig. 6). To facilitate a safe return to work strategy, companies are adopting touchless technologies to lower the health risks in future and creating seamless experiences for employees and visitors.

Fig. 6.

Touchless Mask Station (i) With coins as payment (ii) with contactless payment [30].

3.2. Touchless technology in daily life

In a daily routine life, we share many hard surfaces in public which are a definite source of germ spread and provide a tempting target for technology developers to enable touchless interaction in these daily life usage places.

Withdrawing money from an ATM using a card is an obvious example, along with any use of a fingerprint to be used a biometric interface which can spread germs [31]. The concept of "wave and pay" contactless payments have gained massive adoption during the COVID-19 pandemic. Most banks around the world extended the limits over single tap spending using contactless cards. In the United Kingdom, contactless card payment limit increased from £30 to £45 at the start of the pandemic in April 2020 and further increased to £100 in October 2021 [32].

Since facial recognition and fingerprints already have multiple security issues even before new threats like COVID-19 are included, the idea of smartphone based interaction with the ATM is a logical route to explore (see Fig. 7 ). By using a mobile phone to access an ATM, you can allow for a full security layer between the bank application and ATM [35].

Fig. 7.

(i) Cardless ATM machine, providing touchless interaction with smartphone by scanning a QR code to establish connection [33] (ii) Amazon One palm detection as touchless technology for access control [34].

Amazon introduced Amazon One (see Fig. 7) which is a contactless, convenient way to use the palm of hand to pay at a store, use it as a loyalty card, and get access to different locations more effortlessly without any physical contact. This is an example of making retail stores more friendly for the customers while keeping the health safety measures. The palm recognition technology has an edge over the face recognition and fingerprints as it is more private and not used as official credentials in documents.

Moving vending machines to a touch free interface to cope with the “new normal”, COKE introduced touchless interaction for their vending machines controlled by smartphone with a QR code scan without downloading any app (shown Fig. 8 ). As a similar approach by KFC, touchless technology developed using depth sensors to provide hand interaction with the screen interface of their ordering machines (see Fig. 8). Touchless vending machines work with smartphones through QR code scanning or use NFC as touchless payment technology to keep businesses operating with safe options for their customers.

Fig. 8.

(i) Coke's touchless machines serving chilled drinks [36] (ii) KFC Touchless Kiosk [37].

3.3. Touchless technology for travel industry

COVID-19 was the first pandemic that resulted in the halting of international air travel. In the worldwide airline industry, initial expected losses due to COVID-19 pandemic were 113 billion USD worldwide in 2020. This is around half of the 5-year (2015–2019) cumulative profit of the whole airline industry of 269 USD billions (see Fig. 9 ).

Fig. 9.

Air Traffic comparison between 2019, 2020 and first half of 2021; decline started in March 2020 when COVID-19 pandemic declared as a global pandemic [FlightRadar24].

The airport journey includes surface access to/from the airport, check-in, security, shopping, eating or drinking, getting information, finding terminals, passport control, departure, and arrivals. Touchless or contactless technology can address the pathogen and virus transmission issues during traveling at these access points [38]. It has proven to be a game-changer for the travel industry. The different areas of travel going touchless include:

• ●Vending machines
• ●Kiosks and terminals
• ●Touchless check-in and baggage drop. Biometric recognition eliminates the need to scan boarding passes [39]. Using Elenium's BagDNA, recognizing bags without the need to use any form of bag tags.
• ●Touchless airport security involves advanced body scanners, as well as bag scanners that remove the need of opening bags and placing items on trays.
• ●Touchless shopping where you order via apps, digital vending machines and virtual ‘shopping walls’.
• ●Seat back pockets for menu and TV screens to make them accessible digitally via smartphones using QR codes.
• ●Self-cleaning lavatories such as using ultraviolet lights to disinfect and touchless soap, water and tissues dispensing.
• ●Upon arrival passengers can expect to see touchless immigration and customs procedures

COVID-19 has provided the foundation for developing a contactless airport experience and will play a big role in the aviation industry. From touchless boarding to security screening with facial-recognition, this technology has become a necessity of the travel industry.

The role of biometric technology is crucial in enabling touch-free operations. Emirates airline introduced self check-in (Fig. 10 ) and bag drop for passengers to eliminate the surface touch and provide a contactless journey [40]. This strategy has been adopted by others like Etihad Air, Air Asia etc.

Fig. 10.

(i)Self-checking at airports by Emirates Air [40] (ii) AirAsia self check-in facility [41].

From entry gate to taking a seat, air travel is now moving to a touch-free transformation, this includes the airplane bathroom. As shown in Fig. 11 , a fully touchless lavatories system to keep the passengers safe from germs. Qatar Airways offers touchless technology for Oryx One inflight entertainment system (IFE) [42]. This Zero-Touch technology enables passengers to pair their smartphones with their IFE screen by simply scanning a QR code displayed on the seat-back screen (Fig. 11).

Fig. 11.

(i) Zero-touch IFE to pair smartphone with IFE screen [42] (ii)Touchless lavatories for plane [43].

Push buttons for pedestrian crossing on the roads have wide exposure to germs which can be easily transmitted to other peoples when touching the button throughout the day. The SmartSense touchless push button uses InfraRed technology to activate a crosswalk by simply waving a hand. Using a touchless approach, it helps to reduce the risk transferring germs spreading through touch. Trails of touchless push button pedestrian crossings were run by the Victorian government with initial tests in Melbourne's Royal Women's Hospital and the Royal Melbourne Hospital [44].

In addition to the MagicBand [48], Disney World has introduced a convenient and contactless new access option using mobile phone, MagicMobile to access park, parking points and hotel rooms as Fig. 12 ) shows. To measure the customer experience hygienically, a touchless feedback collection device was introduced where customers can rate their experience without any contact with the device. By holding the hand in front of the chosen answer (emoji), customers can register their feedback.

Fig. 12.

(i) SmartSense touchless push button, helping pedestrians to activate button by waving hand [45] (ii) Disney World's MagicMobile allows visitors to unlock their Disney Resort hotel room doors, enter the theme parks, and make food and merchandise purchases in a contactless way [46] (iii) Zero UI traveller feedback collection system at airport [47].

3.4. Touchless technology in health

With the start of COVID-19, healthcare systems of many well developed countries were completely overloaded for a period. With the shortages of essential resources and patient overflow became one of the most prominent news stories, along with the lack of personal protection equipment for doctors and nurses. The growing use of technology in hospitals and use of shared computerized resources has increased pathogens spreading opportunities which leads to higher demand of Zero UI [49].

COVID-19 has affected the whole operational facilities in healthcare systems which created the need for architects and designers to focus on redesigning of healthcare spaces for heightened cleanliness and contactless environments. The world has seen use of touchless infra-red temperature check becoming the most prominent health care change with COVID-19 and is now becoming a daily sight for many. These innovations will continue to happen in the healthcare industry as it is just a starting point as the need for Zero UI technology has been so clearly demonstrated to the world.

To reduce the need for touching devices in a healthcare setting, a touchless respiratory monitor was developed to measure respiratory flow. With a remote thermal sensor and LiDAR measuring temperature, it can reduce the need for directly touching the patient [50] (see Fig. 13 ). By using remote sensing, a touchless system was introduced for early detection of eye diseases to control blindness [51], however its main purpose is early detection of eye diseases instead of sanitation measures. Similar studies were conducted for Contactless Sleep Monitoring [52], Contactless analysis of heart rate variability [53] and Early Diagnosis of COVID-19 [54]. As this progress will continue on its way even after vaccination, educating healthcare customers is important for medical professionals to know about how and where touch free interaction can be applied to allow for an increase in protection of patients and safety of what can be costly equipment [55]. The telemedicine concept [56] is the way to minimize human contact in the health industry. As with new innovation and access to technology, the global telemedicine market is expected to reach 175.5 billion USD by 2026 from 45.5 billion USD in 2019 [57].

Fig. 13.

Touchless Respiratory Monitor, LiDAR and thermal sensor are completely touchless and are fixed to the bed about 1 m above the subject [50].

3.5. Touchless technology in education

Education is one of those fields which was affected massively due to the complete shut down of education institutes. The strategies to allow for schools to safely return are needed, as there needs to be a redesign of most classroom settings where touchless technology can play a role in solving the new hygiene policy requirements in the new normal [58]. The use of sensor technology in mitigating COVID-19 can help in the future to minimize human contact with potentially contaminated surfaces and objects in educational institutes [59]. This raises the question of a potential new goal in the future where completely touch free interactive learning experiences are possible for students.

One possibility is Augmented Reality(AR) which provides touchless interaction using leapmotion [61], Microsoft kinect [60], azure kinect [62] and latest SDKs as described in section 2.2. It allows for embodied digital learning apps with motion and hand tracking cameras which enables interactions with 3D learning material like a remote laboratory [63] and allows for the avoiding of touching physical learning materials.

In collaborative learning, Augmented Reality with hand gestures and real-time hand interaction create possibilities for touchless learning environments to be achieved. Fig. 14 has listed two case studies of touchless technology reportedly used during the COVID-19 which avoid the user from interaction with the digital devices and surfaces.

Fig. 14.

Use of touchless technology using kinect for children education [60] (ii) Using leapmotion touchless interaction hand for learning PC assembling [61].

3.6. Touchless technology & accessibility

The advancement in human-computer interfaces creates a digital divide when they are not fulfilling the needs of people with physical disabilities. Touchless technologies in all forms, provide options for better accessible interfaces for those people who are suffering with physical disabilities. Touchless technology in the form of gaze tracking and use of voice commands have been shown to allow for a higher level of accessibility [64] than other approaches. These innovations should rightly be celebrated, but it is possible that future touchless technologies could create barriers to people living with physical disabilities. Innovations in this field must always be gauged on their effects in terms of accessibility.

4. Future opportunities

The COVID-19 outbreak has exposed the underlying threat of pandemics in our globalised connected world but has also created historic opportunities for developing a touch-free technology infrastructure to address these threats. Touchless technology is actually much more than just a quick fix. It has changed the world's perspective to touchless technology and dictated a new demand for its development, especially in the service industry. By combining our latest developments in sensor technology and artificial intelligence, the future of touchscreen technology will be touchless.

All types of touchless technologies including IoT sensors [65], gestures, real-time hand interaction [66] and NFC will be part of that future. Seeking the best interaction will be the developers goal and so regardless of the differences between technologies, the benefits of improved safety and convenience using touchless technology will become standard across all industries. For ubiquitous touchless access, technologies will still be under constant development, for example, research on crafting millimeter wave solutions for touchless systems promises to provide large performance benefits compared to current systems. As the touchless trend continues to surge, now is a fascinating time to explore touchless technology options and build a strategy for the future. Eye tracking technology is not much explored yet as touchless interaction technology [67], but it can be a potential option for creating touch free interfaces. These contactless technologies will promote smart cities infrastructures in the near future including touchless fingerprints, iris, and face recognition for identity, temperature sensing, and access control.

It is time to turn towards new solutions as the touchless technology is an evolving process. The innovations in the eye tracking technology and effective use of eye patterns can bring new interaction techniques as standard features in new generation smart devices for human communication. With the recent progress in hand tracking technology with deep learning, touchless typing can become more realistic without the need of wearing any sensors. High initial cost involved with touchless technology products and their implementation is the major bottleneck with adoption but it is decreasing with time. The other major resistance is from the user side, but COVID-19 has helped people to understand the need and most of us understand why avoiding touching in a public place is important.

5. Conclusion

Touchless technology as a rapidly emerging technology has lots of growth opportunities with increasing adoption. This paper provided an overview of recent development progress in touchless technology as Zero UI technology and outlined the crucial role that it will play in our future interactions with digital devices. With the rise of concepts such as Zero UI and ubiquitous computing where every object around us could potentially be have some form of computational capability, at a fundamental level it will change people's perceptions of how they interact with technology. This effect will make our current reign computational paradigm of smartphones and laptops look like a weird form of interactive purgatory as they will appear just as archaic as the large mainframes and dumb terminals of the mid to late 20th century. COVID-19 crisis has created an awareness about the potential invisible risks linked with physical touch points and the goal of a touch-free interaction has become paramount for all major industries where human interaction is involved. The COVID-19 pandemic did not create this situation but it has pushed these concepts that were seen as an increasingly luxurious way of interacting with computation devices in “The Before Times” to now a necessity in our new post-pandemic world.

Author statement

Both authors of this paper have contributed equally in all steps, including participation in the conceptualization, developing methodology, design research, data analysis, writing, proofreading or revision of the manuscript. Authors also certify that any of the material of this paper has not been submitted anywhere else for publication.