Smile analysis in dentistry and orthodontics – a review

ABSTRACT

The desire for an attractive smile is a major reason people seek orthodontic and other forms of cosmetic dental treatment. An understanding of the features of a smile is important for dental diagnosis and treatment planning. The common methods of smile analysis rely on the visual analysis of smile aesthetics using posed photographs, and videos and gathering information about smiles through patient questionnaires and diaries. Recent emerging trends utilise artificial intelligence and automated systems capable of detecting and analysing smiles using motion capture, computer vision, computer graphics, infrared and thermal imaging, electromyography, and optical sensors. This review aims to provide an up-to-date summary of emerging trends in smile analysis in dentistry and orthodontics. Understanding the advantages and limitations of emerging tools for smile analysis will enable clinicians to provide tailored and up-to-date treatment plans.

Introduction

Smiling is one of the standard day-to-day facial expressions governed by the underlying mood state and social environment (Fridlund 2002). Individuals often observe smiling differently from other facial expressions, whereby authentic, spontaneous smiles are perceived as channels for effective communication, happiness, and friendliness (Krys et al. 2016). Healthy smiles are important to the well-being of a person and those they interact with. The desire to ‘improve’ one’s smile is often why many individuals seek dental advice (Messinger et al. 2001; Pabari et al. 2011). Dental professionals have a considerable understanding of what constitutes a healthy smile (Pisulkar et al. 2019).

Smiling is a complex facial expression that can include sparkling of the eyes and curving of the mouth corners (Hertzberg 2009). The major types of smiles may be summarised under three broad categories: enjoyment smiles which comprise smiling with eye constriction, play smiles which involve smiling with an open mouth; and Duplay smiles defined as smiles exhibiting eye constriction as well as open mouth (Messinger and Fogel 2007). On the other hand, some focus has been placed on understanding spontaneous, genuine smiles, which depict happiness. These types of smiles are often termed ‘Duchenne’ smiles, acknowledging Duchenne de Boulogne, a famous neurologist who first drew attention to the anatomophysiological patterns associated with such smiles (De Boulogne 1990). In contrast to non-Duchenne or posed smiles, genuine smiles are often recognised by their simultaneous concomitant activation of two major facial muscles; the zygomaticus major and the orbicularis oculi, pars orbitalis muscles (Ekman et al. 1990).

The interpretation of a smile is intricately woven into the fabric of societal context (Thibault et al. 2012). In ‘non-smiling’ cultures, smiles may be viewed with suspicion or scepticism and potentially interpreted as insincere or masking hidden motives (Krys et al. 2016). Conversely, ‘smiling cultures’ in the United States, for example, utilise frequent smiles as social lubricants, expressing positive affect and acknowledging others (Talhelm et al. 2019). Even the intensity of a smile can hold cultural nuances, and sharing experiences with close friends often evokes pronounced smiles compared to solitary encounters (Jakobs et al. 1999). Similarly, research has shown that context shapes the perceived sincerity (Zaalberg et al. 2004). Participants in that study readily offered genuine Duchenne smiles to funny jokes but held back on ‘polite smiles’ for dull humour, highlighting the dynamic nature of smile interpretation. Another study found that Americans judged wider smiles as more intelligent, while Japanese preferred more subtle expressions (Matsumoto and Kudoh 1993). This showcases the diverse perspectives cultures hold on nonverbal cues. In addition, specific patterns in simultaneous smiling behaviour could be observed among American English and French speakers, noting that French individuals tend to exhibit more synchronised smiling during conversations (Priego-Valverde et al. 2018). Furthermore, Loan et al. (2016) demonstrated how Tokelauan people in New Zealand utilise smiles as a cultural mask to conceal negative emotions and maintain social harmony. These diverse examples underscore the multifaceted nature of smiles and the importance of cultural awareness when interpreting their meaning.

Understanding the quantitative and qualitative characteristics of smiles is therefore important. This information can be gathered through methods such as visual identification, self-reports, and automated systems using motion capture, computer vision, computer graphics, electromyography, infrared and thermal imaging, and sensor-based technologies. The purpose of this review is to introduce the audience to various contemporary mechanisms and recent trends in smile analysis within the literature. Additionally, this review aims to provide clinicians and academics with an up-to-date understanding of the various tools available for smile analysis to help guide treatment planning and decision-making.

How can a smile be assessed?

Visual observation

Visual assessment of smiles relies on the subjective analysis of smile characteristics from direct observation, facial photographs, or video recordings. This method is one of the most utilised techniques reported in the literature (Ambadar et al. 2009). However, the measurement of certain defined smile attributes could make this method more systematic. Clinicians often examine the aesthetic features commonly associated with smiles such as the extent of buccal corridors, amount of gingival display, morphology of the dentition, dentofacial symmetry, midline discrepancy, curvature of smile arc and inclination of anterior teeth amongst others (Machado 2014). Clinicians generally rely on posed photographs and software such as Adobe Photoshop to formulate their treatment plans (Al-Johany et al. 2011; Omar and Duarte 2018). The perceived attractiveness of buccal corridors has been explored using static photographs. Digitally enhanced images of smiles with or without buccal corridors were assessed for perceived attractiveness among dental professionals. Lay personnel and dentists had a significant preference towards broader arch forms displayed during smiling (Roden-Johnson et al. 2005). Excessively wide buccal corridors, gingival display exceeding 2 mm and midline diastemas were all regarded as unattractive by both laypeople and dentists (Abu Alhaija et al. 2011).

Other information can also be derived from the direct visual examination of smiles. For instance, one study investigated age-related changes in older participants and found that they showed fewer teeth during spontaneous episodes of smiling (Van der Geld et al. 2008). It is also expected that individuals with positive overjet would have noticeable changes in the curvature of their smiling line after orthodontic treatment (Mah et al. 2014).

Videographic recordings reproduced via a three-camera setup could capture the dynamics of a smile in high-definition from three different angles (Husain et al. 2017). Research utilising full-frontal videos of participants posing for a smile has demonstrated that midline diastemas are perceived as unattractive (Chaves et al. 2021). Furthermore, the assessment of video clips of participants with malocclusion highlighted significant changes in gingival display during posed and unposed smiling episodes (Rao et al. 2021).

The above methods are only successful in providing basic information about smiles (Kang et al. 2016). Smiling is a complex facial expression and requires an objective and robust method of assessment. For instance, the Facial Action Coding System (FACS) is a method for coding different facial expressions (Ekman et al. 1978). Within FACS, the muscular components contributing to different facial expressions are designated as Action Units (AUs) (Gosselin et al. 2010). This method is labour-intensive and requires the experience of observer-based coders to recognise different AUs and translate them into equivalent facial expressions such as smiling (Yang et al. 2021).

Self-reports

Self-reports are designed to collect qualitative or quantitative data depending on the posed questions, and the nature of answers that are received. Qualitative methods are concerned with experiences and meanings gathered through interviews using open lines of questioning, dialogues and diaries (Castellan 2010). To appreciate the importance of qualitative methods, one study found that teenagers with malocclusion were ashamed to show their teeth when talking to others (Trulsson et al. 2002). However, not all teenagers are sensitive about their teeth when smiling (Twigge et al. 2016). Moreover, qualitative interviews have demonstrated that orthodontic patients in particular, experience anxiousness when requested to pose for a smile despite this seeming like a simple task (Çifter 2018). For those patients, changes in smiles could be associated with the existence of undesirable white spot lesions (Yadav et al. 2019).

Qualitative rich data derives alternate ways to harness data from social media and online platforms. Smiling can be seen to have a positive influence on social media and even as a channel to boost the job recruitment process on these platforms (Oikarinen and Sinisalo 2017). For instance, extracting and analysing Twitter statements revealed that specific dentofacial characteristics play a pivotal role in altering smiling behaviour to avoid bullying (Chan et al. 2017). Moreover, individuals with malocclusion expressed great concerns about displaying their photographs on social media platforms (Patel et al. 2016).

The inclusion of pre-determined questionnaires in smile research may indeed represent an efficient and straightforward method to measure how patients, clinicians, and the public view smiles. For instance, the confidence and consciousness of altered smile aesthetics, and the impact of smile aesthetics in a social context are sub-domains investigated in the Smile Esthetics-Related Quality of Life questionnaire (SERQoL) (Saltovic et al. 2018). Smiling has been shown to impact the oral-health-related quality of life of Spanish schoolchildren (Montero et al. 2016). These results are similar to a New Zealand survey which reported one-quarter of Māori teenagers hide behind their smiles (Broughton et al. 2012). In contrast, the smiles amongst adults can be measured via a short five-item Smile Aesthetics Satisfaction Scale (SASS) questionnaire which is a popular tool for dentists (Lajnert et al. 2018).

Questionnaire-based surveys are often used to analyse, categorise, and differentiate factors that influence various smile characteristics. Questionnaires could include visual analogue scales in which participants self-rate their smile satisfaction. In one study, participants were satisfied when their smiles showed all their teeth and some gingiva, and were dissatisfied when their smiles had asymmetrical gingival display (Van der Geld et al. 2007). Questionnaires focusing on the observation of buccal corridors during smiling revealed that orthodontists and prosthodontists regularly notice smiles with buccal corridors compared to lay people who are less critical (Pisulkar et al. 2019). Another study revealed that Japanese dental students were less tolerant of gummy smiles than orthodontists (Ioi et al. 2010). Other studies investigating the aesthetic appearance of smiles with recession showed that dental professionals were aware of these aesthetic limitations (i.e. the presence of recession) when they rated smiles on a visual analogue scale (Musskopf et al. 2013).

The future: artificial intelligence and automated systems

Manual assessment of smiles can be a daunting process and is: (a) prone to observer-related biases, (b) labour-intensive, (c) requires extensive training, calibration and experience, (d) and is time-consuming (Cohn 2010). The limitations above call for automated systems with reliable analysis of smiling features. There has been a recent paradigm shift to develop and incorporate automated systems in smile analysis. Despite the advances in Artificial Intelligence (AI) technologies, it is important not to disregard manual methods but instead consider both techniques complementary to one another.

Motion capture

Motion capture relies on recording the movement of individuals or objects in space (Menolotto et al. 2020). It uses key markers in recorded videos to simplify the process of computer vision (Mathis et al. 2020). However, the scope of motion capture is not just limited to placing key markers in recorded digital footage; it has far more exquisite features and sophisticated technology. For instance, one study suggested that smiling had a specific response on different vowels using 3D motion capture techniques (Fagel 2010).

Motion capture is also capable of analysing differences in smile symmetry and in particular those participants with unilateral cleft lip using markerless 4D stereophotogrammetrical technologies (Zhao et al. 2021). Participants with repaired cleft lips have shown significant differences in open- and closed-mouth smiles compared to controls using the 4D stereophotogrammetric systems (Seaward et al. 2022).

Computer vision

Computer vision is the science that is concerned with extracting data from visual inputs relying on traditional non-AI computational models as well as state-of-the-art AI systems, and then translating the data into meaningful information (Voulodimos et al. 2018). These computer vision techniques have been utilised by researchers in New Zealand contributing to the development of smile detection algorithms which can detect smiling faces with acceptable accuracy (Mohammed et al. 2023). Also, dental patients may now benefit from AdaBoost algorithms (i.e. adaptive algorithms in Machine Learning) that identify facial features. This allows self-smile training because patients can look over their smiles on a real-time basis (Song et al. 2012). The distinction between spontaneous and posed smiles utilising computer vision remains an area with continuous output in affective computing (Kawulok et al. 2021).

Automated detection of asymmetrical facial expressions may highlight subtle lip excursions and smirks during dynamic facial expressions with acceptable accuracy (Sénéchal et al. 2013). However, previous reports utilising automated recognition systems highlighted that these subtle asymmetrical differences in lip and eyebrow movements were insignificant when individuals deliberately and spontaneously smiled (Schmidt et al. 2006; Valstar et al. 2006).

The use of computer vision and affective computing has limitations. Identification of happiness based on smiling faces could be challenging when automated performance contrasts with human observers. Research has shown that automated recognition systems possess adequate identification capacity to detect and analyse the quantitative features of smiles (Mohammed et al. 2022). However, recognising complex or somewhat blended emotions is still a challenging task for AI (Del Líbano et al. 2018). To overcome this issue, fine-grained image classification techniques can be used to detect the subtle differences between blended emotions and categorise the different types of smiles (Xie et al. 2015). Other limitations include the lack of robustness in the digital recording process, whereby the occurrence of non-standardised conditions (i.e. head tilt, jerky movement, and changes in light conditions) could affect the ideal performance of the detection algorithms (Perusquía-Hernández 2021).

Computer graphics

During the 1980s, elementary computer graphics were developing, enabling dental professionals to better visualise their treatment goals (Vannier et al. 1983). In subsequent years, continued developments in this field gave rise to 3D and 4D (temporal) computer imaging graphics that can create animated facial models and further quantify subtle facial expressions (Popat et al. 2009).

Computer graphics are also used in conjunction with motion capture techniques. These combined techniques can produce animated 3D dynamic models of different facial expressions, including smiles, which can then be analysed for maximal smile intensity and speed (Shujaat et al. 2014). The 3D computer-animated graphics can also be used to examine the spatial and temporal characteristics of smiles such as smile extent and asymmetry (Helwig et al. 2017).

Infrared and thermal imaging

Thermal imaging is a technique in which temperature variations on the skin are recorded (Goulart et al. 2019). Nowadays, advancements in AI thermal feature extraction technologies have reached acceptable accuracy in identifying smiles using unique infrared analysis of skin temperatures (Khan et al. 2009). Differentiation between spontaneous and posed smiles would then rely on the accurate analysis of detected variations in temperatures, whereas those distinct thermal changes in different facial exertions would distinguish such variables (i.e. spontaneous vs posed expressions) (Liu and Wang 2012). Automated detection of smiles from footage obtained through infrared imaging technologies is also plausible through convolutional neural networks (Wu et al. 2020).

Electromyography

Electromyography (EMG) aims to recognise, detect, and analyse electric signals generated from contracting muscles (De Luca 2006). Muscular activity could be tracked with two main types of EMG: surface EMG and intramuscular EMG (Smith and Hargrove 2013). Research highlights that both types could be utilised simultaneously to examine muscular activity during smiling (Hochreiter et al. 2023). As such, some focus has been placed on understanding the characteristics of smiles in terms of their Duchenne and non-Duchenne behaviour (Inzelberg et al. 2018).

Within this paradigm, one EMG study reported that the electric activity of the zygomaticus muscle was highest during smiling by surface EMG (Schumann et al. 2010). Also, wearable EMG devices have demonstrated some success in differentiating spontaneous and posed smiles in individuals based on the characteristic temporal resolution of the electric signal (Perusquía-Hernández et al. 2017). In addition, a team of craniofacial researchers in New Zealand have developed a smartphone-assisted wireless wearable EMG device that is capable of measuring facial muscle activity and a cornerstone for analysing facial expressions using wearable devices in freely moving individuals (Prasad et al. 2019). Nevertheless, many shortcomings can be foreseen with EMG due to the obstructive nature of the electrodes. This issue can be minimised by localising distal wearable electrodes in areas with low mobility to detect smiling behaviour (Gruebler and Suzuki 2010).

Sensor-based technologies

Integrating optical sensors in regular eyewear to study facial expressions in real life is an emerging novel possibility. This would introduce wearable sensors that can detect spontaneous smiles in real-life situations. Optical sensors are able to identify eight facial expressions with nearly 93% accuracy when integrated into eyeglass frames (Masai et al. 2017). Optical sensors embedded into eyewear may also differentiate between spontaneous and posed smiles (Saito et al. 2020). On the other hand, photo-reflective sensors in head-mounted displays are able to map and transfer detectable smiling behaviour alongside other facial expressions to avatars (Suzuki et al. 2017). Progression with such technologies has introduced acoustic interferometry in smile assessment which utilises a set of ultrasonic transducers placed on the skin. They project sound interference patterns inside the body, creating interaction with the facial muscles to study smile intensity (Iravantchi et al. 2019).

Implications for clinical practice

The landscape of dental practice is undergoing a shift with the emergence of automated smile analysis tools. These technological marvels delve into the dynamic interplay of aesthetics and function, capturing the essence of a smile in its nuanced glory. Regarding patient engagement, smile analysis tools act as powerful catalysts, promoting AI-driven simulations and interactive smile visualisation. User-friendly interfaces enriched with aesthetically pleasing graphics and simulations, strive to involve patients in the smile design process. They can envision and manipulate potential treatment outcomes, fostering a sense of ownership and investment in a person's smile makeover.

Beyond engagement, smile analysis tools provide both patients and clinicians with valuable virtual data (Omar and Duarte 2018). For example, individuals who have access to smile analysis software on their smartphones may be able to self-detect a dental problem using these simple and easy-to-use technologies. For clinicians, these tools provide increased accuracy and efficiency which leads to superior diagnostic skills and treatment planning.

The influence of these tools reaches beyond the patient-clinician dyad. Personalised smile analysis reports can foster a deeper connection with the dental patient (Santi et al. 2020). When combined with knowledge about oral health and preventative care, they have the potential to motivate patients towards a healthier smile and indirectly reduce long-term treatment costs. A summary of these different smile analysis tools and their plausible advantages and disadvantages is presented (Table 1).

Table 1.

A summary of the different smile analysis methods.

Smile analysis tools	Description	Advantages	Disadvantages	References
Visual observation	Direct visual analysis of data collected through observation, photos, or videos	–Analysis of certain dentofacial smile aesthetics –Straightforward method –Inexpensive	–Require human expertise and knowledge –Limited accuracy	Kang et al. 2016 Machado 2014 Omar and Duarte 2018
Self-reports	Closed-ended questionnaires Open-ended questionnaires Interviews Diaries	–Simple and straightforward –Inexpensive	–No Artificial Intelligence involved –Limited to perceived data regarding one’s smile	Broughton et al. 2012 Castellan 2010 Lajnert et al. 2018 Van der Geld et al. 2007
Motion capture	Movement capture of objects or individuals using key markers	–Dynamic analysis of smiles in motion –Analysis of smile symmetry –Quantifiable smile data	–Limited scope for analysing static smile aesthetics –Specialised equipment –Visibility –Cost	Fagel 2010 Mathis et al. 2020 Seaward et al. 2022 Zhao et al. 2021
Computer vision	Data extraction from visual inputs using AI systems and non-AI computer vision techniques	–Accuracy of smile detection is continuously improving –Detection and analysis of smile symmetry –Straightforward method –Quantifiable smile data	–Affected by the visual input footage –Limited precision in the differentiation of spontaneous and posed smiles	Del Líbano et al. 2018 Kawulok et al. 2021 Mohammed et al. 2022 Perusquía-Hernández 2021 Sénéchal et al. 2013 Song et al. 2012
Computer graphics	3D computer graphics 4D computer graphics	–Realistic animation and visual display –Dynamic analysis of smiles	–Used in conjunction with other systems –Availability – still emerging	Helwig et al. 2017 Shujaat et al. 2014 Popat et al. 2009 Vannier et al. 1983
Infrared and thermal imaging	Thermal feature extraction Infrared imaging	–Differentiation between spontaneous and posed smiles –Dynamic smile analysis –Quantifiable smile data	–Limited value for analysing certain dentofacial smile aesthetics –Sensitive to temperature variations –Sensitive to lighting conditions –Cost	Goulart et al. 2019 Khan et al. 2009 Liu and Wang 2012 Wu et al. 2020
Electromyography	Surface EMG Intramuscular EMG	–Automated analysis of muscular activity during smiles –Differentiation between spontaneous and posed smiles –Real-time smile tracking – wearable devices –Quantifiable smile data	–Limited value for analysing smile aesthetics –Obstructive nature of electrodes –Visibility –Cost	De Luca 2006 Gruebler and Suzuki 2010 Hochreiter et al. 2023 Inzelberg et al. 2018 Perusquía-Hernández et al. 2017 Prasad et al. 2019
Sensor-based technologies	Optical sensors Photo-reflective sensors	–Differentiation between spontaneous and posed smiles –Real-time smile tracking – wearable eyewear –Transfer smiling behaviour to avatars	–Availability – still emerging –Visibility of eyewear	Iravantchi et al. 2019 Masai et al. 2017 Saito et al. 2020 Suzuki et al. 2017

AI: Artificial Intelligence: EMG: Electromyography; 3D: Three-dimensional; 4D: Four-dimensional.

Current issues and future research directions

While smile analysis tools hold huge potential, several challenges remain to be addressed. One critical concern is the bias inherent in some automated facial analysis algorithms. Their underperformance in accurately detecting and analysing facial features of individuals from diverse ethnic backgrounds has been investigated (Klare et al. 2012). This bias can lead to inaccurate diagnoses and inappropriate treatment recommendations, perpetuating existing health disparities. To address this, research efforts should focus on training algorithms with diverse datasets representative of numerous ethnicities and cultural backgrounds. This requires working together with diverse populations to ethically collect data.

Ethical considerations must be paramount throughout the process and include consultation from relevant committees to ensure informed consent and data privacy (Van Noorden 2020). Beyond ethical considerations, harnessing publicly available data offers exciting possibilities for smile research. Platforms such as social media and research databases hold vast amounts of facial expression data, potentially providing valuable insights into diverse smiles and their cultural context. However, utilising this data ethically and responsibly requires careful consideration. Researchers must ensure data anonymisation, respect individual privacy, and avoid perpetuating existing biases. Additionally, developing robust data analysis techniques that can extract meaningful insights from large and diverse datasets is important. This necessitates collaboration between computer scientists, data analysts, and dental professionals to develop ethical and effective approaches to utilising publicly available data.

An additional challenge lies in the limited availability and accessibility of advanced smile analysis systems. Many are still in development, with high costs hindering their widespread adoption in clinical practice (Silva et al. 2015; Jafri et al. 2020). To overcome this, academic and research institutions can play a crucial role by continuing development efforts and seeking alternative funding sources. Raising awareness among dental professionals about the benefits and potential use of these tools is essential. The increasingly competitive dental market, particularly in some regions, necessitates a transition toward digitalisation, and therefore smile analysis tools can be a valuable asset in this process (Jahangiri et al. 2020).

Conclusions

This review has comprehensively explored the evolving landscape of smile analysis in dentistry and orthodontics. Notably, advancements in AI and automated systems demonstrate promising potential for surpassing human accuracy in smile detection and analysis. This holds significant advantages for clinicians, offering data-driven insights to optimise treatment planning and patient outcomes. However, it is crucial to acknowledge that these technologies are still in their nascent stages. Further research is warranted to refine their accuracy, address potential biases, and develop user-friendly interfaces for seamless integration into clinical practice. By bridging the gap between cutting-edge technology and practical application, future research can empower dentists with new tools for enhanced smile analysis. This, in turn, ultimately benefits both clinical practice and patient satisfaction.

Disclosure statement

No potential conflict of interest was reported by the author(s).