A Prototype Mobile Application for Triaging Dental Emergencies

Abstract

Evidence suggests that dental emergencies are likely to occur when preferred care is less accessible. Communicative barriers often exist that cause patients to receive sub-optimal treatment or remain in discomfort for extended lengths of time. Furthermore, limitations in the conventional approach for managing dental emergencies prevent dentists from receiving critical information prior to patient visits. We developed a mobile application to mediate the uncertainty of dental emergencies. Patient-provided information accompanied by high-resolution images may significantly help dentists predict urgency or prepare necessary treatment resources.

The development and study consisted of a needs analysis and quality assessment of intraoral images captured by smartphones; prototype development; refining the prototype through usability inspection methods; and formative evaluation through usability testing with prospective users.

The developed application successfully guided all users through a series of questions designed to capture clinically meaningful data using familiar smartphone functions. All participants were able to complete a report within 4 minutes and all clinical information was comprehendible by the users.

Our results illustrate the feasibility of patients utilizing smartphone applications to report dental emergencies. This technology allows dentists to remotely assess care when direct patient contact is less practical.

This study demonstrates that patients can use mobile applications to transmit clinical data to their dentists and suggests the possibility of expanding its use to enhance access to routine and emergency dental care. Here, we address how we can enable patients to directly communicate emergent needs to a dentist while relieving them of enduring emergency room visits.

Introduction

The American Dental Association’s Principles of Ethics states “dentists shall be obliged to make reasonable arrangements for the emergency care of their patients”.^[1] Immediate assessments are vital for patients experiencing oral trauma, orofacial pain, infections, or similar conditions, however access to care is often limited by distance and time.^[2] Many dental emergencies occur during the weekend and evening hours when dental offices are usually closed ^[3–7]. With few options, many patients will seek initial treatment at medical emergency clinics where physicians provide symptomatic treatment and may not necessarily address the cause.^[5,8] Emergency departments have experienced a surge in patient visits with dental emergencies where the majority remain untreated and are referred to their dentist for proper care.^[2,9–12] As a result, unnecessary hospital costs accrue while patients remain in discomfort for extended periods of time. In dental practices, dentists must deviate from routine workflow when caring for patients with emergencies. To prepare for these, many dental practices employ telephone-based services to manage out-of-office emergencies. Although effective for notifying dentists of a problem, these services generally do not detail the urgency of a patient’s needs nor adequate information required to effectively estimate the time and resources for emergency care.^[13,14]

Teleconsultations using mobile technologies have proven to be a strong candidate for enhancing inadequate patient-provider communications in medical fields.^[15,16] Ubiquitous smartphone usage among most demographics has created opportunities for clinicians to deliver various health-related interventions in real-time using universally available capabilities.^[17,18] Since emergencies are often subject to delays in care, the smartphone may provide the ability to remotely triage dental patients prior to chairside assessments. For example, the high-resolution camera on smartphones permits visual interpretations of patient cases and avoids misleading verbal accounts native to telephone-based systems. The functions of smartphone cameras (autofocus, anti-movement, and white-balance) permit fine detail and color visualization in dental images.^[19] Transmitting these images appended with patients’ symptoms to a dentist could improve patient-provider communication in real-time and enable dentists to make more timely treatment decisions.

Dentistry has recently begun to explore mobile technologies for a variety of purposes^[20] as the smartphones’ capabilities for information capture and transmission continue to improve. To date, dental mobile applications have included various functions such as data collection,^[21] symptom checking,^[22] and photographing and transmitting high-quality dental images.^[19,23] Additionally, Namakian^[24] has shown that dentists can assess patients remotely, as well as in person if accurate information is available. Therefore, with proper information captured remotely, clinical decision-making such as treatment plans and resource allocations could be supported. Here, we address the development of a smartphone application and how it can enable patients to directly communicate emergent needs to a dentist with the goal of relieving prolonged treatments and enduring emergency room visits. For a health application to be effective, we need to first determine the technical feasibility of patients being able to utilize the application prior to determining how a dentist may choose to utilize the data.

Study goals

Our goal was to explore the feasibility of prospective patients using a smartphone to capture clinical information that help dentists make fundamental clinical decisions when they cannot examine them directly. We explored potential users experiences leveraging the camera and other means of data entry (voice/text) to develop a prototype application for exchanging information about dental emergencies between patients and dentists. We evaluated the prototype to determine the extent to which the application supported information capture by potential users involved in a simulated emergency.

Methods

The study consisted of four phases: 1) needs analysis and quality assessment of intraoral images captured by smartphones; 2) prototype development; 3) refining the prototype through usability inspection methods; and 4) formative evaluation through usability testing with 20 prospective users. Figure 1 provides a detailed description of these phases, which are described in the following sections.

Figure 1.

Prototype design process

Initial development

Needs Analysis

Prior to starting development, we spoke to dentists to understand their needs when patients present with emergency situations. Two dentists who worked in the emergency clinic at the University of Pittsburgh, School of Dental Medicine identified the common conditions for which their patients typically seek urgent care. For each condition, they compiled a list of information they typically review when providing care for these patients. With the request for visual aids, the research team independently evaluated images from different smartphone models as well as the device used in the study and reached consensus that the quality of displaying and identifying oral conditions is sufficient across different manufacturers.

Prototype Development

We first developed a paper prototype to design how information would flow through the application and incorporated all the necessary information identified by the dentists during the needs analysis. We included clinical questions intentionally written at an elementary reading level. Data entry was designed using conventional smartphone data-input methods such as text, radio button selections, voice recording, and capturing/appending photographs. For instance, the prototype application permits users to type additional comments about their symptoms, as well as the ability to leave a voice-message for dentists to respond to. The free-speech audio input provides an additional method to report the incident to a provider. In contrast to traditional telephone services, transmitting images through mobile devices allows a visual inspection of high-definition images and increases the ability to remotely evaluate care.

Design Evaluation

We refined and developed a high-fidelity prototype and evaluated it using two usability inspection methods^[25]: 1) cognitive walkthrough to identify potential usability problems in the application’s functionality^[25]; and 2) heuristic evaluation, to evaluate the extent to which the application’s screen design conformed with established design principles.^[26] We then revised the prototype interface based on the results of these inspections.

The cognitive walkthrough was applied to determine how a novice user would navigate through the application.^[22] The application’s instructions to complete a task are created and evaluated by the developer to determine any difficulty that a novice could experience. Following the instructions provided by the application, we compared the assumed novice user experience with the intended action sequence designed, as outlined in Table 1. Prior to every action, we asked specific questions proposed by Wharton et al^[27] to analyze the success of task-completion. Each question focused on a description of the appropriate action and others that are available, how the action is executed, as well as any modified goals while completing a task. Any ‘no’ answer represented a failure in the sequence and identified a potential problem with completing the task.

Table 1.

Information needs identified as being critical to dentists

• Chief complaint	• Sensations
• Anatomic location	• Presence of swelling
• Onset of discomfort	• Presence of bleeding
• Pain intensity	• Difficulty eating/drinking
• Pain frequency	• Medication taken
• Presence of fever	• Medication effect
• Triggers of pain	• Photograph
• Pain reliefs	• Miscellaneous comments

While cognitive walkthrough identifies potential usability problems, heuristic evaluation evaluates the user interface’s compliance with established design principles.^[28–31] Typically, 2–3 evaluators with knowledge in human-computer interaction examine the interface design layout and navigation to assess whether they comply with the ten heuristics as defined by Nielsen^[32] (Appendix A). Three evaluators (TKS, TPT, CDS) independently inspected the application’s user-interface to identify heuristic violations and compiled all violations into a single list. The three evaluators then removed redundant findings through consensus driven discussion and rated the severity of violations as described below:

• Low —Minor usability or cosmetic problem given low priority

• Medium – Usability or cosmetic problem causing exhaustive thought or assumptions by the user, given medium priority

• High – Major usability or cosmetic problem that may results in catastrophic task errors, given high priority.

Formative Evaluation with Usability Testing

Once the high-fidelity prototype was finalized, we performed formative evaluations with usability testing. An IRB exemption was obtained through the University of Pittsburgh and we recruited a convenience sample of 20 prospective users for the usability testing.^[25] The inclusion criteria for study participants included individuals at least 18 years old, English literate, and had used any smartphone in the last six months. No participants had any previous dental education background.

To observe how users would navigate the application, a researcher (CDS) provided verbal instructions only on how to access the application from the home screen. Additionally, the researcher presented a ‘broken tooth’ scenario to the participant for which they would need to contact a dentist urgently. The researcher then asked the participant to try contacting the dentist using this application. No additional instructions regarding navigation through the application were provided as the application included all further instructions for them to complete the task of submitting a case-report.

Dental emergencies are time-sensitive needs and accordingly, it is essential to assess the time a potential patient may spend while using the application described.^[33] We began the evaluation with a timed task-completion analysis, which examined the time required for completing a case report. ^[34] After initial instructions, participants were asked to create and submit a case report without interruption or assistance. All the participants completed the case report and each sub-task was time-stamped as they completed it.

Once the participant completed the timed task, we conducted usability testing using a retrospective think-aloud (RTA) protocol^[35] to examine their experiences and uncover usability problems.^[36,37] The RTA is described by Balatsoukas et al^[38] as a “sensitive protocol for detecting unique usability problems related to users' cognitive behavior” applying both user verbalization and facilitator observation while users interact with a system interface.

Each participant revisited the screens from the timed-task exercise and recalled their experiences while navigating through the application.^[39] We asked them to describe their thought processes for solving tasks and encouraged them to verbalize any positive or negative experiences with the application.^[37,40] For instance, users described the comprehensibility of the user interface, as well as the difficulty or ease of completing tasks. We recorded detailed notes and subsequently analyzed the transcripts for usability problems experienced by the participants.

At the conclusion of usability testing, each participant completed the validated Questionnaire for User Interaction Satisfaction (QUIS) ^[41] on their experience with the application. The survey contains 39 questions to assess user satisfaction of the screen and layout, terminology and system information, learnability, system capabilities, overall reactions to the application, and an additional section for users to list any positive or negative aspects they perceived. Based on participant feedback during the pilot phase, we modified the language of some QUIS questions and reduced the Likert response scale from nine to six to reduce confusion.

Results

Needs Analysis

Table 1 lists the information we identified that dentists typically gather from patients who report urgent needs. We incorporated this information into the application to elucidate responses concerning the four main areas of emergency dental care: teeth, gums, jaws, and mouth.

We also reached consensus that the quality of smartphone images for displaying oral conditions was sufficient. As displayed in Figure 2, the photographs captured with different smartphone devices clearly show the fine details of color/shade and shape of various tooth and gum conditions. This finding further supported the smartphone as a potential instrument to triage dental emergencies.

Figure 2.

Smartphone image of recurrent decay under composite restoration (photographed with Motorolla RAZR)

Prototype

The prototype application was developed on a Motorola Droid X smartphone with an Android operating system due to the convenience of programming in its open-source platform. At the time of the study, the specifications of the smartphone used included an 8 mega-pixel camera and a 480 x 854 pixel resolution (228 ppi) screen. Image quality continues to increase with each generation with some current smartphone specifications reaching beyond 41 mega-pixels of image resolution and screen resolution greater than 600 x 1024 pixels. A 2015 study by Boissin et al evaluated different smartphone camera images and concluded that Apple, Blackberry, and Android platforms could all replace digital cameras for the purposes of medical teleconsulation. ^[42]

Upon opening the application, the completion of a single report requires navigation through approximately 13 individual screens (outlined in Table 2). Users begin with a standard login screen where a username and a personal identification number are entered. Next, the user selects “create a new report” and is prompted to select a dentist’s name where the reported data would be transmitted. A title screen asks the user “What bothers you the most today?” where the user can select their general area of concern: tooth, jaw, gums, or mouth (Figure 3). The subsequent screen asks the user to identify the most appropriate scenario that describes their discomfort. For example, under the selection for ‘Tooth’, the user is able to select ‘broken tooth’ or ‘displaced/lost dental work’.

Table 2.

Action sequence of completing a case report

Case Report - Action sequence
Screen	Main Task	Screen	Main Task
1	Select 'Start New Report'	9	Select general tooth region
2	Select receiving dentist	10	Select specific tooth location
3	Select general concern	11	Choose to photograph an image
4	Select scenario	11	Photograph an image
5	Answer Primary Evaluation 1	12	Insert text comment
6	Answer Primary Evaluation 2	12	Record a comment
7	Answer Primary Evaluation 3	12	Enter contact information
8	Answer Other Indicators	13	Submit report

Figure 3.

Screen 3: Select General Concern

In the following four screens, the users enter their symptoms and physical conditions using slide indicators, radio buttons, and checkboxes. The user is then asked to upload or photograph two images of their present condition. The user also has the option to enter free text in the comments box, their contact preferences (email, telephone, text-message) and send voice messages. In the final screen, the user is able to submit the report and view a summary of their inputs to acknowledge the completed submission.

Usability Inspection Results

Cognitive Walkthrough

The cognitive walkthrough identified two potential usability problems that could interfere with submitting a report. Two separate screens where users select the regional and proximal location of their discomfort were identified as being too far apart given their similarity. These two screens were later relocated to create a more natural progression of tasks. The second observation prompted a revision to the ‘Likert scale faces’ to reduce ambiguous or indefinite responses. In screens where the user is instructed to select their level of pain, the six Likert-face options displayed were reduced to three (no pain, mild pain, severe pain).

Heuristic Evaluation

Of the twenty-four heuristic violations identified by the three evaluators, the most violated heuristics were ‘Consistency and standards’ and ‘Match between system and the real world’ with six violations for each heuristic. The evaluators rated four violations as high severity (described below) and the remaining as medium or low severity. Most of the medium and low severity violations included texts that required rephrasing and design issues that were reasonably simple to revise.

Consistency and Standards

Users identified the inconsistent use of checkboxes and radio buttons on two screens and were rated as high severity because they could confuse users who are unaware that they can enter more than one response for a question. Conventionally, checkboxes imply multiple-choice responses to selections while radio-buttons are preferred for single-choice responses. The answer choices on these screens were subsequently revised to reflect this standard.

Match Between System and the Real World

Previously, the question, ‘When did the pain begin?’ displayed intervals of time that overlapped each other. We rated this finding a High Severity violation because it is critical to have an accurate response on pain onset. The responses provided were not inclusive of all ranges of time and were inherently confusing.

Error Prevention

The user, when asked to choose a dentist to contact, had the option to select “none”. Clearly, this option is counter-intuitive because the user needs to send the report to a dentist. Therefore, it is a high severity violation and the “None’ option was subsequently removed. Evaluators also recommended clearly indicating mandatory questions to be completed using asterisks to prevent errors by not having complete answers.

Visibility of System Status

The application did not provide any feedback or visual cue to acknowledge that the user submitted a case report to the selected dentist or dental practice. Once this violation was discovered, a final confirmation screen was added into the interface to prevent future confusion.

Assessing End User Usability

We recruited twenty participants (nine male, eleven female) to participate in the usability study between the ages of 18–61 years old. The average age of the participants were 32.4 years old (Standard Deviation: 15.5 years), with 70% between 18–29 years of age. The participants had an average of 4.6 years’ experience (SD: 1.6 years) using smartphone devices. Only one participant was a novice user with one year or less of smartphone experience.

Timed task-completion analysis

All participants successfully completed the simulation of a case report in less than 4 minutes. The mean time for users to complete the report was 3.18 minutes (SD: 34.9 seconds), with a range from 2.28 to 3.88 minutes. Participants took the most time (average 73 seconds, SD: 6.4 seconds) completing information on four screens (screens 5–8) that required 50% of the total data entry. The average completion times for each screen are displayed on Table 3. The screen for ‘applying comments and contact information’ required users to spend the most time (30.75 seconds, SD: 11.17 seconds) for any individual screen and required four separate data elements for the user to input. At this screen, users were able to input both free-text and voice-input as a mode of data-entry.

Table 3.

Timed task-completion

			MEAN	MIN	MAX	SD
Years of participant smartphone experience:			4.60	3	7	1.50
Participant age:			32.40	18	60	15.48
Task	Elements (n)	Simulated tasks to complete 1 case-report	Time till completion (seconds) Non-think aloud
1	1	Initiate new report	4.40	2	6	1.82
2	1	Select receiving dentist	9.35	3	17	5.10
3	1	Select regional concern	9.95	3	25	4.61
4	1	Select proximal scenario	7.65	3	16	4.02
5	2	Answer Pain Severity Questionnaire	14.35	7	25	4.99
6	2	Answer Onset and Fever Questionnaire	19.80	10	37	7.92
7	3	Answer Pain Description Questionnaire	22.00	8	32	7.41
8	5	Answer Other Indicators	16.45	11	32	5.28
9	1	Select regional location	13.35	5	19	5.30
10	1	Select proximal location	12.35	3	20	6.54
11	1	Append photograph	24.05	11	35	6.17
12	4	Apply comments and contact info	30.75	14	58	11.17
13	1	Submit report	6.65	3	20	4.51
Total report completion time (seconds)			191.10	137	233	34.90

Retrospective think-aloud protocol

Of the thirteen screens outlined in Table 2, participants experienced the most difficulty in Screens 9 and 10 , which required the user to document the location of their problem (Screen 10 seen on Figure 4). We identified ten usability problems in these two screens. Here, users first selected the regional location of the simulated discomfort, followed by a larger image to select a more precise location on a mouth model. Seven users recognized this task as being redundant since they had already correctly selected the exact location in the previous screen. The participants recommended using only a single screen and utilizing a magnification function to record the exact location of their concern.

Figure 4.

Screen 10: Select specific tooth location

Participants reported their unfamiliarity with the use of slider indicators on Screens 5 and 6 in ten separate instances. Four users also expressed difficulty with the record-function button in the ‘Apply comments and contact information’ screen. Here, users were able to voice-record their input though these individuals required multiple attempts to complete the task of recording a voice-message. Users requested the words ‘record’, ‘play’, and ‘stop’ be included with the button images. The complete list of these usability problems observed are displayed in Appendix B.

User satisfaction

QUIS assessed users’ satisfaction of the application’s screen layout, terminology and system information, learnability, capability, and their overall reaction. The participants rated each of these categories on a scale of 1–6 with six indicating most satisfied. All participants indicated their satisfaction of the application’s usability with each category receiving scores greater than 5.3 with an average of 5.59 satisfaction. Table 4 shows the cumulative average responses of the twenty participants for each category.

Table 4.

User satisfaction scores of each QUIS component (Scale: 1–6, least to most satisfied)

Component	Number of questions	Mean Satisfaction Score(Scale: 1–6, least to most satisfied	Average Standard deviation
Screen and layout	5	5.58	0.67
Terminology and system information	11	5.56	0.78
System learnability	8	5.77	0.48
System capabilities	8	5.68	0.65
Overall reaction to the software	5	5.36	0.87
Total	37	5.59	0.32

The last two QUIS questions permitted participants to provide open-ended responses about their positive or negative perceptions of the application. However, only 11 (55%) participants responded to either or both questions. Five of the eleven participants (45%) expressed that the application was simple to use and easy to navigate. Other positive opinions included the application’s simply followed directions and its efficiency to use. In contrast, four participants commented that the application was slow or not smooth when transitioning between screens at times. Three users listed repetitive or redundant questions as a drawback of the application.

Discussion

The study results indicate the feasibility of patients using mobile applications to transmit relevant clinical data to their dental providers during an emergency. The results also highlight the ease of using the mobile application among individuals in any age group and with minimal smart phone use experience. The results are significant because they suggest the possibility of expanding mobile applications to enhance patient follow-up after initial dental appointments, patient understanding on seeking dental care and improving access to care for outside office hours’ dental emergencies. The participants’ high user satisfaction rating signifies the importance of performing needs analysis and incorporating user-centered design methods during the early phase of developing the technology than later. Previous studies on mobile applications have explored its use by providers to contact patients and for remotely monitoring patients for various health conditions. ^[15–18] To our knowledge, this is the first study in the United States that explores the development process and potential use of a mobile application that allows prospective patients to contact their dental providers.

Mobile applications have the potential to make a difference where dentistry is striving to make progress. Broken appointments are a huge problem in dentistry especially among socio-economically disadvantaged populations. Lack of transportation, finance and inflexible work schedules are major reasons for broken appointments among Medicaid enrolled patients ^[43–45]. Moreover, dental practices are concerned about the high attrition rate of patients following their initial treatments and the patients’ failure to comply with follow-up visits^[46] . Establishing tele-consults and patient education modules through mobile applications may alleviate these issues to some extent^[47,48]. While these numbers continue to increase, a Pew Research Center survey found that 50% of American low-income adults own a mobile phone and 62% of all mobile phone owners have used their phone to look up health information. An additional 7% of Americans are considered ‘smartphone dependent’ with no other means to access broadband Internet. ^[49]

In terms of health care communication, the paradigm is shifting towards clinicians receiving their clinical data on smartphones in e-mails, calls, and text messages.^[50] For patients, this landscape is rapidly changing with the use of secure messaging through patient health portals that allow patients to access and exchange personal health information. However, dental patients are infrequently awarded the luxury to communicate their health information directly to their providers. The clinical relevance of this study is that we enabled a platform for patients to directly communicate with their dentist rather then going through other medical/emergency room avenues and not receiving the appropriate treatment protocols.

We designed the application through user-centered design methods and by leveraging the capability of a smartphone to take high quality images. The study results indicate the usefulness of our mobile application to contact and convey relevant information to dentists during dental emergencies and after-office hours. The range of participant’s ages (18–61 years old) and smartphone experience (1–7 years) further exhibited its versatility for accommodating users with varying smartphone use experience.

The drawbacks of integrating mobile applications with patient care have previously been acknowledged, due to lack of evaluations that impact both the user’s experience and well-being.^[51,52] Implementation of poorly designed software can have costly repercussions for both providers and patients and valid assessments are necessary prior to clinical use.^[41,53] Furthermore, many have detailed the value of robust design and evaluation techniques that we utilize here.^[54] Specifically, inclusion of end-users is essential while developing patient-centric technologies.^[55,56] Our study was an initial validation of the application’s usefulness with prospective dental patients. As a result of this evaluation, additional tests should assess users’ capabilities to select all oral anatomical locations supported by the application, as well as other approaches for obtaining the same clinical data. Furthermore, the significance of the anatomical data should be evaluated with dentists prior to any interface revisions.

Though smartphones may be useful for supporting the efficacy of remote examinations in dentistry, there is no definitive substitute for a chairside exam. The application described here only intends to exchange clinical information when other means of dental evaluations are less ideal. Our results warrant a clinical study to validate the utility of the clinical data captured, as well as the application’s true implementation in clinical practice. Hypothetically, the application would allow dentists to evaluate how urgent a patient’s conditions are and manage emergent situations in a more efficient manner than conventional methods. Smartphone use as a patient self-help tool may very well augment the current approach for receiving care when access is limited. Exploiting the utility of these mobile applications could relieve unwarranted patient visits, and expedite appropriate patient care.^{[3,4,10,22,57]}

Limitations

The study reports an assessment of the technical feasibility of utilizing smartphones between dentists and patients and does not address the clinical implications. The twenty participants studied here only simulated a patient experiencing dental trauma and were in no state of dental emergency. Actual patients experiencing an emergency scenario may therefore significantly deviate from the reported completion times for navigating through the application.

Furthermore, investigating the application’s clinical utility is necessary to determine the efficacy of implementing a remote triaging service for patients while we only report potential user’s interactions. Further investigation should also assess the success and accuracy of user’s self-reported information, including the images taken of dental discomfort. This would determine the significance and value of the various data elements included in this application for triaging emergency dental care. Eventual real world implementation must also address reimbursement, legal, regulatory and policy issues, including appropriate clinical guidelines and treatment protocols.

Conclusion

We designed and implemented a test mobile application to facilitate optimal information exchange between dentists and patients with the goal of expediting emergency care. The application successfully supported clinical information capture by patients using ubiquitous smartphone inputs such as text and radio buttons, voice recording, and high-resolution photography. All participants successfully completed the instructed clinical tasks and the application was commended for being instructional and simple to use. The results of this study illustrate the feasibility of patients utilizing a smartphone application to self-report emergency dental conditions. Additionally, the results also support the ability for dentists’ to remotely assess care when direct patient contact is less practical.

Appendix A. Nielsen's ten usability heuristics for evaluating user interfaces

Nielsen's Ten Usability Heuristics
Heuristic	Explanation
1 Visibility of system status	The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
2 Match between system and the real world	The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system- oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
3 User control and freedom	The system should support undo and redo.
4 Consistency and standards	Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
5 Error prevention	Even better than good error messages is a careful design, which prevents a problem from occurring in the first place.
6 Recognition rather than recall	Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another.
7 Flexibility and efficiency of use	Allow users to tailor frequent actions.
8 Aesthetic and minimalist design	Dialogues should not contain information that is irrelevant or rarely needed.
9 Help users recognize, diagnose, and recover from errors	Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
10 Help and documentation	Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

Appendix B. Observations identified by users through the retrospective think-aloud protocol

Location	Issue	Frequency
Initiate new report	-	-
Select receiving dentist	Expected 'next' to be automatic	1
Select regional concern	Expected 'next' to be automatic	1
Select proximal concern	-	-
Evaluation metrics 1	Missed slider	3
	Issue with using slider	2
	Selected face (image)	1
	Missed question	1
Evaluation metrics 2	Issue with using slider	5
	Required second look	1
Evaluation metrics 3	-	-
Evaluation metrics 4	Confused by question	1
Select regional location	Required second look	1
Select proximal location	Initially chose correct tooth	7
	Chose incorrect tooth	2
Append photograph	-	-
Apply comments and contact info	Required two tries	2
	Selected 'record button' to stop	2
	Did not select record	2
Submit report	-	-