Evaluating Augmented Reality Head-Mounted Devices in Healthcare: A Review of Hardware, Software, and Usability Approaches

Abstract

Augmented reality head-mounted devices (AR HMDs) are increasingly deployed in healthcare. Given the stringent safety and efficacy requirements of medical settings, proactive quantitative testing of key performance attributes prior to deployment is critical for risk assessment. A systematic performance evaluation framework is essential not only to support clinical adoption but also to secure regulatory approval. This review systematically summarizes hardware, software, and usability assessment methods for AR HMDs in healthcare, analyzes current research and experimental designs, and identifies challenges arising from device heterogeneity, limited coupling with real-world clinical scenarios, and subjective bias. To address these issues, we propose five design principles to guide the development of objective and practical evaluation methods: (1) identify key components based on core functions; (2) prioritize testing by functional contribution; (3) replicate authentic clinical and human-visual conditions; (4) objectify subjective perception; (5) test functionally linked components jointly.

Introduction

Augmented Reality (AR) is a technology that overlays computer-generated virtual information onto the real world to enhance the user’s perception and interaction capabilities. A head-mounted display (HMD) is a device worn on the user’s head that typically overlays virtual images onto the user’s field of view through optical see-through or video see-through methods. It also incorporates built-in cameras and sensors to acquire spatial information, enabling precise registration with the real-world scene.1 AR HMDs commonly integrate display, sensing, tracking, and computing functions in their hardware design. Depending on the level of integration and power configuration, current devices can be categorized into two main types: one type integrates all the functions and power into a single device; the other type offloads computing or power modules to an external mobile terminal, achieving overall functionality through a wired connection. Additionally, AR HMDs are often combined with visual markers or natural feature recognition to ensure that virtual images are displayed at their intended locations and updated in real-time.

With their display capabilities, simple structure, and ease of learning, AR HMDs are rapidly expanding in medicine. Some AR HMD systems have received FDA 510(k) clearance and are being used clinically,2 including spine surgery,3–5 dental treatment,6–8 knee replacement surgery,9,10 hip surgery,11,12 tumor resection,13–15 nasopharyngeal surgery,16,17 and plastic surgery.18 Furthermore, their value in rehabilitation is increasingly evident, with AR HMDs offering a cost-effective and user-friendly alternative for rehabilitation assessment and training.19 In medical training, they can create immersive training environments for physicians and trainees,20–22 enhancing training efficiency and reducing costs. Unlike consumer AR, medical AR participates in patient care and must meet stricter demands: millimeter spatial accuracy, high reliability and stability, patient safety assurance, sterile compatibility, and compliance with medical-device regulations and ergonomic standards.23

Translating medical AR from concept to clinic demands deeper, context-specific optimization. Most systems remain anchored to consumer-grade hardware and software.24 Under the FDA Medical Extended Reality Program,25 current AR HMD performance testing in healthcare confronts the following challenges:26,27

1. Insufficient Hardware Performance Evaluation: The performance of key components of current AR HMDs, such as displays and cameras, has not been adequately clinically validated. For example, displays may suffer from veiling glare, which can impair the presentation of medical image details and reduce diagnostic accuracy.28

2. Insufficient Software Performance Evaluation: The software development platforms relied upon by AR HMDs exhibit significant heterogeneity, and there is currently no unified evaluation standard. For instance, color rendering deviations by virtual engines may lead to color distortions in medical images, affecting doctors’ diagnoses and decision-making.29

3. Lack of Objective Usability Assessment Standards: Current usability assessment studies are mostly based on subjective evaluations, and there is a lack of objective measurement methods for individual differences in the Human Visual System (HVS).30 Additionally, the usability metrics required for some clinical scenarios are not unified, making it difficult to form a common evaluation standard.

These challenges collectively reveal a deeper, systemic bottleneck: the absence of a unified, standardized performance-evaluation framework for medical AR HMDs, resulting in fragmented research, high translational barriers, and constrained industrial growth. This paper centers on evaluation methods for hardware, software, and usability, reviews existing evaluation approaches, and analyzes their challenges and optimization strategies in medical applications. Finally, the paper proposes five principles for the design of evaluation methods, as shown in Figure 1, to build a more scientific and comprehensive performance evaluation system and further promote the application and development of AR HMDs in the medical field.

Figure 1.

Five design principles for constructing a performance evaluation framework for AR HMDs in healthcare.

Search Strategy

A systematic literature search was conducted exclusively within the Web of Science Core Collection, limited to English-language records published between 2015 and 2025. The Boolean query was formulated as follows: (“augmented reality” OR AR) AND (“head-mounted device*” OR “head-mounted display*” OR HMD*) AND (performance OR accuracy OR evaluation OR assessment OR testing OR validation OR regulation) AND (healthcare OR surgery OR rehabilitation OR training).

The search returned 677 unique citations. After automated deduplication and manual cross-checking, two reviewers independently screened titles and abstracts against predefined eligibility criteria (Cohen’s κ = 0.78); disagreements were resolved through consensus, yielding 300 candidate articles. Full-text evaluation resulted in the inclusion of 70 studies that explicitly reported performance testing or validation protocols for AR-HMD systems in healthcare contexts. Despite the systematic approach, the inherent breadth of the query and the interpretive nature of screening may have introduced residual subjectivity.

Hardware Performance Evaluation

The hardware of AR HMDs is the main carrier for functional implementation. The performance of the display, camera, and processor is crucial and directly affects the safety and effectiveness of medical applications.31 The display is used for near-eye presentation of AR images, and its image reproduction performance determines the visualization quality of medical images. The camera is responsible for spatial recognition and tracking, and insufficient accuracy and stability can lead to errors in operation guidance. The processor handles data processing and image rendering, and its computational power and response speed affect the overall efficiency of the system. Therefore, it is important to conduct objective and precise quantitative evaluation of the performance of these core hardware components to provide a basis for the safe and compliant use of these devices in clinical applications.26

Display Image Quality

The display is a core component of AR HMDs and directly affects the visualization quality of medical images,26 which in turn can influence the diagnostic and therapeutic processes of physicians. There are many types of display technologies for AR HMDs,31,32 including Half Mirror, Birdbath, Free-Form Prism, Waveguide, Retinal Scanning, Multiple LCDs, Microlens Arrays, Pinlight, and 3D Holographic Display.33 Although the implementation technologies vary, the goal is to display medical images and information. Therefore, from the perspective of using the display as the evaluation terminal, it is important to prioritize the evaluation of key performance parameters that are common to HMD displays.26,27,32 In addition to basic parameters such as display resolution, FOV, and display frame rate, there are also parameters that manufacturers may not specify but could potentially affect the display quality of medical images,27 including color uniformity, veiling glare, lateral chromatic aberration, and spatiotemporal resolution.31 Given the characteristics of these parameters, each needs to be detected separately, and the most suitable evaluation method is often related to specific hardware evaluation methods. Traditional evaluation methods for medical flat-panel displays do not consider spatial-related image artifacts caused by additional optical components such as HMD lenses,27 so it is necessary to explore evaluation methods that are suitable for HMDs. Additionally, since these devices use near-eye display, it is important to fully consider and simulate the conditions under which the HVS observes during evaluation, including factors such as interpupillary distance (IPD), pupil entry position, convergence angle, and scanning field of view.

Existing evaluation methods for the above parameters mainly include spot measurement methods based on optical sensors, indirect measurement methods based on reflective light paths, and precision measurement methods based on optical platforms. Spot measurement methods typically place measurement tools directly behind the HMD display for evaluation. Tools are not limited to photometers,33 colorimeters,34 spectroradiometers,35 and cameras36 to measure parameters such as the color, brightness, and contrast of the HMD display, as shown in Figure 2(A). However, it is difficult to precisely control the pupil entry position during measurement, which affects the accuracy of the measurement. At the same time, the measurement space behind AR HMDs is narrow and some lenses are curved, making direct measurement difficult. Therefore, some studies have proposed indirect measurement methods based on reflective light paths. By changing the position of the measuring instrument, the difficulty of arranging the measuring instrument is simplified, thereby improving the evaluation accuracy. Some studies have placed a front-surface mirror at a 45° angle at the HMD observation position, reflecting the displayed content completely to the external measurement device,37 improving measurement stability, as conceptually shown in Figure 2(B). However, this measurement process may introduce diffuse reflection errors. A method to avoid this problem is to use a conical measurement probe connected to an integrating sphere, placed behind the HMD to measure the parameters, reducing the interference of bright areas with the measurement signal.36 Precision measurement methods based on optical platforms use high-precision optical platforms to fix the measuring devices and combine them with precision adjustment devices such as electric linear translation stages and universal stages to precisely control the measurement angle and spatial position, ensuring precise and stable measurement of key parameters in the center and edge areas of the FOV, as conceptually shown in Figure 2(C).28 In addition, when designing specific evaluation schemes, it is also necessary to consider the characteristics of the evaluation parameters and build dedicated functional evaluation platforms. For example, to detect the spatial and temporal characteristics (spatiotemporal resolution) of displaying moving images, some studies have combined cameras with silicon photodetectors to detect the spatial resolution, temporal response, and spatiotemporal performance of the display under motion. At the same time, the display content of the HMD can be driven by an external graphics card to eliminate the error of system delay on the evaluation results.38 The main advantage of such methods is that they can significantly improve the reproducibility of experiments, reduce the impact of measurement device placement errors or environmental variable fluctuations, and thus improve the stability and reliability of evaluation data.

Figure 2.

Methods for evaluating AR HMD display performance: (A) Instrument-based direct measurement. (B) Displacement-based measurement. (C) Test bench-based precise measurement; Reprinted from Zhao C, Kim AS, Beams R, Badano A. Spatiotemporal image quality of virtual reality head mounted displays. Sci Rep. 2022;12(1):20235. Creative Commons.37 (D) Simulated IPD measurement; Reprinted from Zhao C, Beams R, Badano A. Radially variant contrast measurement in virtual reality headsets using circular concentric ring patterns. J Soc Inf Disp. 2023;31(5):387–397. This article is a U.S. Government work and is in the public domain in the USA.38

In addition, several evaluation studies have pointed out that attention should be paid to the settings of HVS and lighting conditions during the evaluation of display performance to better match the actual clinical use and reflect the performance status of the HMD under real use conditions. Key factors of the HVS include IPD, pupil entry position, and pupil size. For IPD adjustment, some devices can use the HMD’s own parameter adjustment settings (common in VST devices), as shown in Figure 2(D).39 For OST AR devices, the measurement device can be moved horizontally to achieve the same effect of IPD adjustment. Similarly, the influence of different pupil entry positions can be simulated by moving the evaluation device radially forward and backward. Different pupil sizes can be simulated by adjusting the aperture of the observation device’s camera or installing an iris diaphragm, etc., to simulate the size of the human eye pupil under actual use conditions.40

The image quality evaluation of the display should also appropriately consider the combination with other related hardware. For example, in the operating room environment, OST AR HMDs often use colored sunshades to reduce environmental light interference and enhance the AR effect. However, some studies have found that they may reduce contrast perception in actual use.33 Therefore, it is recommended to include functional implementation-related accessories in the display performance evaluation to improve the reliability of the evaluation experiment design.

Camera Tracking Accuracy

Another key function of AR HMDs in medical application scenarios is spatial positioning and tracking.41 This function mainly relies on the built-in camera of the HMD combined with computer vision algorithms (based on or not based on visual markers) to be realized. The tracking accuracy has a direct impact on the quality of clinical applications, especially in high-risk scenarios such as intraoperative treatment. Therefore, it is crucial to conduct systematic evaluation of the camera tracking accuracy of AR HMDs.42

Currently, the evaluation of HMD camera tracking performance is mainly based on two types of methods. The first type of method assesses the accuracy by measuring the HMD’s ability to track a known spatial location. That is, a target point with a known spatial location in the real world is selected as a reference point. At the same time, the tracking data of the HMD system for this point is recorded as the experimental point, and the spatial distance information between the two points is compared and analyzed for error43 (as shown in Figure 3(A)). This type of method has strong flexibility and can be customized according to specific medical application scenarios. For example, in a simulated incision surgery scenario, subjects can be asked to wear the HMD and perform a pre-set tracking curve task on a tablet to test the accuracy of the tracking system. Finally, the tracking accuracy is assessed by comparing the deviation between the curve trajectory drawn under the AR-assisted state and the pre-set curve trajectory.44 However, this method may be affected by the perceptual bias of the HVS, thereby introducing subjective judgment error.45 To reduce the interference of subjective factors, cameras and other devices can be used to replace the human eye for measurement46 (as shown in Figure 3(B)). Alternatively, tasks that do not require subjective judgment can be designed. For example, a movable grid plate with evenly distributed holes for installing tracking markers recognizable by the HMD can be used. By moving the grid plate at a fixed distance on a slide rail and adjusting the two-dimensional movement distance and longitudinal depth, a quantitative analysis of the three-dimensional tracking performance of the HMD camera can be achieved.43 However, this method is mainly suitable for static tracking accuracy assessment and is difficult to fully reflect the demand for high-precision dynamic tracking ability in medical scenarios.

Figure 3.

Methods for evaluating AR HMD camera tracking performance: (A) Comparison with known real-world locations under AR guidance; Reprinted from Cutolo F, Fida B, Cattari N, Ferrari V. Software framework for customized augmented reality headsets in medicine. IEEE Access. 2019;8:706–720. Creative Commons.43 (B) Camera-based assessment; Reprinted from Ballestin G, Chessa M, Solari F. Assessment of optical see-through head mounted display calibration for interactive augmented reality. 2019. © Copyright 2019 IEEE.45(C) Comparison using an external tracking system; Reprinted from Monica R, Aleotti J. Evaluation of the oculus rift s tracking system in room scale virtual reality. Virtual Real. 2022;26(4):1335–1345. Creative Commons.47

The design of the second type of evaluation method mainly involves introducing an external tracking system. By performing the same task and comparing the tracking data recorded by the HMD and the external system, the tracking performance of the HMD camera can be quantified, as shown in Figure 3(C). This method takes advantage of the non-interference characteristic between Inside-Out and Outside-In tracking methods and the stability of the external independent tracker hardware.48 In specific experiments, recognition markers are usually attached to the HMD and the target object so that they can be precisely tracked in six degrees of freedom by the external tracking system.47 Subsequently, by setting and performing standardized tracking tasks, the measurement data of the HMD system and the external tracking system are compared to quantify tracking accuracy and reliability.49 To meet the demand for high-precision tracking in medical applications, higher-precision tracking systems can also be built or used in the evaluation. For example, using high-power infrared emitters as markers and equipping them with dual photodiode stereo sensors with infrared low-pass filters for tracking can achieve a sampling rate as high as 50kHz and a motion-to-pose latency as low as 28μs.50 In addition, considering that the HMD may operate for a long time in medical applications, the stability of its camera performance also needs to be assessed. Some studies have run the HMD camera for a long time and statistically analyzed the ranging data of the ToF camera at different time intervals (such as 40 minutes after startup, 40–60 minutes, and 1 hour later) to evaluate its long-term operational reliability.51

To adapt to more complex medical application scenarios, such as surgical navigation involving soft tissues, markerless inside-out tracking technology is becoming a research focus.52,53 This technology eliminates the dependence on external markers, thereby enhancing the system’s applicability and clinical operational convenience to a certain extent. However, compared with marker-based tracking systems, the accuracy of markerless tracking technology is more susceptible to environmental lighting, target surface characteristics, and dynamic changes, and there are still relatively limited methods for detecting its accuracy separately. Therefore, further in-depth exploration of the accuracy evaluation of markerless tracking systems is needed to ensure their reliability and feasibility in clinical environments.

Performance Evaluation of Key Hardware Components

In the process of realizing the core functions of HMDs, in addition to the display and camera, other key components that significantly impact the device’s performance also need to be tested accordingly. However, the current heterogeneity of HMD hardware architectures is quite large. Specific tests should be based on the actual composition of the hardware and prioritize evaluation according to their importance in the main function implementation. At the same time, the evaluation methods should match the characteristics of the components to ensure the scientific nature and accuracy of the evaluation. For example, in real-time intraoperative navigation application scenarios, the built-in processor of the HMD plays a key role in system latency and frame rate, which affects the user experience of doctors. Therefore, its performance needs to be tested through experiments. Some studies have used high-speed cameras to capture and analyze frame image changes to quantify performance.54 Similarly, a evaluation platform based on microcontrollers, photodetectors, and microphones can be built to detect the asynchrony of audiovisual stimuli.55 However, research on the functional contribution of other key hardware components of AR HMDs and their corresponding evaluation methods is still limited and requires further exploration and study.

Software Performance Evaluation

As the core component for hardware function scheduling, the software performance of AR HMDs has a decisive impact on medical applications. However, existing AR HMD software is mostly developed based on consumer-grade software development platforms, without fully considering the operational constraints and high-precision requirements of medical environments. Moreover, there is a lack of unified evaluation standards to verify their reliability in medical guidance. The focus of function implementation and performance evaluation lies in the precision of virtual-to-real registration, the fidelity of image rendering, and the medical adaptability of the software framework.

Registration Accuracy of Virtual Models

Virtual-to-real registration refers to the key process of precisely aligning and superimposing virtual models or images with real-world scenes in terms of spatial position, scale, and orientation, which is often jointly realized in combination with the camera hardware of the HMD. The precision of virtual-to-real registration of AR HMDs is one of the important links to ensure the accuracy of medical applications. The mainstream registration techniques currently include manual registration, computer-aided registration, and external registration.48,56 Although these techniques differ in implementation methods, their common goal is to achieve precise alignment between virtual and real models, and evaluation can be based on this feature to establish a standardized evaluation scheme.

The key to registration precision evaluation methods can be understood as calculating the spatial error between the AR registration point and the actual physical reference point. Compared with the performance evaluation of hardware camera tracking, virtual-to-real registration precision evaluation focuses more on the spatial alignment error between the virtual model and the physical model after the software completes the registration. The commonly used method to achieve evaluation is as follows: First, select one or more spatial points to be measured in the real environment and record their coordinates. Then, establish a matching model in the virtual environment and perform registration. The subject wears the AR HMD and uses a tracking pointer to touch the feature points of the AR virtual image under AR guidance, records the coordinates, and compares them with the physical reference points for error analysis.57 Usually, this type of experiment relies on an external optical tracking system to provide a unified coordinate system for measurement. However, in medical scenarios with limited operating space, optical tracking markers may be affected by environmental constraints, limiting their applicability. Therefore, some studies have optimized the external tracking system according to actual evaluation needs, such as using electromagnetic tracking systems to enhance adaptability,58 as shown in Figure 4(A). However, the measurement accuracy in this type of evaluation experiment is easily affected by individual physiological characteristics (such as IPD, VAC, etc)., which reduces the objectivity and consistency of the evaluation. Therefore, developing objective evaluation methods that can avoid subjective errors and improve measurement reliability is the development direction of this type of performance evaluation.

Figure 4.

Methods for evaluating AR HMD registration precision: (A) Subjective evaluation method; Reprinted from Condino S, Turini G, Parchi PD, et al. How to build a patient-specific hybrid simulator for orthopaedic open surgery: benefits and limits of mixed-reality using the microsoft holoLens. J Healthc Eng. 2018;2018(1):5435097. Creative Commons.56 (B) Video-analytical approach using HMD camera captures; Reprinted from Debarba HG, De Oliveira ME, Dermann LA, Chaqu A, S E, Charbonnier C. Tracking a consumer HMD with a third party motion capture system. IEEE. 2018:539–540. © Copyright 2018 IEEE.58

To enhance the objectivity of measurement, one improved approach is to install a small camera on the HMD to capture image data from the user’s perspective, combined with precise displacement control of the HMD using a robotic arm to reduce human operational errors. This method can measure display pattern errors on a standardized evaluation board, providing a new idea for the quantitative assessment of the registration precision of AR HMDs.59 In addition, another method is to record the AR image with the HMD, export the video, and analyze the spatial deviation between the AR image and the target image frame by frame,60 as shown in Figure 4(B). Both types of evaluation methods can enhance the objectivity of evaluation, reduce errors brought by subjective assessment, and provide a reference for the experimental design of objective evaluation of the registration precision of AR HMDs.

Image Rendering Fidelity

In medical applications of AR HMDs, virtual images not only need to be accurately registered with the expected anatomical locations but also should maintain high fidelity in terms of color, shape, and proportion of the digital models to ensure the credibility of diagnostic and treatment decisions. Currently, the rendering of most three-dimensional medical digital models mainly relies on consumer-grade engines (such as Unity, Unreal Engine, etc).22 Since there is no unified rendering standard, different engines may lead to significant differences in model fidelity, thereby affecting the accuracy and stability of intraoperative visualization.

To quantitatively analyze the performance differences of different rendering engines, existing studies have used ColorChecker color cards, TG18-QC test patterns, cell slice images, and grayscale medical images. By controlling a single variable in the rendering engine, the impact of different rendering parameters on color performance is detected, and the CIE 1976 color difference standard is used for quantification.29 Similarly, other rendering parameters that affect the visual perception of the model, such as materials, textures, light sources, and perspective, also need to be included in the evaluation to comprehensively evaluate the rendering quality of the virtual model.61 However, the above evaluation methods are mostly aimed at the performance analysis of rendering parameters within a single platform and cannot better cope with the current challenge of significant differences in HMD development platforms.

Faced with this challenge, a feasible solution is to develop a cross-platform, easy-to-deploy standardized test toolkit to unify test content and indicators.62 The toolkit can include benchmark test images or models with the same parameter characteristics and allow users to customize scenes to evaluate key indicators such as color mapping, grayscale response, display resolution, and spatiotemporal characteristics on different devices and software platforms. To meet the compatibility requirements of most medical AR HMDs, test data can be packaged in a universal format (such as JSON) and distributed over the network to achieve automated and standardized testing. Such methods can provide a more efficient and concise new approach for rendering performance testing of various platforms in medical scenarios.

Medical Adaptability of Software Frameworks

The adaptability of software frameworks refers to the ability of AR HMD software platforms to meet diverse medical needs, including accurate responses and compatibility with different clinical departments, hardware configurations, data security, and operational processes. Since medical scenarios have particularly strict requirements for safety, real-time performance, and scalability, medical AR HMDs lacking sufficient adaptability often struggle to cope with challenges brought by cross-departmental, cross-system, and even subsequent technological iterations. Therefore, before building an efficient and unified software framework, it is necessary to fully explore the adaptability standards that meet medical scenarios and their evaluation methods. Currently, the performance evaluation of the medical adaptability of AR HMD software frameworks mainly focuses on the following two ideas: The first is horizontal comparison of specialized software frameworks: by developing or selecting various software frameworks and applying them to similar medical scenarios for comparative testing of core functions (such as real-time tracking and computational efficiency). For example, in AR HMD-guided surgical scenarios, the fast processing capability of a tracking system based on the CUDA architecture within 0.007 seconds44 can be regarded as an embodiment of high adaptability, capable of meeting the rapidly changing surgical needs. In addition, the selection of different architectures for software frameworks (such as distributed and monolithic) will also affect the adaptability of the framework in multi-department collaboration, telemedicine, or large-scale data processing environments.

The other is comprehensive scoring based on standardized development guidelines: according to the standardized guidelines for AR HMD applications, a series of evaluation indicators covering software design to application construction are established. The core lies in measuring the performance of software in dimensions such as graphical programming, application programming, analysis, and optimization, and assessing whether it supports various medical image formats, and whether it is easy to develop secondary to meet different clinical needs and other “adaptability” ky points.63 Through these multidimensional scores, the generality and scalability of the software in a broader range of medical scenarios can be judged.

When assessing the adaptability of medical AR HMD software frameworks, it is necessary to closely integrate with the needs of the clinical front line. Inviting doctors or relevant medical teams to participate in the assessment helps to set standards for application scenarios and processes that are closer to clinical work, such as the precision requirements for medical image rendering and surgical navigation, the integration needs for multi-department general functions, and the expandability of secondary development.

Usability Assessment

The usability assessment of AR HMDs is crucial for ensuring the success of medical procedures, as it greatly affects the operational experience of medical staff and clinical outcomes.64 When evaluating the usability of AR HMDs, the focus is on considering the real operational environment of medical professionals and the impact of the HVS.26,31 The factors involved mainly include the human information processing system, lighting environment, color environment, human-computer interface design, and multiple other dimensions.65 These factors directly or indirectly affect the terminal operation or judgment of medical users, thereby impacting the quality of medical care. However, the evaluation of these factors often involves subjective perception, with significant individual differences, making it difficult to achieve accurate and objective quantitative evaluation like hardware and software performance testing. Quantitative indicators for measuring these parameters are challenging for assessing usability impact, but this does not prevent the continuous exploration of more precise and practical evaluation methods in the assessment.

Visual Information Processing System

In AR medical applications, the efficacy evaluation of the visual information processing system is a crucial aspect to ensure the usability of AR HMDs. Its core lies in quantifying the user’s egocentric depth perception of the displayed information, a parameter that directly affects the user’s accuracy in integrating virtual information with the real environment. Therefore, the precision assessment of this parameter is particularly critical for AR HMDs in high-precision medical scenarios.

Traditional evaluation paradigms mainly rely on subjective measurement methods, employing a mixed assessment framework of depth estimation tasks (Depth Estimation Task) and Likert scale questionnaires.66 These tasks may include observing and estimating the depth of AR content display, spatial perception through finger movements, etc., as shown in Figure 5(A). However, these methods fall short in terms of precision and reproducibility of experimental results, failing to meet the high standards required in medical scenarios.67

Figure 5.

Methods for evaluating depth perception in AR HMDs: (A) Subjective assessment; Reprinted from Kahl D, Ruble M, Kr U, Ger A. The influence of environmental lighting on size variations in optical see-through tangible augmented reality. IEEE. 2022:121–129. © Copyright 2022 IEEE.67 (B) Objective quantification of subjective judgments using experimental apparatus; Reprinted from Swan JE, Singh G, Ellis SR. Matching and reaching depth judgments with real and augmented reality targets. IEEE Trans Vis Comput Graph. 2015;21(11):1289–1298. © Copyright 2015 IEEE.64

To achieve higher precision in evaluation, some studies have attempted to transform human subjective perception results into quantifiable objective data. A typical approach is to use an improved optical platform, with plastic tubes installed on both sides of the platform: one side for fixing and supporting the framework of the physical model, and the other side equipped with a movable tube with a pointer. The subject moves the tube under the table to operate the pointer to point to the target position, thereby objectively quantifying the depth perception error of the AR HMD,68 as shown in Figure 5(B). This device can measure errors in the millimeter range, approaching the 1 mm or smaller precision required for AR image-guided brain surgery.69 In addition, when evaluating depth perception, multiple depth cues, including model material, lighting environment, shadows, and textures, should be comprehensively considered,61 and the experimental conditions should be consistent with the attributes of the test objects.

Since depth perception assessment is inherently affected by subjective factors and individual differences cannot be completely eliminated, based on existing research, it is recommended to combine high-precision objective measurement platforms with corresponding subjective questionnaires when constructing evaluation methods to form a comprehensive evaluation scheme for AR HMD depth perception assessment,54 which can improve quantification accuracy while also considering the subjective differences of the HVS.

Lighting Environment

The lighting environment is one of the important variables affecting the usability of AR HMDs. Changes in light intensity, direction, and nature can significantly affect the user’s visual perception accuracy of virtual content, thereby directly affecting the effectiveness and safety of medical operations. In specific medical environments such as operating rooms, complex lighting conditions (such as strong light interference from shadowless lamps) can greatly affect the visibility and accuracy of AR display information.

To systematically detect the impact of lighting conditions on AR HMDs, some studies have set up experimental scenarios in different lighting environments and asked subjects wearing AR HMDs to manually adjust the physical model to match the fixed virtual model while using an external tracker to record the matching error between the physical and virtual positions.70 On this basis, more practical elements of medical scenarios, such as zoning settings, personnel movement, and anchor placement, can be integrated,71 and a more comprehensive assessment of lighting effects can be achieved by adjusting light intensity or type. Evaluation indicators can be selected in combination with the real needs of the operating room, such as recording the accuracy of hologram overlay, task completion time, number of registration rejections, real-time overlay delay of holograms, and number of operational failures. It is worth noting that the lighting environment may also indirectly affect device performance, such as the degree of interference with the gesture and voice recognition functions of AR HMDs by surgical gloves of different colors under different lighting conditions.72 Therefore, in the assessment of the lighting environment, the dual impact of lighting on the main performance of AR HMD hardware and software and the overall workflow of the operating room should be comprehensively considered.

Clinical Usability Analysis

With the increasing research on the practical medical applications of AR HMDs, reaching a consensus on therapeutic effects has gradually become one of the important indicators for assessing the usability of AR HMDs. The evaluation criteria for clinical applicability need to be adjusted according to specific application targets, especially for surgical applications, which require higher strictness and comprehensiveness in the usability assessment of HMDs. Current clinical usability studies of AR HMDs mainly focus on two aspects: the results of clinical application performance and the postoperative reports from clinical doctors.

In surgical scenarios involving rigid anatomical structures, the clinical usability research of AR HMDs is relatively mature, and more unified methods and standards for usability evaluation have emerged in some specific surgeries. For example, in spinal pedicle screw surgery, researchers usually analyze the intraoperative or postoperative fluoroscopic images of screw placement and assess the position and angular deviations of pedicle screws or K-wires according to the Gertzbein-Robbins screw placement grading criteria. In addition, some studies also consider clinical indicators such as screw placement time, intraoperative blood loss, and related complications to comprehensively evaluate the application effects of AR HMDs in actual surgeries,3,5,70–73 as shown in Table 1. In the field of dental implant surgery, the accuracy of AR HMD-guided implant placement is often assessed by comparing the deviations in the coronal, apical, and angular aspects between preoperative planning and postoperative implants,7,8 as shown in Table 2. For AR HMD-assisted acetabular cup placement surgery, the position and orientation of the acetabular cup relative to the pelvis after surgery are often used as the evaluation baseline.11 The usability evaluation indicators for clinical application scenarios of AR HMDs are becoming standardized, promoting their promotion and application.12

Table 1.

Summary of Studies Highlighting Clinical Applications in Pedicle Screw Placement Surgery

Author & Year	Study Scale	Equipment	Clinical Evaluation Criteria
Butler, A.J. et al, 20233	606 screws / 164 patients	Xvision Spine system (XVS) (Augmedics Ltd, USA)	Average time for screw placement, intra-operative screw revision rate
Molina, C.A. et al, 202073	113 total implants / 5 cadavers	Xvision Spine system (XVS) (Augmedics Ltd, USA)	Gertzbein-Robbins scale, linear deviation, angular deviation
Liu, A. et al, 202171	205 screws/28 patients	Xvision Spine system (XVS) (Augmedics Ltd, USA)	Gertzbein-Robbins scale
Felix, B. et al, 20225	124 thoracolumbar pedicle screws/ 7 cadavers	VisAR (Novarad Corporation, USA)	Gertzbein-Robbins scale
Chang, C. et al, 202274	24 screws/1 cadaveric	Caduceus (Taiwan)	Gertzbein-Robbins scale
M U Ller, F. et al, 202075	20 K-wires/3 lumbar cadaver spines	HoloLens (Microsoft Corporation, USA)	Translational Error, Angular Error
Yahanda, A.T. et al, 202176	63 screws / 9 patients	Xvision Spine system (XVS) (Augmedics Ltd, USA)	Gertzbein-Robbins scale
Charles, Y.P. et al, 202177	80 pedicle screws / 20 patients	AlluraClarity Flexmove, (Philips. The Netherlands)	Gertzbein-Robbins scale
Farshad, M. et al, 202178	4 pedicle screws /1 patient	HoloLens 2 (Microsoft Corporation, USA)	Angular Deviation, Translational Deviation
Frisk, H. et al, 202270	48 pedicle screws / 4 spine phantoms	Magic Leap1(Magic Leap, USA)	Gertzbein-Robbins scale
Cao, B. et al, 202379	82 pedicle screws / 5 spine cadavers	HoloLens, (Microsoft Corporation, USA)	Gertzbein-Robbins scale
Bhatt, F.R. et al, 202380	222 pedicle screws / 32 patients	xvision Spine system (XVS) (Augmedics Ltd, USA)	Gertzbein-Robbins scale
Ghenbot, Y. et al, 202481	64 pedicle screws / 3 cadavers	HoloLens 2 (Microsoft Corporation, USA)	Gertzbein-Robbins scale
Yanni, D.S. et al, 202182	192 pedicle screws / 24 lumbar spine models	SpineAR (Surgical Theater, Inc).	Gertzbein-Robbins scale
Spirig, J.E.M. et al, 202183	18 k-wires / 2 cadavers	HoloLens (Microsoft Corporation, USA)	3D Distance, Trajectory Angle

Table 2.

Summary of Studies Highlighting Clinical Applications in Dental Implant Surgery

Author & Year	Study Scale	Equipment	Clinical Evaluation Criteria
Tao, B. et al, 20248	121 dental implants/32 phantoms	HoloLens 2 (Microsoft Corporation, USA)	Global coronal, apical, and angular deviations
Fan, X. et al, 20237	102 dental implants /30 patient phantoms	HoloLens 2 (Microsoft Corporation Corp).	Global coronal, apical, and angular deviations
Pellegrino, G. et al, 201984	2 dental implants /2 patients	HoloLens (Microsoft Corporation, USA)	Entry point, apical point and angular deviations
Yotpibulwong, T. et al, 202285	5 dental implants/5 patients	Moverio BT-300, (Seiko Epson Corporation, Japan)	Entry point, apical point, angular deviations
Kivovics, M.A.R. et al, 202286	16 dental implants/4 models	Magic Leap 1(Magic Leap, USA)	Angular deviation, coronal, and apical global deviation.
Gonz A Lez-Rueda, J.O.N. et al, 202387	20 dental implants/4 models	HoloLens 1 (Microsoft Corporation, USA)	Global coronal, apical, and angular deviations
Liu, L. et al, 202388	25 dental implants/25 models	HoloLens (Microsoft Corporation, USA)	Entry point, middle point, apex point, and angular deviations
Elena et al, 202389	207 dental implants/14 models	HoloLens 2 (Microsoft Corporation, USA)	Global coronal, apical, and angular deviations
Marhuenda Ramos et al, 202490	20 dental implants/1 models	HoloLens 2 (Microsoft Corporation, USA)	Global coronal, apical, and angular deviations
Bochet, Q. et al, 202391	2 dental implants/1 patient’s models	HoloLens 2 (Microsoft Corporation, USA)	Entry point, apical point and angular deviations

In contrast, non-rigid surgeries, due to factors such as tissue deformation and unstable tracking, still lack unified usability evaluation criteria. Current research is mostly conducted in simulated in vitro environments, comparing the registration accuracy of AR HMDs with traditional navigation technologies and measuring the rotational and translational errors of markerless tracking in clinical applications.14 Due to the lack of stable marker recognition and predictable tissue deformation patterns, more clinical research data is needed to support these studies.

Beyond surgical scenarios, AR HMDs also show potential in rehabilitation and diagnostic fields. For patients with neurological disorders (such as stroke and Parkinson’s disease), AR technology can enhance patients’ understanding and interest in rehabilitation exercises through real-time visual feedback and immersive interaction.92 Traditional motion analysis relies on optoelectronic systems or inertial measurement units (IMUs), while the built-in sensors and tracking algorithms of AR HMDs provide a low-cost, easy-to-deploy alternative for objectively recording patients’ motion trajectories and assessing their rehabilitation progress.93,94 For stroke patients, AR-assisted real-time monitoring and virtual guidance can help clinicians adjust rehabilitation strategies in a timely manner, bringing more convenient and efficient rehabilitation pathways for patients with impaired movement and cognition.19

Furthermore, AR HMDs have a broad prospect in medical training. Some studies have simulated in vitro training for spinal pedicle screw insertion, evaluating the usability of AR HMDs in medical training by calculating the distance between the virtual navigation screw and the edge of the pedicle bone.95 Other studies have simulated central venous catheter insertion surgery, using an infrared reflection marker-based method to track the movement of non-rigid tissues.96 In life support training, a study has used a questionnaire survey (covering six dimensions: user input, system output, usability, simulation fidelity, immersion, and preference) to evaluate the overall effectiveness of the AR system.97 These studies indicate that the interactivity and immersion of AR HMDs provide a more intuitive and repeatable training platform for medical students and clinical staff. To accommodate the diversity of assessment methods, the complexity of the entire clinical workflow must be considered.98 Tailored to the specific clinical context of AR HMDs, qualitative and quantitative analyses may be conducted via heuristic evaluation, cognitive walkthrough, task analysis, GOMS, usability testing, field studies, structured interviews, think-aloud protocols, or mixed-method approaches.

The clinical usability of AR HMDs needs to be comprehensively evaluated in a multidisciplinary and multi-scenario context. Research on the usability evaluation of rigid structure surgeries has accumulated a relatively systematic set of criteria, while soft tissue surgeries, due to technical limitations such as marker stability and registration accuracy, have fewer clinical studies and await the establishment of unified evaluation standards. The fields of rehabilitation and diagnostics show potential for objective assessment and real-time training, offering more rehabilitation options for those with impaired movement and cognitive functions. In the future, if their safety and effectiveness can be verified in large-scale clinical practice and training, and comprehensive evaluation methods and standards are established, AR HMDs are expected to play a more important role in clinical medicine.

Discussion

Hardware Performance Evaluation

Considering the hardware characteristics of AR HMDs, the following suggestions are made for constructing evaluation schemes: determining the components to be tested based on functional priority; employing standardized evaluation platforms with adjustable high-precision parameters to ensure measurement accuracy; and designing experimental parameters to simulate real clinical environments and human visual characteristics.

Specifically, the hardware evaluation of medical AR HMDs should first clarify the functional importance of key components such as displays, cameras, and processors, and determine the evaluation priority accordingly. For the performance evaluation of specific components, a dedicated experimental measurement platform should be constructed to precisely control the spatial position of the measuring instruments relative to the HMD and flexibly adjust the evaluation parameters. In addition, objective measurement tools should be used as much as possible to reduce errors introduced by human factors, such as replacing human observation with cameras and manual operations with robotic arms. At the same time, experimental conditions should simulate real application scenarios as closely as possible, including characteristics of the HVS such as IPD and pupil size, to better match the actual clinical usage environment.

Current research on hardware performance evaluation of AR HMDs mainly focuses on the two core components of displays and cameras, but there are still some methodological shortcomings. First, hardware performance evaluation methods have not yet formed a unified standard. For example, in the process of display performance evaluation, the spatial relationship between the measuring instruments and the HMD, the angle, the division of the measurement area, and the simulation methods of visual parameters (such as IPD and iris size) have not been unified. At present, only a few studies have attempted standardization through 3D printing or optical experimental platforms.24

Second, the current evaluation methods often fail to fully reflect the real clinical usage environment. Most experimental settings are overly idealized, ignoring actual scene factors such as physiological micro-movements of patients, changes in environmental lighting, interference in marker recognition, and movements of personnel in the operating room. This may lead to discrepancies between the actual performance of the device in real applications and the experimental results.68 Therefore, future research should focus on developing more objective evaluation methods that are closer to real clinical applications.

In addition, considering the diversity of medical applications and the high heterogeneity of AR HMD devices (such as display technologies, camera types and quantities), when conducting hardware evaluation, targeted evaluation scheme design should be carried out according to specific clinical needs. Other key hardware components, such as processors, microphones, speakers, and sunshades, should be selectively tested and evaluated based on their impact on device functionality. Currently, research on hardware evaluation of AR HMDs is still insufficient, and the reason is the diversity of device structures and the differences in manufacturers’ design goals. In the future, research in these areas should be further strengthened to promote the systematic and objective construction of the hardware performance evaluation system for AR HMDs.

Future work must prioritize establishing standardized and systematic frameworks for evaluating AR HMD hardware performance. This requires unified protocols defining consistent measurement environments, device configurations, and parameter settings to ensure reproducibility and comparability across studies. Crucially, evaluation scenarios need higher ecological validity, achieved through testbeds that realistically simulate dynamic visuals, complex lighting, and human movement. Incorporating automated and intelligent assessment tools can significantly enhance objectivity and efficiency. Furthermore, the scope of evaluation should broaden beyond core displays to encompass critical supporting components like processors, microphones, and speakers, focusing on aspects such as multimodal interaction, noise handling, and thermal performance. Finally, fostering cross-institutional collaboration and developing open-access performance databases are essential for the continuous improvement and validation of evaluation methodologies.

Software Performance Evaluation

Software, as the core driving force of AR HMDs, plays an important role in the interaction between virtual and real scenes and the effectiveness of clinical applications, with a focus on registration accuracy, the fidelity of medical image rendering, and the smoothness of clinical applications. However, there are still some limitations in the evaluation methods for these important software performance aspects, mainly concentrated in the following areas: the objective evaluation of virtual-to-real registration accuracy of AR HMDs, where traditional evaluation methods rely on subjective human perception, and individual differences make it complex to establish standardized and objective evaluation methods; the consistency of rendering across heterogeneous platforms, where differences in rendering capabilities of different rendering engines may affect the quality of medical images; and the lack of specialized software frameworks tailored for surgical environments and corresponding evaluation standards. To address these challenges, the following points are suggested when constructing software performance evaluation methods for AR HMDs:

First, establish objective registration accuracy evaluation methods. Traditional approaches rely on user visual judgment, which is susceptible to individual differences. Subsequent HMD registration evaluation can combine machine vision to automatically recognize registration feature points, record spatial coordinates through robotic arm movement, or directly calculate through algorithms. This can replace or reduce the subjective intervention of the human eye and improve the quantification accuracy of registration errors.

Second, evaluate the consistency of rendering across heterogeneous platforms. Although current medical AR applications are mostly based on general consumer-grade rendering engines, these platforms have not yet standardized the rendering of medical images in terms of color, grayscale, and depth perception. It is recommended to develop a unified image test toolkit that includes standard color cards, medical images, and dedicated test patterns to quantify rendering differences across platforms in multiple dimensions, thereby addressing the issue of rendering heterogeneity across platforms. Additionally, construct specialized evaluation software frameworks tailored for medical scenarios. Current software evaluation methods lack a standardized evaluation system specifically designed for medical environments, making it difficult to ensure the effectiveness of software in actual medical workflows.

Future work must also establish specialized software evaluation frameworks for medical AR, addressing the current lack of standardized assessment criteria tailored to clinical settings which hinders validation of real-world effectiveness. Progress requires developing high-fidelity automated methods for assessing registration accuracy under clinically relevant conditions using sensor fusion and computer vision. Concurrently, medical-oriented rendering quality benchmarks, aligned with diagnostic needs and supported by curated datasets and core fidelity metrics, are essential. Furthermore, context-specific validation frameworks—such as those for surgical navigation or rehabilitation—must incorporate real-world variables and undergo multicenter validation. Advancing standardization through universal protocols, open toolkits, and AI-assisted analytics is crucial across these efforts. Together, these developments will enhance objectivity, standardization, and critically, clinical relevance, securing a robust evaluative foundation for the reliability and safety of medical AR software.

Usability Assessment

Usability assessment aims to analyze the efficiency and accuracy of healthcare workers operating AR HMDs in actual diagnosis, treatment, or training processes, involving both subjective experience and objective operational performance dimensions. Unlike hardware and software performance evaluation, usability assessment places greater emphasis on user perception and human factors engineering. Current related research mainly focuses on several dimensions such as the visual information processing system, lighting environment, and clinical usability to evaluate the degree of AR HMD usability. In constructing evaluation methods, the following points should be considered for the common characteristics of usability:

Comprehensive assessment of multi-level perception factors, in addition to the routine subjective questionnaire surveys, objective measurement methods should be added, such as depth perception tests, completion time of operational tasks, and action error rates, to reduce the bias caused by pure subjective evaluation and ensure that the assessment results are more comprehensive and reliable.99

Establish comprehensive evaluation criteria closely linked to clinical endpoints. For different types of surgeries (such as spinal pedicle screw surgery, dental implant surgery, and soft tissue resection), differentiated clinical effectiveness indicators should be developed, such as the Gertzbein-Robbins screw placement grading criteria or dental implant angle error indicators, and compared and analyzed with traditional surgical methods to scientifically evaluate the added value and potential risks brought by AR HMDs.

In addition to the usability evaluation aspects discussed above, there are many other factors that need to be considered in the usability evaluation of AR HMDs. This part mainly involves factors such as human-machine systems, human-machine interface design, and mental workload in human factors engineering. For example, the impact of system latency in human-machine systems on usability can be comprehensively judged by combining subjective questionnaire surveys with objective indicators in specific operations, such as the total number of sutures and the number of incorrect sutures in simulated suture surgery tasks, to assess the actual impact of system latency on surgeons.100 The comfort of using HMDs and the intraoperative load can be quantified in terms of their impact on usability. Studies can use surface electromyography technology to assess the burden on neck muscles or use inverse dynamics models to quantify the load on the cervical spine, to guide the optimization design of the device and comprehensive preclinical testing, and to reduce the fatigue risk and operation error rate of medical staff.101 Other usability factors that need to be considered include, but are not limited to, the impact of the device’s FOV on the senses,102 ease of use,99 privacy and security,103 and human factors design of wearable systems.104 Through comprehensive and systematic usability evaluation, the effectiveness and safety of AR HMDs in medical applications can be ensured, and the experience and satisfaction of medical staff and patients can be further improved.

Future research should establish a standardized, modular evaluation toolkit integrating core technical metrics, user experience, clinical efficacy, and non-technical factors like device-management workflows;Objective measures—including eye-tracking, task performance, and physiological signals—must enhance comprehensiveness and comparability across clinical contexts.105 Longitudinal real-world usability studies examining sustained performance, cumulative fatigue, learning curves, and team collaboration under authentic clinical stressors remain critical. Deepening interdisciplinary collaboration among human factors, clinical medicine, computer science, and hospital administration is essential to develop holistic frameworks balancing technical performance, user experience, clinical benefit, safety, and organizational adoption, thereby fully addressing real-world clinical deployment complexity.

Conclusions

Conventional AR HMDs were engineered for industrial contexts and evaluated mainly through subjective impressions and rudimentary metrics. Their rapid penetration into healthcare has exposed the inadequacy of these methods under stringent medical-device regulations and clinical specificity, manifesting in three limitations:

1. heterogeneity barriers: absence of unified hardware/software standards.

2. clinical detachment: test conditions fail to mimic real medical environments.

3. subjective bias: perceptual metrics lack objective quantification.

To address these gaps, we propose five design principles for AR HMD performance testing:

1. identify key components by core function.

2. prioritize testing by functional contribution.

3. replicate authentic clinical and human-visual conditions.

4. objectify subjective perception.

5. test functionally linked components jointly.

The framework may not cover all edge cases and requires large-scale validation and refinement.

Disclosure

All authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.