Reliability and Validity of 3D Limb Scanning for Ankle Foot Orthosis Fitting

Abstract

Background:

Recent decreases in the cost of 3D scanners and improved functionality have resulted in increased adoption for ankle foot orthosis (AFO) fittings, despite limited supporting data. In order for 3D limb scanning to be a feasible alternative to traditional casting methods, a consistent and accurate representation of limb geometry must be produced at a reasonable cost.

Objective:

To evaluate the repeatability and validity of multiple lower limb measurements obtained using low-cost 3D limb scanning technology.

Study Design:

Prospective, randomized, crossover-controlled, cross-sectional, reliability and validity study.

Methods:

Physical measurements and 3D limb scans were completed for 30 participants. Eleven measurements were selected for comparison based on their relevance to AFO fittings. Validity was assessed by comparison of physical and scan based measures using Pearson’s correlation coefficients and root mean square differences. Reliability was assessed using intraclass correlation coefficients and minimal detectable change values. Bland-Altman plots were generated for data visualization.

Results:

All correlation values were above or equal to 0.80. Most ICC values were above 0.95. MDC values for physical and scan based measurements differed by less than 2.0 mm. Scan MDC values were around or below 4mm for foot and ankle measures, and under 6mm for circumference and length measures.

Conclusions:

The results of this study demonstrate that low-cost 3D limb scanning can be used to obtain valid and reliable measurements of 3D limb geometry for the purpose of AFO fitting, when collected using the clinically relevant standardized conditions presented here.

BACKGROUND

A well-fitting ankle foot orthosis (AFO) is essential for many individuals with below knee functional deficits (e.g. traumatic injury, or peripheral neurologic disease) to participate in activities of daily living. Recent decreases in the cost of 3D optical scanning systems (available for less than $400 at the time of this study) along with improved functionality, have resulted in increased adoption for clinical prosthetic and orthotic fittings.^1–5 Previously available systems were generally complex, economically implausible alternatives to manual casting methods for device fitting.^{3, 4, 6–11} Current limb scanning technology has several potential advantages over manual casting techniques and can lead to the development of improved methods for fitting prosthetic and orthotic devices.^{6, 8, 10, 12–14} Limb scanning can potentially improve the fitting experience for some patients by decreasing the time, mess and effort required to capture limb geometry. Further, the ability to retain scans indefinitely provides a consistent starting point for modifications to improve fit, and allows objective documentation of limb changes over time to support clinical decision making.¹⁴ The digital format of scans allows easy sharing between clinics and providers, and eliminates the space requirements for storing casts and positive limb models. Further, after the modest initial cost of equipment, scanners have the potential to provide cost savings over traditional casting methods by eliminating recurring supply costs.¹² The ability to improve scan based fittings is dependent on an understanding of the limitations of each step of the clinical process, including scanning.

Limb scanning has been widely used in prosthetic fittings, with well-established accuracy and reliability.^{1–3, 6, 7, 9, 10, 14–17} Despite the growing interest and feasibility of this technology for providing AFOs, there is limited evidence in the literature. Unlike residual limbs following amputation, which are generally cylindrical/conical and can be better suited for scanning and reconstruction, the shape captured for an orthosis is more geometrically complex. This limits the ability to infer that the findings from studies related to 3D scanning of residual limbs applies to AFO fitting. Further, previous studies investigating the validity of optical scanners have primarily evaluated limb volumetry using water displacement or inanimate objects of known dimensions as their gold standard for comparison.^{2–4, 6, 7, 14, 16, 17} While measures of limb volumetry provide an objective indication of scan accuracy, volumetry measures have limited applicability for AFO fittings given the need for dimensional accuracy across opposing bony prominences, and do not directly align with the clinical use of limb scanning for orthosis fitting.³ Objective data is needed to understand the accuracy of width and length measures around the foot and ankle. The limited number of studies related to the use of 3D scanning for foot orthosis fitting provide initial promising data for AFO fitting.^{12, 18} Telfer et. al (2012) evaluated the reliability of a limited number of foot measures used in foot orthosis fitting (length, width at forefoot, width at rearfoot, and peak medial arch height), however, only some scan measures met the criteria for excellent reproducibility (ICC > 0.90).¹² While relevant, the study evaluates only a portion of the geometrical parameters relevant to AFO fitting.

The growing accessibility of Additive Manufacturing (AM) for device fabrication using 3D printing, and use of central fabrication, is increasing interest in 3D limb scanning for patient specific AFOs.^{2, 8, 10–13, 15, 19} Combining limb scanning with AM has the potential to provide a repeatable quantitative alternative to the normally qualitative fitting process, while also decreasing fabrication time and cost.¹³ In limited testing, custom orthotic devices manufactured with 3D scanning and AM have achieved improved geometrical fit with similar function compared to prefabricated orthotics.^{10, 11, 13} Schrank and Stanhope (2011) found that 3D fabrication for AFOs can achieve dimensional accuracies within 1mm, however, the reliability and accuracy of the limb geometry captured during scanning is unknown.¹⁹ The ability to use AM approaches to consistently provide quality fitting AFOs is dependent in part on the ability to obtain accurate and reliable limb geometry using a 3D limb scanning system.

Systematically and objectively improving the orthosis fitting process, and ultimately the patient experience, is reliant in part on the consistent and accurate representation of limb geometry at a reasonable cost. The purpose of this study is to evaluate the repeatability and validity of multiple lower limb measurements, pertinent to the AFO fitting process, obtained using low-cost 3D limb scanning technology. Specifically, we will evaluate intrarater test-retest reliability, intrarater and interrater test-retest reliability and criterion validity of 3D limb scanning.

METHODS

Subjects

Thirty healthy, able-bodied individuals (13 male, 17 female) between the ages of 18 to 75 years were recruited for this study. Subjects were on average 34.9 (SD 16.0) years old with a mean body mass and height of 71.5 (SD 11.3) kg and 1.7 (SD 0.1) m, respectively. All subjects provided written informed consent prior to participation in this institutional review board approved study.

Procedure

Eleven measurements were identified for comparison based on clinician input and their relevance to AFO fitting as described in Table 1. These reference points were used for all measurements to ensure direct comparison between physical caliper measurements and software-based measurements. [Table 1] All subjects completed a single collection session. Following consent, eighteen 1.9 cm (0.75 in) diameter round adhesive stickers were placed over anatomical landmarks identified using manual palpation. A 0.5mm black dot was placed in center of each sticker and was used as the reference point for direct comparison of physical measurements and scan-based measurements.

Table 1.

Measurements evaluated and corresponding anatomical reference points for marker placement.

Measurement	First marker	Second marker
Width of the metatarsal heads	Medial aspect of first metatarsal head	Lateral aspect of fifth metatarsal head
Width of the calcaneus	Medial calcaneal tuberosity	Lateral calcaneal tuberosity
Foot height	Most superior point on the foot distal to the tibialis anterior insertion	Plantar surface
Medial arch height	Most superior point on the medial longitudinal arch	Plantar surface
Medial-lateral width between ankle malleoli	Medial malleolus	Lateral malleolus
Foot length	Most posterior aspect of calcaneus	Most distal aspect of phalynx
Minimum circumference above the ankle malleoli	Minimum circumference less than 10 cm proximal to the ankle malleoli
Maximum calf circumference	Maximum calf circumference greater than 5 cm distal to the knee condyles
Medial-lateral width of the knee condyles	Medial aspect of medial condyle	Lateral aspect of lateral condyle
Anterior-posterior width at mid patellar tendon	Mid patellar tendon	Popliteal surface posterior to mid patellar tendon
Distance from bottom of foot to tibial tubercle	Tibial tubercle	Plantar surface

To capture the entire lower limb geometry, participants stood on a translucent 40 cm by 40 cm (16 in by 16 in) acrylic cube with 1.9 cm (0.75 in) thick walls, shown in Figure 1. The cube design included one open side so the scanner could be positioned to visualize the underside of the foot. Subjects stood in a relaxed position with equal weight on both limbs as measurements were taken. [Figure 1] The web-based tool Research Randomizer was used to determine whether scans or physical measurements were completed first for each participant, to account for potential order effects or changes in limb geometry during the session.²⁰

Figure 1.

Translucent acrylic cube used for 3D limb scanning.

The OriginCal IP54 digital caliper (Anytime Inc., Granada Hills CA) was used to take three consecutive physical measurements in millimeters at each identified measurement location. For measurements outside of the caliper’s scope—minimum circumference above the ankle malleoli, maximum calf circumference, and distance from bottom of foot to tibial tubercle—a tape measure was used in place of the caliper. Physical measuring devices were reset to zero between each measure. Three 3D scans of each participant’s lower limb geometry were captured using an Original Structure Sensor scanner (Occipital, Inc., San Francisco CA), using a non-visible infrared structured light projection. The scanning system and corresponding software is currently used in clinical practice without established reliability and validity data. Collection of scans followed procedures currently used in practice at the time of this study (author J.P.). The scanner was connected to an iPad (Apple Inc., Cupertino CA) and the Scanner app by Standard Cyborg, Inc (San Francisco CA) was used to capture limb geometry. In the scanning app, the limb was first identified, and the scanning volume was fit to the subject’s limb. Once the scan was in progress, the rater slowly rotated the iPad around the entire limb to ensure all geometry was captured, with each scan taking approximately 60–90 seconds. During scanning the scanner was often moved slowly over a single location and at different angles, using the visual display to ensure adequate coverage and visually assess for image refinement. The same methods and order were repeated sequentially for each participant to determine test-retest reliability.

After the collection, scans were evaluated using the measurement tools within Design Studio software (Standard Cyborg, Inc., San Francisco CA), a product that currently is used in prosthetic and orthotic clinics. Digital measurements were taken in Design Studio for the three scans at each time point by two raters [Figure 2]. All raters were provided formal training on the software, consisting of practice image evaluation with the primary rater, before performing study measurements. Each rater independently determined the x-y-z orientation of the scan, completed all measurements, and were blind to the other rater’s measurements. If a rater performed physical measurements with a participant, they waited a minimum of 24-hours before completing digital measurements to avoid the potential for recall to influence measurements.

Figure 2.

Screen capture displaying limb geometry with markers obtained using the 3D scanner.

Statistical Analysis

Microsoft Excel 2016 (Microsoft Corp., Redmond, WA) was used to confirm data normality, calculate Pearson product-moment correlations, minimal detectable change (MDC) values, root mean square (RMS) differences, and construct Bland-Altman plots. The Shapiro-Wilk test was used to verify normality of the data along with box plots, histograms, and quantile-quantile plots. Pearson product-moment correlations were calculated using the function PEARSON (array1, array2) and categorized based on the scale of negligible (0–0.30), low (0.30–0.50), moderate (0.50–0.70), high (0.70–0.90), and very high (0.90–1.0) correlation.²¹ Standard error of the measurements (SEM) was calculated using the equation SD x SQRT (1-ICC), where SD is the pooled variance. MDC was calculated using the equation SEM x 1.96 x SQRT (2). SPSS v. 25 (SPSS Inc., Chicago, IL) was used to calculate the intra-class correlation coefficient (ICC) using model (2,k) and categorized based on the scale of poor (0–0.49), moderate (0.5–0.74), good (0.75–0.89) and excellent (0.9–1.0) reliability.²² Intrarater-intersession ICC values were calculated to determine the test-retest reliability of scanning and digital measurement, while interrater-intrasession, and interrater-intersession ICC values were calculated to determine the reliability of digital measurements. Intrarater-intersession ICC values were calculated to determine the reliability of physical measures between sessions.

RESULTS

Correlation Coefficients and Mean RMS Differences

Pearson’s correlation coefficients, mean RMS differences, and MDC values for all measurements are presented in Table 1. All physical and scan between measure correlations were greater than or equal to 0.80 indicating high correlation. The average correlation for foot and ankle measurements used in AFO fittings (width of metatarsal heads, width of calcaneus, foot height, medial arch height, medial-lateral width between ankle malleoli) was 0.88. Circumferential and length measurements (foot length, minimum circumference above the ankle malleoli, maximum calf circumference, medial-lateral width of the knee condyles, anterior-posterior width at midpatellar tendon, and distance from bottom of foot to tibial tubercle) had an average correlation of 0.96. The mean RMS difference between caliper and scan foot and ankle measurements was 3.2 mm and 5.8 mm for circumferential and length measurements, respectively.

Bland-Altman Plots

Bland-Altman plots with a 95% confidence intervals (see file, Supplemental Digital Content 1) between physical and scan measures indicated a mean percentage error of −0.42±4.01 % for foot and ankle measures and −1.81±1.52 % for circumferential and length measures. Mean difference values between physical and scan measures for foot and ankle measures were −0.60±1.96 mm and −2.59±1.78 mm for circumferential and length measures. Mean percentage errors for scan intrarater test-retest (see file, Supplemental Digital Content 2) were 0.32±1.32 % for foot and ankle measures and −0.41±0.41 % for circumferential and length measures. Mean difference values for scan intrarater test-retest were −0.03±0.39 mm for foot and ankle measures and −0.62±0.89 mm for circumferential and length measures.

ICC

ICC values and confidence intervals for the rater and session assessments of caliper and scan reliability are presented in Table 2. Most ICC values in table 2 were within the excellent range (0.90–1.00), with a minimum ICC value of 0.84. The lowest ICC values were for the interrater-intersession comparison of foot height and medial arch height measures, at 0.84 and 0.85 respectively. ICC values lower than 0.95 in Table 2 are underlined.

Table 2.

Pearson’s correlation coefficient and mean RMS difference values presented for between caliper and scan measures. MDC values are presented for: caliper intrarater-intersession, scan intrarater-intersession, scan interrater-intrasession, and scan interrater-intersession.

Measurement	Pearson’s Mean RMS difference (mm)		MDC (mm)
	Caliper vs. Scan		Caliper intrarater-intersession	Scan intrarater-intersession	Scan interrater-intrasession	Scan interrater-intersession
Width of the metatarsal heads	0.93	2.4	1.1	2.8	2.2	3.1
Width of the calcaneus	0.93	4.2	0.8	2.8	1.1	3.0
Foot height	0.80	3.4	2.1	2.8	4.9	5.9
Arch height	0.85	3.2	2.3	3.6	3.4	5.8
Medial-Lateral width between ankle malleoli	0.87	2.8	1.0	2.6	1.2	2.7
Foot length	0.99	3.0	1.5	3.2	2.3	4.3
Minimum circumference above the ankle malleoli	0.93	8.7	1.4	7.5	1.6	8.1
Maximum calf circumference	0.96	7.2	4.3	5.4	2.2	5.8
Medial-lateral width of the knee condyles	0.98	3.9	3.4	4.2	1.4	4.2
Anterior-posterior width at mid patellar tendon	0.93	6.0	4.8	3.7	1.5	3.7
Distance from bottom of foot to tibial tubercle	0.97	5.8	3.6	5.2	6.3	7.9
Average:
Foot and ankle	0.88	3.2	1.5	2.9	2.5	4.1
Circumference and length	0.96	5.8	3.2	4.9	2.5	5.7

MDC

The average MDC for foot and ankle measures using calipers and scanning differed by less than two millimeters on average. Caliper intrarater-intersession, scan intrarater-intersession, scan interrater-intrasession, and scan interrater-intersession MDC values were 1.5 mm, 2.9 mm, 2.5 mm, and 4.1 mm, respectively. For circumferential and length measurements average MDC values for caliper intrarater-intersession, scan intrarater-intersession, scan interrater-intrasession, and scan interrater-intersession were 3.2 mm, 4.9 mm, 2.5 mm, and 5.7 mm, respectively.

DISCUSSION

The results of this study demonstrate that 3D limb scans obtained using this low-cost system can provide a valid representation of superficial limb geometry. Further, 3D scanning-based limb measurements can be obtained with excellent reliability. This includes excellent intrarater reliability for the entire process and interrater reliability for scan-based measurements. Custom AFOs often require a precise fit around the bony geometry of the foot, ankle, and proximal cuff. As a result, more accurate measurements and better reliability is needed around the foot and ankle relative to areas where the device contacts compliant soft tissues. Data from this study demonstrates generally higher ICC and lower MDC values for measures around the bony prominences of the foot and ankle. Further, the minimal detectible change values for manual measurement using precise calipers and 3D limb scans differed by less than 2.0 mm on average.

Limited data is available describing the validity of limb geometry measurements obtained using low-cost 3D scanning, slowing the adoption of scanning for the AFO fitting process. Although scanning system manufacturers provide specifications for their systems under optimal conditions, the extent to which these values represent the accuracy of clinical limb scans is unknown. Most available validity data from previous 3D limb scanning investigations indicate limb volume can be accurately assessed, ^{2, 7, 9, 16} however, volumetric data has limited relevance to the AFO fitting process. Rather, validity data is needed for width, length, and circumferential measures. In the present study we directly compared physical measurements and scan-based measurements on a real limb, collected in a manner consistent with clinical limb scanning techniques. The physical and 3D scan-based measurements were highly associated for foot and ankle measures with average correlation coefficients of 0.88 and a mean RMS difference of 3.0 mm. Similarly, the average correlation coefficient was 0.96 for circumferential and length measures, with a mean RMS difference of 5.8 mm.

Limited data is available displaying Bland-Altman plots for distance or circumference measures obtained using low-cost 3D scanning. Previous studies evaluating similar measures have mostly assessed volumetric and circumferential measures of the limb for prosthetic applications.^{3, 5–7, 16} Mean differences in the data presented here trend in the negative direction for the scan and physical measure comparison, showing that scan measures are slightly larger than physical measurements. Given potential fit and comfort issues with AFOs that are too small, a larger rather than smaller device may be desirable if measurement error is present.

Few studies evaluating the reliability of 3D limb scanning provide ICC values for measures relevant for AFO fitting. ICC values can provide insight into how reliable measurements are across different raters and sessions. Most previous reliability studies, which generally demonstrate good to excellent ICC values, primarily evaluate limb volume. ^{1, 3, 6, 7, 16} Telfer et al. assessed scanning and casting based approaches for evaluating foot geometry demonstrated excellent reliability for most distance measures obtained from a foot scan. ¹² The study used multiple measures collected in a relaxed standing position, similar to those used here, including foot length, forefoot width, heel width, and medial arch height. Foot related measurements in the present study had consistently higher ICC values and generally lower MDC values than those presented in the Telfer study.

Few studies provide MDC data related to the 3D scanning of limb geometry. The MDC measure is important because it’s the minimal amount of change that a measurement must show to be greater than the within subject variability and measurement error, in the unit of measurement.²³ Clinically there is no established objective range of acceptable measurement deviations. Creating a compliant or tight-fitting device is dependent on individual patient preference. Telfer et al. provided MDC values for foot length, forefoot width, heel width, and medial arch height, which ranged from 1.9 mm to 21.3 mm. ¹² The MDC values presented in Table 1 are generally similar to, or lower than, the limited number of MDC values presented by Telfer. Based on the results of the present study and experience using the system in clinical practice (author J.P.), clinicians can expect 3D scanned foot and ankle geometries to differ by less than 5mm within raters for scan collection and assessment, and between raters for scan assessment.

Limitations

This study evaluated the surface limb geometry and not underlying bony structure, and further investigation is warranted to determine if and how the geometries obtained using 3D scanning differ from manual casting techniques. Further, subsequent studies could evaluate the consistency of limb geometry obtained using manual casting and 3D limb scanning. The collection of scans could have been made easier by using a scanning apparatus higher than 40cm off the ground to allow easier access to the plantar surface. Although limb geometry is determined in the software, the interrater reliability of collecting the scan was not evaluated, therefore consistency of this step between individuals was not evaluated. Additional evaluation of interrater test-retest reliability for the scanning process is recommended to further support clinical use.

CONCLUSION

The results of this study demonstrate that valid and reliable measurements can be obtained using low-cost 3D limb scanning for the purpose of AFO fitting, when collected using the clinically relevant standardized conditions presented here. High agreement was found between physical and scan-based measurements. Excellent interrater reliability was obtained for scan-based measures. Excellent intrarater test-retest reliability was established for the scanning process. MDC values for intrarater test-retest reliability were typically around or below 4mm for foot and ankle measures, and under 6mm for circumference and length. MDC values changed by approximately one millimeter on average when adding additional raters or comparing over time, demonstrating the stability of scan-related measures and further supporting the use of 3D scanning in clinical practice.

Supplementary Material

SDC 2

Supplemental Digital Content 2. docx

SDC 1

Supplemental digital Content 1. docx

Table 3.

ICC values are presented for: caliper intrarater-intersession, scan intrarater-intersession, scan interrater-intrasession, and scan interrater-intersession. Values less than 0.95 are underlined.

Measurement	ICC (95% CI)
	Caliper intrarater-intersession		Scan intrarater-intersession		Scan interrater-intrasession		Scan interrater-intersession
Width of the metatarsal heads	0.99	(0.99–0.99)	0.98	(0.95–0.99)	0.99	(0.97–0.99)	0.97	(0.94–0.99)
Width of the calcaneus	0.99	(0.99–1.00)	0.98	(0.95–0.99)	0.99	(0.99–0.99)	0.97	(0.94–0.99)
Foot height	0.97	(0.94–0.99)	0.97	(0.93–0.98)	0.89	(0.77–0.95)	0.84	(0.67–0.92)
Arch height	0.98	(0.96–0.99)	0.94	(0.88–0.97)	0.95	(0.90–0.98)	0.85	(0.68–0.93)
Medial-Lateral width between ankle malleoli	0.99	(0.99–1.00)	0.96	(0.92–0.98)	0.99	(0.98–0.99)	0.96	(0.91–0.98)
Foot length	0.99	(0.99–1.00)	0.99	(0.99–1.00)	0.99	(0.99–1.00)	0.99	(0.98–0.99)
Minimum circumference above the ankle malleoli	0.99	(0.99–1.00)	0.98	(0.95–0.99)	0.99	(0.99–1.00)	0.97	(0.94–0.99)
Maximum calf circumference	0.99	(0.99–1.00)	0.99	(0.98–0.99)	1.00	(0.99–1.00)	0.99	(0.98–0.99)
Medial-lateral width of the knee condyles	0.99	(0.97–0.99)	0.98	(0.96–0.99)	0.99	(0.99–1.00)	0.98	(0.96–0.99)
Anterior-posterior width at mid patellar tendon	0.96	(0.92–0.98)	0.98	(0.96–0.99)	0.99	(0.99–1.00)	0.98	(0.96–0.99)
Distance from bottom of foot to tibial tubercle	0.99	(0.99–1.00)	0.99	(0.99–1.00)	0.99	(0.98–0.99)	0.99	(0.97–0.99)
Average	0.98		0.98		0.98		0.95