An Overview of Activities at NIST Towards the Proposed ASTM E57 3D Imaging System Point-to-point Distance Standard

1 Introduction

Terrestrial laser scanners (TLSs) are a class of 3D imaging systems that measure the 3D coordinates of an object in their work volume. They capture the 3D coordinates using a ranging unit mounted on two rotation stages that are orthogonal to each other as illustrated in Figure 1. They are used for a variety of applications, e.g., large scale assembly, surveying, forensics, reverse engineering etc. The Dimensional Metrology Group (DMG) at the National Institute of Standards & Technology (NIST), along with various other organizations has been involved in the development of a documentary standard for 3D imaging instruments that acquire data in spherical coordinate system. NIST led this effort and possesses unique expertise for this task. This expertise comes from NIST’s prior experience on standardization activities related to laser trackers which also use spherical coordinate system and have very similar error sources.

Figure 1.

Schematic of a terrestrial laser scanner (TLS)

TLSs were being used at NIST in several research projects starting in the early 2000s. It soon became clear that there was need to evaluate the instruments and determine the uncertainty of the measurements obtained. In this context, NIST organized three workshops between 2003 and 2006 which convened instrument manufacturers, end users and organizations like NIST to determine the needs of all the stake holders [1,2]. During these workshops the participants agreed upon the need for standard terminology, artifacts and standardized protocols that are needed for evaluating TLSs. Based on these workshops, an ASTM subcommittee on 3D imaging systems was established in 2006, and a working group started work on evaluating the fundamental measurement of 3D imaging systems, their range. In 2015, this working group published the ASTM E2938-15 standard for 3D imaging systems to evaluate relative-range. [3].

The scope of the ASTM E2938-15 standard was limited to the evaluation of the relative-range of 3D imaging systems that acquire data in both the spherical and non-spherical coordinate systems. As this standard was being balloted, another working group was established in 2013 [4] within the E57.02 sub-committee that started addressing the performance evaluation of these instruments over their entire work volume. The proposed standard titled “Standard test method for evaluating the point-to-point distance measurement performance of spherical coordinate 3D imaging systems in the medium range” was submitted for balloting in the spring of 2017.

Challenges in developing the proposed standard included the lack of commercially available high quality targets, methods to obtain ground truth measurements and information about the sources of error. Other challenges included target mounting methods, data collection and data post-processing algorithms and to develop the performance criteria. The methods in the proposed standard were realized and tested at NIST before being incorporated into the document.

This paper will present an overview of the work, the procedures and recommendations for evaluating TLSs that found consensus among the members of the working group and were included in the proposed standard.

2 ASTM standard for point-to-point distance evaluation

2.1 Motivation for the new standard

At the outset of this work and as of the writing of this paper, no standard exists for evaluating TLS systems over their entire work volume. TLS manufacturers typically specify the instrument performance based on non-standard targets using characteristics and data throughput rates that are not consistent. Therefore, it is difficult for an end user to make an informed decision about their TLS purchase for technical, acceptance or warranty related purposes. As the ASTM E2938-15 standard only evaluates the relative-range performance, the end user does not have a standardized way to evaluate the overall instrument performance. Such an evaluation is required because most real-world applications involve measurements that span the entire instrument work volume.

Another motivation for the proposed standard is the specific application/requirement from the industry. For example, one large scale US manufacturer approached NIST and was interested in determining the performance of their TLSs from technical and commercial perspectives. In this context, NIST worked with them under a cooperative research & development agreement (CRADA). Under this agreement, various concepts were explored to understand the issues with evaluating such instruments. Two other TLS manufacturers collaborated with NIST to evaluate their instruments as NIST was in the process of developing TLS performance evaluation procedures.

2.2 Development of the standard and the run-off meeting at NIST

To address the lack of standardization of TLSs, an ASTM working group was constituted under the leadership of NIST to develop this new standard. The participants included manufacturers, experts, end users from the industry, and researchers from NIST and the National Research Council (NRC) of Canada. The mode of development involved bi-weekly internet based teleconference meetings that started in the summer of 2013 and extended into the spring of 2017. Participants of these meetings discussed a variety of topics, ideas, solutions, tested them at their facilities to evaluate their feasibilities and presented their findings. A substantial amount of raw data and information was exchanged in this process. Almost all the methods considered for this standard were tested before incorporation into the proposed standard.

In the spring of 2016, a four-day run-off meeting was held at NIST to evaluate the initial proposed methods. Five TLS manufacturers, representing most of the TLS market share, participated in this run-off meeting and tested their instruments using the proposed standard. The purpose of this run-off meeting was to verify whether the tests could be successfully performed by a variety of instruments and to obtain feedback in terms of feasibility and efficiency. After the tests concluded, the manufacturers gave considerable feedback that resulted in several major changes to the proposed standard. Some of the changes include reducing the number of tests while maintaining the rigor of the evaluation and retaining the use of the flat plate target for the relative-range tests. Details of this run-off meeting were documented by Muralikrishnan et al. [5,6].

2.3 Scope of the proposed standard and realization of the tests

The primary sources of errors in TLS measurements were deemed to be due to the instrument construction, target characteristics and data processing. Though environmental conditions could affect the results, they were not considered as sources of errors in the proposed standard. This is due to the fact that the tests were performed within rated operating conditions, and the environmental effects are addressed in the test uncertainty. Subsequently, the scope of the standard was restricted and the rationale is detailed in the next few sub-sections.

2.3.1 Type of instruments and the measurand

2.3.1.1 Spherical coordinate system instruments

There are many commercial systems that capture 3D data using different technologies (e.g., flash lidar, TLSs etc.). The ASTM E2938-15 standard included the evaluation of all such instruments. The scope of this new proposed standard however was limited to systems that acquire data in a spherical coordinate frame. This was primarily done for the following reasons.

1. Researchers at NIST have been involved in documentary standards for other instruments like laser trackers (e.g., ASME B89.4.19 and ISO 10360-10 standards) that use spherical coordinate systems which are similar in construction to TLSs. The sources of errors for such instruments were extensively studied and NIST researchers understood the issues with spherical coordinate 3D imaging systems.

2. The large volume 3D imaging systems, typically required for surveying and manufacturing assembly, use a spherical coordinate frame for acquiring data. Other instrument designs require performance tests that are sensitive to their error sources.

2.3.1.2 Derived point to derived point distance evaluation

TLSs capture data without the necessity of a cooperative target. They measure the distance of an object/target based on the reflected light from the object/target. Depending on the properties of the target surface, the noise in a single point measurement can be large. More importantly, most TLSs cannot measure only a single point as they are meant to operate in the scanning mode. To enable a standardized method of comparison and evaluation of these instruments, the concept of a derived point was introduced. A derived point is a point computed using multiple measured points on a target surface. It is a point that corresponds to the 3D point cloud of an object and may be one geometric parameter of that object. E.g., the center of a sphere derived from a scan of a sphere or the apex of a pyramid obtained from intersecting the planes of a pyramid. Use of a derived point enables the characterization of instrument construction errors by suppressing the effect of noise associated with a single point.

2.3.2 Targets

Though TLSs do not require a cooperative target, like a laser tracker, they can still benefit from using specialized targets for obtaining consistent, high quality data required for performance evaluation. The characteristics of the target such as its geometry, color, reflectivity and the associated data processing methods affect the results. The data quality can be improved by choosing a target that minimizes target induced errors. A variety of targets were considered for this process and were studied in detail at NIST [7,8]. Some of these targets included flat plate targets, contrast/checkerboard targets and spheres. Of these, metallic aluminum spheres that were media blasted to give a dull matte finish were found to perform well for some of the proposed tests (two-face and non-ranging point-to-point distance tests) and vapor blasted aluminum plates for relative-range tests. Some of the reasons for choosing these targets will be discussed next.

2.3.2.1 Spheres

Spheres can be suitable targets for evaluating 3D imaging instruments as their geometry appears the same regardless of the direction of the scan. They are typically used for registering multiple scans when a single scan cannot cover the entire region of interest. There are however issues pertaining to the data quality that influence their use in performance evaluation. The derived point for a sphere is its center, and determining the center of a sphere from TLS data is challenging. Recently, Rachakonda et al. [9] studied sphere data sets from several TLSs extensively and proposed novel algorithms and data segmentation techniques to obtain the sphere center. Due to various sources of errors from both the TLS and the sphere target, determining a sphere center is fraught with uncertainty. This is important because the target induced errors may incorrectly be attributed to the instrument construction.

It was observed that a major component of the error in determining the sphere center is in the ranging direction of the TLS [9]. Hence using such targets for relative-range evaluation would lead to a larger test uncertainty. However, sphere targets are suitable for non-ranging point-to-point distance tests that involve measuring targets at approximately the same distance. Sphere targets are also suitable for two-face tests because any ranging direction error introduced due to the spherical geometry is common-mode between the front-face and back-face measurements and is therefore not of any consequence. For these reasons, spherical targets were chosen for all the non-ranging length tests.

To enable a reference length measurement for the point-to-point distance tests (described in the later sections) a special custom sphere was commercially procured. This target, referred to as the “integration sphere^*” (Figure 2), is a partial sphere fabricated with a kinematic nest inside and a scannable surface on the outside. A 1.5 in (38.1 mm) diameter sphere or spherically mounted retroreflector (SMR) mounted in this nest is concentric with the center of the outer partial sphere. These “integration spheres” allow the measurement of their centers using a single measurement from a laser tracker.

Figure 2.

An “Integration sphere” used for point-to-point distance evaluation of TLSs

2.3.2.2 Flat plates & hybrid targets

The ASTM E2938-15 standard mandates the use of planar targets for relative-range tests. The recommendations from that standard have been adopted into this proposed standard based on several tests and studies performed at NIST [10]. In this process, several planar targets were designed and are presented in the appendices of the proposed standard. These targets allow for more efficient realization of the test. These implementations are not mandatory in the proposed standard, but have been included as methods for consideration by the user.

The derived point for such a planar target is its geometric center. Measuring the geometric center of a planar target with both the reference instrument (RI) and the instrument under test (IUT) can be challenging. Any misalignment of the target with respect to the line joining the RI and IUT will result in an abbe error which may be incorrectly attributed to the instrument construction. Various designs of planar targets were explored to minimize such errors due to the alignment process. These targets are depicted (front and back) in Figure 3 and are described next:

1. Flat plate target with SMRs on the back: This target, depicted in Figure 3a, is of approximate dimensions 300 mm × 300 mm × 40 mm and has a media blasted front surface with a flatness of ≈ 20 μm on the front and five kinematic nests to mount SMRs on the back. The front scannable surface was designed per the ASTM E2938-15 standard and the back surface was designed to aid in the alignment process using the RI. Because there are no fiducials on the front surface to locate a point on the plane, different segmentation and data processing methods may result in a different point being calculated as the derived-point of the plate. If the plate is not carefully aligned, and this derived-point does not coincide with the point measured by the RI, then there is an error associated with the measurement that appears as relative-range error, but is in fact due to the test setup.

2. Plate-sphere artifact with 100 mm diameter “integration sphere” at the center: This artifact, depicted in Figure 3b, was designed so as to enable the RI and the IUT to measure a common point in space, thus overcoming the alignment problem that affects the target in Figure 3a. The flat plate is of approximate dimensions 450 mm × 450 mm × 40 mm. A flat plate target results in a derived point with low uncertainty in the ranging direction, but not in the non-ranging direction. A sphere on the other hand results in reliable measurements in the non-ranging direction, but not in the ranging direction. A plate-sphere artifact combines these two geometries and overcomes the inadequacies of either one and results in a better estimate of the target derived point. When an “integration sphere” is used, obtaining the reference value becomes easier. Though this target design was promising, it had a drawback in that the sphere was too small to be measured at longer distances (> 20 m) with some TLSs.

3. Plate-sphere artifact with 200 mm diameter sphere at the center: This artifact, depicted in Figure 3c, was designed to overcome the issue of the 100 mm diameter sphere not being measurable by some TLSs. A larger 200 mm diameter “integration sphere” was used and additional recesses were added around this sphere to minimize the multi-path scattering that affects the determination of the sphere center. The size of the flat plate was approximately 600 mm × 600 mm × 40 mm. This target was scanned by five different scanners during the run-off meeting held at NIST. Though the results were satisfactory, the participants in the meeting expressed concern about the large angle of sweep required to scan this artifact and the lack of data/features at the center of the artifact. A large angle of sweep introduces errors caused by the angular encoders into a relative-range test and the results will not be representative of the relative-range performance of the IUT.

4. Flat plate target with fiducials: To allay the concerns expressed by the participants in the run-off meeting about the previous target design, a new target was designed. The flat plate target depicted in Figure 3a was modified to include two 100 mm diameter “integration sphere” targets to lower the uncertainty associated with derived point calculation. The addition of these two spheres help to reliably segment the data associated with the flat plate without obstructing the planar region of interest. Such a target design could be used with larger diameter “integration spheres” to enable IUT measurements at longer distances.

Figure 3.

Targets considered for relative-range tests

2.4 Test methods, positions and their implementation

The scope of the proposed standard is to evaluate the instrument in its entire work volume. The number of point-to-point distances in a TLS’s work volume is infinite and it will be practically impossible to test an instrument for all possible point-to-point distances. To limit the number of positions, it was important to identify the instrument related errors that contribute most to the performance of an instrument. To understand this, Muralikrishnan et al. [11] identified 18 sources of error that affect TLS performance and developed a detailed error model. All these sources of error are a result of the construction of the instrument, i.e., the imprecision in either fabrication or assembly of these instruments. Examples of these error sources are unintended beam & mirror offsets and tilts, angular encoder related errors, etc. The error model provides an insight into the positions at which certain systematic errors are more prominent than others. The test positions in the proposed standard are based on this error model. It should be noted that any test position may reveal the effect of a combination of error sources and the contribution of each of those error sources may be mathematically deduced [12]. However, such an exercise is not the purpose of the proposed documentary standard.

The purpose of this error model in the context of the standardization activity is to reduce the number of test positions while maintaining sensitivity to all the sources of errors. Based on the error model and the feedback from the manufacturers, the proposed standard mandates 32 tests, namely 12 two-face tests and 20 point-to-point distance tests. The point-to-point distance tests include eight symmetric tests, six asymmetric tests, one inside test, three relative-range tests and two user-defined tests. The two user-defined tests may be performed at the discretion of the end-user in collaboration with the manufacturer. The next sub-sections will describe these tests in detail.

2.4.1 Two-face tests

Two face tests are designed for instruments that can scan both in the frontsight and backsight mode. These modes enable a TLS to rotate its head only 180° to obtain a 360° view by combining scans obtained from both the sides. Not all instruments may have such a mode, however for instruments that have this capability, the two-face test is mandatory per the proposed standard. These tests are sensitive to only 12 of the 18 sources of errors described in [11] and all the 12 sources of error manifest as angular errors in the derived point measurement.

To perform these tests, three spheres are mounted on a vertical column in such a way that one sphere is at an elevation angle ϕ = 0° (level with the origin of the scanner). The second sphere is placed at an ϕ = 45° ± 10° and the third sphere is placed at ϕ = −45° ± 10°. This is illustrated in Figure 4. The derived point of a single sphere measured using the IUT’s frontsight mode will be compared to the derived point of the same sphere measured by the IUT’s backsight mode. No reference length measurement is needed as the stationary spheres are assumed to be not affected by their mounting or environment in the short duration of the test. In essence, a zero-distance length is being measured by scanning the same sphere twice, once using the frontsight and the second time using the backsight of a TLS.

Figure 4.

Setup to perform two-face tests

To perform these tests the scanner is placed at a distance d from the target, with its elevation angle at ϕ and an azimuth angle θ. The various values of d, ϕ and θ are listed in Table 1. Here, the elevation angle is measured with respect to the horizontal plane.

Table 1.

Test positions for two-face tests

Test	Distance of target from IUT, d	Elevation angle $\varphi$ of target	Azimuth angle $\theta$
TF1	Not more than 10 m	45° ± 10°	0°
TF2	Not more than 10 m	0° ± 10°	0°
TF3	Not more than 10 m	−45° ± 10°	0°
TF4	Not more than 10 m	45° ± 10°	90°
TF5	Not more than 10 m	0° ± 10°	90°
TF6	Not more than 10 m	−45° ± 10°	90°
TF7	At least 20 m	45° ± 10°	0°
TF8	At least 20 m	0° ± 10°	0°
TF9	At least 20 m	−45° ± 10°	0°
TF10	At least 20 m	45° ± 10°	90°
TF11	At least 20 m	0° ± 10°	90°
TF12	At least 20 m	−45° ± 10°	90°

Though they are not comprehensive tests, two-face tests offer the following advantages compared to the point-to-point distance tests described in the next sub-section.

1. Simple setup (as illustrated in Figure 4).

2. No reference length measurements are required.

3. Simple sphere targets may be used. They do not need to be “integration spheres”.

4. Two-face tests are quicker to perform than the point-to-point distance tests.

Any significant deviations from the expected values could point to instrument related errors. This may also indicate that some of the point-to-point distance test results may have a similar outcome and these deviations are best addressed before proceeding with further testing.

2.4.2 Point-to-point distance tests

The purpose of the point-to-point distance tests is to evaluate the IUT by measuring the distance between two targets and comparing it with a reference length. The reference length is obtained using an RI that can measure the reference length with an uncertainty value that is at least four times lower than that of the IUT. In the tests performed at NIST, the RI was a laser tracker and the target was an “integration sphere” for the non-ranging tests and a flat plate target for the relative-range tests.

2.4.2.1 Non-ranging point-to-point distance tests

The setup to perform most of these measurements could be either a grid of “integration spheres”, a scale bar with two spheres at the ends, or any other setup where the distances can be realized and oriented in the required position with respect to the IUT. These tests are sensitive to all the 18 sources of error and give a near-complete picture of the instrument performance.

A test facility was set up at NIST to implement the procedures that could realize these tests. Per the proposed standard these tests can be implemented in any manner, but the setup depicted in Figure 5 was chosen to be implemented at NIST for the following reasons:

1. Commercial availability of the “integration sphere” made it easier to determine reference lengths by measuring only a single point, compared to probing the sphere at multiple points and deducing its center (SMR walking method [13]).

2. An implementation using scale bars, though simple, may introduce additional sources of uncertainty as the entire artifact is moved from one position to another.

3. An implementation using a grid of spheres mounted on a wall will require the use of the SMR walking method (or other methods) to obtain the reference lengths, which is a cumbersome process.

4. The DMG at NIST has environment controlled laboratories that are large enough to accommodate this grid setup where the RI can measure the reference length from one side of the grid and the IUT can scan the spheres from the other side. The large space is necessitated by the fact that the “integration spheres” are partial spheres and have only a limited line-of-sight to an SMR mounted inside it.

Figure 5.

Setup to implement the most of the point-to-point distance tests

The point-to-point distance tests involve a series of symmetric and asymmetric length measurements. The various target positions are labeled in Figure 5 and the test distances and positions are listed in Table 2. Here, α is the angle subtended by the test length at the IUT’s origin, and θ is the IUT’s azimuth angle. These tests assume that the IUT’s elevation angle ϕ ≈ 0° with the sphere at the center of the grid (sphere E in Figure 5) before scanning any sphere.

Table 2.

Test positions for non-ranging point-to-point distance tests

Test	Description	Azimuth angle $\theta$
PP1	α at least 80° while measuring DF	0°
PP2	α at least 80° while measuring DF	90°
PP3	α at least 80° while measuring BH	0°
PP4	α at least 80° while measuring BH	90°
PP5	α at least 80° while measuring CG	0°
PP6	α at least 80° while measuring CG	90°
PP7	α at least 80° while measuring AI	0°
PP8	α at least 80° while measuring AI	90°
PP9	α at least 40° while measuring either ED or EF	0°
PP10	α at least 40° while measuring either ED or EF	90°
PP11	α at least 40° while measuring either EB or EH	0°
PP12	α at least 40° while measuring either EB or EH	90°
PP13	α at least 40° along the azimuth and at least 40° along the elevation directions while measuring either DH or HF	0°
PP14	α at least 40° along the azimuth and at least 40° along the elevation directions while measuring either DH or HF	90°
PP15	See details for inside test

The inside test is a test illustrated in Figure 6, and involves two spheres placed at a certain distance (d₀) from the IUT in such a way that they are equidistant and collinear from the IUT, but diametrically on opposite sides of the IUT. The spheres are also placed in such a way that the IUT scans it at a mean elevation angle of 0°.

Figure 6.

Setup to implement one of the point-to-point distance tests called the “Inside test”

While the azimuth angles listed in Table 1 and Table 2 are nominally 0° and 90°, they can be any pair of angles that are 90° apart to within ± 10°.

2.4.2.2 Relative-range tests

The procedures from ASTM E2938-15 required to perform the relative-range evaluation were retained in the new proposed standard, but the mandated number of tests was increased to three, as illustrated in Figure 7. For each length test, a plate target is placed at a location close to the IUT and is measured by both the IUT and RI (e.g., position A in Figure 7). The target is then moved away from the IUT to a second location, collinear with the IUT and the previous location (e.g., any position B through D in Figure 7). The target is scanned again at this location by both the IUT and RI. The scan data is processed to obtain the test length and the RI measurements are processed to obtain the reference length. The positioning of the RI with respect to the targets and IUT depends on the design of the target [8]. This process is repeated to perform tests for two other lengths.

Figure 7.

Relative-range tests

Since the non-ranging point-to-point distance tests use spheres, using spheres and other geometries for relative-range tests was explored [14]. Though these targets performed acceptably for a single distance, they involve systematic errors when used at multiple distances. The spherical geometry introduces a range dependent error in the ranging direction due to the squishing/flaring effect [8,9] which does not reflect the actual performance of the instrument. For this reason, planar targets were retained as the targets to be used in these tests.

2.5 Data processing

Data processing methods used to obtain the derived points in the proposed standard may introduce errors that may be incorrectly attributed to the instrument. The two geometries that are used in the proposed standard are spheres and planes. The methods to obtain the derived point for both the geometries involve a cleanup procedure where the data corresponding to the object of interest (plane or sphere) is separated from its surroundings.

The procedure for obtaining the derived point of a flat plate is the same as described in ASTM E2938-15[3]. The procedure to obtain the derived point of a sphere is described in detail by Rachakonda et al. [9] and was tested against thousands of datasets from several instruments. This process uses an iterative cone-cylinder exclusion routine, illustrated in Figure 8. In this method, an initial center of the sphere is calculated and is then refined by iteratively excluding the points outside a conical and cylindrical region constructed around the sphere data. The detailed procedures of obtaining the derived point is beyond the scope of this paper, however the recommendations from the studies by Rachakonda et al. [9] have been incorporated into the proposed standard. Some of these recommendations for a sphere target are listed below:

1. Calibration: The radius of the sphere target along with its uncertainty shall be determined through a calibration procedure.

2. Target size: The minimum sphere target size should be specified by the IUT manufacturer, and shall be sufficient to yield a minimum of 300 points after point selection process (cone-cylinder method) described in the proposed standard.

3. Circularity: The circularity of the sphere target shall not exceed 20 % of the smallest point-to-point distance maximum permissible error (MPE) of the IUT.

4. Segmentation: The derived-point coordinate of the sphere is refined through a “cone-cylinder” method described in the proposed standard.

Figure 8.

Illustration of the cone-cylinder method in 2D [9]

2.6 Metrics for evaluation

To evaluate the instrument, the datasets are reduced to derived points and certain metrics are proposed for each of these tests. These metrics are described next and are in a coordinate system that is coincident with the IUT’s origin.

2.6.1 Two-face tests

The two-face error is the root-sum-square of the components along the azimuth and elevation directions of the target and is given by:

$$E_{\mathit{two}-\mathit{face}}=r_1\sqrt{{\left({\varphi }_1-{\varphi }_2\right)}^2+{\left[\left({\theta }_1-{\theta }_2\right)\mathit{cos}\left({\varphi }_1\right)\right]}^2}$$ 1

where, ( $r_1,{\theta }_1,{\varphi }_1$) is the derived-point of the target measured in the frontsight and ( $r_2,{\theta }_2,{\varphi }_2$) is the derived point of the target measured in the backsight (both in a spherical coordinate system).

2.6.2 Point-to-point distance tests

For all the point-to-point distance tests (both non-ranging and the relative-range tests), the metric adopted is the error in the measured distance. This is the difference between the distance measured by the IUT and that measured by the RI and is given by:

$$E_{\mathit{distance}}=D_{\mathit{meas}}-D_{\mathit{ref}}$$ 2

where, $D_{\mathit{meas}}=\sqrt{{\left(x_1-x_2\right)}^2+{\left(y_1-y_2\right)}^2+{\left(z_1-z_2\right)}^2}$ is the distance between the targets as measured by the IUT and $D_{\mathit{ref}}=\sqrt{{\left(X_1-X_2\right)}^2+{\left(Y_1-Y_2\right)}^2+{\left(Z_1-Z_2\right)}^2}$ is the distance between the same targets as measured by the RI. Here, (x₁, y₁, z₁) & (x₂, y₂, z₂) are the coordinates of the derived-point of the targets measured by the IUT and (X₁, Y₁, Z₁) & (X₂, Y₂, Z₂) are the coordinates of the derived point of the same target measured by the RI.

For the relative-range tests, the dispersion of the residuals is also calculated for reporting. This value is the root-mean-square of the residual distance (q_i) to the plane representing the target, calculated using N points, and is given by:

$$\mathit{RMS}=\sqrt{\frac{{{\displaystyle \sum }}_{i=1}^N{q}_i^2}{N}}$$ 3

3 Summary and conclusion

The working group within the ASTM E57 subcommittee on Test Methods developed a new standard for evaluating the ability of spherical coordinate, 3D imaging systems to measure derived point to derived point distance within the instrument work volume. The working group consisted of instrument manufacturers, stakeholders, subject matter experts and researchers. Many test methods were considered and tested rigorously with many TLSs to ensure that the tests can be reasonably performed and objective metrics can be calculated. Several papers [5–14] have been published by NIST researchers which document the issues regarding TLS performance evaluation. Decisions were made on many issues such as: a) number and types of test methods, b) targets to be used for these tests and c) methods and considerations to process the data. This paper gives an overview of all the work that was performed that culminated in the proposed standard that is presently being balloted^**[15] in the ASTM E57 committee.