Towards Estimating the Uncertainty Associated with Three-Dimensional Geometry Reconstructed from Medical Image Data

Abstract

Patient-specific computational modeling is increasingly used to assist with visualization, planning, and execution of medical treatments. This trend is placing more reliance on medical imaging to provide accurate representations of anatomical structures. Digital image analysis is used to extract anatomical data for use in clinical assessment/planning. However, the presence of image artifacts, whether due to interactions between the physical object and the scanning modality or the scanning process, can degrade image accuracy. The process of extracting anatomical structures from the medical images introduces additional sources of variability, e.g., when thresholding or when eroding along apparent edges of biological structures. An estimate of the uncertainty associated with extracting anatomical data from medical images would therefore assist with assessing the reliability of patient-specific treatment plans. To this end, two image datasets were developed and analyzed using standard image analysis procedures. The first dataset was developed by performing a “virtual voxelization” of a CAD model of a sphere, representing the idealized scenario of no error in the image acquisition and reconstruction algorithms (i.e., a perfect scan). The second dataset was acquired by scanning three spherical balls using a laboratory-grade CT scanner. For the idealized sphere, the error in sphere diameter was less than or equal to 2% if 5 or more voxels were present across the diameter. The measurement error degraded to approximately 4% for a similar degree of voxelization of the physical phantom. The adaptation of established thresholding procedures to improve segmentation accuracy was also investigated.

Introduction

Computational modeling with anatomy reconstructed from medical image data combines mathematics, physics and computer science to study anatomy, disease states, and complex biophysical problems in silico. Three dimensional (3D)-image based geometries are increasingly used for patient-specific visualization, anatomical/pathological measurement, physics-based simulation, and additive manufacturing. Clinically relevant examples include surgical treatment planning for cardiovascular [1] and orthopaedic implants [2], radiotherapy treatment planning [3], tailored magnetic resonance imaging (MRI) coil configurations for patients with active implanted devices [4], and patient-specific implant designs [5]. Non-clinical applications such as defect detection in microelectronics [6], measuring density variations in pharmaceutical tablets [7], and non-destructive testing of various products [8] are also increasingly relying on image data. In the field of geology, recent studies have focused on the ability of segmentation to obtain the pore structure of rocks [9, 10]. See Chiffre et al. [11] for a comprehensive review of the industrial applications of computed tomography (CT) technologies for evaluation of various aspects of industrial manufacturing.

Medical image data is composed of a collection of two-dimensional images that form a three-dimensional image space. Each two-dimensional image is subdivided into a collection of regularly distributed rectangles called pixels. The empty space between each two-dimensional image inherits the properties of the adjacent pixels, forming a volumetric element called a voxel. In addition to their spatial (x, y, z) location, each pixel is assigned a numeric “greyscale” value that represents the interaction between the physics of the imaging process and the imaged material at each voxel location. Structures comprised of a single material will have similar greyscale values throughout, assuming beam hardening and other imaging artifacts are not present. However, structures containing multiple materials will interact differently with the imaging physics [12], resulting in significant variation in greyscale values at material boundaries. These gradients in greyscale values between structures enables the delineation of boundaries between the various structures. The analysis and delineation of boundaries between structures is referred to as image segmentation.

X-ray CT and MRI are the most common imaging modalities used to develop 3D geometric representations of objects for which the computer-aided design (CAD) geometry does not exist. CT is similar to MRI in that they both produce a stack of 2D greyscale images of a 3D volume. These images are typically distributed in the DICOM file format for clinical applications and the more recent DICONDE format for industrial applications. MRI uses non-ionizing radio-frequency (RF) radiation to detect the magnetic resonance of hydrogen atoms, while CT uses ionizing radiation and measures the attenuation of X-rays. Therefore, the two techniques have different areas of application: MRI is useful in examining materials with high water content, e.g. soft tissues, while CT is useful for examining materials with variable density. Typical applications include bones and tumors. Contrast agents are also used to enhance the visibility of specific tissues for a given imaging modality.

There are many sources of variability when acquiring image data, including vendor-specific hardware and reconstruction algorithms, varying imaging protocols (including pitch, scan time, and dosage), and instrumental drift [13]. In addition to the sources of variability associated with image acquisition, there is also error associated with the segmentation process. A common segmentation technique is thresholding, where a range of image intensity values is used to selectively extract the structure of interest. This process is typically based on a greyscale analysis of the relevant structure and the surrounding background material. As a tissue/material should be represented by similar greyscales, regions of very high greyscale gradient typically denote the boundary between materials. The exact position of the boundary between materials within this region of high greyscale gradient is, however, open to user interpretation. The exact position of a boundary between two materials will typically be determined by a user’s parameter choices in the algorithms used to perform the segmentation. There are also several variables that can influence the intensity value throughout a structure, such as noise artifacts, pathological conditions, use of contrast agents, and others. Taken together, these can significantly impact the final shape of the extracted object. For example, Figure 1 shows how the shape of an abdominal aortic aneurysm (AAA) is affected by the threshold value used to isolate the contrast agent within the blood stream. Notice how detail in the distal side-branches decreases as they typically have less perfusion of contrast agent relative to the major vessels. This degree of variability may be acceptable if the goal of the imaging study is to detect the presence of the aneurysm, but may not be sufficient if the imaging data is used to estimate rupture risk or to provide geometric parameters that would assist with selecting a device with the appropriate structural shape and size to provide the best fit to the patient’s anatomy.

Figure 1.

The relationship between threshold value and the shape/volume of an abdominal aortic aneurysm (AAA). Note how threshold not only affects the segmentation extents, but also the predicted aneurysm volume (data shown in the table). These models were segmented by thresholding, followed by a flood fill filter to remove parts of the segmentation disconnected from the AAA and associated vessels. An ISO-surfacing approach is used to generate a smoothed preview of the raw voxel segmentation (see [14]).

One of the primary artifacts inherent in medical imaging is the “partial volume” effect, which occurs because of the limited resolution of the imaging system relative to the length scale of the structure of interest. This effect also occurs because the physical boundaries between materials do not necessarily coincide with voxel boundaries. The result is voxels that contain multiple materials (see the voxels along the diagonal in Figure 2a). The attenuation that occurs within a partially filled voxel is a weighted average of the attenuation of the X-rays in the background and in the material. Therefore, the surface is a combination of the value of attenuation of X-rays partially filled with material in a surrounding medium. A similar effect is observed in images derived from an MRI scan. Comparing the images in Figure 2c and d, higher resolution scans can increase the accuracy of the interfacial reconstruction since the volume of partially filled voxels decreases as scan resolution increases. But a higher resolution scan may not be available clinically because of patient safety concerns. It is therefore critical to develop image segmentation procedures that not only lead to high confidence in the segmentation, but also quantify the uncertainty associated with each segmentation study.

Figure 2.

(a) The partial volume effect is shown for a scenario where voxels along the diagonal of the square are partially filled with background (greyscale = 0) and material (greyscale = 255). Intermediate greyscale values are assigned to the interface between the two regions. (b) A glass model of an aneurysm scanned using a (c) low resolution scanner with approximately 3 pixels spanning the wall thickness and (d) a higher resolution scanner with approximately 50 voxels spanning the wall thickness.

Researchers typically utilize the ISO50 threshold-based segmentation methodology as a repeatable methodology that estimates the location of the interface between the various materials in an image. This approach defines the material boundary as the middle greyscale value between the background and material peaks in the greyscale value histogram [14]. But this methodology has limitations since it relies on ideal CT data, which would have a single greyscale value associated with each segmented structure, a single greyscale value associated with the background, and a single layer of partial volume voxels at the interface between these regions. This places an unrealistic expectation on real-world data, where imaging artifacts (such as beam hardening) result in less than perfect scans, with a range of greyscale values associated with each structure and with the background. The potential variation in greyscale values between multiple structures and within each individual structure suggests that a single ISO50 value may lead to local inaccuracies in iso-surface positioning [14]. To the authors’ knowledge, no study has examined the impact of the ISO50 approach on the accuracy of the segmentation process.

The accuracy of CT segmentations has been a subject of continuing interest. In medical imaging, a foundational paper concerned with growth rates of tumors showed that CT could reproduce the volume of deformable objects to within 3% under ideal imaging conditions [15]. Efforts to produce 3D segmentations of tumors followed [16]. The issue of reproducibility of these segmentations was studied shortly thereafter [17, 18]. Subsequent studies concentrated on the relation of CT images to an external ground truth. Levine et al. [19] developed a reference phantom to help control for variations in scanner settings, noise, and artifacts. The phantom consisted of three spheres of known diameter and spacing supported in a LEGO matrix. Their study showed that centroids could be recovered in medical CT to within 0.2 mm, but the observed ball diameter varied by 1 mm out of 6.35 mm as the threshold was varied from 0 HU to 500 HU, where HU is a measure of X-ray attenuation in tissue. A study of ellipsoids printed using additive manufacturing found that volumes could be reproduced to within 2% with 95% confidence [20]. Later, a multi-center phantom study concluded that volume changes of about 25% could be obtained reliably under “real-world” conditions of a mix of vendors, software, and scanning protocols [21]. The work was also extended to low-contrast phantoms and showed that it is possible to estimate volumes to within about 15% [22].

Extending previous research, this manuscript summarizes an investigation of the effects of imaging system resolution on the uncertainty associated with three-dimensional geometries reconstructed from both idealized and actual CT images. The segmentations were based on either an idealized spherical reference phantom or actual CT data. The idealized phantom was generated in CAD format using the Simpleware CAD module (Synopsys, Inc., Mountain View, CA, USA). A CAD voxelization process was then used to convert the CAD sphere into 3D greyscale images at various clinically and industrially-relevant resolutions and these were segmented and meshed using the commercial segmentation software Simpleware ScanIP (v7.0, Synopsys, Inc., Mountain View, CA, USA). These datasets were used to study the relationship between image resolution and the accuracy of a sphere diameter measurement. Additionally, a new image stack of the National Institute of Standards and Technology (NIST) reference phantom of Levine et al. [19] was generated using a SkyScan 1173 Micro-CT scanner and resampled to various resolutions to allow for an accuracy study similar to that of the idealized phantom.

The rest of the paper is organized as follows. The next section provides a description of the methodology used to acquire the idealized and NIST phantom data. A description of the ISO50 methodology and segmentation and analysis methodologies is also provided. The diameter results for each sphere are summarized in the results section for both the reference and physical phantoms. The discussion section provides some perspectives on the results, including implications of this work for computational model verification and validation studies.

Methods

This section outlines the methods used to develop and analyze the image datasets of the idealized CAD phantom and NIST reference phantoms. The segmentation procedure used to extract the sphere diameters from the image data is also described.

Idealized Phantom

A spherical phantom with a diameter of 6.350 mm was generated in CAD format using the Simpleware CAD module (v7.0, Synopsys Inc., Mountain View, CA, USA). This diameter corresponded to the nominal value specified by the manufacturers of the spheres used in the NIST reference phantom [20]. A CAD voxelization process was then used to convert the CAD sphere into three-dimensional greyscale images at various clinically and industrially-relevant resolutions. This process resulted in an image dataset that was devoid of artifacts. Partial volume voxels were present to accurately represent the position of the sphere surface after the conversion. The Simpleware CAD module uses a distance function to create the 3D images. In this process, the perpendicular distances from the CAD surface to the voxel points are measured and used to encode the partial volume voxels [23]. The 3D images were segmented and meshed using the commercial segmentation software ScanIP.

It is important to note that each image stack was created by voxelizing the CAD model of the sphere at each resolution to avoid potential resampling errors that could occur if one high resolution image stack was used as the source of the lower resolution data.

Reference (NIST) Phantom

While the procedure used to develop the image stack of the idealized sphere produces a realistic set of images for the segmentation process, the resulting images are somewhat idealized in that there are no reconstruction artifacts. Therefore, spherical phantoms based on the work of Levine et al. [19, 20] were also investigated. Their phantom consisted of three PTFE spheres supported in a LEGO matrix (see Figure 3). The manufacturer reported nominal diameters of 6.350 mm +/− 0.025 mm for each sphere, where the uncertainty range refers to the manufacturer’s tolerance. The LEGO support matrix was chosen because of the low cost, consistent mechanical tolerances, and similarity of the polymer material Hounsfield unit values to those of the human body. Also, PTFE has a good contrast (about 500 HU) versus the LEGO structures, so segmentation is relatively straightforward. Note that the glass spheres analyzed by Levine et al. are not considered in this study.

Figure 3.

(a) Schematic diagram of the NIST reference phantom and (b) a 3-D image of the NIST phantom. The phantom consists of three PTFE spheres (red) identified as A, B, and C supported in a LEGO matrix (blue). Note that the matrix geometry is not analyzed in this study.

Sphere Diameter Determination (NIST Phantom)

A coordinate measurement machine (CMM) measures the geometrical characteristics of an object. NIST employed a Cordax RS-5 CMM, which uses a contact probe technology to determine the spacing between spheres and sphere diameters. The spheres were measured based on 15 points distributed over the exposed portion of the sample. The diameter was calculated based on a least-squares fit to the 15 measured points. The resulting diameters for spheres A, B, and C were 6.368 mm, 6.367 mm, and 6.401 mm, respectively. The uncertainty at 95% confidence in the CMM measurement was reported to be +/− 0.095 mm. These measurements provided the ground truth data for comparison to the reconstructed spheres.

Micro-CT Scanning

A micro-CT scan was performed on the reference phantom using a SkyScan 1173 Micro-CT scanner. The selected parameters were optimized to acquire a series of high resolution two-dimensional (2D) projection images at 24.99 μm pixel size with high signal and low noise levels, which were then reconstructed into an isotropic cross-sectional image stack. The distance between slices was 24.99 μm, i.e. the voxels were isotropic. The micro-focus X-ray emission source was set with an applied maximum voltage of 70 kV and a current of 100 μA operating with a spot size of < 5 μm. A 1.0 mm aluminum filter was used to minimize the presence of beam hardening artifacts associated with polychromatic X-ray sources used in clinical and laboratory CT systems. Projection images were acquired using a 2240 × 2240 pixel flat-panel detector at every 0.2 degrees over a 360-degree rotation, for a total of 1800 projection images. At each rotational step, 8 frames were averaged with an exposure time of 900 ms.

Image Reconstruction

The series of 2D projection images were reconstructed using NRECON GPU Version: 1.6.9.18 (Bruker MicroCT N.V. Kontich, Belguim), using a modified Feldkamp back-projection algorithm. Projection Pixel Shift Correction was employed by use of a reference post scan as a means to correct for small changes in the projection image due to shifts in the environment temperature. Reconstructed images were output in 16-bit TIFF format.

Segmentation

The commercial segmentation software ScanIP was used to perform all segmentations. The ISO50 approach¹ was utilized to identify the interface between the spherical boundaries and the image background,

$$ISO50=\frac{{\langle grey\rangle }_{mat}+{\langle grey\rangle }_{bg}}{2},$$

where <grey>_mat is the average greyscale value of the sphere material and <grey>_bg is the average greyscale value of the surrounding medium. As shown in Figure 4a, masks were created and analyzed as part of the segmentation process, where the voxels inside the yellow mask are used to determine the sphere material greyscale, and the voxels inside the red mask are used to determine the greyscale of the background medium. Note that the masks were eroded at the interface between the sphere and the background so as to exclude any partially filled voxels from the averaging. The peaks in a histogram of pixels versus greyscale value identifies the average greyscale of each material and the background (Figure 4b).

Figure 4.

(a) An image from the NIST reference phantom scan dataset showing the masks for the spherical phantom (yellow) and background (red). The unsegmented dark grey regions in the image are the LEGO support matrix. Erosion at the interface between the sphere (light grey) and other structures (black) separates the masks and minimizes the potential for partial volumes affecting the averaging calculation. NOTE: The unsegmented portion (dark grey) correspond to the LEGO support matrix. (b) An x-y plot of the number of pixels in an image slice as a function of voxel brightness. The peak at 2000 corresponds to air/background, near 18000 corresponds to the LEGO support, and near 39000 corresponds to the sphere.

The steps for performing an ISO50 segmentation of each sphere were as follows and are summarized graphically in Figure 5:

1. Crop the image to a local region surrounding the sphere of interest

2. Perform initial segmentation by estimating a lower threshold value with no calibration, set upper threshold value to maximum

3. Erode the segmentation to remove partial volume voxels from the mask

4. Analyze the segmentation to determine the average greyscale value of the sphere <grey>_mat and the surrounding background material <grey>_bg.²

5. Repeat the sphere thresholding using the ISO50 protocol where:

   Lower threshold = defined by the ISO50 equation

   Upper threshold = maximum greyscale value³

6. Remove any segmented pixels associated with the LEGO matrix using an Open Filter and Flood Fill operation. Ensure additional processing did not affect the surface except in the local regions where the holder artifacts were present.

7. Generate a triangulated surface model, with no additional smoothing options enabled. Export surface model in STL format. The Simpleware ScanIP software used for the STL generation uses a marching cubes approach to extract the smoothed triangulated surface from the raw voxel segmentation, utilizing partial volume voxels to ensure high surface accuracy, see [24] for details.

Figure 5.

(a) Initial segmentation of a sphere. (b) Erode surface by several voxels to ensure partial volume voxels are not included in the greyscale analysis. (c) Perform final segmentation based on the ISO50 protocol. Segmentation artifacts around the support structure interface were common (highlighted by red boxes). (d) Final sphere geometry with segmentation artifacts removed.

The STL representation of each sphere was the basis for all geometric analyses. The sphere diameter was estimated using three measurement approaches:

1. A sphere fitting method which uses a weighted least-squares approach, where the weight of each point is the sum of the areas of its neighboring triangles. Analysis performed using Materialise 3-Matic software (v.11, Materialise NV, Leuven, BE).

2. A volumetric method where the diameter is derived from the volume of the reconstructed sphere. Analysis performed in ScanIP.

3. A pointwise calculation where the diameter is determined for all nodal points of the STL reconstruction and then averaged. Analysis performed in ScanIP.

The maximum error in the diameter was also extracted from the STL data. This error was defined as the vertex with the largest deviation from the reference diameter. In practice, the vertex could be at a minimum or maximum distance from the centroid.

Taken together, these metrics provide both global and local estimates of the error in the reconstructed geometry.

Results

CAD Phantom Analysis

The image stacks created when voxelizing the CAD phantom were segmented and analyzed in ScanIP as outlined in the Methods section. Table 1 summarizes the relationship between scanner resolution and the percent error in the reconstructed diameter relative to the ground truth diameter for all three metrics. There is no error attributed to the segmentation process (to the level of precision studied) for scanner voxel sizes ≤ 0.128 mm. The error is observed to steadily increase for voxel sizes ≥ 0.32 mm.

Table 1.

Error in the segmentation process for the CAD sphere as a function of scanner voxel size

Voxel Size [mm]	Sphere Diameter [pixels]	Reconstructed Diameter [mm]	% error (fitted sphere)	% error (volumetric)	% error (pointwise)
0.025	256	6.350	0.00	0.00	0.281
0.032	200	6.350	0.00	0.00	0.283
0.064	100	6.350	0.00	0.01	0.279
0.128	50	6.350	0.00	0.08	0.284
0.320	20	6.346	0.06	0.66	0.338
0.640	10	6.333	0.27	2.62	0.553
0.800	8	6.314	0.57	7.78	0.847
1.067	6	6.283	1.06	10.18	1.341
1.600	4	6.194	2.46	15.81	2.751

NIST Phantom Analysis

As described in the Methods section, the NIST phantom was scanned using a SkyScan 1173 Micro-CT scanner at 24.99 μm pixel size (see Figure 6). The typical scan from a clinical MRI or CT scanner is on the order of 0.5 mm pixel size, therefore the micro-CT dataset was down-sampled using ScanIP to mimic the effect of a lower resolution scanner. A new image stack was created and exported at voxel sizes ranging from 0.025 mm to 1.60 mm. Each image dataset was analyzed according to the ISO50 segmentation protocol outlined in the Methods section. Similar trends were observed when the diameter estimation procedure was performed in each of the three coordinate directions individually (data not shown).

Figure 6.

Selected images from the micro-CT scan of the NIST phantom. Note the high degree of contrast between the PTFE spheres and the supporting LEGO structure.

Table 2 summarizes the relationship between scanner resolution and the percent error in the reconstructed diameter for the NIST phantom for all three metrics. ISO50 segmentation was performed for all three spheres. Similar to what was observed for the CAD phantom, the error is initially constant for scanner voxel sizes less than 0.128 mm and then increases for coarser resolutions.

Table 2.

Error in the segmentation process for each sphere in the NIST phantom as a function of scanner voxel size

Voxel Size [mm]	Sphere Diameter [pixels]	% error (fitted sphere)			% error (fitted sphere)	% error (volumetric)	% error (pointwise)
		Sphere A	Sphere B	Sphere C	Average	Average	Average
0.025	256	0.50	0.30	0.05	0.28	0.85	0.47
0.032	200	0.51	0.30	0.05	0.28	0.85	0.47
0.064	100	0.54	0.31	0.01	0.29	0.86	0.48
0.128	50	0.65	0.40	0.01	0.35	1.05	0.54
0.320	20	1.18	0.79	0.30	0.76	1.21	0.89
0.640	10	1.44	1.95	1.47	1.62	2.85	2.62
0.800	8	2.94	2.81	2.25	2.66	4.96	3.46
1.067	6	3.97	3.58	3.10	3.55	5.09	4.73
1.600	4	4.75	3.71	3.46	3.97	1.65	5.91

Figure 7 summarizes the relationship between percent error in the diameter of the segmented spheres and scanner voxel size for the idealized CAD phantom and the NIST reference phantom. The percent error is greater when analyzing a physical model versus the idealized phantom for all voxel sizes studied.

Figure 7.

Comparison of the percent error in the diameter of the segmented spheres of the idealized CAD phantom (red data points) and NIST reference phantom (blue data points) using the three measurement approaches outlined in the Methods section. Results provided in terms of both the voxel spacing (a, c, e) and number of voxels across the sphere diameter (b, d, f). The blue dots correspond to the averged sphere diameter data from Table 2.

Discussion

This study outlines a methodology that provides insight into the error associated with image segmentation and geometry reconstruction as a function of scanner resolution. The results were based on the analysis of an idealized geometry (sourced from CAD) and physical image datasets acquired from a CT scanner. Similar spherical geometries were analyzed in both cases to enable a comparison of the impact of image source on reconstruction accuracy.

For the idealized image dataset, a measurement accuracy of at least 2% was observed for voxel sizes finer than 1 mm when using the fitted sphere approach (see Table 1). The error steadily increased as the voxel size increased above 0.3 mm, which corresponds to 50 or fewer voxels spanning the sphere diameter. The volumetric and pointwise analyses resulted in higher relative geometry errors. Indeed, the reconstructed surface shape can be highly irregular, especially at low resolutions (not shown). Therefore, the decreased error observed with the fitted sphere approach represents the ability of this metric to “average out” local errors. Additionally, the error based on the volume metric was higher than the fitted sphere and pointwise approaches. This agrees with theory since the error in the volume calculation is proportional to the third power of the diameter.

It is also important to note that the idealized spherical geometries were derived from iso-surfaces, i.e. not from the raw voxel segmentations. This precludes the impact of positioning effects on the reconstruction of the idealized sphere. Thus, there is no limitation that the centroid of the sphere must be at the center, corner, or edge of a voxel. As such, the position of the idealized sphere does not influence the diameter measurements. Similarly, there is no requirement for the sphere diameter to be an integer multiple of the image resolution. The use of partial volume voxels allows for sub-voxel accuracy in the measured diameters.

Higher relative error was also observed when segmenting the NIST phantom versus the idealized phantom (see Figure 7), especially at the lower voxel spacings. Interpolating the data in Table 2, a scanner voxel size of approximately 0.7 mm is required to maintain a 2% accuracy for the NIST phantom. The increase in resolution required to achieve similar accuracy for the NIST phantom is primarily attributed to the fact that the CAD phantom can be considered a “clean” dataset. This means there are no reconstruction artifacts present in the images. These artifacts, which may be associated with either the physical object or the scanning process, can degrade the accuracy of the segmentation process. In spite of this, a 4% error was maintained in the reconstruction of the NIST phantom at the coarsest scanner resolution, which corresponds to only 4 voxels across the sphere diameter. The low error at this (and other) resolutions is attributed to the significant contrast differences between the sphere and background and the fixed position of the phantom in both space and time, factors which are admittedly atypical of many biological scenarios. Therefore, the results found here may be viewed as a lower bound on the error associated with image acquisition, reconstruction, and segmentation of medical image data.

There is also a persistent discrepancy between the diameter calculated during the reconstruction of the NIST images versus the ground truth, even at the highest resolutions studied. One source of this error could be related to the choice of segmentation threshold, i.e., the average of the sphere and background greyscale values. Using a value of 50 to determine the boundary location assumes the interface is located at the mid-point between the two materials after erosion. Also, the spatially averaged greyscale values used in the ISO50 protocol may not apply globally since the local threshold value can vary spatially. Figure 8 summarizes a study of the impact of the lower threshold value on segmentation accuracy. In this case, the determination of the lower threshold (ISOXX) is as follows:

$$ISOXX=\frac{\left(100-XX\right)*{\langle grey\rangle }_{mat}+XX*{\langle grey\rangle }_{bg}}{100},$$

where XX is the weighting value used to calculate the threshold. Using this definition, a value of 0 places full weighting on the average greyscale for the material (ISO00), while a value of 99 weights the lower threshold towards that of the background (ISO99). Focusing on Sphere A, a study of the relationship between segmentation threshold and error in sphere diameter was conducted (see Figure 8). Results reveal that for the highest resolution images, an ISO value of 68 provide a better estimation of the sphere diameter for this system. Applying this new threshold value to the data shows a reduction in error to less than 0.01 % for the lowest voxel spacing (see Figure 9).

Figure 8.

Comparison of the error in the segmentation of NIST Phantom sphere A (red) and the actual diameter of sphere A when varying the ISO value used to threshold the background.

Figure 9.

Results for using the ISO68 value for calculating the diameter of the spherical geometries. Results provided in terms of both the number of (a) voxels across the sphere diameter and (b) voxel spacing. At the highest resolution, the percent error decreases to less than 0.01% when applying the ISO68 threshold value.

There are many reasons an adjustment to the threshold value may be required. For example, interaction between the measurement system and the material of interest can lead to artifacts in the image data. An example is beam hardening, which is a common CT artifact associated with polychromatic X-ray beams. Beam hardening occurs because lower energy X-rays are preferentially absorbed when X-rays travel through an imaged object. Reconstruction algorithms typically assume that X-ray absorption is a property of the material and is independent of the energy level of the X-ray photons. The mismatch between this assumption and reality results in errors in the estimation of the attenuation coefficient of the material. This typically manifests itself in the reconstructed images as the edge of the imaged object appearing brighter than the interior [25], which leads to surface positioning inaccuracies as the ISO50 greyscale analysis cannot compensate for this local greyscale variation. Other examples of image artifacts are the “cupping artifact” [26] and the “streak artifact” [27]. The cupping artifact is usually ascribed to beam hardening, which is a change in the absorption coefficient that arises as a polychromatic X-ray beam passes through matter and undergoes changes in the spectrum of the non-attenuated X-rays. The streak artifact commonly arises when reconstruction algorithms are not sufficiently robust to describe high-contrast materials.

This work provides critical input for verification and validation and uncertainty quantification (VVUQ) studies. The ASME V&V 20 Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer [28] describes three categories of errors associated with a computational modeling activity:

• δ_model: errors due to modeling assumptions and approximations (also called “model form” error)

• δ_num: errors due to numerical discretization and solution

• δ_input: errors due to error in the simulation input parameters

Each of these errors may have contributions from multiple sources. For example, the errors associated with model input parameters are due to a lack of knowledge about the geometry, material properties, and boundary conditions. The error estimate from this investigation therefore represents one element of the error in the geometry. Furthermore, this error is considered an epistemic (i.e., systematic) error since it is associated with limitations of the instrument used to acquire the geometric data, which could be reduced, for example, through improvements to scanner technology and/or refinements to the scan protocol.

Previous work has analyzed the sensitivities and uncertainties in patient-specific hemodynamic flow models. For example, flow predictions in the carotid bifurcation were shown to have a three-fold (or greater) dependence on geometry variations versus viscosity [29] and inflow boundary conditions in the [30]. Uncertainty in the minimum diameter of coronary vessel stenoses (constrictions) has also been identified as the primary contributor to total uncertainty in pressure loss predictions [31, 32]. These investigations underscore the need for increased emphasis on the accuracy of reconstructed anatomical structures over other model input parameters.

Due to the highly controlled nature of this investigation, it is the authors position that the results presented here provide an estimate of the lower bound on the contribution of geometric error to δ_input. In other words, the geometric error is expected to increase in less idealized situations. This is especially true in clinical settings where patient safety dictates that lower resolution scanners must be used, and therefore may not resolve tissue interfaces with the high degree of accuracy encountered in this research. Future research could further refine this error estimate for clinical applications by using anatomically accurate geometries constructed from glass or by additive manufacturing.

Conclusions

This manuscript summarized a first step towards estimating the epistemic error associated with three-dimensional geometry reconstructions. It was shown that the reconstruction accuracy is a function of voxel size. The relationship between segmentation threshold and reconstruction accuracy was also investigated. It is important to note that the proposed methodology to improve reconstruction accuracy by varying the segmentation threshold may not be generally applicable, especially when multiple structures are present in the data. The inherent heterogeneity of biological materials also brings additional complexity. And while this study represents a detailed analysis of the ability of a phantom to provide reference lengths for scanner calibration, further study of the impact of scanner resolution on the uncertainty associated with the imaging, reconstruction, and segmentation of 3D objects is warranted.