Simultaneous Registration of Location and Orientation in Intravascular Ultrasound Pullbacks Pairs via 3D Graph-Based Optimization

Abstract

A novel method is reported for simultaneous registration of location (axial direction) and orientation (circumferential direction) of two intravascular ultrasound (IVUS) pullbacks of the same vessel taken at different times. Monitoring plaque progression or regression (e.g., during lipid treatment) is of high clinical relevance. Our method uses a 3D graph optimization approach, in which the cost function jointly reflects similarity of plaque morphology and plaque/perivascular image appearance. Graph arcs incorporate prior information about temporal correspondence of the two IVUS sequences and limited angular twisting between consecutive IVUS images. Additionally, our approach automatically identifies starting and ending frame pairs in the two IVUS pullbacks. Validation of our method was performed in 29 pairs of IVUS baseline/follow-up pullback sequences consisting of 8,622 IVUS image frames in total. In comparison to manual registration by three experts, the average location and orientation registration errors ranged from 0.72 mm to 0.79 mm and from 7.3° to 9.3°, respectively, all close to the inter-observer variability with no difference being statistically significant (p = NS). Rotation angles determined by our automated approach and expert observers showed high correlation (r² of 0.97 to 0.98) and agreed closely (mutual bias between the automated method and expert observers ranged from −1.57° to 0.15°). Compared with state-of-the-art approaches, the new method offers lower errors in both location and orientation registration. Our method offers highly automated and accurate IVUS pullback registration and can be employed in IVUS-based studies of coronary disease progression, enabling more focal studies of coronary plaque development and transition of vulnerability.

I. Introduction

When studying natural course of plaque development in human coronary arteries in vivo using intravascular ultrasound (IVUS), registration of axial location and circumferential orientation of each image frame in corresponding baseline and follow-up IVUS pullback data is of paramount importance if location-specific quantitative comparisons between the two timepoints are to be determined [1]–[5]. With large patient cohorts involved in baseline/follow-up studies (e.g., 506 patients in [4]) and large numbers of IVUS image frames for each pullback (e.g., 80–133 frames after ECG gating [3]), manual registration of location and orientation is an extremely tedious task and requires expert knowledge. As a result, current clinical baseline/follow-up studies are not utilizing all available information due to undetermined frame location/orientation correspondences. Consequently, studies of mechanical forces, morphology and composition of lesions are only examined in culprit locations or over long vessel segments and are typically circumferentially averaged [1]–[4], [6]. Such approaches limit our ability to develop better understanding of the focal baseline/follow-up associations since the coronary artery disease evolves frequently in a focal and eccentric manner [4], [7], [8].

Automated location/orientation registration methods for IVUS pullbacks have attracted substantial research interest in recent years. Despite several methods reported [9]–[13], no perfect solution emerged delivering reliable registration results. The registration task is indeed very challenging [12], [13]. The challenges associated with motion artifacts, longitudinal oscillations, artifactual angular twisting, stuck/accelerated IVUS catheter during pullback acquisitions, vessel-morphologic changes due to plaque progression/regression at follow-up, etc. Fig. 1 provides an example of IVUS pullback registration with seven frame pairs manually registered. Clearly, in focal analyses of coronary artery disease progression [5], complete one-to-one correspondences between all baseline and follow-up frames would be preferred.

Fig. 1.

Example of IVUS pullback registration. From top to bottom: Baseline pullback in longitudinal view; follow-up pullback in longitudinal view; baseline landmark frames in cross-sectional view; corresponding IVUS frames; corresponding IVUS frames in proper orientation. In this case, the correspondences were determined visually by a human expert.

For automated registration of frame locations, the most intuitive approach is linear fitting (distance normalization) between two identified landmarks (e.g., side-branches) [9], [10]. This method works well when the transducer pullback exhibits a constant speed, which is not always the case in clinical settings. Non-rigid temporal alignment methods [11]–[13] (e.g., dynamic time warping, DTW [14]) treat the alignment task independently from the orientation registration task – consequently, the alignment results may not be optimal in the circumferential direction. In addition, these methods only rely on vessel morphologic features (area or shape) or plaque components, which may change for a long-term follow-up as a result of plaque progression/regression. For automated registration of orientation, a cross-correlation based method (rotating each follow-up image locally [9]) and a Catmull-Rom spline based method (rotating all the follow-up images globally [10]) have been reported. These two approaches achieve sub-optimal rotation results for two already location-aligned IVUS image sequence. In addition, the former method depends on Virtual Histology (VH)-IVUS defined plaque constituents and needs manual interaction to determine the rotational registration angle. The latter method suffers from artifactual angular twisting between consecutive image frames.

More generally, in the fields of computer vision and medical imaging, the problem of spatio-temporal registration/alignment of image sequences has been investigated for several years. Related work mainly followed two lines of approaches, including feature-based approaches [15]–[17] appropriate when the appearance varies from sequence to sequence, and direct approaches [15], [17]–[20] that fit similar intensities between sequences. The spatio-temporal registration/alignment algorithms use a particular transform to achieve optimality, typically based on gradient descent [15], [18], Powell methods [17], [20] or dynamic programming [21].

Compared with previous IVUS pullback location/orientation registration methods that treat the registration of location and orientation as separate tasks, we propose a novel joint spatio-temporal approach for IVUS pullback registration. By combining these two aspects in one global optimization task of finding an optimal path in a 3D graph, the location and orientation are registered simultaneously. To increase the robustness to the changes of vessel morphologic features, our method combines advantages from feature-based and direct approaches by incorporating plaque thickness and plaque/perivascular pixel similarities which provides more robust correspondence in two time points. To ensure geometrically feasible registration, graph arcs incorporate prior information about baseline–follow-up correspondences of the two IVUS sequences as well as information about limited-range angular twisting between consecutive IVUS image frames. To initialize the registration procedure, global and local similarities of two IVUS pullbacks are extracted from the 3D graph to automatically identify the most proximal and most distal image pair correspondences. To the extent of our knowledge, our approach is the first in the literature to simultaneously establish temporal (location) and spatial (orientation) correspondences between two different image sequences of two similar dynamic scenes by using a graph-based method, which has been traditionally used in image segmentation to detect boundaries or surfaces [22]–[26] or in deformable image registration to obtain the displacement field [27]–[29]. Partial results of this work have been reported in [30]. The current study provides more sophisticated methodology including regularization graph arcs, concise cost function, and automated model initialization, as well as a more comprehensive evaluation in a larger patient population.

Our system registers image sequences in a highly automated fashion as long as sufficiently accurate lumen and external elastic lamina (EEL) segmentations are available for all frames of both image sequences. This means that there is no need to provide a variety of manually obtained guiding information, such as branch locations/orientations [9]–[12], plaque characterization [9], [11], [12], matching anatomic landmarks [9], [10], and co-registration angles [9]. IVUS segmentation suitable for further analysis is invariably required for quantitative IVUS-based studies of coronary atherosclerosis [1]–[6], [8], [31], [32].

II. Methods

A. 3D Graph Construction

Given a pair of baseline and follow-up IVUS pullbacks, a 3D directed graph is constructed and searched for the optimal path (Fig. 2). In the 3D graph, each node represents the possible location and orientation correspondences of a baseline and a follow-up image frame pair. For example, the uppermost green graph node with coordinates (bl = 4, θ = 212°, fu = 3) in Fig. 2 represents correspondence of a follow-up frame #3 with 212° rotation with a baseline frame #4. The complete registration is defined by a sequence of nodes, which for each (bl, θ, fu) coordinate triplet forms a path in the 3D graph defining a possible location and orientation correspondence of baseline and follow-up image frames. Costs assigned to each node in the 3D graph reflect similarities of IVUS image data appearance and plaque morphology between the baseline and follow-up. An optimal path is defined as the path with the lowest global cost, which in an overall sense defines the frame-to-frame location and orientation registration of two image sequences. Clearly, such a registration is optimal with respect to the employed cost function.

Fig. 2.

Construction of a 3D graph used for solving the registration problem. This illustrative example assumes a baseline pullback (4 frames) and a follow-up pullback (3 frames). Green lines depict the optimal path through the graph determining the registration solution.

To calculate the lowest cost path, a dynamic programming algorithm is employed [21], [33]. A three-dimensional cumulative cost matrix C is computed according to node costs c(i, j, k) (bl = i, θ = j°, fu = k, see Section II-B for details) and node connections. The design of the node connections assumes limited angular twisting between consecutive image pairs [9], [34]. In each stage of the dynamic programming, the best one of the N = 3 × (2 × θ_twist + 1) possible preceding nodes is selected, where θ_twist is the rotational constraint of angular frame-to-frame twisting. The node connection diagram is visualized in Fig. 3. Nodes at any location (e.g., the red, green, or blue shaded nodes in Fig. 3) in the 3D graph, can be reached from any nodes in the other three rows (nodes connected by red, green, or blue arcs in Fig. 3), which potentially correspond to preceding baseline/follow-up frame pairs. As a result, each element C(i, j, k) (bl = i, θ = j°, fu = k) in the matrix C is computed as

Fig. 3.

Node connections for three different graph locations are shown as red, green, and blue dashed lines. This illustrative example assumes θ_twist = 3°. Note that the graph node connections seamlessly connect nodes at any location, including connections across the graph boundaries.

$$C(i,j,k)=c(i,j,k)+min\{\begin{array}{l}min(C(i-1,j+\mathrm{\Delta }\theta ,k)+f_p(\mathrm{\Delta }\theta ))\hfill \\ min(C(i,j+\mathrm{\Delta }\theta ,k-1)+f_p(\mathrm{\Delta }\theta ))\hfill \\ min(C(i-1,j+\mathrm{\Delta }\theta ,k-1)+f_p(\mathrm{\Delta }\theta ))\hfill \end{array},$$ (1)

where i is the i-th baseline image, j is the rotation angle of the k-th follow-up image, Δθ∈{−θ_twist, …, θ_twist}. Note that when j + Δθ≤ 0 and j + Δθ > 360, the connections will link to 360 + j + Δθ and j + Δθ − 360, respectively, shown as the connections to purple nodes in Fig. 3. f_p(Δθ) is the smoothness penalty function for different angular twisting Δθ. The angular twisting between consecutive IVUS images depends on the geometrical vessel morphology and vessel movement in the 4D case. It can be analytically determined using fusion with X-ray angiography [34]. Under ideal circumstances, such twisting should be identical between the two pullbacks and thus their differences should be zero. While this cannot be assured in clinical setting when artificial catheter movements may add an unpredictable component, the relative frame-to-frame differences should be small but they are cumulative. Therefore, we increasingly penalize twisting Δθ in either direction as follows

$$f_p(\mathrm{\Delta }\theta )=\omega ·\frac{\mathrm{\Delta }\theta }{{\theta }_{\mathit{twist}}}·c(i,j,k),$$ (2)

where ω controls the penalty weight. The elements C(i, j, 1) in the first layer (follow-up frame 1) and the elements C(1, j, k) in the first rows of each layer (baseline frame 1) are initialized as

$$C(i,j,1)=min(C(i-1,j+\mathrm{\Delta }\theta ,1)+c(i,j,1)),$$ (3)

$$C(1,j,k)=min(C(i,j+\mathrm{\Delta }\theta ,k-1)+c(1,j,k)).$$ (4)

The initialization condition is set as

$$C(1,j,1)=c(1,j,1).$$ (5)

As is typical for dynamic programming, the optimal path through the 3D matrix C is determined by backtracking from element min(C(I, j, K)) to C(1, j, 1) by following the minimum values of the neighboring elements, where I and K represent the last frames in baseline and follow-up, respectively.

B. Cost Function Design

Design of node-associated costs is always problem-specific. For IVUS pullback registration, Each cost c(i, j, k) reflects the similarity between the i-th baseline image and the k-th follow-up image with j-degree rotation. The similarity is designed as a combination of feature-based and direct approaches based on three terms: 1) perivascular tissue, including pixels outside of the EEL border (the green region in Fig. 4(c)); 2) plaque appearance, including the pixels between the EEL and lumen borders (the orange region in Fig. 4(c)); and 3) plaque thickness, defined as the distance between lumen and EEL borders at 360 circumferential wedges centered at the lumen centroid. Such a cost function design is depicted in Fig. 4 and formulated as

Fig. 4.

Cost function designed for IVUS pullback registration. (a) Longitudinal and cross-sectional views of a baseline IVUS pullback with expert-defined segmentation (lumen - yellow, EEL - red). (b) Longitudinal view of corresponding follow-up IVUS pullback. (c) Cost function of 1 − Corr_pv, 1 − Corr_pa, and 1 − Corr_pt. Costs are calculated by comparing the IVUS image frame given in panel (a) with all follow-up IVUS image frames in panel (b). Each value on a curve corresponds to the minimal value with respect to all possible rotations of a follow-up frame. In the cross-sectional view, the perivascular tissue and plaque are overlaid with green and orange colors, respectively. (d) Cost function c considering all follow-up images.

$$c(i,j,k)=\alpha ·(1-{\mathit{Corr}}_{pv}(i,j,k))+\beta ·(1-{\mathit{Corr}}_{pa}(i,j,k))+\gamma ·(1-{\mathit{Corr}}_{pt}(i,j,k)).$$ (6)

Calculation of Corr(i, j, k) is based on the normalized cross-correlation approach as detailed in [9], α, β, and γ are the weights for the perivascular tissue, plaque appearance, and plaque thickness terms, respectively, satisfying α + β + γ = 1. In [9], the calculation of Corr_pa relies on the VH-derived indexed values to describe plaque component similarity. In our implementation, IVUS image pixel intensities are directly used to reveal the plaque appearance similarity while avoiding the dependency on VH. Since the Corr_pt used in [9] cannot discriminate between two similar plaque shapes with different sizes, a size factor was added to represent differences of mean plaque thickness between baseline and follow-up:

$${\mathit{Corr}}_{pt}(i,j,k)=\frac{min(\overline{f_i},\overline{g_k})}{max(\overline{f_i},\overline{g_k})}·\frac{1}{n-1}\sum _{rd=1}^n\frac{(f_i(rd)-\overline{f_i})(g_k(rd,j)-\overline{g_k})}{{\sigma }_{f_i}{\sigma }_{g_k}},$$ (7)

where j is the rotation angle of a follow-up image being registered, $\overline{f_i}$ and $\overline{g_k}$ are average plaque thickness values of the i-th baseline and the k-th follow-up images, respectively, σ_fi and σ_gkare their respective standard deviations, n is the number of plaque thickness values which were set to 360, and rd is the plaque thickness index calculated in the radial directions. Note that when computing these Corr values, coinciding lumen centroids for the baseline and rotated follow-up image pairs are provided by our automated segmentation and operator-guided refinement algorithm [26]. This cost function design follows the rules for characteristic calcifications, perivascular landmarks, and plaque shape, which cardiologists use to match baseline and follow-up IVUS pullbacks [31].

C. Identification of Starting and Ending Frame Pairs

For the baseline and follow-up IVUS acquisition in the same vessel, the starting and ending positions of pullbacks are not always the same. Therefore, before applying the proposed registration method, we should identify the most proximal and most distal corresponding image frame pairs. Our strategy combines global and local costs embedded in the 3D graph-based framework. Fig. 5 demonstrates identification of the ending frame pair, which is determined by the following equation:

$$E(i,k)=min\{\begin{array}{l}{arg}_{j,k}min(G_c(I,j,k)+L_c(I,j,k))\hfill \\ {arg}_{i,j}min(G_c(i,j,K)+L_c(i,j,K))\hfill \end{array},$$ (8)

where G_c(i, j, k) is the global cost. G_c(i, j, k) corresponds to mean cost of the optimal path (green solid line in Fig. 5), which backtracks from element (bl_i, θ_j, fu_k) (green point in Fig. 5) in the 3D cumulative cost matrix C (cube in Fig. 5), and is defined as:

$$G_c(i,j,k)=\frac{C(i,j,k)}{n_e}.$$ (9)

Fig. 5.

Schema of the proposed global and local method for identification of ending pair. Given a baseline and follow-up IVUS sequence pair, our method constructs a 3D graph and follows two tracks: In the first track, the last baseline frame is fixed and the follow-up frame iteratively shifted by the minimal required length of baseline–follow-up overlap (25 mm in our case). In each iteration, global and local costs embedded in the 3D graph are extracted and combined. By fixing the last follow-up frame and iteratively shifting the baseline frame, another series of costs are obtained in the second track. The frame pair providing minimal cost is chosen as the ending pair.

C(i, j, k) is computed according to Eq. (1); n_e is the number of elements which form the optimal path. Such a design of the global cost represents the average similarity between two registered IVUS image sequences with (i, k) as the ending frame pair. L_c(i, j, k) is the local cost, associated with the similarity between the i-th image in baseline and the k-th image with rotation angle θ_j in follow-up, as defined in Eq. (6). I and K represent the last frames in baseline and follow-up, respectively. After the ending pair is identified, both the 3D cumulative cost matrix C and the cost matrix c are inverted and the same global and local approach is used to identify the starting pair with a searching range constrained by the minimal required lengths of the baseline/follow-up pullback overlap (25 mm overlap required in our studies).

Subsequently, the aforementioned selection order is inverted by identifying the starting pair first followed by obtaining the ending pair. This, two start/end pairs are available and the final solution is identified by choosing the lower total cost (global and local) from these two solutions. The 3D graph is subsequently constructed and the optimal path is determined, simultaneously registering the IVUS pullback with respect to location and orientation.

D. Post-processing

In clinical data, the pullback lengths (number of frames) of the imaged vessel segment at baseline and follow-up may differ dramatically. Reasons come from many aspects, including primary heart rate, different segment covered, resistance to transducer withdrawal, anatomic considerations, problems with the pullback device, etc. [32]. As a result, our registration of baseline and follow-up may generate one-to-many or many-to-one correspondences. To achieve one-to-one correspondence among baseline and follow-up, the registered frame pair with the minimal cost (Eq. (6)) is selected and used to define the overall correspondence of the two pullback IVUS frame pairs.

III. Experimental Methods

A. Patient Data

The performance of the proposed method was evaluated in 29 serial IVUS-VH studies of patients with stable coronary artery disease enrolled in the PREDICT study (ClinicalTrials.gov Identifier: NCT01773512) at the Charles University Hospital, Prague. The PREDICT study aims at prediction of extent and risk profile of coronary atherosclerosis and their changes during lipid-lowering therapy based on non-invasive techniques. From 33 patients in total, 29 patients satisfied inclusion criteria [6], [32] and were selected for this baseline/follow-up registration study.

IVUS imaging was performed in the standard fashion using the IVUS phased-array probe (Eagle Eye 20 MHz 2,9F monorail, Volcano Corporation, Rancho Cordova, California), IVUS console, Gold standard software, and motorized pullback at 0.5mm/s (research pullback device, model R-l00, Volcano Corporation, Rancho Cordova, California). After 8–14 months patients underwent repeat cardiac catheterization and IVUS of the same coronary artery.

The Volcano IVUS imaging system provided EKG R-wave gated IVUS image sequences with 8,622 gated frames in total (72 to 235 frames per pullback), 0.27 to 0.67 mm distances between adjacent frames depending on the heart rate, and 500 × 500 pixels per image frame. Both baseline and follow-up pullbacks were at least 30 mm long, with at least 25 mm long mutual overlap.

B. Independent Standard

Three experts defined the ground truth for baseline/follow-up IVUS pullback registration. The experts were blinded to the results of the computer registration and had the following levels of expertise:

• Expert 1 (TK): Cardiologist with 16 years clinical experience in coronary imaging (mainly IVUS).

• Expert 2 (AW): IVUS analyst with 17 years of IVUS experience.

• Expert 3 (ZC): IVUS analyst with 4 years of IVUS experience.

For each frame of each IVUS pullback, luminal and EEL surfaces were automatically segmented using our fully three-dimensional LOGISMOS graph-based approach [23], [24], which shows excellent performance compared to the state-of-the-art algorithms on a public available IVUS image database [35]. Automatically determined surfaces were reviewed and algorithmically refined by an expert cardiologist using our previously reported computer-aided refinement approach [26]. Examples of the final segmentation can be seen in Fig. 4.

To evaluate the performance of the automated registration, 262 frame pairs with well-identifiable landmarks (baseline ↔ follow-up) were determined by Expert 1. The 262 follow-up frames were further rotated by Expert 1 to make their circumferential positions coincide with their corresponding baseline frames, where 5° steps were used. Expert 1 devoted about 50 minutes to each baseline/follow-up registration; 24 hours of expert effort were needed to define the ground truth in all 29 baseline/follow-up pairs. The remaining two experts (Experts 2 and 3) were provided the 262 landmarks in baseline pullbacks and were asked to find corresponding follow-up frames and properly rotate them to determine inter-observer variability of manual registration.

C. Registration Accuracy and Statistical Analysis

The registration results were first qualitatively evaluated by visually checking for gross inaccuracies. To quantify the success rate of the locational registration, distances (mm) between the expert-identified follow-up frames and the automatically registered frames were used as the performance measure (Eq. (10)). To quantitatively evaluate the orientation registration performance, rotational registration angles of corresponding frames were compared with those determined by the expert (Eq. (11)). All results are reported as the mean error ± the standard deviation.

$$\mathrm{\Delta }L=\Vert L_x-L_y\Vert ,$$ (10)

$$\mathrm{\Delta }O=min(\mid O_x-O_y\mid ,360°-\mid O_x-O_y\mid ),$$ (11)

where L_x and O_x are the automatically registered location and orientation, and L_y and O_y denote the independent standard.

For statistical analysis, differences between manual and computer registered locations were determined using a Wilcoxon signed-ranks test [36] to reflect non-normal distribution, while angular differences between orientations were determined using a paired Students t-test, both with p < 0.05 deemed statistically significant. Linear regression analysis [37] and Bland-Altman plots [38] were used to compare the rotation angle differences. To reveal the true angular difference between two rotations, angular values were first transformed to the range [−180°, 360°].

D. Comparison with Other Methods

Performance of our method was compared to the inter-observer variability of manually-performed registration, the work of Timmins et al. [9], our earlier-reported side-branch guided (SBG) approach [10], the DTW framework [14] with area-based costs [11], [12], and DTW framework [14] with correlation-based costs (Eq. (6)). For the method in [9], the location registration is actually the same as SBG (piecewise distance normalization) [10], the manual co-registration angle was derived from the first corresponding side-branch pairs obtained in [10]. Other parameters were set as in [9]. In the DTW framework, due to the sliding window approach [11], the method is not designed to register pullbacks of different length; the extremes of path search strategy [12] did not work well on our data set in a preliminary experiment (e.g., failed in 8 out of the 29 pullback pairs). Therefore, we used our automatically-determined start/end frame pairs to initialize the DTW framework for a relatively fair comparison of the registration performance. For DTW framework with area-based costs, the first four features used in [11], [12] were employed and the areas of calcified and fibro-lipidic plaque were derived from VH in our implementation, and the directional penalty [11], [12] was set as 0.05 by optimizing the accuracy using Expert 1 landmark frame registration as the ground truth. For DTW framework with correlation-based costs, we first optimized the orientation for each frame pair followed by locational registration in the DTW framework for which the orientation was used as a single fixed cost.

E. Computational Resources

The proposed system was tested on an HP Z400 workstation with 3.33 GHz Xeon W3680 CPU, 24 GB of RAM, running Windows 7 SP1 Enterprise. The mean execution times are reported in terms of the complete registration process, the identification of starting/ending frame pairs, and the frame-by-frame registration, respectively.

F. Registration Method Parameters

According to our earlier studies, the node connection parameters θ_twist used for the registration graph construction need not exceed 15° [34]. Others reported that the average rotation between consecutive EKG-gated IVUS image frames is 5.74°±0.15° [9]. Generally, the registration solution is more constrained for lower values of θ_twist and thus less computationally demanding. Here, θ_twist = 6° was experimentally set as a trade-off between accuracy and efficiency and its exact value was not critical for the method’s performance. The smoothness penalty weight ω was set as 0.1. Considering that perivascular landmarks (side-branch, small vein, myocardium proximity, etc.) can provide more consistent correspondence than plaque appearance and plaque thickness between baseline/follow-up, α was set to 0.5 to emphasize the importance of the perivascular tissue term; both β and γ were set to 0.25 to let the plaque appearance term and plaque thickness term contribute equally.

IV. Results

A. Qualitative Assessment of Registration Performance

Fig. 6 shows the automated registration of the pullback pairs shown in Fig. 1. Prior to registration, the reconstructed baseline and follow-up longitudinal views looked quite different (Fig. 1). After registration (Fig. 6), both the longitudinal and cross-sectional views show good correspondences between baseline and follow-up. The automatically (fourth row in Fig. 6) and manually (fifth row in Fig. 1) registered follow-up cross-sectional images are visually comparable in both appearance and orientation. Fig. 7 shows a 3D reconstruction of the lumen and EEL surfaces before and after automated registration. It can be visually appreciated that two segments become narrower (blue arrows in Fig. 7) and that plaque rupture is shown at follow-up (red circle in Fig. 7).

Fig. 6.

Automated registration of pullback pair from Fig. 1. From the first to the fourth row: Registered baseline pullback; registered follow-up pullback; baseline landmarks selected by Expert 1; registration results obtained by our automated method. Optimal path (green path) in 3D (bl, θ, fu) space and its projections (red paths) on 2D (bl, fu) plane and 2D (θ, fu) plane generated by our method for this example are shown in the upper-right panel.

Fig. 7.

Three-dimensional visualization of IVUS pullback registration. (a) Original pullback pair. (b) Automatically registered pullback pair. Lumen shown in orange, EEL surface in green. Notice that two segments become narrower (blue arrows) and that plaque rupture is visible at follow-up (red circle, in which two registered IVUS image pairs are shown on the right side).

Fig. 8 shows six examples of registration results obtained by a) our new method and b) three experts. When visually comparing our results with expert registration, good agreement was reached across a variety of image-induced challenges and morphology changes (note matching of the following clues in Fig. 8): side-branches (cases B, D, E, F), shadows (cases A, E), big and small calcifications (cases E, A, C), small veins (cases B, F), and myocardium proximity (cases C, F).

Fig. 8.

IVUS pullback registration. From the first row to the seventh row: our baseline/follow-up registration results in longitudinal views, baseline landmarks selected by Expert 1, corresponding follow-up registration obtained by our method, Expert 1, Expert 2, and Expert 3. White dotted lines indicate positions of the landmark locations shown in cross-sectional views.

B. Quantitative Assessment of Registration Performance

Table I shows the location and orientation registration errors of our 3D graph based method and the inter-observer variability. Compared to three expert registration results, the reported 3D-graph-based method achieved mean distance errors ranging from 0.72 mm to 0.79 mm with the mean angle errors ranging from 7.3° to 9.3°. There were no significant differences between our method and experts in location and orientation registration (p=NS). The maximal distance and angle errors generated by our method were 8.4 mm and 147°, respectively, comparable to the maximum disagreements between experts. For the 262 baseline landmarks selected by Expert 1, our method failed to determine eight corresponding follow-up frames; 7 of these were the most distal landmarks from 7 patients and 1 was the most proximal landmark. Similarly, Expert 2 failed to identify three of the most distal landmarks in three patients, stating that the corresponding landmarks were beyond the coverage of the follow-up pullbacks.

TABLE I.

Distance and rotational registration errors of inter-observer variability and our method. LM: landmarks. No differences were statistically significant

User	ΔL(mm)	Max ΔL(mm)	ΔO(°)	Max ΔO(°)	No. LM
Exp. 1 vs. Exp. 2	0.67 ± 1.28	10.73	9.47±16.53	130	259
Exp. 1 vs. Exp. 3	0.62 ± 0.95	6.00	9.17±15.39	120	262
Exp. 2 vs. Exp. 3	0.66 ± 1.31	10.73	6.53±14.37	175	259
3D graph vs. Exp. 1	0.75 ± 1.22	8.40	9.27±13.52	103	254
3D graph vs. Exp. 2	0.79 ± 1.17	7.93	7.33±10.67	125	253
3D graph vs. Exp. 3	0.72 ± 1.16	8.40	7.70±11.51	147	254

For statistical analysis of orientation registration, linear regression analysis (Fig. 9(a)–(b)) demonstrated that the automatically registered frame orientations exhibited high correlation with expert registrations (R² ranging from 0.97 to 0.98), comparable to inter-observer variability. The Bland-Altman plots (Fig. 9(c)–(d)) revealed that the rotation angles determined by our method and experts agreed well with bias values ranging from −1.57° to 0.15°, comparable to inter-observer variability ranging from −1.64° to 1.08°.

Fig. 9.

Linear regression analysis and Bland-Altman plots comparing frame orientation differences among experts (a,c), and between our method and experts (b,d).

Table II summarizes the location and orientation registration errors of Timmins method [9], SBG approach [10], DTW framework with area-based costs [11], [12], DTW framework with correlation-based costs, and of our new method. The proposed 3D graph based method achieved lower errors in both the location and orientation registration.

TABLE II.

Comparison of location and orientation registration errors of the Timmins method [9], SBG [10], DTW-area [11], [12], DTW-correlation (which uses the DTW framework of [14] with our correlation-based costs), and our 3D graph method. Note that our 3D graph method shows consistently lower registration errors. Note also that DTW-area [11], [12] was not designed for orientation registration.

Standard	Timmins method [9]		SBG [10]		DTW-area [11], [12]		DTW-correlation		3D graph
	ΔL(mm)	ΔO(°)	ΔL(mm)	ΔO(°)	ΔL(mm)	ΔO(°)	ΔL(mm)	ΔO(°)	ΔL(mm)	ΔO(°)
Exp. 1	1.35±1.55	19.40±23.81	1.35±1.55	11.70±15.42	1.23±1.62	-	0.79±1.19	18.57±35.26	0.75±1.22	9.27±13.52
Exp. 2	1.35±1.76	18.15±22.95	1.35±1.76	11.00±14.39	1.13±1.56	-	0.82±1.18	17.53±36.05	0.79±1.17	7.33±10.67
Exp. 3	1.19±1.49	18.32±22.95	1.19±1.49	10.90±14.11	1.14±1.57	-	0.73±1.08	17.32±34.72	0.72±1.16	7.70±11.51

The mean execution time for the complete automated registration process was 3.2 ± 1.5 minutes per pullback pair. Identification of starting/ending frame pairs and performing complete 3D frame-by-frame registration required 2.3 ± 1.1 and 0.9 ± 0.4 minutes per pullback pair, respectively. While the computation of the cost function was time consuming (about 6–24 hours per patient, depending on the length of the respective IVUS pullback) and was computed in parallel on a computer cluster in about two days for all 29 patients, such training is only performed once and its computational demands do not affect the analysis speed.

V. Discussion

A. Importance of Automated IVUS Pullback Registration

As mentioned in Section I, current clinical studies of the natural course of plaque progression and regression rely on tedious, time-consuming, and subjective registration of location and orientation of longitudinal IVUS pullbacks. We have reported in Section III-B that registering IVUS baseline and follow-up IVUS pullbacks by an expert cardiologist (Expert 1) required 50 minutes per pullback pair – and only landmark-frame pairs were registered. Clearly, it is extremely time-consuming to establish frame level correspondence manually, and almost impossible when the patient cohort is large. Given the variety of challenges presented in IVUS pullback data [12], [13], [32], manual registration requires relatively high-level of experience and expert interpretation. It is also subject to considerable intra- and inter-observer variability. As a result, current clinical studies mostly rely on comparisons of integral features determined over relatively long vessel segments and thus combining large tissue areas. This regional rather than local approach hinders our ability to achieve better understanding of local plaque progression and local plaque vulnerability [5], [9].

B. Comparison with Other Methods

To automatically register frame location, some of the previous methods simply normalize the frame distances between manually detected side-branches [9], [10]. As demonstrated in Table II, the distance registration errors of this type of method are relatively large mainly because of the irregularities in catheter speed in IVUS acquisitions. Notably, such errors were observed even when many frames with branches (already manually identified in these approaches [9], [10]) were used as landmarks. Other methods only make use of morphology-based features [11]–[13] such as vessel/plaque area and shape context, thus overlooking important image-based features such as plaque and perivascular tissue appearance, which demonstrated excellent performance in terms of location registration errors (see ΔL of “DTW-correlation” in Table II). Therefore, the registration results of such approaches [11], [12] are not robust enough to properly register follow-up frames with similar vessel areas or shape properties. Other reasons causing the DTW framework with area-based costs [11], [12] not to be sufficiently accurate on our data are likely attributed to: 1) portion of patient data used in [11], [12] were repeatedly acquired shortly after each other and at the same disease and intervention stages, thus without any morphologic changes. In contrast, our follow-up data were acquired after 1 year with possible plaque progression-induced changes of morphology; 2) although some data used in [11], [12] suffered from significant morphologic changes, only side-branch locations were used as landmarks in their evaluation; 3) in our data, the numbers of IVUS frames in baseline and follow-up sequence pairs are frequently quite different, which makes the registration task more challenging.

To automatically register frame orientation, previous solutions [9], [10] ignore the problem that appears when a stuck/accelerated IVUS pullback occurs during acquisition. Additionally, for methods relying on local optimization [9], if there is an incorrect locationally registered frame pair, the rotation angle may also be incorrect and using this incorrect angle to constrain consecutive image pair orientation registration will affect the result (refer to the relatively large ΔO of [9] in Table II). Furthermore, due to the lack of prior information about limited angular twisting, the DTW framework with correlation-based costs approach exhibited large orientation registration errors (refer to ΔO of “DTW-correlation” in Table II). In our globally optimal approach, the final orientation registration was not affected in this way.

C. Advantages of the Reported Method

An important advantage of the reported method is that it simultaneously registers locations and orientations of all frames in the two longitudinal IVUS pullbacks in a geometrically feasible manner. Subsequently, it is more robust than methods which register location or orientation separately. Furthermore, our system is designed to be highly automated as long as segmentation of lumen and EEL borders is available for all frames of the two image sequences. While our method was only tested on IVUS data, it may be easily extended to other intravascular imaging modalities solely by modifications of the objective function, including intracororonary optical coherence tomography (OCT) [39] or OCT-IVUS data co-registration [40] [41].

The contributions of our new method can be summarized as follows.

1. To the best of our knowledge, this is the first approach to simultaneously establish frame-to-frame correspondence in location and orientation between two image sequences using a 3D graph-based method.

2. The proposed 3D graph structure allows incorporating rotation-related features and catheter twisting prior for registration, and the designed clinical knowledge-based cost function is robust to vessel morphologic changes.

3. The reported method can automatically identify the starting and ending frame pairs in two image sequences.

D. Limitations

Similar to most of previous IVUS pullback registration approaches, our method relies on the segmentation of lumen and EEL, which is used to determine plaque and perivascular regions and also provides a common lumen centroid for both the baseline and follow-up image pairs to calculate the correlation terms in Eq. (6). Inaccurate or ambiguous segmentation may result in incorrect final registration. Sensitivity of the registration to such segmentation inaccuracies need further evaluation in our future work. A potential solution for avoiding segmentation dependency may be to identify discriminative regions to measure the structural similarity between two images as used in video-volume registration for endoscopic 3D motion tracking [20]. Nevertheless, accurate IVUS segmentation is a common pre-requisite to any quantitative IVUS-based studies of coronary atherosclerosis and as such must always be accomplished regardless of the registration needs [1]–[6], [8], [31], [32].

Even though our proposed method outperforms previous methods especially in handling irregular catheter speeds and dealing with morphologic change, it may not perform well if the IVUS pullback include images with a pronounced frame-to-frame “jump,” in which a large number of image frames is missing in one of the paired pullbacks either due to gating issues or a sudden movement of the catheter (e.g., after being slowed down by local plaque’s resistance). A possible approach to solve this problem is to increase the node-connection distance during the graph construction. The trade-off, however, may be a loss of image sequence continuity. Alternatively, utilizing branch detection algorithms either automatically [42] or semi-automatically [43] may help constrain the range of frames considered for registration.

Another limitation of our method is that the computation burden of cost function calculation may increase noticeably with increasing length of the registered IVUS sequences. Potentially, such a problem could be overcome by employing an acceleration algorithm [44]. However, integration of such a speed-up delivering solution is beyond the scope of this work and may be attempted in the future. Recently, we have developed an efficient, integrated, side-branch constrained framework that is not subject to length-dependent computational complexity increases [45].

VI. Conclusion

The presented results demonstrate that our automated 3D graph-based registration method outperforms recent semiautomated approaches and achieves registration accuracy closely approaching inter-observer variability. The presented 3D graph-based framework yields globally optimal registration of location and orientation of baseline and follow-up IVUS pullbacks, which is achieved in a single optimization step by using a 3D dynamic programming algorithm. By incorporating rotation-related features and catheter twisting prior in the graph structure, designing a comprehensive knowledge-based cost function, and extracting global and local graph information for identification of starting/ending image pairs, our work delivers highly automatic, accurate and robust IVUS pullback registration. The new method was tested in 29 in vivo IVUS pullback pairs from 29 patients. The experimental results yielded location and orientation registration errors that were close to the inter-observer variability. Our method has a potential to enable large-volume focal studies of natural course of plaque development in human coronary arteries in vivo.