Open Access
Add to BibSonomy
Abstract
Coronary stenting or percutaneous coronary intervention (PCI) is widely used to treat coronary artery disease. Improper deployment of stents may lead to post-PCI complication, in-stent restenosis, stent fracture and stent thrombosis. Intravascular optical coherence tomography (OCT) with micron-scale resolution provides accurate in vivo assessment of stent apposition/malapposition and neointima coverage. However, manual stent analysis is labor intensive and time consuming. Existing automated methods with intravascular OCT mainly focused on stent struts with thin tissue coverage. We developed a deep learning method to automatically analyze stents with both thin (≤0.3mm) and very thick tissue coverage (>0.3mm), and an algorithm to accurately analyze stent area for vessels with multiple stents. 25203 images from 56 OCT pullbacks and 41 patients were analyzed. Three-fold cross-validation demonstrated that the algorithm achieved a precision of 0.932±0.009 and a sensitivity of 0.939±0.007 for stents with ≤0.3mm tissue coverage, and a precision of 0.856±0.019 and a sensitivity of 0.874±0.011 for stents with >0.3mm tissue coverage. The correlation between the automatically computed and manually measured stent area is 0.954 (p<0.0001) for vessels with a single stent, and is 0.918 (p<0.0001) for vessels implanted with multiple stents. The proposed method can accurately detect stent struts with very thick tissue coverage and analyze stent area in vessels implanted with multiple stents, and can effectively facilitate the evaluation of stent implantation and post-stent tissue coverage.
© 2021 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Coronary artery disease (CAD) is the leading cause of mortality and morbidity worldwide [1–4]. The etiology of CAD is mainly associated with atherosclerosis, an inflammatory process with cholesterol accumulation leading to hardening and narrowing of coronary arteries [5]. Both genetic and lifestyle factors contribute to individual-level risks of CAD [6]. For patients whose coronary diseases are less complex, percutaneous coronary intervention (PCI) with stent implantation is recommended [7]. Evaluation of stent deployment and post-PCI tissue coverage is crucial to assess the long-term risks associated with stent implantation.
Intravascular optical coherence tomography (OCT) is an increasingly popular light-based imaging modality with high axial resolution (10-15$\mu$m), and can be used for in vivo evaluation of stent implantation in coronary arteries [8]. The principle of OCT is to measure the echo time delay produced by the backscattering of light from the vessel wall based on interference of light [9]. Metallic stents strongly reflect light emitted from the fiberoptic catheter, exhibiting a bright spot followed by a dark shadow in OCT images. The obvious features can help physicians to quickly locate the position of stent struts and calculate tissue coverage thickness. However, with OCT, the dataset of one pullback consists of hundreds of cross-sectional images. Manual identification of stent struts is very time consuming and labor intensive. Moreover, manual analysis will intrinsically lead to intra- and inter-observer variability [8]. This is particularly significant for stent struts with very thick tissue coverage shown in Fig. 1(a), where the shadow features of the struts are usually very weak or even absent due to the strong attenuation of light by the thick neointima. Automatic stent detection algorithms and quantitative measurements can drastically alleviate physicians’ workload and reduce inter-observer variability.
Fig. 1. Stent struts with very thick tissue coverage in OCT images. (a): Stent struts from a single stent. (b): A vessel with two stents implanted. Blue markers indicate stent struts. Red/green: inner/outer stents. Arrow: guide wire.
Prior to this research, many algorithms were reported for automatic detection of metallic stents [8,10–20]. Early research mostly focused on hand-crafted design of strut features using 2D cross-sectional images for stent detection, such as maximum reflected intensity [10] in each A-line, intensity profile of individual A-lines [8], etc. Later, algorithms based on machine learning were proposed, such as those adopting continuous wavelet transform (CWT) with feature extraction and classification using probabilistic neural network (PNN) [12] and fuzzy C means (FCM) [13], K-nearest neighbor method with feature vectors [14], and classification with bagged decision trees with feature engineering [15,16]. Based on the previous method on stent detection [15], Lu et al. [21] further developed a support vector machine (SVM) classification algorithm to assess tissue coverage, and a graph growing algorithm to identify clusters of uncovered struts. Utilizing 3D knowledge of stent structure by graph search, Wang et al. [17] proposed a Bayesian network and achieved promising performance on a large validation dataset containing 103 pullbacks from 72 patients. A highly automated, comprehensive OCT image visualization and analysis toolkit for stent detection and assessment (OCTivat-Stent) [22] based on the above mentioned methods [15,17,21] provided a reliable analysis platform with improved statistical power for comparing designs. Neural network methods were subsequently introduced for stent detection. An artificial neural network [18] with one hidden layer and ten nodes was used to classify strut candidates obtained by the first and the second gradient along the strut axial direction. Based on the detected stent struts, quantitative analysis of stent strut apposition and coverage has been extensively studied and solved by previous methods [8,17,20–22].
Benefiting from the emergence of deep learning and convolutional neural network (CNN) technologies [23–27], significant breakthrough was made in the field of medical image processing [28]. A deep learning model based on 2D image segmentation for stent analysis was introduced by Guo et al. [19], demonstrating superior performance to conventional image processing and machine learning. Subsequently, a pseudo 3D segmentation network [20] was adopted to capture the 3D information of stent continuity.
However, nearly all previous methods were designed to detect stent struts with thin tissue coverage. There is a lack of research for automating the analysis of stent struts with very thick tissue coverage such as shown in Fig. 1(a), where the task is intrinsically more challenging as the shadows following the stent struts are less obvious, and the signals of stent struts are not strong enough either. In addition, for PCI patients with long-term follow-up, there could be multiple stents implanted in the same vessel to treat in-stent restenosis (ISR) (Fig. 1(b)). It is crucial to detect stent struts with very thick tissue coverage in vessels both with single and multiple stents to correctly measure the tissue coverage thickness and stent area for patients with long-term follow-up to understand the re-stenotic status of the vessel and optimize subsequent treatment. In multiple stenting cases, the earlier and the later implanted stents can form different layers. Here, we refer the inner stent (red points in Fig. 1(b)) to the one close to vessel lumen boundary, and the outer stent (green points in Fig. 1(b)) as the one away from the lumen. How to best perform quantitative measurements such as stent area and tissue coverage thickness is less obvious. However, accurate identification of different stent layers and quantification of the inner stent area using intracoronary imaging are vital for the optimal treatment of patients with ISR [29]. To the best of our knowledge, there is no research reporting automated methods for analyzing multi-layer stent struts. We experimented existing methods reported in the literature, and found the conventional methods [8,10–18] based on shadow and pixel intensity did not work anymore for such cases, and it is also challenging for CNN methods.
This research aimed to fulfill the unmet clinical need to achieve automatic detection of stent struts with both thin and thick tissue coverage, and accurately assess the stent area for vessels implanted with multiple stents. We proposed a CNN method to achieve the above goals, and extensively validated the method by comparing the algorithm performance with two experienced analysts as well as with other state-of-the-art methods. We also evaluated the accuracy of quantitative measurements such as stent area. Finally, we discuss the significance and limitations of our method in a clinical context.
2. Materials and methods
2.1 Materials
A total of 56 OCT pullbacks from 41 patients with metallic stents were collected, containing 25203 cross-sectional images and 144712 struts. The images were acquired by a commercially available OCT system (Abbott C7-XR$\rm {^{TM}}$ coronary OCT imaging system, Westford MA). The system runs at a pullback speed of 20 mm/s with a frame rate of 100 images/s. All pullbacks were acquired in polar coordinates consisting of 496 A-lines each with 992 pixels, and were transformed to Cartesian coordinates of 1024$\times$1024 pixels for display. The tissue coverage and malapposition distance of stent struts were calculated using the method proposed by the previous publication [20]. Stent struts with >0.3mm tissue coverage account for 26.75% of the total dataset present with stents. This study was approved by the local IRB office of authors’ institutions. Patients’ information was de-identified before analysis.
The dataset was divided into three subsets to conduct three-fold cross-validation to evaluate the performance of stent detection algorithm. The principle of data division is to keep the proportion of images with thin and thick tissue covered stents relatively close among the three subsets, as shown in Table 1. In particular, stents with >0.3mm tissue coverage in the three subsets respectively account for 26.05%, 30.12%, and 24.79% (Table 1). In this study, struts with negative tissue coverage (i.e. malapposition) were categorized as $\le$0.3mm. For each fold, only the images with stents from the first two subsets were used to train neural networks, but all images in the third subset were included for evaluating stent analysis, including the images with stent struts from the stented vessel segments, and other images without stent struts outside the stented area. No image was excluded for any reason, representing a full spectrum of possible cases encountered in clinic with varying degree of image quality and various imaging artifacts such as residual blood and eccentric catheter position. Because both stent struts and guide wires have similar bright spots and shadow features in OCT images (Fig. 1), all struts and guide wires were labeled with circular binary masks covering stent struts or guide wire blooming by a trained analyst at pixel-level using an open source software (Labelme, MIT, Cambridge, MA). OCT images in polar coordinates were resized into 512$\times$512 before image analysis.
Table 1. Data assignment for three-fold cross-validation
In our dataset, a total of 13 pullbacks containing 733 OCT images were with multiple stents. The locations of stent struts in polar coordinates were transformed into Cartesian coordinates, which reflect the real structure of the stents and are more in line with physicians’ interpretation convention, and transforming the locations of stent struts into the Cartesian coordinates did not generate any data loss. We further generated corresponding binary strut masks as the input for developing the subsequent multi-layer stent classification model. Because there are insufficient number of binary images with multiple stents for training CNNs with three-fold cross validation, we used 623 binary images from subset 1 and 2 containing multiple stents as the training set to train a CNN model for multi-layer stent strut classification, and used binary images with single stent and multiple stents extracted from subset 3 for testing. We specifically refer subset 3 as the test set for this regard, and also used it for validating stent area measurement, and inter-observer variability in later experiments. These binary images were labeled in the same way but with each layer of struts marked with a different pixel value. The 623 binary images were augmented to 1869 images by image rotations (90°, 180°, 270°) to improve the diversity of the data for training. In the experiments, we performed multi-layer stent classification with all images of a pullback as the input on all cases with single and multiple stents without excluding a single image.
2.2 Methodology overview
The overall algorithm consists of stent strut detection, multi-layer stent classification and stent area measurement (Fig. 2). At first, original OCT images in polar coordinates acquired from OCT imaging systems are used as input of the stent detection model, which generates the output segmentation maps which are then transformed into binary images in Cartesian coordinates, and sent to a multi-layer stent classification model (a separate CNN) to distinguish between inner and outer stent struts. The multi-layer stent classification model is essentially a segmentation model to assign different layers of struts into different pixel values. Finally, at the stage of stent area measurement, only the inner stent struts are used to generate a smooth stent area curve by applying cubic spline interpolation with the information of lumen boundary.
Fig. 2. Flow diagram illustrating the overall methodology for stent strut detection, multi-layer stent classification and stent area measurement. The original OCT images in polar coordinates are resized into 512$\times$512 and input into a CNN model. The output segmentation maps are converted into binary images, which are taken as the input of the second separate CNN model. After applying cubic spline interpolation, a smooth stent curve (white) passing through all inner layer struts (red points) is obtained, and the region enclosed by the stent curve is the stent area. Red/green: inner/outer layer struts.
2.3 Stent detection with very thick tissue coverage using deep learning
Inspired by the widely used encoder-decoder structure in U-Net [30] and other CNN segmentation models [27,31], we designed a CNN model with an encoder-decoder architecture for stent detection as shown in Fig. 3. The feature encoder module is based on ResNet-50 and can extract high-level semantic information from the original input, and gradually shrink the spatial dimension of feature maps to encode more abstract and global information. The shortcut mechanism of ResNet [24] utilized in the encoder blocks can effectively avoid gradient vanishing and accelerate convergence of network training. The feature decoder module is used to gradually recover spatial resolution reduced in the feature encoder. Inspired by [32], we further adopt a dense upsampling convolution (DUC) module to upsample the feature maps instead of using linear upsampling layer or deconvolution layer in conventional CNNs [30,31]. In DUC, a feature map with a dimension of h/2$\times$w/2$\times$c is fed into a 1$\times$1 convolution with 4c output channels, generating a feature map with a dimension of h/2$\times$w/2$\times$4c, which is then reshaped to h$\times$w$\times$c. The DUC block is able to capture and decode more detailed information, which is essential for small object detection such as the deeply covered struts in OCT images. In addition, we use four skip connections fusing the feature maps from the encoder and decoder module to remedy resolution loss caused by max-pooling and strided convolutional layers.
Fig. 3. Architecture of our proposed CNN model. The model consists of a feature encoder and a feature decoder module. The feature encoder module uses ResNet-50 as the backbone network. The feature decoder module adopts a dense upsampling convolution (DUC) block for more effective feature decoding. Each convolution operation of decoder blocks is followed by a batch normalization layer and a ReLU activation layer.
After obtaining stent segmentation results, we transform the segmentation maps into binary images. The center of each connected component in the segmented strut (blooming area) is extracted one by one as the precise location of each stent strut. The stent segmentation maps are further used as the input of a multi-layer stent classification model. In this research, the CNN model with the same architecture as that for stent detection is used to achieve multi-layer stent classification, but with different weights. The multi-layer stent classification model predicts segmentation output of strut layers with the same size as that of the input binary image. With the segmentation maps of pixel-level strut classification results, we can differentiate inner and outer layer stent struts, while the classification makes no difference to cases with a single stent.
2.4 Stent area measurements
With the detected stent struts, several quantitative measurements can be performed, such as stent area, malapposition area, percentage of uncovered stent struts, tissue coverage area and thickness, and etc. These measurements require a precise delineation of vessel lumen. In this study, stent detection does not require lumen segmentation as typically required in previous reports [8,10,12,13,15–17]. Instead, an independent lumen segmentation module is adopted. This way, any possible accumulated errors can be avoided between lumen and stent detection.
For lumen segmentation, we constructed a dataset with 8051 OCT images as the training set and 2684 images as the test set. Another state-of-the-art CNN model U-Net [30] is used to directly locate vessel lumen boundaries in OCT images in Cartesian coordinates. The centroid and border coordinates of the segmented lumen, together with the center of stent struts, are used for quantitative stent measurements.
For stent area measurements, we firstly exclude outer layer struts based on the multi-layer stent classification model whenever multiple stents are present. In this research, we adopt two stage interpolation scheme [17] to generate stent contour. The first interpolation is to generate evenly-spaced virtual stent points (“anchor points") between detected stent struts with a consistent distance away from the vessel boundary based on their adjacent strut locations. The second interpolation generates a smooth stent contour curve using cubic spline interpolation by ensuring the contour traversing all the inner struts and the virtual anchor points. After that, the stent area enclosed by the inner stent struts can be calculated. An example of applying multi-layer stent classification and stent area measurement is shown in Fig. 4.
Fig. 4. An example of multi-layer stent area measurement. (a) When we directly calculate stent area using all detected stents, there could be significant errors in estimating the real stent area. (b) After distinguishing the inner stent struts from the outer stent struts, only the inner stent struts are used to calculate the stent area, which can better estimate stent expansion and vessel re-stenotic status. Blue points: automatically detected struts. White: stent contour for stent area measurement. Red/green: inner/outer layer struts.
2.5 Experimental details
The backbone of ResNet-50 [24] was used to achieve multi-class segmentation to differentiate stents, background and guide wires. We adopted a combination of Dice loss (1) and multi-class cross entropy (2) loss for training, as shown in (3):
2.6 Algorithm evaluation and statistical analysis
The algorithm performance for stent detection was evaluated on strut-level instead of pixel-level. Detected struts were classified as true positives (TP), false positives (FP) and false negatives (FN) with the human annotated struts as the gold standard. The locations of detected struts and labeled struts were compared in Cartesian coordinates to address the issue of wrap-around ($0^{\circ }$ vs $360^{\circ }$). If the distance between the centers of a predicted strut and centers of any of labeled stents is less than 94$\mu$m (10 pixel= 94$\mu$m in Cartesian coordinates), the predicted strut is classified as TP. Performance of stent detection was evaluated using the metrics of precision and sensitivity, computed as follows:
Dice coefficient was used to evaluate performance of lumen segmentation and compare inter-observer variability of stent area measurement:
The results of stent detection were evaluated on the whole test set, instead of computing sensitivity and precision of each image and averaging them. Comparison with other methods was evaluated using paired Student’s t-test, and a p-value<0.05 indicates statistical significance.
It is inevitable that different analysts may have different personal perceptions and experiences in analyzing OCT images and therefore introduce inter-observer variability. Meanwhile, the features of the deeper struts with tissue coverage >0.3mm are inherently of low contrast and are inconspicuous, which increase the possibility of generating inconsistent stent annotations. Therefore, to evaluate the discrepancy between analysts, 4 randomly selected cases from the test set (subset 3) containing 832 OCT images were labeled by another independent image analyst blinded to the results from the first analyst and the algorithm. The stent struts with >0.3mm tissue coverage in the dataset account for 25.23%, which is close to the distribution in the whole training set and test set. In addition, another randomly selected 110 OCT images in vessels implanted with multiple stents from the test set were utilized to assess the inter-observer variability of multi-layer stent area measurement between algorithm and analysts, and between the analysts. The area enclosed by the predicted inner stent struts was compared with the area generated by the manually identified inner stent struts. The agreements between the algorithm and analysts, and between the analysts were compared using Pearson correlation coefficient and Bland-Altman analysis.
3. Results
3.1 Stent detection and segmentation
Based on three-fold cross-validation, the proposed method achieved a precision of 0.932$\pm$0.009 and a sensitivity of 0.939$\pm$0.007 for stents with $\le$0.3mm tissue coverage, and a precision of 0.856$\pm$0.019 and a sensitivity of 0.874$\pm$0.011 for stents with >0.3mm tissue coverage (Table 2). Comparing with a representative conventional method using a Bayesian network [17] and other widely used CNN methods, including Deeplab V3+ [27], U-Net [30], CE-Net [31], and a newly proposed TransUNet [34], our proposed method showed a compelling performance and short inference time. The performance on the deeper struts (>0.3mm) is slightly worse than that of the shallow struts ($\le$0.3mm), because the training data for deeper struts are relatively limited and their features are less obvious due to the strong signal attenuation of OCT imaging. Figure 5 illustrates representative stent detection results containing some challenging cases. The stent detection model is able to accurately detect stent struts with thin tissue coverage (Fig. 5(a)), and locate the stent struts with thick tissue coverage in follow-up OCT cases (Fig. 5(c, d)) and in cases present with multi-layer stent struts (Fig. 5(h)). Residual luminal blood (Fig. 5(b, f)), stent malapposition (Fig. 5(e)) and plaque rupture (Fig. 5(g)) are three typical challenging scenarios for automated detection of struts, but are handled well using our proposed method. Capturing those unobvious stent struts in single stent (Fig. 5(c, d)) and multiple stents (Fig. 5(h) is vital for stent area measurement in the general case. Figure 6 shows some typical failure examples caused by image artifacts, echo reflections and extremely low-contrast (Fig. 6(c)). The image artifacts (Fig. 6(a)) and echo reflections (Fig. 6(b)) have similar features of stent struts and generated some false positives. Incomplete blood flushing (Fig. 6(d)) caused absence of artery lumen and shadow feature and also contributed errors.
Fig. 5. Stent detection results by the proposed algorithm. For each case, the top row shows original images and the bottom row illustrates the corresponding detected stent struts. (a): Clear lumen with thin-medium tissue coverage. (b): Significant residual luminal blood. (c) Thrombus and very thick tissue coverage. (d): Struts with very thick tissue coverage. (e): Malapposition. (f): Significant luminal blood and low image contrast. (g): Plaque rupture. (h): Multiple stents.
Fig. 6. Four typical examples of stent detection failure. (a): A false positive generated by shadow artifact (red arrow). (b): Echo reflections may contribute to the source of false positives (red arrow). (c): Struts without bright reflections and high-contrast shadows may be missed by the algorithm and generate false negatives (green arrows). (d): A stent strut (green arrow) is missed due to the lack of shadow feature and residual blood artifacts.
Table 2. Algorithm performance and comparsion with state-of-the-art
3.2 Ablation experiments
In ablation experiments, we individually removed the skip connections (Ablation 1) and replaced the DUC modules (Ablation 2) with the widely used de-convolution layers to upsample feature maps. As shown in Table 3, skip connections and DUC modules can effectively improve network performance. Moreover, resizing the input images into higher resolution (1024$\times$1024) resulted in a small gain in performance but with a significant increase of inference time.
Table 3. Ablation experiments for our proposed method
3.3 Multi-layer stent classification and stent area measurement
We evaluated the performance of the classification model using 110 images with multiple stents from subset 3. The proposed multi-layer stent classification model achieved a precision of 0.975 and a sensitivity of 0.958 for classifying inner stent struts, and a precision of 0.901 and a sensitivity of 0.940 for classifying outer stent struts. Figure 7 shows stent contour generation for stent area measurement for single stent (Fig. 7(a)) and multiple stents (Fig. 7(b, c, d)). For multi-layer stent struts, the stent area enclosed by inner stent struts is calculated after applying multi-layer stent classification. Misclassifying few inner struts as outer struts for single stent (Fig. 8(a)) and multiple stents (Fig. 8(b)) hardly affects stent area measurement because our algorithm can automatically interpolate virtual strut points based on adjacent inner strut information and vessel boundaries to make correct area measurement. However, for multiple stents, misclassifying outer struts as inner struts may cause errors in stent area measurement, such as shown in Fig. 8(c). The lumen segmentation model achieved a Dice coefficient of 0.988 $\pm$ 0.045 on the test set, providing an accurate reference for stent area measurement. But residual luminal blood (Fig. 8(d)) may deteriorate lumen segmentation results, resulting in significant deviation of stent area contour with erroneously interpolated virtual stent points.
Fig. 7. Stent area measurements after applying multi-layer stent classification. (a): Stent area measurement is not affected for single stent. (b)-(d): Inner and outer layer stent struts are correctly distinguished in a typical OCT image containing multiple stents. The stent area is computed using only the inner layer stent struts.
Fig. 8. Examples of stent area measurement with misclassification of multi-layer stent struts or errors in lumen segmentation. (a): A single stent strut is misclassified as the outer strut, but with the stent area measurement unaffected. (b): An inner stent strut is misclassified as the outer strut but hardly affects stent area measurement. (c): An outer stent strut is misclassified as the inner strut, resulting in larger stent area. (d): Wrong lumen segmentation due to significant luminal blood will cause errors in stent area measurement (Red points: inner struts; green points: outer struts; white points: virtual stent points; blue contour: vessel boundaries).
We individually calculated the correlation between automatic and manual stent area measurement. A strong correlation (r=0.954, p<0.0001, Fig. 9(a)) for single stent area measurement was achieved between automated and manual analysis. Bland-Altman plot shows overall good agreement, but with a slight offset between the algorithm and the analyst (95% limits of agreement -0.16$\pm$1.32 $mm^2$, Fig. 9(b)). For multiple stents, our algorithm also shows good correlation (r=0.918, p<0.0001, Fig. 9(c)) and overall good agreement (95% limits agreement of -0.11$\pm$0.51 $mm^2$, Fig. 9(d)). To assess the difference of stent area measurement with different tissue coverage, we calculated the average coverage thickness in images from vessels implanted with a single stent, and compared the stent area for stents with thick tissue coverage (average tissue coverage >0.3mm) vs. thin coverage ($\le$0.3mm). The stent area measurement for struts with thick tissue coverage generated a lower correlation (r=0.925, p<0.0001) and a slightly worse agreement (95% limits agreement of -0.24$\pm$1.52 $mm^2$) compared with that with thin tissue coverage (r=0.962, p<0.0001, 95% limits agreement of -0.13$\pm$1.25 $mm^2$).
Fig. 9. Pearson correlation coefficient and Bland-Altman analysis. Correlation and corresponding Bland-Altman plot between automatic and manual stent area measurement in vessels implanted with a single stent ((a)-(b)), and multiple stents ((c)-(d)).
3.4 Comparison of stent detection and stent area measurement between different analysts
As shown in Fig. 10, the agreement between the algorithm and analyst is very close to that between the analysts for both categories. The precision ($\le$0.3mm) and sensitivity ($\le$0.3mm) of [algorithm VS analyst A, algorithm VS analyst B, analyst A VS analyst B] respectively are [0.962, 0.949, 0.948] and [0.953, 0.939, 0.949]. The precision (>0.3mm) and sensitivity (>0.3mm) of [algorithm VS analyst A, algorithm VS analyst B, analyst A VS analyst B] respectively are [0.901, 0.909, 0.876] and [0.899, 0.862, 0.919]. Figure 10 shows that the algorithm achieved the best agreement with analyst A. This is because the algorithm was trained on the dataset annotated by analyst A. For stent area measurement, the Dice coefficient of [analyst A VS analyst B, algorithm VS analyst A, algorithm VS analyst B] respectively is [0.970$\pm$0.035, 0.972$\pm$0.027, 0.968$\pm$0.032], indicating that the algorithm achieved a very close agreement with the two analysts.
Fig. 10. Inter-observer variability of stent detection and stent area measurement. (a): Stent detection comparison between the algorithm and two analysts. The metrics are stratified to different tissue coverage categories. The differences between algorithm and analysts are close to that between the analysts. (b): Comparison of stent area measurement between algorithm and analysts for cases with multiple stents.
3.5 3D reconstruction of multiple stents
Figure 11 shows 3D rendering of a vessel with two stents implanted. The red and green points represent inner and outer layer of stent struts respectively. It can be clearly observed that the inner and outer stent struts are distributed at different depth.
4. Disscusion
In this study, we proposed a computer algorithm based on a novel deep convolutional neural network to achieve automatic stent detection and classification of multi-layer stent struts. Our method outperformed existing methods for detecting struts with very thick tissue coverage using a test set containing all OCT images from 18 pullbacks regardless of whether there are stents. Testing a full spectrum of possible cases encountered in clinic is attractive for clinical applications. Moreover, to the best of our knowledge, we are the first to propose the methodology for classification of multiple stents and correctly calculate the stent area enclosed by the inner stent struts. In our comparison experiments, close agreement was achieved between the algorithm and analysts both for stent detection and stent area measurement.
Fig. 11. 3D reconstruction of a vessel implanted with two stents. (a): Longitudinal section view of a coronary artery with two stents implanted. (b): Lateral view of 3D reconstruction. (red/green: inner/outer layer struts). (c): Fly-through view shows depth information of inner and outer layer stent struts.
In-stent restenosis (ISR) is defined as luminal narrowing with >50% diameter stenosis of a stented coronary segment at follow-up angiography [35]. Although drug-eluting stents (DES) can significantly reduce the incidence of ISR compared with bare-metal-stents (BMS) [36], the prevalence of ISR is estimated to vary from 5% to 10% for patients with second-generation DES [29], and 10% to 20% of patients receiving DES need to treat ISR recurrence, leading to recurrent in-stent restenosis (R-ISR) [37]. Therefore, ISR remains a clinical challenge for long-term patient management. The mechanism and main contributors of ISR include stent under-expansion or fracture, aggressive neointimal proliferation with inflammation and neoatherosclerosis [29]. According to the mechanism and contributors of ISR and other factors, there are some alternative treatments for ISR, including balloon angioplasty, cutting and scoring balloon, atherectomy and laser ablation, drug-coated balloons (DCB) and repeated DES [35]. Therefore, it is vital to correctly assess stent-related measurements and individualize the treatment of ISR to optimize clinical outcomes. For patients with ISR and R-ISR, the tissue coverage of stent struts in OCT images is often thicker than 0.3mm, and there is a lack of clinical tool for analyzing such cases. Based on the specific cause of ISR, an algorithmic approach guided by intracoronary imaging for the treatment of DES-ISR is proposed in previous clinical research [29]. Our proposed algorithm for stent detection of single and multiple stents with very thick tissue coverage can be incorporated into the approach to provide insight into the mechanism of ISR, guiding treatment decision-making for ISR.
The algorithm we proposed can be applied universally to detect stents both with thin and with very thick tissue coverage, and can accurately compute stent area in both cases. Comparing with previous studies [11–13,15–17,20] that can only calculate stent area for vessels implanted with a single stent, our method is highly significant for evaluating complex stenting cases, especially for vessels with multiple stents to evaluate the risk of restenosis and re-stenting. Integrated stent information with unobvious outer stent struts is beneficial to improve the performance of the multi-layer stent classification model, ensuring accurate stent area measurement. Moreover, the short inference time of the proposed method can help physicians obtain real-time quantitative measurements for treatment decision planning during the PCI procedure.
Since OCT images are acquired with a pullback, a pseudo-3D CNN [20] with consecutive OCT slice input was proposed to improve the performance of stent detection. However, we have not found such a strategy could improve our model performance. A truly 3D CNN is computationally intensive and harder to train with our limited data. Moreover, the comparison experiments (Fig. 10) demonstrated that our method already achieved human level performance.
This study has some limitations. First, only a few patients in our database have experienced two PCIs with stent implantation at the same segment of the same vessel to treat re-stenosis. Therefore, it is challenging to train the classification model for multi-layer stent struts. The performance of the model can be further improved by using more training data. Secondly, we have not directly compared our proposed algorithm with previous proposed CNN methods [19,20] for stent detection because there is no standard OCT image database. Moreover, those algorithms are not open-source yet. However, we did try our best to implement the best known state-of-the-art CNN methods by ourselves and thoroughly compared their performance with our proposed method using our test data. It is worthy of note that previous studies did not consider stent area measurements for multi-layer stent struts. Although our proposed method has some limitations, we believe that our work has a great potential to motivate more clinical research for follow-up stent analysis for patients with restenosis and re-stenting.
5. Conclusion
In summary, we proposed a novel CNN method and achieved automated detection of stent struts with both thin and thick tissue coverage (>0.3mm), as well as a new method for analyzing stent area for vessels implanted with multiple stents. Validation studies demonstrated that our proposed method achieved good accuracy with human analysts in stent detection, stent area measurement and multi-layer stent analysis, and has the potential to be used in clinic to drastically reduce the workload of physicians.
Funding
National Natural Science Foundation of China (62075033, 62135002, 61921002); Sichuan Province Science and Technology Support Program (2020YFS0076).
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. C. J. Murray and A. D. Lopez, “Mortality by cause for eight regions of the world: global burden of disease study,” Lancet 349(9061), 1269–1276 (1997). [CrossRef]
2. C. D. Mathers and D. Loncar, “Projections of global mortality and burden of disease from 2002 to 2030,” PLoS Med. 3(11), e442 (2006). [CrossRef]
3. T. A. Gaziano, A. Bitton, S. Anand, S. Abrahams-Gessel, and A. Murphy, “Growing epidemic of coronary heart disease in low-and middle-income countries,” Curr. Probl. Cardiol. 35(2), 72–115 (2010). [CrossRef]
4. I. Abubakar, T. Tillmann, and A. Banerjee, “Global, regional, and national age-sex specific all-cause and cause-specific mortality for 240 causes of death, 1990-2013: a systematic analysis for the global burden of disease study 2013,” Lancet 385(9963), 117–171 (2015). [CrossRef]
5. B. H. Rice, “Dairy and cardiovascular disease: a review of recent observational research,” Curr. Nutr. Rep. 3(2), 130–138 (2014). [CrossRef]
6. A. V. Khera, C. A. Emdin, I. Drake, P. Natarajan, A. G. Bick, N. R. Cook, D. I. Chasman, U. Baber, R. Mehran, and D. J. Rade, “Genetic risk, adherence to a healthy lifestyle, and coronary disease,” N. Engl. J. Med. 375(24), 2349–2358 (2016). [CrossRef]
7. S. Deb, H. C. Wijeysundera, D. T. Ko, H. Tsubota, S. Hill, and S. E. Fremes, “Coronary artery bypass graft surgery vs percutaneous interventions in coronary revascularization: a systematic review,” Jama 310(19), 2086–2095 (2013). [CrossRef]
8. G. J. Ughi, T. Adriaenssens, K. Onsea, P. Kayaert, C. Dubois, P. Sinnaeve, M. Coosemans, W. Desmet, and J. D’hooge, “Automatic segmentation of in-vivo intra-coronary optical coherence tomography images to assess stent strut apposition and coverage,” Int. J. Card. Imaging 28(2), 229–241 (2012). [CrossRef]
9. H. G. Bezerra, M. A. Costa, G. Guagliumi, A. M. Rollins, and D. I. Simon, “Intracoronary optical coherence tomography: a comprehensive review: clinical and research applications,” JACC Cardiovasc. Interv. 2(11), 1035–1046 (2009). [CrossRef]
10. G. T. Bonnema, K. O. Cardinal, S. K. Williams, and J. K. Barton, “An automatic algorithm for detecting stent endothelialization from volumetric optical coherence tomography datasets,” Phys. Med. Biol. 53(12), 3083–3098 (2008). [CrossRef]
11. C. Xu, J. M. Schmitt, T. Akasaka, T. Kubo, and K. Huang, “Automatic detection of stent struts with thick neointimal growth in intravascular optical coherence tomography image sequences,” Phys. Med. Biol. 56(20), 6665–6675 (2011). [CrossRef]
12. S. Tsantis, G. C. Kagadis, K. Katsanos, D. Karnabatidis, G. Bourantas, and G. C. Nikiforidis, “Automatic vessel lumen segmentation and stent strut detection in intravascular optical coherence tomography,” Med. Phys. 39(1), 503–513 (2011). [CrossRef]
13. K. Mandelias, S. Tsantis, S. Spiliopoulos, P. F. Katsakiori, D. Karnabatidis, G. C. Nikiforidis, and G. C. Kagadis, “Automatic quantitative analysis of in-stent restenosis using FD-OCT in vivo intra-arterial imaging,” Med. Phys. 40(6Part1), 063101 (2013). [CrossRef]
14. N. Bruining, K. Sihan, J. Ligthart, S. de Winter, and E. Regar, “Automated three-dimensional detection of intracoronary stent struts in optical coherence tomography images,” in 2011 Computing in Cardiology (IEEE, 2011), pp. 221–224.
15. H. Lu, M. Gargesha, Z. Wang, D. Chamie, G. F. Attizzani, T. Kanaya, S. Ray, M. A. Costa, A. M. Rollins, and H. G. Bezerra, “Automatic stent detection in intravascular oct images using bagged decision trees,” Biomed. Opt. Express 3(11), 2809–2824 (2012). [CrossRef]
16. H. Lu, M. Gargesha, Z. Wang, D. Chamie, G. F. Attizani, T. Kanaya, S. Ray, M. A. Costa, A. M. Rollins, and H. G. Bezerr, “Automatic stent strut detection in intravascular oct images using image processing and classification technique,” in Medical Imaging 2013: Computer-Aided Diagnosis, vol. 8670 (International Society for Optics and Photonics, 2013), p. 867015
17. Z. Wang, M. W. Jenkins, G. C. Linderman, H. G. Bezerra, Y. Fujino, M. A. Costa, D. L. Wilson, and A. M. Rollins, “3-d stent detection in intravascular oct using a bayesian network and graph search,” IEEE Trans. Med. Imaging 34(7), 1549–1561 (2015). [CrossRef]
18. H. S. Nam, C. S. Kim, J. J. Lee, J. W. Song, J. W. Kim, and H. Yoo, “Automated detection of vessel lumen and stent struts in intravascular optical coherence tomography to evaluate stent apposition and neointimal coverage,” Med. Phys. 43(4), 1662–1675 (2016). [CrossRef]
19. Y. Guo, L. Bi, A. Kumar, Y. Gao, R. Zhang, D. Feng, Q. Wang, and J. Kim, “Deep local-global refinement network for stent analysis in ivoct images,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, 2019), pp. 539–546.
20. P. Wu, J. L. Gutieérrez-Chico, H. Tauzin, W. Yang, Y. Li, W. Yu, M. Chu, B. Guillon, J. Bai, and N. Meneveau, “Automatic stent reconstruction in optical coherence tomography based on a deep convolutional model,” Biomed. Opt. Express 11(6), 3374–3394 (2020). [CrossRef]
21. H. Lu, J. Lee, S. Ray, K. Tanaka, H. G. Bezerra, A. M. Rollins, and D. L. Wilson, “Automated stent coverage analysis in intravascular oct (ivoct) image volumes using a support vector machine and mesh growing,” Biomed. Opt. Express 10(6), 2809–2828 (2019). [CrossRef]
22. H. Lu, J. Lee, M. Jakl, Z. Wang, P. Cervinka, H. G. Bezerra, and D. L. Wilson, “Application and evaluation of highly automated software for comprehensive stent analysis in intravascular optical coherence tomography,” Sci. Rep. 10(1), 18491–13 (2020). [CrossRef]
23. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems (2012), pp. 1097–1105.
24. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2016), pp. 770–778.
25. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2015), pp. 1–9.
26. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: unified, real-time object detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2016), pp. 779–788.
27. L. C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in Proceedings of the European Conference on Computer Vision (ECCV) (2018), pp. 801–818.
28. G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” Med. Image Anal. 42, 60–88 (2017). [CrossRef]
29. E. Shlofmitz, M. Iantorno, and R. Waksman, “Restenosis of drug-eluting stents,” Circ. Cardiovasc. Interv. 12(8), e007023 (2019). [CrossRef]
30. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, 2015), pp. 234–241.
31. Z. Gu, J. Cheng, H. Fu, K. Zhou, H. Hao, Y. Zhao, T. Zhang, S. Gao, and J. Liu, “Ce-net: context encoder network for 2D medical image segmentation,” IEEE Trans. Med. Imaging 38(10), 2281–2292 (2019). [CrossRef]
32. P. Wang, P. Chen, Y. Yuan, D. Liu, Z. Huang, X. Hou, and G. Cottrell, “Understanding convolution for semantic segmentation,” in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) (IEEE, 2018), pp. 1451–1460.
33. L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFS,” IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2018). [CrossRef]
34. J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and Y. Zhou, “Transunet: transformers make strong encoders for medical image segmentation,” arXiv preprint arXiv:2102.04306 (2021).
35. F. Alfonso, R. A. Byrne, F. Rivero, and A. Kastrati, “Current treatment of in-stent restenosis,” J. Am. Coll. Cardiol. 63(24), 2659–2673 (2014). [CrossRef]
36. M. Megaly, M. Glogoza, I. Xenogiannis, E. Vemmou, I. Nikolakopoulos, M. Omer, M. Saad, L. Willson, D. J. Monyak, and P. Sullivan, “Coronary intravascular brachytherapy for recurrent coronary drug-eluting in-stent restenosis: a systematic review and meta-analysis,” Cardiovasc. Revasc. Med. 23, 28–35 (2021). [CrossRef]
37. F. Alfonso, J. García, M.-J. Pérez-Vizcayno, L. Hernando, R. Hernandez, J. Escaned, P. Jiménez-Quevedo, C. Bañuelos, and C. Macaya, “New stent implantation for recurrences after stenting for in-stent restenosis: implications of a third metal layer in human coronary arteries,” J. Am. Coll. Cardiol. 54(11), 1036–1038 (2009). [CrossRef]
