1. Introduction

Colorectal cancer (CRC) was the second most deadly cancer and third most common malignancy, which was estimated to account for approximately 9.4% of cancer-associated mortality worldwide and posed a serious threat to human health [1]. It often stemmed from small benign polyps progressing over time to gradually transform into malignant. Early diagnosis and excision of such diseased polyps could substantially decrease the occurrence and mortality of CRC and had become a worldwide public health priority [2]. In clinical practice, colonoscopy was frequently adopted means of examination and considered a gold standard for detecting colorectal lesions known as polyps or adenomas [3]. During the colonoscopy, physicians entailed visual inspection of the bowel by means of an endoscope to access location information and boundary details of polyps, whose accuracy heavily relied on the ability and tedious effort of physicians. Even skilled clinicians may also fail to reach an agreement on the segmentation for the same polyp image. Hence, a viable solution was to propose automated and accurate polyp segmentation strategies, which could provide physicians with great assistance for precisely locating and segmenting polyp areas to make further diagnosis.

Unfortunately, it still faced many challenges as a result of the following factors. First, high variations in polyp’s appearance, e.g., color, size, shape, and complex morphological features of polyps, complicated the segmentation task, even if they were of the same type. Second, the fuzzy boundary contrasts between polyps and their surrounding mucosas made the polyps camouflaged against other endoluminal structures, while spot interference occurred when the light source of the lens was reflected on the tissue fluid of intestinal mucosa. Third, the inconsistency in scanning equipments and parameter standards leaded to a domain shift in polyp imaging across different medical institutions. These issues easily contributed to polyp segmentation negatively. Given these challenging factors, there was a growing demand for an automated and precise segmentation method that was ability of identifying nearly all potential polyps at their early stages from colonoscopy images, which could guide physicians to perform quick localization of polyp area and precisely delineate its boundary. To this end, this work intended to establish a polyp segmentation network which could represent polyp features better in a medical scenario for improving polyp segmentation performance and generalization capability with respect to domain shift.

In this context, with the backing of computer vision technologies, numerous automated polyp segmentation methods had been presented and obtained remarkable progress in the past several years. Among diverse polyp segmentation approaches, early learning-based approaches [4–7] resorted to hand-crafted low-level features, including color, shape, texture, appearance, and some combination of these characteristics, etc. Yet, they usually yielded low-quality segmentation results and suffered from poor generalizability to complex scenarios, principally owing to the restricted representation ability of hand-crafted features when handling the high intra-class variations of polyps as well as low inter-class variations between polyps and hard mimics. Benefit from the rapid development of deep learning techniques in the field of medical image analysis, the automated polyp segmentation had progressively evolved from the early learning-based approaches to the deep learning methods, which could roughly fall into deep convolutional neural network (DCNN) based method [8–31], Transformer based technique [32–37], and hybrid architectures of Transformer and DCNN [38–44]. In DCNN-based methods, a fully convolutional network (FCN) with a pre-trained VGG model was utilized for recognizing and segmenting polyp regions through pixel-level predictions, while a modified FCN combining the patch selection strategy was designed to increase the precision in polyp segmentation. Nevertheless, FCN approaches only depended on low-resolution characteristics to perform the final predictions, leading to coarse segmentation results and blurred edges. Currently, the advanced methods were largely based on U-shaped encoder-decoder CNN network, which had become prevalent in the segmentation of polyp images. For instance, the U-Net++ [12] extended the U-Net structure by inserting efficient and densely connected nested-decoder subnetworks while introduced a deep supervision mechanism to enhance the use of aggregative features across multiple semantic scales. Later, ResUNet++ [13] integrated residual computation, squeeze and excitation (SE), and attention mechanism to further boost segmentation ability. In recent studies, SFA [19] adopted a selective feature amalgamation structure along with a boundary-sensitive loss function to recovery the sharp boundary between a polyp and its surrounding mucosa. PraNet [21] merged high-level characteristic information for generating global feature maps and under their guidance, a parallel reverse attention block was utilized to progressively delve the boundary cues and build the correlations between regions and boundary clues. In spite of improving the segmentation accuracy, most of these methods directly used element-wise addition or concatenation operators to incorporate features at different levels gradually, resulting in diluting really useful features by excessive redundant information and weakening the complementary features among different levels. Moreover, the receptive field of the aforementioned methods [8–31] were restricted and difficult to effectively summarize the global-level contextual information for handling size-varying polyps. Although some methods focused on capturing information from multiple scales and multiple receptive fields on the basis of the atrous spatial pyramid pooling (ASPP) block or DenseASPP, they would produce many extra parameters and computations, not to mention over-fitting. Different from DCNN-based methods, Transformer-based methods such as Polyp-PVT [33], SSFormer [35], and ESFPNet [37] etc., were able to perceive semantic distinctions from a global aspect via adaptively modeling long-distance relationships, and possessed stronger feature representation abilities by way of capturing global contextual information. Notwithstanding demonstrating impressive performance over DCNN-based alternatives, grabbing long-distance dependencies could destroy portion of task-specific critical local features and fail to model interactive features from the neighborhood, which could cause overly smooth segmentation for small/tiny polyps and vague boundaries between polyps. Apart from this, attention dispersion also occurred with the deepening of the Transformer model, whilst they often suffered from a dramatic performance decline on unseen out-of-distribution polyp data when domain shift issues existed, whose generalization capability remained lacked. The hybrid architectures of Transformer and DCNN including TransUNet [38], TransFuse [39], HS-Net [41], FCBFormer [42], TranSEFusionNet [43] and ECTransNet [44] etc., aimed to fuse the advantages of both models in a single architecture so as to strengthen underlying local features and restrict attention dispersion. However, feeding local feature information directly into the Transformer could not precisely deal with local contextual relationships, which could cause the local feature details to be overwhelmed by the dominant global context and generate inferior results in small/tiny polyp segmentation task. Another significant limitation of these hybrid architectures [39] fusing Transformer with DCNN was that the predicted segmentation maps were frequently lower in resolution compared with the input images, namely not full-size segmentation map. While the ECTransNet [44] had shown certain improvements in segmentation accuracy, there was still room for enhancement in building global contextual semantic relationships among pixels in the polyp feature maps.

To address the above-mentioned issues, inspired by encoder-decoder structures [11–14], Transformers [32,45,46], residual learning [47] and the work by Zhang [41], we developed a novel improved dual-aggregation polyp segmentation network, termed as Dua-PSNet, to enhance feature representation ability in network fusion and global information modelling by combining both the Transformer branch and the FCN branch in parallel, thereby achieving automatic and accurate full-size polyp segmentation from colonoscopy images. Our solution adapted to both inductive bias and powerful representation of global context, catching local and long-range characteristics adequately on the polyp. Specifically, in the Transformer branch, we exploited the B3 variant of pyramid vision transformer v2 (PVTv2-B3) [46] as the hierarchical encoder for capturing multi-scale features and establish global correlations between them. Then, we constructed a novel multi-stage feature fusion decoder (MFAD) to put emphasis on important local feature cues and selectively extend them into global features. In the MFAD, we made an attempt to design an adaptive feature aggregation (AFA) block for aggregating adjacent high-level feature representations of different scales derived from PVTv2-B3 encoder in a stepwise adaptive fashion, generating critical global semantic features. After that, we developed the ResidualBlock module to strength local boundary details camouflaged in low-level features while devised the selective global-to-local fusion head (SGLFH) module to selectively merge them with the global semantic features for eliminating the semantic gap between low-level and high-level features and constraining attention dispersion, enhancing the efficiency of feature fusion across different levels. In the FCN branch, we exploited the designed ResidualBlock module instead of the original residual block (RB) module to refine highly merged multi-scale features at full-size and supplement output of Transformer branch into full-size prediction mask. These two branches influenced and complemented each other for refining the semantic information of the network to improve the recognition of boundary ambiguous features and boost the network’s performance in dealing with the scale variations of polyps. Our major contributions were summarized as follows:

2. Related works

In this part, we presented an overview of recent efforts in relation to our work briefly from the following two perspectives: polyp segmentation, as well as multi-scale and multi-level amalgamation.

2.1 Polyp segmentation

Nowadays, polyp segmentation had been progressed by leaps and bounds and numerous exceptional works had paid attention to this area. Generally, the current polyp segmentation approaches were mainly divided into four types, including traditional learning-based approaches [4–7], DCNN-based methods [8–31], Transformer-based techniques [32–37], hybrid architecture combing Transformer and DCNN [38–44].

Traditional learning-based methods. Early polyp segmentation solutions highly depended on hand-crafted low-level features, such as color, shape, texture, appearance, or some combination of these characteristics [4–7]. Following the acquisition of manually designed characteristics, a classifier was often trained for detection or segmentation of polyp region from its surrounding tissue. For instance, Ameling et al. [4] extracted texture features with Grey-Level-Co-occurence and Local-Binary-Patterns for conducting polyp segmentation. Karkanis et al. [5] leveraged the covariances of the second-order textural measures computed over the wavelet frame decomposition of different color bands to extract color image features for representing different polyp areas, while performing classification using a step-wise linear discriminant analysis (LDA). Mamonov et al. [6] utilized the information about both the texture and the geometry to acquire a binary classification algorithm with pre-selection. Further, Tajbakhsh et al. [7] applied context and shape information to get rid of non-polyp structures and achieve reliably polyp location. Yet, these traditional learning-based methods often suffered from unsatisfactory segmentation results and lacked generalizability to real-world scenarios as a result of the poor representation capacity of hand-crafted characteristics.

DCNN-based methods. As the continuous development of DCNN, it had gained considerable attention with excellent learning ability and greatly promoted the development in polyp segmentation field. The introduction of FCN [8] provided a true solution to the issue of polyp segmentation at the pixel level. For example, Akbari et al. [9] adopted an FCN fusing image patch selection strategy and Otsu thresholding for polyp segmentation, yielding better segmentation results against traditional learning-based solutions. Similarly, Brandao et al. [10] converted VGG and ResNets into FCNs, and fine-tuned them for achieving polyp segmentation, which incorporated depth information restored using shape-from-shading strategy to provide a richer feature representation. However, most of these FCN methods [8–10] only resorted on low resolution feature information to carry out the final prediction of polyp regions, suffering from fuzzy boundaries and rough segmentation results. As a milestone work, U-shaped encoder-decoder structures instead of a single encoder in FCN had gradually come to be the mainstream network architecture with superior performance, e.g., U-Net [11], U-Net++ [12] and ResUNet++ [13]. Yeung et al. [14] introduced a dual attention-based deep neural network called Focus U-Net to promote selective learning on polyp characteristics by way of combining efficiently spatial and channel attentions into the sole focus gate block. Nevertheless, these skip connections operated with feature maps yielded by only a specific level of the encoder, which was not conducive to information transmission between the encoder and decoder and degraded the segmentation performance. To address this problem, Zhou et al. [12] reconstructed the dense skip connections to integrate feature maps between the encoder and the decoder in U-Net and established U-Net++, obtaining promising segmentation performance. Further, Tomar et al. [15] presented a dual decoder attention network built upon ResUNet++ to segment polyp regions. Yet, they mostly exploited a fixed size of kernel at each layer and only considered the feature maps from final decoder layer to conduct the reconstruction, which constrained the model’s ability of exploring representational characteristics from different receptive fields. Based on the encoding-decoding network architecture, Sun et al. [16] introduced a dilated convolution for extracting and aggregating high-level semantic characteristics without resolution reduction, thereby improving the encoder network. Banik et al. [17] incorporated 2-D dual tree complex wavelet transform pooling with multiple skip connections to establish an enriched version of CNN named Polyp-Net for automatic polyp segmentation. Mahmud et al. [18] integrated depth dilated inception (DDI) module, deep fusion skip module (DFSM) and deep reconstruction module (DRM) into PolypSegNet for precise automated segmentation of polyp areas. Song et al. [24] proposed a parallel medical segmentation framework-AMNet to gain an improvement in polyp segmentation performance. Lin et al. [25] designed a bit-slicing context attention network (BSCA-Net) for improving performance in polyp segmentation by combining bit slice context attention (BSCA) module, split-squeeze-bottleneck-union (SSBU) block, multipath concatenation attention decoder (MCAD) as well as multi-path attention concatenation encoder (MACE). Shen et al. [26] presented a multi-scale coded colon polyp segmentation network, in which the attention mechanism integrating both spatial and channel dimensions was inserted into the encoding and decoding modules to enable the model to focus more on small polyp segmentation task. Lately, Li et al. [27] built the multiple feature association network (MFA-Net) with global attention mechanisms for improving the gains in segmentation performance of polyps. Further, more other U-shaped encoder-decoder based works related to polyp segmentation could be found in these literatures [11–24]. Notwithstanding achieving remarkable segmentation effect, the vast majority of these methods [11–22] lacked spatial details following pooling, and modeled long-range interdependencies insufficiently after performing multiple convolution operators. At the same time, most of these methods were prone to overfitting and suffering from weak generalization ability. Besides, there were also some methods [19–22] built upon U-Net, which recovered edge cues for polyp segmentation through establishing the association between the edge and area characteristics. Murugesan et al. [20] designed Psi-Net containing three parallel decoders to segment polyps, one of which was used for prediction, whereas the other two decoders were responsible for detecting contour and estimating distance map for attaining polyp shape and boundary information. But the correlation between them was not fully explored. Fang et al. [19] put forward a selective feature aggregation network (SFA) consisted of a shared encoder together with two reciprocally restrained decoders, to speculate areas and boundaries of polyps. In contrast, Fan et al. [21] developed a parallel reverse attention network (PraNet) for segmenting polyps, which utilized a parallel partial decoder (PPD) to produce a global map to serve as the guidance region along with a reverse attention (RA) block to probe the contour details and build the relationships between areas and edges. More recently, Song et al. [24] used a parallel attention block together with a reverse fusion component to establish associations between areas and edges for refining the edge information and improving polyp segmentation accuracy. Zhou et al. [28] constructed a cross-level feature aggregation network (CFA-Net) incorporating a boundary prediction module using a layer-wise strategy, to enhance hierarchical features to refine segmentation maps on polyps. Li et al. [31] utilized channel and spatial fusion block in conjunction with spatial and channel attention mechanism, feature complementary module, and shape block to construct a multi-scale channel spatial fusion network (MCSF-Net) for real-time polyp segmentation, enhancing lesion boundary feature extraction and the fusion of multi-scale features together with exhibiting excellent segmentation and real-time performance. However, the boundary information derived from these methods [19–22] was often ambiguous, resulting in sub-optimal performance in polyp segmentation task. In addition, these DCNN-based methods [8–31] were good at capturing local neighboring feature details by means of a local receptive field, yet could be powerless to establish long-distance interdependencies effectively, with restricted feature representation capacity to tackle size-varying polyps.

Our network combined both a Transformer branch built upon U-shaped encoder-decoder architecture with a FCN branch in parallel to capture the long-range contexts and local low-level feature details from different viewpoints, which provided a comprehensive insight into polyp characteristics and brought a segmentation performance improvement.

Transformer-based techniques. Different from DCNN, the Transformer consisting of self-attention gained similarities between all pairs of patches via calculating the dot product between their respective vectors so that features between all patches could be adaptively extracted and mixed, which enabled it to possess a strong capability of modeling long-term relations and reduce inductive bias. As the Transformer model emerged, Visual Transformer (ViT) [32] regarded each image as a sequence of patches (tokens) with a fixed size and subsequently forwarded them into multiple Transformer layers for seeking the association between each other. Due to only a single scale of output feature map with low resolution, it was challenging to directly adapt it to polyp segmentation task. Based this consideration, several methods [33–37] incorporated the pyramid structure in CNN into the design of Transformers, presenting a hierarchical Transformer with different stages. Dong et al. [33] utilized a pyramid vision Transformer (PVT) as backbone encoder and presented a polyp segmentation architecture called Polyp-PVT. Tang et al. [34] proposed a Dual-Aggregation Transformer Network (DuAT) to segment polyp regions, which adapted the PVT as the encoder for capturing richer feature cues. Wang et al. [35] proposed SSFormer comprising a PVT encoder and progressive locality decoder (PLD) to enhance robust and generalization ability for polyp segmentation. Nachmani et al. [36] proposed ResPVT framework to segment polyp, which used Transformer as an encoder to capture more feature representations about polyps. Chang et al. [37] established the ESFPNet architecture for segmenting polyp lesions, which exploited the Mix Transformer (MiT) as the backbone encoder and an efficient stage-wise feature pyramid (ESFP) in the decoder to enhance the usage of high-level semantic information. Despite manifesting notably impressive results, the Transformer network struggled to establish interactive characteristics from the neighborhoods adequately and represent fine-grained feature details accurately, as a result of the low spatial inductive bias and strong global receptive field. In addition, as the Transformer model was deepened, the global feature information was persistently mingled and accumulated, susceptible to attention diffusion. In our network design, we upgraded multi-stage feature aggregation decoder (MFAD) to adapt to the PVTv2-B3 backbone encoder, forming the Transformer branch, together with the FCN to underline local contextual relationship and constrain attention divergence.

Hybrid architectures combing Transformer and DCNN. Aiming at exploiting strengths of both designs, substantial works sought to improve model’s ability of global contextual relations while maintaining a strong extraction of local detailed features by integrating Transformer and DCNN. For example, TransUNet [38] was developed by leveraging ResNet50 network and ViT as the encoder of U-Net for feature extraction and global interactions. But, it could result in a complex and bloated architecture which was inclined to overfitting. Zhang et al. [39] constructed TransFuse network to combine the Transformer and CNN parallelly so that global dependencies along with local spatial detailed information could be efficiently collected. Cai et al. [40] combined a shallow CNN encoder with a deep Transformer encoder for extracting richer feature details, and erected the PP-guided self-attention in the decoder so as to guide self-attention with the help of prediction maps, enhancing the model’s sensitivity to polyp boundaries. Zhang et al. [41] investigated a hybrid semantic network (HSNet) that incorporated both the Transformer and CNN with a dual-branch framework to improve polyp segmentation, which introduced a hybrid semantic complementary module to capture long-term relationships and local appearance cues. Sanderson et al. [42] combined Transformer and FCN into a single structure to form FCN-Transformer hybrid network (FCBFormer) for achieving polyp segmentation. Zhang et al. [43] took full advantage of the combination of the CNN and Transformer on top of the U-Net structure to build a hybrid CNN-Transformer architecture called TranSEFusionNet, improving accuracy in polyp lesion segmentation. Li et al. [44] presented a novel polyp image segmentation network called ECTransNet by incorporating the local feature extraction ability of DCNN with the global contextual semantic correlation construction capability of Transformer, which achieved promising segmentation accuracy and generalization performance. Yet, there was still a need for further improvement in capturing long-range dependencies among pixels. In these methods [39,41], it could be imprecise to deal with local contextual relations by feeding local features directly into the Transformer, which made the local information be deluged by the dominant global context. Concurrently, they only combined the structures in a simple manner and overlooked the interaction between the two semantic representations, whereas down-sampling operations of these methods could cause the loss of local information, resulting in blurred boundary. What’s more, most of these approaches [39] were not able to perform full-size segmentation map prediction. Inspired by this [40,42], we introduced Transformer and FCN in a parallel way to generate dual-aggregation polyp segmentation network (Dua-PSNet) for full-size polyp prediction. In our Dua-PSNet, the Transformer branch was developed to learn critical global semantic information, whereas the FCN branch was used for emphasizing fine local boundary details. These two branches influenced and complemented each other, enhancing the distinguished capability for polyps and achieving full-size segmentation.

2.2 Multi-scale and multi-level amalgamation

Multi-scale feature representation offered a viable mean for tackling variations in scales with respect to segmentation task, whilst multi-level fusion provided multiple granularities for semantic segmentation. For instance, TransFuse [39] created a BiFusion module to incorporate multiple levels of features from both Transformer and CNN branches. SFA [19] embedded selective kernel module (SKM) into convolutional layers for dynamically extracting multi-scale and multi-receptive-field features from kernels of different sizes, and added up-concatenations between encoder and decoder in U-Net structure to aggregate these features. UACANet [23] conducted the atrous convolution with regard to high-level features, which could enlarge the perceptual field and degrade the loss of global spatial feature information to a certain degree, yet the boundary space information of subtle polyps could be lost owing to null convolution. PraNet [21] aggregated multiple high-level features extracted from Res2Net-based backbone network via a PPD component. MSNet [29] used the subtractive units for yielding different characteristics between contiguous levels of the network, which provided different perceptual fields concerning different subtractive units in a feature pyramid way to acquire abundant feature differences of multiple scales. M²SNet [30] equipped with both the inter-layer and intra-layer subtraction structures to collect multi-scale complementary information among different levels, and effectively aggregated specific level features and multi-path cross-level distinctive features for generating the final prediction, thereby amplifying the perceptions of polyp regions. Yet, it was prone to lose the boundary details of small/tiny polyps and affected segmentation accuracy. AMNet [24] built a multi-scale interactive fusion network for obtaining richer local and global feature details on polyps via merging high-level and low-level characteristics of different scales working together to supplement the position and spatial information, and introduced the parallel network of attention mechanisms for augmenting the model’s segmentation capability. MFA-Net [27] integrated a parallelly dilated convolutions arrangement (PDCA) block between the encoder and the decoder to excavate critical feature representations and introduced a multi-scale feature restructuring module (MFRM) to reorganize and merge semantic information at different scales from the encoder, while cascaded global attention stacking (GAS) block in the decoder to enhance global attention perception and guide shallower features through the use of deeper features. Polyp-Net [17] amalgamated 2-D dual-tree complex wavelet transform pooling with local gradient weighting-embedded level-set method to achieve multi-model pixel-level fusion, suppressing high-intensity false area and ensuring smoothness of polyp contour. PolypSegNet [18] utilized deep fusion skip module (DFSM) for fusing different scales of features produced by different levels of the encoder, and created deep reconstruction module (DRM) to restore and optimize decoded feature maps of multiple scales from different levels of decoders, enabling skip inter-connections establishment between encoder and decoder as well as the semantic gap between them reduction. Segformer [48] predicted features of different scales and depths individually by simple up-sampling, and then parallelly fuse them using a multi-stage feature aggregation algorithm. DuAT [34] used global-to-local spatial aggregation (GLSA) module to simultaneously mine local spatial details and global spatial semantic information, and constructed selective boundary aggregation (SBA) module for fine-tuning polyp boundaries. SSFormer [35] used a pyramid Transformer encoder for multi-scale feature extraction and progressive locality decoder for multi-stage feature fusion, enabling characteristics of different depths and representation capabilities to guide reciprocally. The outstanding performance illustrated that the decoder approach of multi-stage feature amalgamation was helpful to boosting the performance of the Transformer in the task of dense prediction. Lately, MCSF-Net [31] introduced the channel and spatial fusion module to amalgamate high-level multi-scale features through using channel and spatial attention mechanisms while developed a feature complementary module to fuse low-level multi-scale feature maps, which not only effectively captured the spatial positional information of polyps but also accurately preserved lesion boundaries. ResPVT [36] implemented PVT as backbone encoder to extract multi-stage feature information and adopted fusion module (FM) to meld the high-level features. ECTransNet [44] embedded an edge complementary module to fuse and complement polyp feature maps at multiple resolutions for effectively mining crucial edge information of polyps, and employed a residual-based feature aggregation decoder to adaptively merge high-level and low-level features, thereby better retaining spatial position information and reinforcing the model’s ability to segment local details. On a similar note as [31,36,44], our Transformer branch employed a stepwise adaptive method to generate global semantic information and focus on local edge details, and then selectively aggregated them, improving the feature information processing capability of different levels and achieving more effective feature fusion. Besides, more studies on multi-scale and multi-level amalgamation could refer to these literatures [8–16,20,22,25,26,28]. The vast majority of these methods combined different level features by utilization of simple element-wise addition or concatenation operations, which tended to create substantially excessive information to attenuate the really useful characteristics and weaken the complementary feature cues between different levels.

In our network design, we utilized the PVTv2-B3 as backbone encoder for capturing different levels of features from input images and explore the correlations between them. In the decoder, we constructed the AFA module to aggregate resultant high-level characteristics from the encoder to form global semantic features in a stepwise adaptive manner, while designed the ResidualBlock module to enhance extraction capability of local boundary details from low-level features. Further, we proposed the SGLFH module to selectively inject local boundary information into global semantic features, which could enable characteristics of different representation abilities to guide mutually and the fusion more efficient.

3. Methodology

In this part, we initially provided an overview of the presented Dua-PSNet for full-size polyp segmentation. Then, we presented two key branches, including the Transformer branch and FCN branch. In the Transformer branch, we introduced PVTv2-B3 encoder and multi-stage feature aggregation decoder (MFAD), respectively, while elaborately depicted the AFA, ResidualBlock and SGLFH components in the MFAD. At last, we developed the entire loss function.

3.1 Overall architecture

The whole architecture of the developed Dua-PSNet for polyp segmentation was illustrated in Fig. 1(a). The model involved two key parallel branches starting from a $H \times W$ input images, one of which was the Transformer branch and returned reduced-size $\frac{H}{4} \times \frac{W}{4}$ feature maps, the other one of which was an FCN branch and returned full-size $H \times W$ feature maps. Specifically, a colonoscopy image with a resolution of $h \times \omega $ was fed into Transformer and FCN branches simultaneously. In the Transformer branch, we leveraged PVTv2-B3 as an image encoder to derive 4 pyramid features ${F_i} \in {R^{\frac{H}{{{2^{i + 1}}}} \times \frac{W}{{{2^{i + 1}}}} \times {C_i}}}$ and model long-range dependencies between different features, where $i \in \{{1,2,3,4} \}$ and ${C_i} \in \{{\textrm{64},128,\textrm{320},512} \}$. For accurate polyp segmentation, we constructed the MFAD to conduct stepwise adaptive aggregation on high-level features from PVTv2-B3 encoder for extracting the most important global semantic features as well as focus more on local boundary cues from low-level features. In the MFAD, we used convolution units to reduce the number of channels from the last 3 feature maps ${F_2}$, ${F_3}$ and ${F_4}$ to 64, while fed them into the developed AFA module to implement high-level feature fusion in a progressive adaptive way generating global feature map ${O_1} \in {R^{\frac{H}{4} \times \frac{W}{4} \times 64}}$. Meanwhile, was forwarded into the designed ResidualBlock module producing ${O_2} \in {R^{\frac{H}{4} \times \frac{W}{4} \times 64}}$, which enhanced local edge perception ability from low-level features. Afterwards, we presented the SGLFH module to selectively aggregate feature maps ${O_1}$ and ${O_2}$, resulting in the feature map ${F_T} \in {R^{\frac{H}{4} \times \frac{W}{4} \times 64}}$, which could encourage semantic information of different expression powers to guide mutually and bridge the semantic gap between high-level and low-level features. The resulting feature map ${F_T} \in {R^{\frac{H}{4} \times \frac{W}{4} \times 64}}$ was up-sampled into full-size $F_T^{{\prime}} \in {R^{H \times W \times 64}}$ and concatenated with the feature map ${F_c} \in {R^{H \times W \times \textrm{32}}}$ generated by the FCN branch along the channel dimension. Ultimately, the concatenated feature map was tackled into the final full-size polyp segmentation map through the prediction head module.

 

Fig. 1. Overall architecture of the developed Dua-PSNet for polyp segmentation (a), including two critical parts: Transformer branch (b) and FCN branch (c). AFA: adaptive feature aggregation module; SGLFH: selective global-to-local fusion head module.

Download Full Size | PDF

3.2 Transformer branch

Our implementation of the Transformer branch was shown in Fig. 2, which used a pre-trained PVTv2-B3 on ImageNet [49] as an image encoder and designed the AFA, ResidualBlock and SGLFH modules to constitute a MFAD. The PVTv2-B3 encoder produced a 4-level feature pyramid, and then served as the input to the MFAD. In the MFAD, three high levels of feature pyramids were coped with by the AFA module in a stepwise adaptive manner to learn important global semantic context, whereas the remain one low-level pyramid was processed by the ResidualBlock module which improved the ability in representing local fine edge characteristics. In the following, the resulting local features were selectively injected into global semantic features through the SGLFH module, thereby filling the semantic gap between different level features and restricting attention dispersion.

 

Fig. 2. The architecture of the Transformer branch, including the PVTv2-B3 encoder and multi-stage feature aggregation decoder (MFAD). AFA: adaptive feature aggregation module; SGLFH: selective global-to-local fusion head module.

Download Full Size | PDF

3.2.1 Transformer encoder

To ensure our model enough generalization and powerful multi-scale feature extraction capability in polyp segmentation task, we utilized the PVTv2-B3 (See Fig. 2(a)) built upon a progressive shrinking pyramid structure as the image encoder. In contrast with traditional Transformer, the PVTv2-B3 used a linear spatial reduction attention (SRA) layer for reducing the computational burden, and an overlapping patch embedding via strided convolution to obtain more consistency of spatial information, while introduced zero-padding position encoding into the PVT in convolutional feed-forward network to more easily handle inputs with varying resolutions. In specific, the entire PVTv2-B3 encoder was split into 4 stages, each of which embodied both an overlapping patch embedding layer and a linear-layer Transformer encoder. The feature pyramid with four stages was generated from a rough level (4-stride) to a fine level (32-stride) in a progressive way. In the first stage, given an input image with the size of $H \times W \times 3$, the image was firstly separated into $\frac{{HW}}{{{4^2}}}$ patches, and each of size was $4 \times 4 \times 3$. Then, the flattened patches were provided into a linear projection to acquire embedded patches of size $\frac{{HW}}{{{4^2}}} \times 64$. Subsequently, these embedded patches together with corresponding position embedding were fed into a linear-layer Transformer encoder with $L1$ layers, and the output was reshaped to a feature map ${F_1} \in {R^{\frac{H}{4} \times \frac{W}{4} \times 64}}$. In the same way, taking the feature map resulting from the preceding stage as the input, we could derive the following three feature maps of different scales ${F_i} \in {R^{\frac{H}{{{2^{i + 1}}}} \times \frac{W}{{{2^{i + 1}}}} \times {C_i}}}$, where $i \in \{{2,3,4} \}$ and ${C_i} \in \{ 128,\textrm{320,512\} }$. Among these feature maps, ${F_2}$, ${F_3}$ and ${F_4}$ provided high-level semantic information related with polyps, whilst ${F_1}$ contained low-level local details.

3.2.2 Multi-stage feature aggregation decoder (MFAD)

Considering the most existing Transformer models for polyp segmentation lacked local low-level detailed feature processing capability, we proposed a novel MFAD for feature pyramids to stepwise aggregate multiple stages of features. The MFAD mainly consisted of the AFA, ResidualBlock and SGLFH modules, which could be depicted in Fig. 2(b).

3.2.2.1 Adaptive feature aggregation (AFA) module

Taking inspiration from the work [29,30,41], we created the AFA module to gather more varied and distinctive high-level semantic information by stepwise adaptively aggregating three high-level feature maps from Transformer encoder, as shown in Fig. 3. Specifically, we firstly reduced the number of channels to 32 for ${F_2}$, ${F_3}$ and ${F_4}$ by use of local emphasis (LE) module comprising $3 \times 3$ convolution operators with padding of 1, group normalization (GN) and SiLU activation functions in sequence, resulting in $F_2^{{\prime}}$, $F_3^{{\prime}}$ and $F_4^{{\prime}}$, respectively. At the same time, our model could also refocus attention on neighboring features for decreasing attention dispersion by utilization of the local receptive field within the LE module, thus amplifying the weights of the neighboring patches associated with the center patch and highlighting critical local characteristics of each patch. Then, we fused the feature maps of $F_3^{{\prime}}$ and $F_4^{{\prime}}$, as follows: we up-sampled $F_4^{{\prime}}$ by 2 and forwarded the resulting feature map through the LE module, generating $F_4^{{{\prime\prime}}}$. Meanwhile, $F_4^{{{\prime\prime}}}$ was multiplied by $F_3^{{\prime}}$, and then the resultant feature map was further concatenated with $F_4^{{{\prime\prime}}}$. After that, we processed the concatenated feature via the LE module and then performed double up-sampled operation, obtaining the aggregated feature map ${M_2}$. The above processing steps could be represented by the follows:

 

Fig. 3. Detailed structure of the developed adaptive feature aggregation (AFA) component. LE: local emphasis module.

Download Full Size | PDF

Similarly, we aggregated the feature maps of $F_2^{{\prime}}$, $F_3^{{\prime}}$ and $F_4^{{\prime}}$, as follows: we conducted a 4-fold and 2-fold up-sampling on $F_4^{{\prime}}$ and $F_3^{{\prime}}$, separately. And then, the resulting feature maps were streamed into the LE module, leading to feature maps $F_4^{{{\prime\prime\prime}}}$ and $F_3^{{{\prime\prime}}}$, individually. Further, we carried out an element-wise multiplication operation on $F_2^{{\prime}}$, $F_3^{{{\prime\prime}}}$ and $F_4^{{{\prime\prime\prime}}}$ to obtain the feature map ${M_1}$, thus the fused feature map could be represented as follows:

At last, we concatenated between ${M_1}$ and ${M_2}$, and the resultant feature map was passed through the LE module for obtaining the fused global feature map ${O_1}$, which could be depicted by:

Note that through the aforementioned procedures, the AFA module was able to aggregate adjacent high-level features from the Transformer encoder in a stepwise adaptive fashion, generating the most important global semantic information.

3.2.2.2 ResidualBlock module

In order to boost the extraction ability of local low-level feature details for Transformer feature pyramid, we constructed residual blocks named ResidualBlock module through the incorporation of the LE modules as well as a skip connection with convolutional layer with $1 \times 1$ kernel (denoted as ${C_{1 \times 1}}$) and GN, as displayed in Fig. 4(a). The LE module consisted of $3 \times 3$ convolution, GN and SiLU activation function in an order arrangement. Compared to original RB module (See Fig. 4(b)) in FCBFormer network, we made small changes in the order of $3 \times 3$ convolution along with adding $1 \times 1$ convolution and GN in residual connection. This sequence was designed to optimally leverage the effects of these three operations. We first performed $3 \times 3$ convolution operation on high-resolution low-level features from PVTv2-B3 encoder to extract the local boundary information related to polyp lesions. Then, GN was applied immediately to help in normalizing the distribution of the resulting features and mitigating internal covariate shift. Finally, we introduced a SiLU activation function to the normalized features, fostering local edge feature representation power. Through equipping with two units (LE modules) consisting of $3 \times 3$ convolution, GN and SiLU activation function in sequence, the boundary details of low-level features could be emphasized. In addition, we placed $1 \times 1$ convolution and GN in the residual connection for strengthening cross-channel interaction of boundary information in high-resolution low-level features and scaling their feature channel dimensions. By doing so, it could more efficiently cope with edge details of polyp lesions. Concretely, we first provided the low-level feature map ${F_1}$ into two sequential LE modules and an unit with $1 \times 1$ convolutional layer and GN, respectively, generating feature maps $F_1^{\prime}$ and $F_1^{^{\prime\prime}}$. Next, we directly implemented an element-wise addition operation on the resulting feature maps followed by a SiLU activation function, obtaining the feature map ${O_2}$, which could be expressed as follows:

As such, more fine boundary details from low-level features could be underlined and the detailed information processing capability of our model could be strengthened.

 

Fig. 4. The proposed ResidualBlock module (a) and the original RB module in FCBFormer (b).

Download Full Size | PDF

3.2.2.3 Selective global-to-local fusion head (SGLFH) module

With the goal of making effective integration between high-level and low-level characteristics, we implemented selectively fusion on feature maps ${O_1}$ and ${O_2}$, by designing the selective global-to-local fusion head (SGLFH) module. In the SGLFH module, we utilized two symmetric attention units (AUs) to adaptively pick up mutual representations from two inputs ($O_1^{\prime}$, ${O_2}$) before fusion, thereby exhibiting its selective characteristic, as shown in Fig. 5. The low-level and high-level feature information was fed into the two AUs in different ways for compensating the missing critical spatial edge information of the high-level semantic features and the loss of semantic information of low-level features, followed by concatenation operation to aggregate the outputs of two AUs. As given in Fig. 5, the AU function $PAU$ process could be expressed as follows:

 

Fig. 5. The flowchart of the proposed selective global-to-local fusion head (SGLFH) module. AU: attention unit.

Download Full Size | PDF

In addition, the feature map ${F_T}$ was further implemented 4-fold up-sampling operation to generate $F_T^{\prime}$, which maintained consistent size with the input image.

3.3 FCN branch

The FCN branch (See Fig. 1(c)) was characterized as a composition of the ResidualBlock module, strided convolutional layers for down-sampling, nearest neighbour interpolation for up-sampling, and dense U-Net type skip connections. This design encouraged extraction of highly merged features of different scales required for matching outputs of the Transformer branch into full-size segmentation maps. Through the encoder of FCN branch, the feature map ${F_j} \in {R^{\frac{H}{{{2^j}}} \times \frac{W}{{{2^j}}} \times {C_j}}}$ was attained in sequence from the down-sampling layer 1 to 5, where $j \in \{{1,2,\ldots ,5} \}$ and ${C_j} \in \{{32,32,64,128,512} \}$. On the contrary, by way of the decoder of FCN branch, the feature resolution was progressively restored by a factor of 2 from the up-sampling layer 1 to 5, whereas the number of feature channels was reduced consecutively by a factor of 2 ranging from the second to fifth up-sampling layer.

3.4 PredictionHead (PH) module

Compared with the PredictionHead (PH) module in FCBFormer network [42] (See Fig. 6(b)), we made small modification on it including substituting the designed ResidualBlock module for the RB module, as illustrated in Fig. 6(a). The PH module received a full-size feature tensor derived from stitching output tensors from the Transformer branch and FCN branch to perform polyp segmentation mask prediction at full-size. Concretely, the stitched full-size feature tensor was passed through two consecutive ResidualBlock modules followed by $1 \times 1$ convolutional layer and SiLU activation function to yield a full-size segmentation map. Each layer of PH module produced 64 channels, apart from the prediction layer ($1 \times 1$ convolution + SiLU) returning a single channel. Through aggregating complementary features generated by the Transformer branch along with the FCN branch, the PH module was able to achieve accurate prediction of polyp segmentation map.

 

Fig. 6. The PredictionHead (PH) module in the Dua-PSNet (a), and the FCBFormer (b).

Download Full Size | PDF

3.5 Loss function

It was reported by works [50] that using joint loss functions with adaptive weight coefficients could boost the model’s performance with faster convergence speed. Hence, we combined weighted binary cross-entropy loss ($L_{BCE}^\omega $) with the weighted Dice loss ($L_{Dice}^\omega $) for supervision of the network training. The entire loss function $L({S,G} )$ for the proposed Dua-PSNet could be formulated as follows:

4. Experiments and results

In this part, we provided the details on datasets and evaluation indicators, as well as implementation details. Then, we performed comparison between the developed Dua-PSNet with the cutting-edge polyp segmentation methods from quantitative and qualitative perspectives. After that, we measured our model’s generalization ability across different datasets. Finally, aiming at clarifying the effectiveness of the principal components in the proposed Dua-PSNet, we carried out a series of ablation experiments.

4.1 Datasets and evaluation indicators

4.1.1 Datasets

In order to assess the ability of the presented Dua-PSNet to segment polyps, we conducted a series of experiments on five benchmark datasets, each of which was described in detail as follows:

Kvasir-SEG [51]: This dataset was created by the Vestre Viken Health Trust of Norway, and contained 1,000 polyp images gathered from multiple colonoscopy video sequences, where the image resolution distribution varied between 332 × 487 and 1920 × 1072. It provided a more fine-grained pixel-level segmentation annotation validated by experienced gastroenterologists.

CVC-ClinicDB [52]: This dataset was provided by the Hospital Clinic of Barcelona and comprised 612 images with a resolution of 288 × 384, which were captured from 23 colonoscopy video sequences of 13 different patients.

CVC-ColonDB [6]: It consisted of 380 images with a fixed resolution of 500 × 570 extracted from 15 colonoscopy videos with a sample of 20 frames at random from each sequence, in which each image was accompanied by a binary mask for putting emphasis on the polyp regions.

ETIS-LaribPolypDB [2]: ETIS was an early polyp segmentation dataset, which involved 196 polyp images with a resolution of 966 × 1225.

CVC-300 [53]: It was a cross-domain dataset, and embraced 60 polyp images with a resolution of 500 × 574.

4.1.2 Evaluation protocols

Following recommendations from common methods for polyp segmentation [21,42], we utilized mean Dice ($mDice$), mean IoU ($mIoU$), mean absolute error (MAE) [54], weighted F-measure ($F_\beta ^w$) [55], S-measure (${S_\alpha }$) [56], and mean E-measure ($m{E_\phi }$) [57] as our assessment indicators for comprehensively investigating our model’s performance. Among these indicators, $mDice$ and $mIoU$ were regional level similarity measures and mainly highlighted the internal consistency of segmented objects. MAE was a pixel-by-pixel comparison metric, while $F_\beta ^w$ comprehensively considered the precision and recall. ${S_\alpha }$ concentrated on the structural similarity at the region and object level, and ${E_\phi }$ paid attention to the segmentation performance at the pixel and image level.

4.2 Implementation details

The developed Dua-PSNet was built under the PyTorch framework, which was trained utilizing a single NVIDIA RTX3060 GPU with 12 GB memory. Different data augmentation strategies were applied during training, including a gaussian blur, color jitter, horizontal and vertical flips, and affine transforms. Aiming to assess the learning and generalization power of the proposed Dua-PSNet, following the same settings as in [21,42] we separately partitioned Kvasir-SEG and CVC-ClinicDB into training, validation and test sets with a ratio of 8:1:1 in our experiment A. Similarly, we randomly chose 1,288 and 162 images from CVC-ClinicDB and Kvasir-SEG to form the training and validation sets, respectively, while an overall 798 images (100 images selected from Kvasir-SEG dataset, 62 images chose from CVC-ClinicDB benchmark, 380 images collected from CVC-ColonDB benchmark, 196 images gathered from ETIS-LaribPolypDB dataset, and the remain 60 images from CVC-300) from the used five benchmark datasets were adopted as the test set in our experiment B. The specific data partitioning was illustrated in Table 1. For the ablation studies, we also exploited the Kvasir-SEG and CVC-ClinicDB to evaluate the baseline performance of FCN along with PVTv2-B3, and then investigate the effect with successive additions of PVTv2-B3, AFA, ResidualBlock, and SGLFH modules. Besides, we adjusted the size of the input images into 352 × 352, whilst normalized the RGB values into a range between -1 and 1. The values of ${\lambda _1}$ and ${\lambda _2}$ for loss function were assigned into 1 and 0.2, empirically. The AdamW [58] optimizer with an initial learning rate of $1e - 4$ was adopted for optimization of the proposed model during the training. In order to accelerate the model’s convergence, the learning rate was stepped down by a factor of 2 when the $mDice$ on the validation set did not further boost over 10 epochs until hitting a minimum of $1e - 6$. Eventually, the model with the highest validation $mDice$ was considered as the final one. All models were trained with a mini-batch size of 4 for more than 200 epochs, unless otherwise specified. During the testing stage, our model predicted full-size binary segmentation maps for RGB images with $H \times W$ spatial dimensions.

Table 1. Datasets used in this study.

View Table | View all tables in this article

4.3 Comparison with state-of-the-art approaches

4.3.1 Comparison approaches

In an effort to investigate the effectiveness of the developed Dua-PSNet, we compared it with a variety of current state-of-the-art counterparts in the domain of polyp segmentation, including U-Net [11], U-Net++ [12], PraNet [21], MCSF-Net [31], SSFormer [35], FCBFormer [42], SFA [19], MSNet [29], SANet [22], M2SNet [30], Polyp-PVT [33], CFA-Net [28], ESFPNet [37], TranSEFusionNet [43] and ECTransNet [44]. Note that, for these methods, we performed retraining and testing using their publicly available codes and recommended parameter settings as well as default data augmentation methods reported in their original literatures. Among data augmentation strategies used in these models and our approach, there only existed slight differences. These slightly different augmentation methods did not significantly affect the comparison of models, which could ensure the relative fairness of comparisons.

4.3.2 Quantitative comparison

Table 2 provided the quantitative comparison results between our model and five advanced methods on both CVC-ClinicDB and Kvasir-SEG benchmark datasets in experiment A. On the CVC-ClinicDB benchmark, we could observe that our model outperformed all compared approaches by a large margin in terms of $mDice$ and $mIoU$ metrics. Specifically, in comparison with DCNN-based approaches (such as U-Net [11], U-Net++ [12], and PraNet [21]), our model had more obvious advantages. For instance, our model brought considerable performance improvements of 3.7%, 10.6%, 1.6% and 4.3%, 15.2%, 2.2% in $mDice$ and $mIoU$, against U-Net, U-Net++ and PraNet, respectively. When compared to other Transformer-based methods (including SSFormer [35] and FCBFormer [42]), our model also manifested excellent learning ability, observing an increase of 2.0% and 3.1% in $mDice$ and $mIoU$ over SSFormer, and an improvement of 1.5% and 1.4% against FCBFormer. In the Kvasir-SEG dataset, our model also consistently gained the best segmentation results in terms of $mDice$. For example, our model achieved 0.6% and 1.3% gains over SSFormer with respect to $mDice$ and $mIoU$, respectively, while performed competitively ($mDice$: 0.9253 vs 0.9231, $mIoU$: 0.8746 vs 0.8752) against FCBFormer. In addition, FCBFormer and SSFormer obtained relatively better segmentation results compared to the other DCNN-based approaches on both datasets. According to these results, the effectiveness of our model could be demonstrated.

Table 2. Quantitative comparison results with different advanced approaches for polyp segmentation on the CVC-ClinicDB and Kvasir-SEG benchmarks in experiment A. The bold fonts denoted the best results.

View Table | View all tables in this article

Table 3 summarized the quantitative comparison results of our model against thirteen cutting-edge methods on the CVC-ClinicDB and Kvasir-SEG benchmarks in experiment B. From the comparison results in Table 3, it could be seen that our model likewise achieved considerable performance gains over other segmentation methods. In particular, comparing to SSFormer, our model attained 3.3%, 4%, 2.3%, 2.2%, and 1.8% score improvements on average in terms of $mDice$, $mIoU$, $F_\beta ^w$, ${S_\alpha }$, and $m{E_\phi }$ on the CVC-ClinicDB dataset. Among all evaluation indicators, it performed the best on $mDice$, $mIoU$, $F_\beta ^w$, and ${S_\alpha }$ metrics, reaching up to 0.939, 0.895, 0.936 and 0.950, respectively. On the Kvasir-SEG dataset, our model gained overall performance increases of 1.8%, 2.9%, 2.9%, 0.9% and 1.8% against FCBFormer, respectively, concerning $mDice$, $mIoU$, $F_\beta ^w$, ${S_\alpha }$, and $m{E_\phi }$. In contrast, the MAE score of our model reduced by 0.5%. Here, our model manifested best results ($mDice$:0.922 and $mIoU$:0.874) in $mDice$ and $mIoU$, exceeding the Polyp-PVT by 0.9% and 1.1%, separately. Even when compared to recently developed state-of-the-art DCNN-based M2SNet, Transformer-based ESFPNet and hybrid CNN-Transformer-based TranSEFusionNet, our model still obtained consistently best segmentation results in all evaluation metrics on both datasets. Notably, our model was slightly inferior to recently proposed CFA-Net ($m{E_\phi }$:0.983 vs 0.989 and MAE:0.008 vs 0.007 on the CVC-ClinicDB dataset; $m{E_\phi }$:0.960 vs 0.962 and MAE:0.024 vs 0.023 on the Kvasir-SEG dataset) in $m{E_\phi }$ and MAE indices, but it still maintained advantages over the CFA-Net in $mDice$, $mIoU$, $F_\beta ^w$, and ${S_\alpha }$. Further, on the CVC-ClinicDB dataset, our model brought significant gains of 1.6% and 1.7% in $mDice$ and $mIoU$ as comparison with the ECTransNet, respectively, whereas maintained excellent segmentation performance consistent with the MCSF-Net ($mDice$: 0.939 vs 0.941, and $mIoU$: 0.895 vs 0.895). On the Kvasir-SEG dataset, our model consistently surpassed the ECTransNet and the MCSF-Net, with 2.1% and 1.1% increases in $mDice$ and 2.7% and 1.3% boosts in $mIoU$, respectively. On the basis of these results, we could conclude that our model had relatively better overall segmentation performance against the highly competitive ECTransNet and MCSF-Net. Again, these results indicated that our model had strong feature learning power, proving its superiority.

Table 3. Quantitative comparison results with different advanced approaches for polyp segmentation on the CVC-ClinicDB and Kvasir-SEG benchmarks in experiment B. The bold fonts denoted the best results.

View Table | View all tables in this article

4.3.3 Qualitative comparison

Figure 7 depicted the qualitative comparison results of our model over five cutting-edge polyp segmentation approaches in experiment A. On the basis of the visualizable comparison results, it could be observed that the results of our model were in most agreement with the ground truths, while surpassed the other compared approaches under different challenging conditions. Concretely, for small size of polyps in the first and second rows, our model could perform accurate segmentation on them. Nevertheless, U-Net++ completely failed to segment them, and U-Net produced some over-segmented regions. In this case, FCBFormer, SSFormer and PraNet generated several un-related or over-segmented areas, resulting from the neglect of the edge feature details from the deep characteristics. With respect to polyps with variable shapes and large sizes in the third and fourth rows, U-Net and U-Net++ manifested worse than the other methods, while FCBFormer, SSFormer and PraNet produced some errors with un-related or under-segmented regions. In the case of blurred boundaries between the polyps and background in the fifth and sixth rows making it more challenging to identify them, our model was able to focus more on the boundary details and segment them more accurately by contrast to the other methods. On the contrary, U-Net and U-Net++ conducted the worst among these methods. And yet, the predicted edges by PraNet were overly smooth, whilst FCBFormer and SSFormer missed some fine details. In a nutshell, the qualitative comparison results in experiment A demonstrated that our model had a good capability to deal with different challenging cases with respect to polyp segmentation.

 

Fig. 7. Qualitative comparisons of polyp segmentation results obtained from different cutting-edge approaches in experiment A, encompassing FCBFormer [42], SSFormer [35], PraNet [21], U-Net++ [12] and U-Net [11].

Download Full Size | PDF

Moreover, we also displayed prediction results of polyp segmentation from our model and recent cutting-edge methods in experiment B, as shown in Fig. 8. As can be seen in Fig. 8, our model provided consistently best performance under various challenging cases, which kept the polyp boundary segmentation intact and clear while decreasing the miss segmentation inside the polyp. Among these advanced methods, SFA performed almost the worst and completely failed to segment polyps under different challenges. When polyp regions were small (such as in the second row), TranSEFusionNet, ESFPNet, Polyp-PVT, M²SNet, FCBFormer, SSFormer and PraNet predicted some un-related segmentation areas. In segmentation tasks of polyps with irregularity shapes and large sizes in the third and fourth rows, SSFormer and SANet completely failed in segmenting them, whereas CFA-Net, Polyp-PVT, M²SNet, and FCBFormer yielded some incorrect segmentation regions. As for TranSEFusionNet and ESFPNet, there still existed some incomplete and false positive situations. When the edges of the polyp areas were similar to the surrounding tissues in the fifth and sixth rows, ESFPNet encountered some mis-segmentations and PraNet produced over-segmented areas, while CFA-Net, M²SNet and MSNet experienced fuzzy boundaries resulting from the loss of fine edge feature details. In this context, TranSEFusionNet manifested incomplete segmentation and Polyp-PVT only discriminated a portion of polyp area and suffered from under-segmentation, whilst FCBFormer as well as SSFormer neglected some fine detailed feature information. Conversely, the segmented mask predicted by our model held a best agreement with the ground truth under all these challenging conditions. Simultaneously, we could also notice from Fig. 8 that when coping with extremely small polyp lesions (such as in the second row), the ECTransNet and MCSF-Net produced some un-related segmentation regions. In addition, when facing polyps with irregularity shapes in the third row or accompanied by excessive mucus along with the indistinct demarcation with the surrounding tissues (See the sixth row), the ECTransNet tended to overlook a small portion of polyp lesion area while the MCSF-Net failed to correctly outline polyp samples, leading to incomplete segmentation. In contrast, our model delineated more accurate segmentation regions of polyp lesions with fine boundaries in tackling different challenging scenes, which closely aligned with the ground truth maps and showcased its superior identification ability. In summary, the qualitative results of polyp segmentation in experiment B signified again that our model was owned to the outstanding strength in accurately predicting the polyp segmentation masks under varied size, shape and contrast scenarios.

 

Fig. 8. Qualitative comparisons of polyp segmentation results generated by our model and thirteen different state-of-the-art methods in experiment B, containing ECTransNet [44], MCSF-Net [31], TranSEFusionNet [43], ESFPNet [37], CFA-Net [28], Polyp-PVT [33], M2SNet [30], FCBFormer [42], SSFormer [35], SANet [22], MSNet [29], PraNet [21] and SFA [19].

Download Full Size | PDF

4.4 Generalization ability

We measured the generalization ability of the proposed Dua-PSNet using two cross-dataset testing schemes. The one was to use the CVC-ClinicDB/Kvasir-SEG training set to train our model and test on the corresponding unseen Kvasir-SEG/CVC-ClinicDB benchmarks, which was referred to testing scheme A (See Table 4). The other one was to utilize the training set built from part of the CVC-ClinicDB and Kvasir-SEG benchmarks for training, while evaluate the trained model on the unseen CVC-ConlonDB, ETIS, and CVC-300 benchmark datasets, respectively, which was used as testing scheme B (See Table 5). As can be seen in Table 4, U-Net and U-Net ++ owed poor generalization ability on both datasets, especially U-Net++ decreased sharply in $mDice$ and $mIoU$ metrics. Relative to the SSFormer, our model attained considerable performance improvements, with $mDice$ by 7.9% and $mIoU$ by 9.5% on the Kvasir-SEG dataset, and $mDice$ by 12.4% and $mIoU$ by 13.5% on the CVC-ClinicDB benchmark. In addition, our model exhibited the best generalization power on the CVC-ClinicDB dataset, reaching up to 0.9203 $mDice$ and 0.8582 $mIoU$. These results illustrated that our model exhibited particular advantages in handling images from a somewhat divergent distribution over that considered during training.

Table 4. Generalizability comparisons of the proposed Dua-PSNet with current mainstream approaches in the testing scheme A. The bold fonts denoted the best results.

View Table | View all tables in this article

Table 5. Generalizability comparisons of the proposed Dua-PSNet with current mainstream approaches in the testing scheme B. The bold fonts denoted the best results.

View Table | View all tables in this article

In Table 5, it should be noticed that our model consistently outperformed other mainstream models in terms of nearly all indicators on the CVC-ConlonDB, ETIS, and CVC-300 benchmark datasets. For instance, compared with the FCBFormer, it provided 2.3% improvement in $mDice$, 1.8% improvement in $mIoU$, 3.0% improvement in $F_\beta ^w$, 1.9% improvement in ${S_\alpha }$, and 1.4% improvement in $m{E_\phi }$, respectively, on the CVC-ConlonDB dataset. Yet, the MAE of our model reduced by 0.4%. On the CVC-300 dataset, our model exceeded the Polyp-PVT by a large margin, improving $mDice$ from 0.899 to 0.910 (1.1% improvement), and $mIoU$ from 0.831 to 0.845 (1.4% improvement), respectively. Even in the most challenging ETIS dataset, it also improved the state-of-the-art performance, with a $mDice$ score of 0.794, a $mIoU$ score of 0.719, a $F_\beta ^w$ score of 0.750 and a ${S_\alpha }$ score of 0.882. Furthermore, our method was consistently superior to recently proposed best-performing M2SNet, CFA-Net, ESFPNet, and TranSEFusionNet for nearly all evaluation indicators on these unseen datasets. However, another one notable finding was that SFA and TranSEFusionNet exhibited a dramatic decline in segmentation performance on these unseen datasets, partially evidencing that their generalization abilities were weak. When compared with the ECTransNet and the MCSF-Net, our model manifested better generalization ability. For instance, in terms of $mDice$ and $mIoU$, our model obtained substantial improvements of 5.5% and 5.4% over the ECTransNet on the CVC-ColonDB dataset, while 6.6% and 6.4% on the ETIS-LabribPolypDB dataset, respectively. Even on the CVC-300 dataset, our model still outperformed the ECTransNet with 0.3% higher $mDice$ and 0.5% higher $mIoU$. In contrast, our model significantly exceeded the MCSF-Net by 5.6% $mDice$ and 4.9% $mIoU$ on the CVC-ColonDB dataset, 2.9% $mDice$ and 2.9% $mIoU$ for the ETIS-LabribPolypDB dataset, and 0.9% $mDice$ and 1.1% $mIoU$ for the CVC-300 dataset. These results validated that our model owned excellent segmentation capability even when confronting with different source datasets. Again, these results revealed that by contrast with the current state-of-the-art approaches, our models still manifested remarkable generalization capability in tackling images with a different distribution from that utilized for training, which could be an efficient option for real clinical practice with considerable variations in data.

4.5 Ablation studies

In this part, we verified the effectiveness of each critical component in our Dua-PSNet through a series of ablation experiments on the CVC-ClinicDB and Kvasir-SEG datasets, and the quantitative and qualitative results were tabulated in Table 6 and Fig. 9, respectively. In the ablation experiments, we selected the FCN and PVTv2-B3 as the baseline (No.1 and No.2) and used the ResidualBlock module instead of RB module in FCN (No.3) to evaluate the role of ResidualBlock module, and then investigate the effect with successive additions of PVTv2-B3 (No.4), AFA module (No.5), ResidualBlock module (No.6) and SGLFH module (No.7).

 

Fig. 9. Visualizable comparison results for verifying the strength of different key components.

Download Full Size | PDF

Table 6. Quantitative segmentation results from different primary modules in the ablation study utilizing the CVC-ClinicDB and Kvasir-SEG benchmarks.

View Table | View all tables in this article

Effectiveness of ResidualBlock Module. We investigated the validity of the ResidualBlock module. As shown in Table 6, compared No.1 with No.3, it could be clearly observed that No.3 using the ResidualBlock module could significantly improve the segmentation performance by 1.1% $mDice$ and 1.6% $mIoU$ on the CVC-ClinicDB dataset, and 1.3% $mDice$ and 2.1% $mIoU$ on the Kvasir-SEG dataset. In addition, from No.6 vs No.5, similar trends could be inspected. From the visualizable comparison results in Fig. 9, it could be witnessed that the perception of more local information was enhanced. In short, these results suggested that the ResidualBlock module was benefit for boosting model’s performance.

Effectiveness of combination of FCN and PVTv2-B3. In order to evaluate the combination of FCN and PVTv2-B3 branches, we tested the performance of No.4 (FCN + PVTv2-B3). As can be seen from Table 6 and Fig. 9, No.4 brought dramatic performance improvements against No.3 and No.2 on both datasets. Specifically, by comparing No.4 with No.3 in Table 5, the $mDice$ and $mIoU$ obtained an increment of 2.2% and 4.2% on the CVC-ClinicDB dataset, while 2.3% and 2.8% on the Kvasir-SEG dataset. In contrast to No.2, No.4 attained a performance gain of 1.2% and 2.1% with respect to $mDice$ and $mIoU$, on the CVC-ClinicDB dataset, whilst 1% and 1.4% on the Kvasir-SEG dataset. These improvements confirmed that the work together of both branches rather than only using a single branch structure could enable our model to more accurately recognize true polyp regions, revealing that complementary hierarchical semantic information could be employed for boosting the model’s segmentation performance, which was also demonstrated by the visualizable comparison results in Fig. 9.

Effectiveness of Adaptive Feature Aggregation (AFA) Module. We further investigated the importance of AFA module. According to the No.4 vs No.5 in Table 6, we could notice that No.5 showed a significant increase in the performance, improving $mDice$ score from 0.9220 to 0.9346 (1.3% ascension) and $mIoU$ score from 0.8701 to 0.8861 (1.6% ascension), on the CVC-ClinicDB benchmark, and $mDice$ score from 0.8926 to 0.9065 (1.4% ascension) and $mIoU$ score from 0.8267 to 0.8478 (2.1% ascension), on the Kvasir-SEG benchmark. This indicated the significance of the AFA module on segmentation performance enhancement. Likewise, the visualizable comparison results in Fig. 9 also implied that the devised AFA block could contribute to the improvement in the performance of polyp segmentation.

Effectiveness of Selective Global-to-Local Fusion Head (SGLFH) Module. Finally, we assessed the contribution of SGLFH component. In Table 6, it was apparent that No.7 integrated the SGLFH module achieved better segmentation performance than No.6 on all the evaluation criterias, especially with $mIoU$ metric reaching 0.9083 and 0.8746 on the CVC-ClinicDB and Kvasir-SEG benchmarks, respectively, which were 1.2% and 1.9% higher than those gained with No.6. This demonstrated that the introduction of the SGLFH module helped improve segmentation accuracy a lot, highlighting the effectiveness of the presented SGLFH module, which was achieved by allowing the model to dynamically aggregate local feature details and the global semantic features for narrowing information gap between them. As displayed in Fig. 9, some edge detailed information could not be precisely identified without utilizing the developed SGLFH module.

5. Discussion

In this work, we developed a novel architecture Dua-PSNet for accurate and automated polyp segmentation at full-size from colonoscopy images in many challenging cases. One of the strengths was that it took full use of the advantages of the Transformer with global-receptive-field on modeling long-range relationship and the FCN with local-receptive-field on establishing local spatial correlation in dense prediction. Combining them using two parallel branches allowed our model to pay attention to global semantic information without missing local spatial details. In this way, they could be reciprocally constrained and complemented for enabling more accurate full-size predictions. In the Transformer branch, we constructed the ResidualBlock module to deeper excavate polyp edge details disguised in low-level features and improve the recognition of edge ambiguous features. Further, we proposed two feature fusion modules, AFA module and SGLFH module, which could aggregate high-level features in a stepwise adaptive fashion and selectively incorporate edge cues into global semantic information, bridging semantic gap between high-level and low-level features. The Transformer was a sequence-to-sequence prediction model which attended to global contextual information. However, their predicted segmentation maps were typical types of a lower resolution than the input images, lacking detailed localization information. Our FCN branch allowed for the extraction of highly merged multi-scale fine boundary characteristics at full-size, compensating for output of the Transformer branch into full-size segmentation map. From quantitative and qualitative comparison results with state-of-the-art methods (See Table 2 and Table 3 as well as Fig. 7 and Fig. 8) together with generalizability analysis (See Table 4 and Table 5), it was demonstrated that our model consistently exceeded these advanced methods [11,12,19,21,22,28–31,33,35,37,42–44] and located polyp areas more accurately with clear boundary contours, even though they were varied, which put emphasis on its powerful strengths in learning and generalization capability. Hence, we could conclude that the proposed Dua-PSNet had great potential to act as an “extra pair of eyes” for endoscopists providing additional objective diagnostic information during colonoscopy, and aid in making a feasible decision for further treatment.

The developed Dua-PSNet performed better than a variety of current advanced methods, comprising DCNN-based methods [11,12,19,21,22,28–31], Transformed-based approaches [33,35,37] and hybrid architecture combining Transformer and CNN [42–44], in the task of polyp segmentation. As reported in Table 1, our model had best segmentation accuracy among all five methods, on the CVC-ClinicDB dataset, with $mDice$ score of 0.9514 and $mIoU$ score of 0.9083. Despite the $mIoU$ value was not highest on the Kvasir-SEG dataset, it was slightly inferior to second-best FCBFormer ($mIoU$: 0.8746 vs 0.8752). In Table 2, our model also attained best scores in nearly all evaluation indicators compared with SFA [19], PraNet [21], MSNet [29], SANet [22], M2SNet [30], SSFormer [35], ESFPNet [37], and TranSEFusionNet [43] on both datasets. In addition to this, the $F_\beta ^w$ value of Polyp-PVT [33] on the Kvasir-SEG was 0.1% marginally higher than our model, whereas the MAE value of Polyp-PVT was 0.1% slightly lower. Through comparison between our model and CFA-Net, a similar observation also existed ($m{E_\phi }$:0.960 vs 0.962; MAE:0.024 vs 0.023) in terms of $m{E_\phi }$ and MAE. We believed the reason for this result was due to too small dataset and abundant noise, which made it difficult to learn sufficient feature details and have an adverse impact on our model. In contrast, the $mDice$ and $mIoU$ scores of our model were higher by 0.9% and 1.1%, 0.7% and 1.3%, respectively. To show our model intuitively, we also provided segmentation results on different testing datasets, and compared testing performance with several representative advanced methods. From visualizable prediction results of different methods in Fig. 7 and Fig. 8, it could also be well visualized that the proposed Dua-PSNet identified a more comprehensive range of polyp regions with smooth and clear boundaries. Even for the challenging scenarios such as irregular shape, varied size as well as blurred boundary between the polyp and its surrounding tissue, it could also deal with well and produce considerably better segmentation mask, proving its strong learning ability. These excellent segmentation results could be attributed to the fact that our model had a powerful ability to fully mine the multi-scale feature representation and dynamically fuse different level of features for tackling the scale variations of polyps, and simultaneously provide fine local edge cues to guide for locating its boundary, thereby boosting the segmentation performance. With the aid of our model, an accurate and operator-independent estimate of the polyp size could be provided to assist in making feasible decisions required during colonoscopy.

Considering stitching features directly using the skip connection of U-Net may cause some relevant information lost, we first restored the feature sizes of three high levels to the same size and then aggregated them in a stepwise adaptive way through the constructed AFA block in the Transformer branch, highlighting important and coarse global contextual features. As listed in Table 6 (No.5 vs No.4), the model adding the AFA component yielded better segmentation results than one without this module, with 1.3% and 1.4% higher $mDice$ and 1.6% and 2.1% higher $mIoU$ on the CVC-ClinicDB and Kvasir-SEG benchmarks, respectively. Notwithstanding preserving most information in global semantic features, the transmission of these semantic information from high-level to low-level was still weakened by simple up-sampling operation. Accordingly, we constructed ResidualBlock module to emphasize critical local boundary details, and proposed SGLFH module to selectively add these fine local details into global semantic features again at the end of Transformer branch, such that the original characteristics of images could be conserved to the maximum extent. From the visualizable results in Fig. 9, it could be visually perceived that the overall segmentation effect on polyps of Fig. 9 (No.3) with ResidualBlock module was relatively better than that of Fig. 9 (No.1) using only FCN with RB module. From No.1 vs No.3 in Table 6, the same observation was obtained. When further adding the SGLFH block, the identification of focus regions was more precise in Fig. 9 (No.7 vs No.6), and the boundary of Fig. 9 (No.7) was sharper and smoother compared to Fig. 9 (No.6). In the segmentation task of polyps, low contrast between the target features and the background made it hard to accurately distinguish them, and most Transformer-based methods only predicted a lower resolution segmentation map (not full-size) for the input image. To enhance the discrimination capability and perform full-size prediction, we combined the Transformer and FCN branches in parallel, where the fine-grained features extracted by FCN branch as well as important and rough features outputted by Transformer branch complemented each other to enhance the target information and weaken the background characteristics for dense prediction at full-size. From the segmentation results in Table 6 (No.2 vs No.4, and No.3 vs No.4) together with Fig. 9 (No.2), Fig. 9 (No.3) and Fig. 9 (No.4), we could find that the model combining two branches stimulated segmentation performance gains by a significant increment (On the CVC-ClinicDB dataset, No.4 vs No.2: 1.2% $mDice$ and 2.1% $mIoU$ improvements, No.4 vs No.3: 2.2% $mDice$ and 4.2% $mIoU$ improvements; On the Kvasir-SEG dataset, No.4 vs No.2: 1.0% $mDice$ and 1.4% $mIoU$ boosts, No.4 vs No.3: 2.3% $mDice$ and 2.8% $mIoU$ boosts). The gradual superposition of each component provided consistent improvement of performance, and combining the presented all components together allowed our model to achieve the highest performance in all evaluation metrics. This justified that all components we devised contributed to performance boost in the entire Dua-PSNet, and excluding any one of them would lead to a decline in the performance of polyp segmentation.

The model’s generalizability was also a challenge in the domain of medical image analysis. We argued that the most important angle of the model’s generalizability lied in its ability to adapt to different types of datasets. Most of existed methods for polyp segmentation were only evaluated on a single dataset, which was not capable of directly reflecting the model’s generalizability. On the contrary, cross-dataset evaluation could be implemented for investigating the generalizability of different networks. In view of the generalizability of U-Net, the Dua-PSNet was also built upon this model for innovation, and the segmentation results on the unseen three datasets implied its superior generalization ability. As can be seen in Table 5, on both the CVC-ColonDB and CVC-300 datasets, our model exhibited best segmentation results, exceeding the second-best Polyp-PVT by 2.0% and 1.6% with respect to $mDice$ and $mIoU$ for the CVC-ColonDB dataset, while 1.1% and 1.4% for the CVC-300 dataset. Even on the most challenging ETIS benchmark, our model was also better in comparison with the latest Polyp-PVT, FCBFormer and ESFPNet apart from $m{E_\phi }$ indicator, which was only slightly lower by 0.7%, 0.2%, and 0.1%, respectively. It was marginally 0.1% higher than CFA-Net in MAE indictor, yet exhibited obviously better in other evaluation metrics. Across all datasets, the proposed Dua-PSNet generalized well with consistently accurate segmentations, which made it more suitable for practical applications in which wide variations of data happened frequently. Furthermore, we also displayed the comparison results on $maxDice$ and $maxIoU$ in Fig. 10, which further evidenced the stability and leadings of our model. The excellent generalizability may be owed to the successful mixture of the merits of the Transformer and FCN in the proposed Dua-PSNet, resulting in the main region of polyp being handled by the Transformer branch whilst a reliable and fine full-size edge around this main area being guaranteed by the FCN branch.

 

Fig. 10. Model generalizability validation using Max Dice (a) and Max IoU (b) metrics. The bold fonts denoted the best results.

Download Full Size | PDF

By comparison with the ECTransNet [44] and MCSF-Net [31], two latest segmentation methods proposed recently, the outstanding segmentation performance of our model was demonstrated as reported in Table 3. With respect to ECTransNet and MCSF-Net, we performed retraining and testing using their publicly available codes and recommended parameter settings as well as default data augmentation methods reported in their original literatures. Among data augmentation strategies used in these models and our approach, there only existed slight differences. Moreover, we also conducted an additional experiment to retrain and test ECTransNet and MCSF-Net based on the released codes and the recommended parameters using our data augmentation techniques in experiment B, and determined the p-values for comparisons of the Dice and IoU metrics, in which five repetitions were adopted for statistical significance evaluation. The gains in Dice and IoU for ECTransNet and MCSF-Net using our data augmentation method (compared to counterparts using respective data augmentation strategies reported in their original literatures) were statistically insignificant with p-values far greater than 0.05, which indicated that slightly different data augmentation methods did not significantly affect the comparison of these models. Even in some cases, the Dice and IoU values for ECTransNet and MCSF-Net using our data augmentation method were relatively lower than those using their respective data augmentation techniques. Under this circumstance, we leveraged their best scores attained using their default data augmentation methods to avoid the bias introduced in model re-training, which could guarantee relatively fairness of comparisons of the performance of different models. Concretely, our model gained the highest 0.922 $mDice$ and 0.874 $mIoU$ scores on the Kvasir-SEG dataset. Notably, on the CVC-ClinicDB dataset, our model achieved a $mDice$ value of 0.2% lower slightly than the MCSF-Net, whereas maintaining the same best level as that in the $mIoU$ metric. And we believed that this difference was very small and its advantage in polyp segmentation accuracy in the face of challenges arising from variations in size and morphology was still substantiated. Moreover, in the case of the unseen datasets CVC-ColonDB, ETIS-LaribPolypDB, and CVC-300, our model achieved the consistently better segmentation performance across nearly all metrics than the ECTransNet and MCSF-Net, as listed in Table 5. For the CVC-300 dataset, the improvements of our model in $mDice$ and $mIoU$ values were not obvious, with 0.3% and 0.5% increases over the ECTransNet, and 0.9% and 1.1% improvements over the MCSF-Net. Perhaps, utilizing domain adaptation technique or data augmentation strategy could contribute to further generalization capability improvements. Figure 8 also provided a visual comparison between our model and these two methods. We could see that the proposed Dua-PSNet consistently predicted relatively accurate segmentation maps under various challenges, whilst the ECTransNet and MCSF-Net occasionally generated un-related or incomplete segmentation effects. Different from the ECTransNet and MCSF-Net, our Dua-PSNet combined both Transformer branch and FCN branch in a parallel pattern for modeling multi-scale global contextual semantic relationships and extracting fine low-level features to relieve the loss of useful information. Additionally, the model incorporated the AFA block in the multi-scale fusion process of high-level features to progressively refine global semantic information, thereby enhancing the representation ability of such characteristics and obtaining spatial and location information related to polyps. The ResidualBlock module enabled the extraction of richer local boundary cues of low-level features. The SGLFH module aggregated the resulting local boundary information into global semantic features for delineating more fine-grained contours of polyp lesions and recalibrating their locations. As a result, our Dua-PSNet was able to achieve strong learning and generalization capability.

Despite generating promising performance, this work had also some limitations and rooms for further enhancement. Firstly, during the model training, we needed to resize the input images into a uniform size of 352 × 352 so that the model’s complexity could be decreased, inevitably resulting in the loss of some information in the images and influencing the overall performance of the model. A future study will attempt to explore the data augmentation (such as Generative Adversarial Network (GAN) [59] or Conditional Variational Autoencoders (CVAE) [60] or SAM [61]) and feature extraction technology [62] for alleviating information loss. Secondly, the introducing of the Transformer increased the model’s complexity. In the future, we plan to seek more efficient strategies [63] for parameter reduction to reduce the computation complexity of our model without compromising its performance. Finally, our network lacked a comprehensive evaluation under a real-world clinical scenario. The future work is to consider new data from complex real clinical settings to further test the model’s generalization ability and thoroughly investigate its potential of practical clinical applications.

In conclusion, we developed a novel improved dual-aggregation polyp segmentation network called Dua-PSNet. One of advantages of this model was the usage of both parallel branches characterized by Transformer and FCN to work together for enhancing polyp segmentation performance. Beyond that, in the decoder (MFAD) of Transformer branch, we also introduced the AFA module to aggregate multi-scale high-level semantic characteristics and added the ResidualBlock module to focus on local fine edge information, whilst designed the SGLFH component to selectively incorporate low-level fine boundary cues with high-level semantic information. Simultaneously, the FCN branch extracted highly merged multi-scale characteristics at full-size, and compensated for outputs of the Transformer branch into full-size prediction. The experimental results on five publicly available benchmark datasets revealed that our model achieved better segmentation performance while exhibiting higher generalization capabilities compared to existing state-of-the-art approaches. We expect that this work will furnish a new perspective on network architecture design for polyp image segmentation and be further extended for other relative fields of medical image segmentation.