1. INTRODUCTION

Since the depth of focus is limited in optical lenses of conventional cameras, it cannot obtain an image that contains all the objects in focus. To address this issue, image fusion technology has been developed, more accurately, multi-focus image fusion technology, in which multiple images taken with diverse focal length are integrated to obtain an all-in-focus image [1]. The main purpose for image fusion is that all of the important salient visual information of input images should be well preserved in the fused image. In particular, for multi-focus image fusion, combine two or more images of the same scene taken with different focus settings into a single all-in-focus image with extended depth of the field [2].

By far, many multi-focus image fusion algorithms have been proposed [1–8]. Most of these methods make use of the information within the local regions of input images when performing fusion. This usually can work well for input images with perfect registration. Nonetheless, input images are not well registered under normal circumstances; some isolated regions with pregnant detail information are usually regarded as defocusing content and selected during the fusion process. This will result in an unpleasing effect in the fused results, such as spatial inconsistency or block artifacts that will appear. Further, the mis-registration often occurs within the set of multi-focus images because of the relative movement between the object and camera or their difference of focal points. In this paper, we will propose a new multi-focus image fusion method to address the fusion issues about the mis-registration of multi-focus images. In the proposed method, the information from each local and spatial neighboring region will be collaboratively applied to distinguish the focused regions from the multi-focus images. In this paper, we will present a dictionary learning with rolling guidance filter (DLRGF)-based focus measurement model to address the fusion of mis-registration and block effects. For the recently proposed multi-focus methods [9–14] based on dictionary learning or sparse representation, most of these commonly obtain fused images via fusing the spare coefficients directly, generally introduce unpleasing artifacts and lead to spatial inconsistencies occurring in the fused images for obviously mis-registered input images.

In this paper, we show a new direction to understand image blur (e.g., image defocusing) via dictionary-based sparse representation based on external data. Specially, we found that when decomposing local image patches into dictionary atoms in an additive manner, a clear-image dictionary and a blur-image dictionary show visually different results. The salient difference demonstrates that dictionary atoms can characterize structure in noticeable blur images thus amplifying the inherent difference between slight blur and clear regions. It is very similar to the relationship between focused regions and defocused regions. Motivated by this feature, we propose a simple but expressive salient feature extraction model with the purpose of measuring the focused regions in multi-focus images. Our method is different from the presented fusion methods [15–17] based on sparse representation and dictionary-based sparse representation in two aspects. First, the sparse coefficients are used to determine the focused and defocused regions in the multi-focus images, unlike these existing methods, using the inverse transform of the fused sparse coefficients to obtain fused images. Second, we use the sparse representation obtained from a learned dictionary trained by several classical multi-focus images blurred via rolling guidance filter, to achieve feature extraction and classification of pixels as focused or unfocused ones. To testify to the feasibility and effectiveness of our proposed method, several experiments are conducted using several sets of classical data under five objective quality metrics. Experimental results demonstrate that our proposed method can compete with the current state-of-the-art methods and outperform the classical multi-scale transform (MST) and multi-scale transform sparse representation (MST-SR) methods for registered and mis-registered images, in terms of visual and quantitative evaluations.

The rest of this paper is organized as follows. Section 2 briefly reviews some related work. Section 3 details the proposed focus measurement model and its application to multi-focus image fusion. Experimental results and conclusions are given in Sections 4 and 5, respectively.

2. RELATED WORK

During the past few years, many multi-focus image fusion methods have been proposed in the literature. These are mainly divided into transform and spatial domain methods, respectively [1]. Among these methods, the MST is one of the most popular kinds of methods [15]. The classical image fusion methods include pyramid decomposition [16] wavelet transform [17], shearlet transform [18], and non-subsampled contourlet transform (NSCT) [2]. In recent years, a new MST method was presented based on hybrid dual-tree complex wavelet transform (HDTCWT) [19] and hybrid dual-tree complex wavelet transform and support vector machine (HDTCWT-SVM) [20]. In contrast, this new fusion method generally achieves better results than the classical MST methods for the input images with well registration and mis-registration.

For the spatial-domain-based methods, the earliest fusion method calculates the average of source images pixel by pixel, which usually caused undesirable artifacts in the fusion results [1]. In recent years, some block and region-based fusion methods have also been proposed [21–23]. In particular, some state-of-the-art pixel fusion methods based on guided filtering (GF) [24], image matting [3], dense scale invariant transform (DSIFT) [1], Quadtree and weighted focus-measure (Quadtree) [4], and guided-filter-based difference image [25] have been presented. Generally, these methods perform well for extracting image details and providing spatial consistency.

In addition, with the high application value of the SR and dictionary learning theories, they are widely used in image processing, including multi-focus image fusion [9–14,26,27]. By exploring the characteristics of the sparse coefficients of source images, Yang et al. [9] took the first step for applying the sparse representation theory to image fusion. To overcome the related disadvantages of the MST- and SR-based methods, Liu et al. proposed a general framework MST-SR for image fusion by taking the complementary advantages of MST and SR [10]. Namely, in this MST-SR frame, the low-pass MST bands were merged with the SR-based scheme while the high-pass bands were fused using the conventional “max-absolute” rule. As a result, the contrast in the fused image is improved and the difficulty in determining decomposition level can be well solved. Moreover, they also presented an adaptive sparse-representation-based image fusion method in [11]. In [12–14], models of dictionary-based sparse representation are also presented. Experimental results have verified that the SR- and dictionary-learning-based fusion methods usually are superior to the traditional MST-based fusion methods in the literature.

3. PROPOSED MULTI-FOCUS IMAGE FUSION METHOD

The human visual system has the ability to distinguish blur defocus regions (or patches) from focus regions (or patches) in a multi-focus image. It implies a potential foundation to construct a system based on multi-focus image examples in focus or defocus states, which can automatically distinguish the defocus from the focus regions. We follow the dictionary-learning-based sparse representation of several classical multi-focus images to determine the defocus regions from the focus regions in multi-focus images. Furthermore, we propose a multi-focus image fusion method based on DLRGF; the overall framework is shown in Fig. 1. From Fig. 1, we can clearly see the main process of the proposed DLRGF-based fusion method. First, we use several classical multi-focus images blurred by a rolling guidance filter to learn a dictionary, which is constructed via offline learning. Then we employ the learned dictionary on the input images to obtain focus region feature maps. Subsequently, we obtain the initial decision map via comparing the differences of the focus regions feature map. The latter is to optimize the initial decision map to obtain a final optimized decision map. Finally, the fused image is obtained via the final decision map.

 

Fig. 1. Overall framework of the proposed multi-focus image fusion.

Download Full Size | PDF

A. Rolling Guidance Filter and Sparse Dictionary

1. Rolling Guidance Filter

Images contain many levels of important structures and edges. In the majority of existing research about edge preserving filters, scale-aware local operations were seldom considered in a practical approach. To address this problem, Zhang et al. [28] proposed the rolling guidance filter to filter images with the complete control of detail smoothing under a scale measure. It is based on a rolling guidance implemented in an iterative manner. Assume that the input image is represented by $I$, $p$, and $q$ index pixel coordinates in the images. The rolling guidance filter is expressed as

2. Sparse Dictionary

Sparse representation [29] commonly is described as follows. Given a set of $N$ signals $Y=\{y_1,\cdots ,y_N\}\in {\mathbb{R}}^{d\times N}$, each signal $y_i$ will formally be written as a sparse number of dictionary atoms as

In natural image decomposition, we collect overlapped image patches as input. Each image patch is vectorized as $y_i$ in Eq. (5), and a dictionary $D$ is trained on all the image patches decomposed from the input image. Based on the constructed dictionary, each image patch is decomposed into a few atoms and their corresponding non-zero coefficients, forming the reconstructed feature $D{x}_i$ via Eq. (5).

Considering inherent discrepancy between the two types of dictionaries trained by clear images and blurred images, we propose a new blur metric and apply it into multi-focus image fusion. First, we learn a blur-image dictionary $D$ via Eq. (5). The dictionary $D$ is learned over 60 patches randomly cropped from four multi-focus images blurred by the rolling guidance filter. Since the filtering results of the classical multi-focus images are blurred by the rolling guidance filter, they have similar visual effect with the corresponding defocused regions (or patches) in the multi-focus images, as shown in Fig. 2. In addition, it can control the level of details during filtering and automatically refine edges that can be preserved in order to preserve large-scale structures optimally as well. We believe learning a dictionary from images blurred by a rolling guidance filter can efficiently represent the defocused regions in multi-focus images. In the stage of learning the dictionary in our proposed method, the spatial and range weights of rolling guidance filtering with ${\sigma }_s=3$, ${\sigma }_r=2$, and iteration number $t=5$, respectively, the $k_0$ corresponding to the used dictionary atoms is set to 5 in patch decomposition and the total dictionary size is $64\times 256$, the patch size is $8\times 8$. We have also tried other choices, including increasing and decreasing these six parameters. It is found that this configuration is sufficient for our feature construction. Moreover, the classical multi-focus images used to learn the dictionary $D$ is shown in Fig. 3.

 

Fig. 2. Three examples of rolling guidance filter. The top row is input images and the bottom row is the corresponding filtered images (${\sigma }_s=3$, ${\sigma }_r=2$, $t=5$).

Download Full Size | PDF

 

Fig. 3. Four source images filtered by a rolling guidance filter served as learning sources.

Download Full Size | PDF

After $D$ is learned, it is applied to all image patches of input images, both blur and clear (defocus and focus), for defocus identification. For each new patch $y_i$, we use another sparse representation to decompose it into basic atoms. It can be described as follows:

B. Image Classification via Sparse Representation

Given the input multi-focus images $I_1$ and $I_2$, the proposed multi-focus image fusion method based on dictionary learning with rolling guidance filter is described as follows. The focus feature map $W_{O,1}$ can be calculated as follows:

In this subsection, we choose the parameters as ${\sigma }_s^{\prime }=6$, ${\sigma }_r^{\prime }=3$, and $t^{\prime }=2$.

C. Decision Map Regularization

After the good classification map $W_{O,2}$ is obtained via Eq. (10), the initial decision fusion map $W$ can be easily calculated as follows:

Since the decision map $W$ is also not aligned with object boundaries and contains some small focused regions as well as some small holes surrounded by the focused region, we applied a simple post-processing approach to process these warts. Namely, the closing operator of mathematical morphology imclose with a disk structuring is used to obtain an ideal fusion decision map $W^{*}$, and it can be described as follows:

Given the ideal decision map $W^{*}(x,y)$, the fused image $I_F$ can be calculated by

4. EXPERIMENTS

In this section, we will confirm the effectiveness and superiority of the proposed DLRGF fusion method for both registered and mis-registered images. Three sets of representative multi-focus images are selected to conduct experiments, as shown in Fig. 4. The first row of Fig. 4 displays three different multi-focus images in which the focus of attention is on the left part of each image. The second row in Fig. 4 displays the corresponding input images with the focus of attention on the right part. Among these three sets of images, the first set [Figs. 4(a1) and 4(b1)] is perfectly registered images, and the other two pairs are mis-registered images. Figures 5, 7, and 9 show the fusion results obtained by different methods on the three sets of input images in Fig. 4. Moreover, Figs. 6, 8, and 10 give the corresponding difference images between each of the fused images in Figs. 5, 7, and 9 and the input image in Figs. 4(a1), 4(b2), and 4(b3), respectively. In addition, we will compare our proposed method with other existing representative or state-of-the-art fusion methods based on NSCT [2], adaptive sparse representation (ASR) [10], NSCT-SR [11], GF [24], and DSIFT [1]. As well, to evaluate the performance of the proposed method and the compared methods, five representative image fusion metrics are selected as follows: $Q_P$ [31], $Q_G$ [32], $Q_{MI}$ [33], $Q_Y$ [34], and $Q_{CB}$ [35]. The default parameters used in the related papers are adopted for these fusion metrics. $Q_P$ computes how well the salient features of the source images are preserved. $Q_G$ is used to evaluate how much edge information is transferred from the source images to the fused image. $Q_{MI}$ measures how the original information from the source images is preserved in the fused image. $Q_Y$ measures how well the structural information of the source images is preserved. $Q_{CB}$ is a human-perception-based fusion metric that utilizes the major features in the human visual system model. Furthermore, we also provide the normalized difference $I_{ND}^{*}$, defined by Eq. (14). Here $I_{ND}$ denotes the difference between the fused image and one of the input images. $I_{\max }$ and $I_{\min }$ denote the maximum and minimum values of the difference image $I_{ND}$:

 

Fig. 4. Three pairs of multi-focus images. (a1), (a2), and (a3) are the input images with the focus on the left part. (b1), (b2), and (b3) are the corresponding input images with the focus on the right part.

Download Full Size | PDF

 

Fig. 5. Fusion results for Figs. 4(a1) and 4(b1). (a) NSCT, (b) ASR, (c) NSCT_SR, (d) GF, (e) DSIFT, (f) DLRGF.

Download Full Size | PDF

 

Fig. 6. Normalized difference images between Fig. 4(a1) and each of the fused images in Fig. 5.

Download Full Size | PDF

 

Fig. 7. Fusion results for Figs. 4(a1) and 4(b1). (a) NSCT, (b) ASR, (c) NSCT_SR, (d) GF, (e) DSIFT, (f) DLRGF.

Download Full Size | PDF

 

Fig. 8. Normalized difference images between Fig. 4(b2) and each of the fused images in Fig. 7.

Download Full Size | PDF

 

Fig. 9. Fusion results for Fig. 4(a3) and 4(b3). (a) NSCT, (b) ASR, (c) NSCT_SR, (d) GF, (e) DSIFT, (f) DLRGF.

Download Full Size | PDF

 

Fig. 10. Normalized difference images between Fig. 4(b3) and each of the fused images in Fig. 9.

Download Full Size | PDF

To ensure the reliability of our proposed method, the multi-focus images used in our experiments are selected based on some public image datasets. These images have been used in many related public papers [1,3,21,24] and are available at [36]. Furthermore, the images used in Figs. 2 and 3 are available at [36] and [37].

A. Experimental Results and Discussion

From the three tests of Figs. 6, 8, and 10, it is obvious that the DLRGF method is superior to the others when the input images are either well registered or not. For an example of well-registered input images, as shown in Fig. 5, serious spatial inconsistencies are still introduced in the fusion results obtained by NSCT, ASR, NSCT-SR, GF, and DSIFT [see the rectangular regions in Figs. 6(a)–6(e)]. However, hardly any spatial inconsistencies are introduced in the fusion result obtained by DLRGF, as shown in Fig. 5(f). In addition, Figs. 7 and 9 give two examinations of input images with mis-registeration, as well.

From Fig. 7, it can be clearly see that some defocused image regions are mistakenly introduced into the fusion results obtained by NSCT, ASR, and NSCT-SR. Therefore, the fused images also produce some spatial inconsistency, especially on the boundaries of the mis-registered objects [e.g., the right clock; see the rectangular regions in Figs. 8(a)–8(c)]. The fusion results obtained by GF and DSIFT do not produce spatial inconsistency in the regions of the “right clock.” However, other regions of “the desk” also produce slight spatial inconsistency in the fusion results obtained by these two methods, as shown in the rectangular regions in Figs. 8(d) and 8(e). By contrast, these artifacts are not introduced in the fusion result obtained by DLRGF [see the rectangular regions in Fig. 8(f)].

At the same time, we can obviously see that spatial inconsistency is contained in the fusion results obtained by the NSCT, ASR, NSCT-SR, and GF methods from Fig. 9, especially in the regions of mis-registered objects [e.g., the head of the student; see in the quadrate regions in Figs. 10(a)–10(d)]. In addition, the fusion result obtained by DSIFT also produces slight spatial inconsistency [see in the quadrate regions in Fig. 10(e)]. However, the fusion result obtained by the DLRGF method may not introduce any spatial inconsistency, as shown in the quadrate regions in Fig. 10(f).

Moreover, Table 1 gives the experimental data. The highest values in Table 1 are shown in bold. From Table 1, it can be seen that DLRGF is more robust to input images with registration and mis-registration, and also shows that the DLRGF is superior to the classical fusion method (e.g., NSCT) and can compete with the state-of-the-art fusion method (e.g., GF, DSIFT) in terms of significant extracted information and spatial consistency. As a whole, it can be concluded that, based on our experiments, the proposed method presents competitive fusion performance as compared with previous methods, both in visual comparisons and objective evaluations.

Table 1. Performance of Different Fusion Methods on Multi-Focus Images with Registration and Mis-Registration

View Table

5. CONCLUSION

In this paper, we propose a new multi-focus image fusion method based on dictionary learning with a rolling guidance filter for well-registered and mis-registered images. In the proposed method, via the dictionary learning approach, several classical multi-focus images blurred by the rolling guidance filter served as the learning images. Then the learned dictionary is performed on the input images to construct a focus measurement model to produce a focus regions map. Subsequently, we compare the focus regions maps of the input images to determine the initial decision map. Finally, the initial decision map is optimized by a morphology operator to obtain a satisfied decision map. Experimental results demonstrate that the proposed method is competitive with the current state of the art and superior to some representative methods when input images are well registered and mis-registered. However, how to improve the efficiency of the proposed method and extend the proposed method to other image fusion fields, such as infrared and visible image fusion and medical image fusion, are the future research directions.