Detail enhancement for infrared images based on Relativity of Gaussian-Adaptive Bilateral Filter

Xiongwei Feng; Zhongliang Pan

doi:10.1364/OSAC.434858

1. Introduction

Infrared imaging technology is closely related to modern society and has been widely used in surveillance systems [1], military defense [2], astronomy [3]. However, due to the limitations of hardware equipment itself and environmental interference, the quality of imaging is not good. It is often with low contrast, blurred details and background noise [4]. In order to let the output infrared image more suitable for being observed by human eyes and for the subsequent target recognition and tracking, it is necessary to further study the processing of the infrared image. Obviously, the algorithm in this paper is optimizing the image contrast for a better display of infrared images.

An updated literature review reflecting recent work on infrared images enhancement is provided [5] by us. According to that paper’s classification of infrared images enhancement algorithms, there are three types of algorithms, mapping-based algorithms, image stratification-based algorithms, and gradient-domain-based algorithms. Early wide dynamic infrared image approaches are mainly mapping-based algorithms. The advantage of the mapping-based algorithms is that the details of the infrared image are greatly enhanced, which can show more detailed information and improve the contrast, such as histogram [6], contrast limited adaptive histogram equalization (CLAHE) [7], dynamic histogram equalization (DYHE) [8]. Lai et al. [9] proposed an improved platform histogram equalization algorithm. Lin proposed an adaptive histogram algorithm for remote monitoring of infrared images [10]. However, after being processed by the mapping-based algorithms, the gray level of the image is missing, part of the image’s details are lost, and the contrast of the image is excessively enhanced. According to the classification of Zhou’s literature [5], infrared images enhancement algorithms proposed by researchers are mainly based on image stratification [11–13,16,17,20–21]. Zuo et al. [13] proposed a novel algorithm for infrared images based on the bilateral filter [14], which holds the best visual effect when comparing with other algorithms. After that, the researchers designed a large number of enhanced algorithms based on this framework to improve the details of infrared images. However, since the bilateral filter requires exponential calculations, it has poor real-time performance. At the same time, the algorithm based on bilateral filter shows a halo at high temperature area.

In 2014, for improving the running time of the algorithm based on bilateral filter, Liu and Zhao designed the algorithm framework based on the guided image filter (GF&DDE) for the first time [15–16], which meets the expectation of shortened running time. However, it has inherent defects include the halos at the edge of the image and the visual effect that is inferior to the algorithm based on bilateral filter. In terms of detail enhancement, the infrared image is improved significantly by the algorithms based on image stratification, but they will produce a halo and gradient inversion. What's worse, the algorithms based on image stratification runs slowly with low efficiency. Therefore, an algorithm based on gradient-domain [18,19] is proposed by the researcher for eliminating the phenomenon of halo, gradient inversion, blur, oversaturation, and so on. However, the effect of detail enhancement is inferior to the algorithm based on image stratification.

In 2020, Xie and Liu [20] proposed a new infrared image enhancement algorithm that uses two-dimensional convolution to divide the image into two parts: the base layer and the detail layer. The acceleration method of this algorithm is to first replace the two-dimensional convolution with one-dimensional convolution, and finally convert the one-dimensional convolution into the iterative calculation. Compared with other classical algorithms, its main advantage is that the running time is greatly shortened. Recently, Chen et al. [21] proposed a new method, which includes adaptive gamma transformation, multi-scale guided filtering, and image fusion. This method significantly enhanced image details and improved contrast.

Yu Xiao and Zhou Zijie proposed a new infrared image extraction algorithm combined with adaptive growth immune field, it can lower the over growth and improve the accuracy of target extraction in complex environment [22]. Nie et al. [23] proposed a new infrared and visible image fusion algorithm based on a total variation with joint norms, it can clearly display the infrared objects and the scene details simultaneously. Huang et al. [24] proposed a method based on the progressive super-resolution generative adversarial network, it can achieve better results in single image super-resolution (SISR) compared to the SR methods. Lv et al. [25] present a detail enhancement approach for low-light images, two main innovations of this method are based on multi-branch convolutional neural network and a large scale low-light simulation dataset.

To address the aforementioned problems, we proposed a novel algorithm for edge/structure image smoothing, which is inspired by GTV regularization [26] and Gaussian-adaptive bilateral filter (GABF) [27]. We called this method the Relativity-of-Gaussian-adaptive bilateral filter (ROGABF). In this case, we proposed a novel algorithm for the infrared image based on the ROGABF, which is inspired by the successful application of the bilateral filter, guided image filter, and local edge-preserving smooth filter in infrared image processing. To present a better visual than the classical infrared image enhancement algorithms, we also proposed a new weight coefficient and simplified the process of enhancement for the detail layer. The proposed algorithm can enhance effectively the detail for infrared images. It should be noted that our algorithm is much faster than other algorithms to maintain the image detail to the maximum extent while avoiding excessive edge brightness.

2. Principle of our research

In this section, the processing process of the proposed algorithm is clearly shown in Fig. 1.

Fig. 1. The process flow chart of ROGABF&DDE

Download Full Size | PDF

As shown in Fig. 1, the whole algorithm processing process consists of three main steps.

In the first step, the input image is divided into two parts, the base layer and the detail layer. The base layer is the filtered result of the ROGABF. We obtain the detail layer by subtracting the filtered result from the input image. In the second step, histogram projection is performed on the base layer to stretch the contrast of the image. For the detail layer that contains weak details of the image, it needs to be enhanced by using the weight coefficient which is proposed by us. We have simplified the approach of gain mask enhancement of Ref. [13,16]. Here multiplied the detail layer and the weight coefficient together and we acquired the enhancement of the detail layer. In the third step, the results of the two parts in the second step are recombined to output and display. Here the process of this step is very simple. We can give a summary: the output is obtained by adding the result of two parts in the second step and mapped the output to the 8-bit domain for display.

Next, we will introduce the working principle of ROGABF and the process of accelerating the processing speed of the algorithm, and give the suggested values of the parameters. Finally, we will give the experiment results of the 8-bit gray images with different filters.

2.1 Relativity-of-Gaussian-Adaptive Bilateral Filter

As mentioned in the introduction of our paper, the ROGABF is a novel edge/structure-preserving image smoothing. The principle of ROGABF is similar to the [26]. We combined Gaussian-adaptive bilateral filter and improved the method in [26] a lot to present the ROGABF. The ROGABF is used to decompose an infrared image into the base layer and the detail layer. The main content of it is from Eq. (5) to Eq. (11). As shown in Eq. (5) to Eq. (11), F_GAB denote the Gaussian-adaptive bilateral filter, so the ROGABF can be expressed as ROF_GAB. We will introduce the GABF first. The principle of the GABF is shown in Eq. (1) to Eq. (4). To reduce the noise of the image further and achieve a better performance in edge-preserving image smoothing, Chen et al. [27] presented a novel BF-based method, which is the Gaussian-adaptive bilateral filter. Different from the bilateral filter, the guidance g and input image I are nonidentical. Where g is obtained by Gaussian blur process on I, this process is expressed as follows:

(1)$$f\left( i \right) = \mathop \sum \nolimits_j W_{i,j}^g\left( {\boldsymbol g} \right){{\boldsymbol I}_j}$$

where i is the center position of the input image, $W_{i,j\; }^\textrm{g}$ is expressed as follows:

(2)$$W_{i,j}^g(g )= \frac{1}{{{k_i}}}exp\left( { - \frac{{\parallel i - j{\parallel^2}}}{{\sigma_s^2}}} \right)$$

where k_i denotes a normalizing factor. In Eq. (2), the second term $\textrm{exp}\left( { - \frac{{\parallel i - j{\parallel^2}}}{{\sigma_\textrm{s}^2}}} \right)$ is the Gaussian spatial kernel, σ_s denote the sizes of the window for extension of the neighborhood. The Gaussian-adaptive bilateral kernel is defined as:

(3)$$W_{i,j}^{gabf}({{\boldsymbol I},{{\boldsymbol g}^ - }} )= \frac{1}{{{k_i}}}exp\left( { - \frac{{\parallel i - j{\parallel^2}}}{{\sigma_s^2}}} \right)exp\left( { - \frac{{{\vert\vert{\boldsymbol I}_i} - {{\boldsymbol g}^ - }{\vert\vert}^2}}{{\sigma_r^2}}} \right)$$

where ${{\boldsymbol g}^ - }$ is obtained by Eq. (1). In Eq. (3), the third term $\textrm{exp}\left( { - \frac{{{\vert\vert{\mathbf I}_i} - {{\boldsymbol g}^ - }{\vert\vert}^2}}{{\sigma_\textrm{r}^2}}} \right)$ is the range kernel. σ_r denote the variation in intensities of the amplitude of an edge. The final output f(i) of the GABF can be expressed by:

(4)$$f\left( i \right) = \mathop \sum \nolimits_j W_{i,j}^{gabf}\left( {{\boldsymbol I},{\boldsymbol {\bar{g}}}} \right){{\boldsymbol I}_j}$$

The previous filters are concerned with local information and can’t eliminate textures well to keep edge structure smooth. The proposed method can process textures of any size after global optimization. Moreover, this method can remove textures and preserve edge structure. The ROF_GAB algorithm consists of two parts: local regularization and global optimization. Local regularization, realized by ROF_GAB, which is different from Difference-of-Gaussian (DOG) [28], can selectively smooth the gradient. It can be shown by the following formula:

(5)$${\rm R} = \left|{\frac{{F_{GAB}^{\sigma 1}\ast \nabla {T_{x,y}}}}{{F_{GAB}^{\sigma 2}\ast \nabla {T_{x,y}}}}} \right|$$

where T is the smoothing image, $\nabla {T_{x,y}}$ is the gradient of T in the x or y direction. Equation (5) is subject to $\sigma 1 < \sigma 2$. $\sigma$ is an adjustable parameter, which belongs to the local Gaussian spatial kernel of Eq. (2). F_GAB referred to Gaussian-adaptive bilateral filter, which is introduced in detail above. The formula in Eq. (5) consists of two parts: a small scale feature $|{\textrm{F}_{\textrm{GAB}}^{\sigma 1}{\ast }\nabla {T_{x,y}}} |$ and a large scale feature $|{\textrm{F}_{\textrm{GAB}}^{\sigma 2}{\ast }\nabla {T_{x,y}}} |$, the former contains gradients, the latter contains strong edges/structures. The cross-scale relative value R determines the variance on a scale.

Global Optimization: in order to further improve the effect of image smoothing, the global optimization function is as follows:

(6)$$ar{g_T}min\parallel T - F\parallel_2^2 + u\left( {\bigg \vert}{\bigg \vert} {{{\frac{{F_{GAB}^{\sigma 1}\ast \nabla {T_{x,y}}}}{{F_{GAB}^{\sigma 2}\ast \nabla {T_{x,y}}}}}{\bigg \vert}{\bigg \vert}_1} + {{\bigg \vert}{\bigg \vert}{\frac{{F_{GAB}^{\sigma 1}\ast \nabla {T_{x,y}}}}{{F_{GAB}^{\sigma 2}\ast \nabla {T_{x,y}}}}}{\bigg \vert}{\bigg \vert}_1}} \right)$$

where F represents the input image, and u is a parameter larger than zero. The first term ${\parallel} T - F\parallel _2^2$, which is an L2-norm fidelity term, is to minimize the distance between the result T of the ROF_GAB and the input image F. The solution of the ROF_GAB norm can be obtained by the iterative reweighted least square method [29]. The ROF_GAB normalization in the X direction can be rewritten as follows:

(7)$$\begin{aligned} &{\bigg \vert}{\bigg \vert}{{{\frac{{F_{GAB}^{\sigma 1}\ast \nabla {T_x}}}{{F_{GAB}^{\sigma 2}\ast \nabla {T_x}}}}{\bigg \vert}{\bigg \vert}_1} = {\bigg \vert}{\bigg \vert}{{\frac{{{{({F_{GAB}^{\sigma 1}\ast \nabla {T_x}} )}^2}}}{{({F_{GAB}^{\sigma 2}\ast \nabla {T_x}} )({F_{GAB}^{\sigma 1}\ast \nabla {T_x}} )}}}{\bigg \vert}{\bigg \vert}_1}}\\ &{ \approx F_{GAB}^{\sigma 1/2}\ast \frac{1}{{F_{GAB}^{\sigma 1}\ast max({{{({{\nabla_x}T} )}^2},\varepsilon } )}}{\vert\vert}{\nabla _x}T{\vert\vert}_2^2} \end{aligned}$$

It is difficult to solve Eq. (6) for the L1-norm in the ROF_GAB regularization. To overcome this problem, we improved the ROF_GAB and make the approximation on the first line in Eq. (7). Where $\varepsilon $ is set as 0.001 to avoid zero. A non-linear weight in Eq. (7) can be defined as follows:

(8)$${W_{x,y}} = F_{GAB}^{\sigma 1/2}\ast \frac{1}{{F_{GAB}^{\sigma 1}\ast max({{{({{\nabla_{x,y}}T} )}^2},\varepsilon } )}}{\vert\vert}{\nabla _{x,y}}T{\vert\vert}_2^2$$

Initialize T° = F, and rewrite the global optimization function in the form of the matrix, which is as follows:

(9)$${({\boldsymbol T} - {\boldsymbol F})^T}({{\boldsymbol T} - {\boldsymbol F}} )+ u({{{\boldsymbol T}^T}{\boldsymbol D}_{\boldsymbol x}^T{{\boldsymbol W}_{\boldsymbol x}}{\boldsymbol T}{{\boldsymbol D}_{\boldsymbol x}} + {{\boldsymbol T}^T}{\boldsymbol D}_{\boldsymbol y}^T{{\boldsymbol W}_{\boldsymbol y}}{\boldsymbol T}{{\boldsymbol D}_{\boldsymbol y}}} )$$

where T and F are vector forms of T and S, respectively. Wx,y contains a diagonal matrix of the weight W_x,y, and matrices D_x and D_y are discrete differential operators. Global optimization is an iterative cycle. The solution of the k-th iteration for vector T is shown as following:

(10)$${{\boldsymbol T}^\textrm{k}} = {({{\boldsymbol E} + u{{\boldsymbol L}^{K - 1}}} )^{ - 1}}{\boldsymbol F}$$

(11)$${{\boldsymbol L}^K} = {\boldsymbol D}_x^T{\boldsymbol W}_x^K{{\boldsymbol D}_x} + {\boldsymbol D}_y^T{\boldsymbol W}_y^K{{\boldsymbol D}_y}$$

E is an identity matrix in Eq. (10), where L^k is a sparse five-point Laplacian matrix [30] in Eq. (11). The fast solution for Eq. (10) can be acquired by the preconditioned conjugate gradient (PCG) [31].

For the parameters of ROF_GAB, the parameter $\sigma 1$ and $\sigma 2$ is usually set as 1.5 and 3, respectively. We setσr as 1. Although K is unlimited when its value is more than zero, the maximum iteration K is choosing between [0,1] in this paper. The positive parameter u is also choosing between [0,1]. In the next section 3, we choose K and u as 0.1 and 0.001, respectively.

Table 1. The Pseudo Code of ROF_GAB

View Table | View all tables in this article

A fast solver is used to speed up the processing the ROF_GAB [31]. As is shown in Table 1, the whole algorithm process is very simple. The first step is to initialize the value of input I, the second step is to calculate the weight coefficient of the k-th iteration operation, and the third step is to return the value of the output S to be updated.

Figure 2 shows the results of the 8-bit gray image processed by different filters. Figure 2(a) is the 8-bit gray image. Figure 2(b) is the result of Gaussian filtering, Fig. 2(c) is the result of the bilateral filter, Fig. 2(d) is the result of guided image filter, and Fig. 2(e) is the result of ROF_GAB. It is obvious that the visual effect of Fig. 2(b) is the worst, and it is blurred, and its details cannot be recognized. The results show that the visual effect of Fig. 2(c) is significantly improved, the contrast is greatly improved, and there is no halo at the edge. As is shown in Fig. 2(e), the proposed method can smooth the image effectively. At the same time, it gives people a real and natural sensibility, and overcomes the defect of the guided image filter to generate a halo in Fig. 2(d).

Fig. 2. The results of different filters: (a) 8-bit gray image (b) gray image processed by Gaussian filter (σ=0.5); (c) gray image processed by bilateral filter ((${\sigma _s}$=10, $\; {\sigma _r}$=25); (d) gray image processed by guided image filter (w=3, $\varepsilon $ = 500); (e) gray image processed by the ROF_GAB ($\sigma 1$=1.5, $\sigma 2$=3,σr = 1, K=0.1, u=0.001).

Download Full Size | PDF

2.2 Weight coefficient of the detail layer

The input image is filtered by the ROF_GAB, and then we obtained the detail layer by subtracting the filtered result from the input image. As shown in Fig. 1, the detail layer is enhanced by multiplying the weight coefficient, while the base layer needs histogram projection for equalization. As is mentioned above, the weight coefficient is applied to enhance the detail effectively. It is the key to choose the appropriate weight coefficient for detail enhancement. Usually, the selected weight coefficient can accurately reflect the richness of image details. The widely used noise visibility function [32] is as follows:

(12)$$f\left( {x,y} \right) = \frac{1}{{a{M_0}\left( {x,y} \right) + 1}}$$

where a is an adjustable parameter, M₀(x,y) represents the local variance of the image to be processed. When the details of the image are very rich, the noise visibility function is close to 0, and there is almost no noise. When the image area changes slowly and the noise visibility is close to 1, the noise is very large. Based on this, the noise visibility function can be selected as the weight coefficient. Here we proposed a new weight coefficient, which can enhance the details of the image and accurately measure the spatial detail, and its expression is as follows:

(13)$$\textrm{w}({i,j} )= \left( {\frac{D}{{A{V_0}({i,j} )+ B}}} \right)\textrm{exp}\left( { - \frac{{{{({{m_\textrm{c}}} )}^2} + {{({{n_\textrm{c}}} )}^2}}}{{2{\sigma^2}}}} \right)$$

The above expression represents the weight coefficient of the detail layer. From the left, the first term is a function based on the local variance of the image and the second term is the Gaussian low-pass filter function.

Among them, ${m_c} ={-} \frac{{{w_1} - 1}}{2}, \ldots , - 1,0,1, \ldots \frac{{{w_{1 - 1}}}}{2}$;${n_c} ={-} \frac{{{w_1} - 1}}{2}, \ldots , - 1,0,1, \ldots \frac{{{w_{1 - 1}}}}{2}$; w₁ is the window size of the Gaussian low-pass filter. $\sigma $ is the variance of the Gaussian function. Here we set $\sigma $ as 1. It is well known that images processed by a Gaussian low-pass filter are blurred at the edges and lose details [33]. The first term contains the local variance of the image, which will protect the edge details. Where A, B and D are all adjustable parameters, and V₀ (i, j) is the local variance of the image, which can reflect the richness of image details. After a large number of experiments, the empirical value of D is 1. The different values of parameters A and B have different effects on the image display. The larger the value of parameter A is, the more obvious the enhancement of image details will be. When it is too large, the edge of the image will be very bright, resulting in excessive enhancement and serious image distortion. Parameter B has little effect on the detail layer when its value larger than 1. We show the different values of AB and the corresponding results in Fig. 3. We can know that when A=1 and B=100, the result of the detail layer is satisfied, so we can use it as the value of AB to enhance the infrared image.

Fig. 3. The results of the detail layer with different values for A and B. The input image is the 8-bit gray image.

Download Full Size | PDF

2.3 Histogram projection of the base layer

After the input image is divided into the base layer and the detail layer, the base layer needs to stretch the contrast of the processed image through histogram projection technology. In order to make the final output 8-bit image more suitable for being observed by human eyes, we select the histogram projection with a threshold value (HP) [13]. The original histogram of the base layer is binarized with a threshold T. When the number of the same grayscale is larger than the value of the threshold, it is 1, otherwise, it is zero, which is expressed as follows:

(14)$$H(x )= \left\{ {\begin{array}{c} {0,\quad ({{n_x} < T} )}\\ {1,\quad ({{n_x} > T} )} \end{array}} \right.$$

In Eq. (14), n_x represents the number of pixel values that are also x, where T is a threshold (which is generally one thousandth of the total pixel of the image). There are two benefits to setting a threshold. One is to significantly improve the global contrast. Second, a small number of outliers will not affect the global gradient distribution. Calculate the cumulative distribution function, which is written in the following equation:

(15)$$B\left( x \right) = \left\{ {\begin{array}{cc} {0,}&{\left( {x = 0} \right)}\\ {\frac{{\mathop \sum \nolimits_{y = 0}^{x - 1} H\left( {x,y} \right)}}{{{n_{\textrm{valid\; }}}}},}&{\; \textrm{(other)}\; } \end{array}} \right.$$

In the above formula, n_valid represents the number of effective gray value (In fact, n_valid is the number of H(x,y) equal to 1 in In Eq. (15)), thereby converting the gray level x of the base layer mapped to R*B(x), where the maximum value of the output image is R = min(n_valid, D), and D represents the display range of the output image.

3. Experimental results

To verify our method in enhancing the details of infrared images, twenty-five groups of experiments are carried out to test the performance of the algorithm. The test platform is a personal computer equipped with an Intel i5-6300 processor and 16GB RAM of MATLABR2019a version software. We use four algorithms to process the infrared images in Fig. 4–Fig. 7, which are DYHE, BF&DDE, GF&DDE, and the new algorithm. The parameters of the reference algorithm are the suggested values of the corresponding Refs. [8,13,16]. The parameters of the proposed algorithm are the suggested values in section 2. To validate the reliability of the algorithm, five infrared images of different scenes were analyzed in detail. The results of the other twenty infrared images are presented in Fig. 8 and Fig. 9. The structure of this section is as follows, and it is divided into four parts. The first part is a Dataset. The second part is to demonstrate the improvement of visual effect in Fig. 4–Fig. 9. The third part is three quantitative indexes and a subjective assessment index: the Root-Mean-Square Contrast (RMSC) [34], the running time of the algorithm, Image Structure Similarity (SSIM) [35], and the mean opinion score (MOS) [36]. The fourth part is ablation study.

Fig. 4. The result of Image 1: (a1) eight-bit grayscale image; (a2) DYHE; (a3)BF&DDE; (a4) GF&DDE; (a5) the proposed algorithm.

Download Full Size | PDF

3.1 Dataset

Five groups of infrared images from Fig. 4 to Fig. 7 were selected as representative samples in Dataset 1 (Ref. [37]) with a resolution of 324×256 pixels. The five images tested in Section 3.2 are from Image1 to Image5 in Fig. 4–Fig. 7. The twenty infrared images of Fig. 8 and Fig. 9 used in our paper can obtain from Dataset 2 (Ref. [38]).

3.2 Visual effect comparison

In this part, we select twenty-five groups of images from different scenes to verify the visual effect of the algorithm. Since human eyes cannot see the original 16-bit infrared image, we chose an 8-bit grayscale image of the original infrared image to replace the infrared image display from Fig. 4 to Fig. 7. The main evaluation indicators are visual effect, details of image and contrast in this section.

Figure 4(a1) is the grayscale image of the original infrared image, which is very dark, and you can hardly see any details. After the infrared image is processed by four algorithms, many details of the image are exposed. Figure 4(a2) is the result of DYHE, Fig. 4(a3) is the result of BF&DDE, Fig. 4(a4) is the result of GF&DDE, and Fig. 4(a5) is the result of the algorithm proposed in this paper. Among the four images, the visual effect of our algorithm is the best. In Fig. 4(a2), the DYHE algorithm improves the contrast of the image, and the brightness is also significantly increased, but in the area marked by the yellow arrow, it is so bright that we cannot see image details. In Fig. 4(a3), the BF&DDE algorithm significantly improves the overall brightness and contrast of the image, but in the yellow marked area, it is still impossible to observe more detailed information. In Fig. 4(a4), the GF&DDE algorithm has the same effect as the BF&DDE algorithm, but the overall brightness of the image is a little higher. Like the above two algorithms, the contour lines of the yellow marked area are still not observed. In Fig. 4(a5), after the image is processed by the new algorithm, the overall image looks more real and natural, without over-enhancement and over-brightness, and the contour lines can be seen in the area marked by the yellow arrow. Compared with the previous three classic algorithms, the new algorithm can show more image information with clearer details.

In Fig. 5, we select an image that contains a white concrete floor, railings and buildings. Among the four algorithms, only DYHE and the new algorithm have satisfactory processing effects, we can see the railing in Fig. 5(b2) and Fig. 5(b5) clearly. In Fig. 5(b2), we carefully observe the yellow arrow area processed by DYHE, we can see the railing in the red boxes clearly, but the brightness of the cement floor is too high, resulting in a halo, which is not real. In Fig. 5(b5), the yellow arrow area of the image processed by the new algorithm avoids the halo phenomenon, and the details are shown clearly without excessive enhancement. In Fig. 6(b3), the image is processed by the algorithm based on the bilateral filter, the details are very blurry. In Fig. 5(b4), due to inherent flaws of the GF&DDE algorithm, a halo is prone to appear in the edge area, so the area marked by the yellow arrow is very bright, resulting in blurry details and poor visual effects.

Fig. 5. The result of Image 2: (b1) eight-bit grayscale image; (b2) DYHE; (b3)BF&DDE; (b4) GF&DDE; (b5) the proposed algorithm.

Download Full Size | PDF

Fig. 6. The results of Image 3: (c1) eight-bit grayscale image; (c2) DYHE; (c3) BF&DDE; (c4) GF&DDE; (c5) the proposed algorithm.

Download Full Size | PDF

In Fig. 6, to compare the performance of the four algorithms, we choose an image that contains only houses. On the whole, in Fig. 6(c2), the processing result of the dynamic histogram is very bright and unnatural. In Fig. 6(c3) and Fig. 6(c4), the processed results of BF&DDE and GF&DDE are blurred, and the enhancement effect is a little worse. In Fig. 6(c5), the result of the new algorithm looks clearer and more realistic and is closer to the eight-bit grayscale image of the original infrared image.

In Fig. 7, we select two sets of infrared images of different scenes, one set contains complex areas of buildings, pedestrians, white floors, and trees. The other set contains only a few trees, railing and chairs. From the results, we can get similar conclusions as before: as mentioned in Ref. [16], the algorithm based on guided filtering produced a halo in the edge area of the image, resulting in a poor visual effect overall. After processed by the DYHE, the brightness of the image is over-enhanced overall and leads to the loss of details. For algorithms based on bilateral filtering, it needs to be improved in detail enhancement for image, the performance is not as well as the proposed method.

Fig. 7. Two sets of images are processed by four algorithms. The first row and the second row are Image 4 and Image 5, respectively.(d1)eight-bit grayscale image;(d2)DYHE;(d3)BF&DDE;(d4)GF&DDE;(d5)the proposed algorithm. (e1)eight-bit grayscale image;(e2)DYHE;(e3)BF&DDE;(e4)GF&DDE;(e5)the proposed algorithm.

Download Full Size | PDF

Fig. 8. The results of four algorithms (ten input images).

Download Full Size | PDF

Compared with other reference algorithms, our approach has a better performance in terms of details and avoiding halo. Intuitively speaking, our method can display details of the image more clearly while avoiding the halo phenomenon at the edge. The original infrared image is so dark and blurry that we can't observe any detail information. After processed by the new algorithm, the result of the 16-bit infrared image has higher contrast. The image details are enhanced significantly and the overall brightness has been improved properly.

To further evaluate the performance of the proposed algorithm, there are a sample of twenty images selected from one hundred infrared images in Fig. 8 and Fig. 9. The number of input images in Fig. 8 is ten. The number of input images in Fig. 9 is the same as that of Fig. 8. In Fig. 8, from the first line to the fourth line, are the results of histogram equalization, GF&DDE, BF&DDE and the proposed method, respectively. The parameters of the reference algorithm and our method are the same as those in Fig. 4. The Fig. 9 is similarly.

Fig. 9. The result of four algorithms (ten input images).

Download Full Size | PDF

3.3 Objective evaluation index comparison

In order to evaluate the effect of the algorithm in this paper more objectively, a wide range of evaluation criteria, the Root-Mean-Square Contrast (RMSC) index, is selected to evaluate the degree of image denoising and enhancement. The value indicates that the proposed algorithm effectively increases the contrast of infrared images to a certain extent. The larger the value of RMSC, the higher the contrast of the image will be. The formula of RMSC is as follows:

(16)$$RMSC = \sqrt {\frac{1}{{MN}}\mathop \sum \nolimits_{i = 0}^M \mathop \sum \nolimits_{j = 0}^N {{(I\left( {i,j} \right) - \bar{I}\left( {i,j} \right))}^2}}$$

where $\bar{I}$ is the average intensity of all pixel values of the experiment image. M is the image’s height, and N is the width.

As shown in Fig. 10, through five sets of experiments, the same infrared image is processed by four algorithms to detect their RMSC, and the experimental data in Fig. 10 is obtained. The result of Fig. 10 shows that the value of the new algorithm is the highest, that is, compared with other algorithms, the proposed method can improve the contrast of infrared images better. We conducted experiments on the running time of each algorithm. The running time of each algorithm is shown in Fig. 11. It can be concluded from Fig. 11 that the running time of the new algorithm is the fastest among the four algorithms. Compared to other methods based on image stratification (BF&DDE and GF&DDE), the running times of the proposed method are significantly reduced, with a mean value of around 0.2 seconds for five infrared images. In contrast, the running time of BF&DDE and GF&DDE is around 4 seconds and 1.8 seconds, respectively. The new algorithm requires a computationally comparable time, so this algorithm is suitable to be applied in real-time, which is also a huge advantage of the new algorithm which is expected to be promoted to the industrial field.

Fig. 10. Comparison of RMSC of different algorithm results

Download Full Size | PDF

Fig. 11. Comparison of running time of different algorithm results(s)

Download Full Size | PDF

For a more comprehensive analysis, SSIM (Image Structure Similarity) is introduced in this paper, which is an indicator to measure the similarity of two images. The five reference images are used for the SSIM computation in Table 2 are the eight-bit grayscale image of Image1-Image5 in Fig. 4-Fig. 7.

Table 2. SSIM Results of Different Algorithms

View Table | View all tables in this article

SSIM can evaluate the image quality after compression, so SSIM is widely used as a quantitative evaluation index of the infrared image algorithm. The larger its value is, the better the image quality is. As can be seen from the experimental data in Table 2, from Image 1 to Image 5, the value of the algorithm's SSIM in this paper is slightly higher than that of the other three classical algorithms, and the algorithm in this paper also performs well in terms of quantitative indicators. Table 3 shows the mean opinion score (MOS) [36] for different methods. MOS is a subjective assessment index. In this experiment, the enhancement results are presented with the original images to 10 observers having image processing knowledge, and scores with confidence intervals are provided for the truthfulness, target detail, artificiality and contrast. Finally, we adopted the average of four evaluation criteria (truthfulness, target detail, artificiality and contrast) as the MOS results. The possible MOS scores range from 1 to 5, indicating very bad, bad, ordinary, good, and excellent, respectively. The test images are from Image1 to Image5 in Fig. 4–Fig. 7. It is shown clearly that the proposed algorithm outperforms the other algorithms in Table 3. That is to say, the visual of the infrared images processed by our method is improved over the other classical methods in section 3.1.

Table 3. MOS Results of Different Algorithms

View Table | View all tables in this article

3.4 Ablation study

To show the effect of two main innovations in our algorithm, ablation experiments are performed to evaluate the effectiveness of ROGABF and the weight coefficient of the detail layer. Similarly, this section contains two parts. The first part demonstrate the visual effect of the different components for our algorithm. The second part is two quantitatively indexes. In the second part, we choose the RMSC and the SSIM to evaluate the performance of different components in our algorithm.

In Fig. 12, we select five infrared images of Section 3.1. From the first line to the third line are the results of the proposed method, without ROGABF and without the weight coefficient of the detail layer, respectively. The visual effect of the proposed method is the best. The visual effect of the image is greatly affected without ROGABF or without the weight coefficient of the detail layer. Which will lead to blurred image details and loss of image details. Such as the results of IMA 1 and IMA3 in the third line, the observer cannot see any image detail.

Fig. 12. The results of different components for our algorithm.

Download Full Size | PDF

To evaluate the effect of two different components in our algorithm objectively, RMSC and SSIM are considered. The input images of Table 4 are the five images of Fig. 12. The values of RMSC and SSIM are the average of five images. As described in Section 3.3, the RMSC can indicate the contrast of image and the SSIM can evaluate the image quality. In Table 4, the values of RMSC and SSIM for our approach are much higher than the other two.

Table 4. The Values of Quantitative Evaluation for Different Components

View Table | View all tables in this article

Infrared images are often 16-bit or more level, but observer cannot distinguish more than 128 levels of gray (7-bit) in an image [39]. The original 16-bit infrared image is very dark as IMA 3 in third line of Fig. 12. Hence the proper method is applied to the infrared image processing. As shown in Fig. 1, the infrared image is filtered by the ROGABF, we can obtain the base layer. If we removal the ROGABF, we can only acquire the enhanced detail layer in the second line of Fig. 12. Similarly, as described above in Fig. 1, the detail layer contains weak details. If the detail layer is not enhanced (without the weight coefficient of the detail layer), people can only observe the image without the enhanced detail.

Table 5 is a notation-list to explain the meanings of mathematical symbols used in this paper.

Table 5. The meanings of mathematical symbols

View Table | View all tables in this article

4. Conclusions

To overcome the shortcomings of the classical algorithms mentioned above, this paper proposes a new infrared image enhancement algorithm based on the relativity of Gaussian-adaptive bilateral filter. The method combined GABF and edge-preserving image smoothing. We proposed a new method to enhance the detail layer effectively. The experimental results show that the proposed method can effectively enhance the image details and improve the overall contrast. The proposed algorithm is superior to the reference algorithm in visual and objective indicators. Since the infrared image detail enhancement technology described in this paper has a significant impact on the target detection and recognition of thermal imaging systems, this technology will gradually move to various application fields of thermal imaging technology.

Funding

Guangdong Province Science and Technology Project (2016B090918071); Natural Science Foundation of Guangdong Province (2019A1515010793); Guangzhou Science and Technology Project (201904010107).

Disclosures

No conflicts.

Data availability

The Dataset 1 used in this work is available in Ref. [37]. The Dataset 2 used in this work is available in Ref. [38].

References

1. X. Hu, X. Wang, X. Yang, D. Wang, P. Zhang, and Y. Xiao, “An infrared target intrusion detection method based on feature fusion and enhancement,” Def. Technol. 16(3), 737–746 (2020). [CrossRef]

2. X. Kong, L. Liu, Y. Qian, and Y. Wang, “Infrared and visible image fusion using structure-transferring fusion method,” Infrared Phys. Technol. 98, 161–173 (2019). [CrossRef]

3. M. K. Dougherty, “Saturn in the infrared,” Nat. Astron. 1(9), 579 (2017). [CrossRef]

4. H. I. Ashiba, H. M. Mansour, H. M. Ahmed, M. F. El-Kordy, M. I. Dessouky, and F. E. A. El-Samie, “Enhancement of Infrared Images Based on Efficient Histogram Processing,” Wirel. Pers. Commun. 99(2), 619–636 (2018). [CrossRef]

5. Y. Zhou, Y. Zhu, B. Zeng, J. Hu, H. Ouyang, and Z. Li, “Review of high dynamic range infrared image enhancement algorithms,” Laser Technol. 42(5), 1–18 (2018). [CrossRef]

6. R. C. Gonzalez and R. E. Woods, Digital Image Processing. Prenticer Hall, London (2002), 72–83.

7. K. Zuiderveld, “Contrast limited adaptive histogram equalization,” in Graphics gems IV., 474–485 (1994).

8. M. Abdullah-Al-Wadud, M. H. Kabir, M. A. Akber Dewan, and O. Chae, “A dynamic histogram equalization for image contrast enhancement,” IEEE Trans. Consum Electron. 53(2), 593–600 (2007). [CrossRef]

9. R. Lai, Y. T. Yang, B. J. Wang, and H. X. Zhou, “A quantitative measure based infrared image enhancement algorithm using plateau histogram,” Opt. Commun. 283(21), 4283–4288 (2010). [CrossRef]

10. C. L. Lin, “An approach to adaptive infrared image enhancement for long-range surveillance,” Infrared Phys. Technol. 54(2), 84–91 (2011). [CrossRef]

11. A. Polesel, G. Ramponi, and V. John Mathews, “Image enhancement via adaptive unsharp masking,” IEEE Trans. Image process. 9(3), 505–510 (2000). [CrossRef]

12. F. Branchitta, M. Diani, G. Corsini, and M. Romagnoli, “New technique for the visualization of high dynamic range infrared images,” Opt. Eng. 48(9), 096401 (2009). [CrossRef]

13. C. Zuo, Q. Chen, N. Liu, J. Ren, and X. Sui, “Display and detail enhancement for high-dynamic-range infrared images,” Opt. Eng. 50(12), 127401 (2011). [CrossRef]

14. C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Sixth International Conference on Computer Vision, 839–846 (1998).

15. K. He, J. Sun, and X. Tang, “Guided image filtering,” IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013). [CrossRef]

16. N. Liu and D. Zhao, “Detail enhancement for high-dynamic-range infrared images based on guided image filter,” Infrared Phys. Technol. 67, 138–147 (2014). [CrossRef]

17. Q. Song, Y. Wang, and K. Bai, “High dynamic range infrared images detail enhancement based on local edge preserving filter,” Infrared Phys. Technol. 77, 464–473 (2016). [CrossRef]

18. F. Zhang, W. Xie, G. Ma, and Q. Qin, “High dynamic range compression and detail enhancement of infrared images in the gradient domain,” Infrared Phys. Technol. 67, 441–454 (2014). [CrossRef]

19. Raanan Fattal, Dani Lischinski, and Michael Werman, “Gradient domain high dynamic range compression,” in Proceedings of the 29th annual conference on Computer graphics and interactive techniques. (2002), 249–256.

20. J. Xie and N. Liu, “Detail enhancement of infrared image based on BEEPS,” In Second Target Recognition and Artificial Intelligence Summit Forum (Vol. 11427). International Society for Optics and Photonics (2020), 1142730–1142744.

21. F. Chen, J. Zhang, J. Cai, T. Xu, G. Lu, and X. Peng, “Infrared Image Adaptive Enhancement Guided by Energy of Gradient Transformation and Multiscale Image Fusion,” Appl. Sci. 10(18), 6262 (2020). [CrossRef]

22. Y. Xiao and Z. Zijie, “Infrared image extraction algorithm based on adaptive growth immune field,” Neural Process Lett. 51(3), 2575–2587 (2020). [CrossRef]

23. R. Nie, C. Ma, J. Cao, H. Ding, and D. Zhou, “A Total Variation with Joint Norms for Infrared and Visible Image Fusion,” IEEE Trans. Multimedia 99, 1 (2021). [CrossRef]

24. Y. Huang, Z. Jiang, R. Lan, S. Zhang, and K. Pi, “Infrared Image Super-Resolution via Transfer Learning and PSRGAN,” IEEE Signal Process. Lett. 28, 982–986 (2021). [CrossRef]

25. F. Lv, Y. Li, and F. Lu, “Attention guided low-light image enhancement with a large scale low-light simulation dataset,” Int. J. Comput. Vis. 129(7), 2175–2193 (2021). [CrossRef]

26. S. Hao, X. Han, Y. Guo, X. Xu, and M. Wang, “Low-light image enhancement with semi-decoupled decomposition,” IEEE Trans. Multimedia. 22(12), 3025–3038 (2020). [CrossRef]

27. B. H. Chen, Y. S. Tseng, and J. L. Yin, “Gaussian-adaptive bilateral filter,” IEEE Signal Process. Lett. 27, 1670–1674 (2020). [CrossRef]

28. D. G. Lowe, “Distinctive image features from scale-invariant key points,” Int. J. Comput. Vis. 60(2), 91–110 (2004). [CrossRef]

29. E. J. Candes, M. B. Wakin, and S. P. Boyd, “Enhancing sparsity by reweighted 11 minimization,” J. Fourier Anal. Appl. 14(5-6), 877–905 (2008). [CrossRef]

30. D. Krishnan, R. Fattal, and R. Szeliski, “Efficient preconditioning of laplacian matrices for computer graphics,” ACM Trans. Graph. 32(4), 1–15 (2013). [CrossRef]

31. Richard Barrett, Michael Berry, Tony F Chan, James Demmel, June Donato, Jack Dongarra, Victor Eijkhout, Roldan Pozo, Charles Romine, and Henk Van der Vorst, Templates for the solution of linear systems: building blocks for iterative methods, SIAM, 1994.

32. A. K. Katsaggelos, J. Biemond, R. W. Schafer, and R. M. Mersereau, “A regularized iterative image restoration algorithm,” IEEE Trans. Signal Process. 39(4), 914–929 (1991). [CrossRef]

33. G. Deng and L. W. Cahill, “An adaptive Gaussian filter for noise reduction and edge detection,” in 1993 IEEE Conference Record Nuclear Science Symposium and Medical Imaging Conference (1993), 1615–1619.

34. E. Peli, “Contrast in complex images,” J. Opt. Soc. Am. A 7(10), 2032–2040 (1990). [CrossRef]

35. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13(4), 600–612 (2004). [CrossRef]

36. Y. Chen, J. U. Kang, G. Zhang, J. Cao, Q. Xie, and C. Kwan, “Real-time infrared image detail enhancement based on fast guided image filter and plateau equalization,” Appl. Opt. 59(21), 6407–6416 (2020). [CrossRef]

37. FLIR. “Thermal Infrared Dataset,” figshare (2014), https://projects.asl.ethz.ch/datasets/doku.php?id=ir:iricra2014.

38. Jim Gao, “ROGABF Dataset,” figshare (2021), https://github.com/hopeprincess/helloword.

39. J. Silverman, “Display and enhancement of infrared images,” in International Conference on Image Processing and its Applications, pp. 345–348 (1992).

Algorithms	Image 1	Image 2	Image 3	Image 4	Image 5
DYHE	0.6294	0.8701	0.6889	0.8659	0.7999
BF&DDE	0.833	0.8671	0.8961	0.8954	0.9139
GF&DDE	0.7526	0.7998	0.8745	0.8887	0.9135
Proposed	0.9089	0.9016	0.9124	0.9008	0.8206

Algorithms	Image 1	Image 2	Image 3	Image 4	Image 5
DYHE	3.1	4.3	3.4	4.3	3.9
BF&DDE	4.2	4.3	4.4	4.2	4.3
GF&DDE	3.8	4.0	4.3	4.4	4.1
Proposed	4.6	4.5	4.6	4.7	4.6

Condition	RMSC	SSIM
Proposed	86.6549	0.8889
Without ROGABF	37.6001	0.2255
Without w(i,j)	39.2529	0.0421

Mathematical Symbols	Meanings
I	The input image
g	Where g is obtained by Gaussian blur process on I
ki	A normalizing factor in Eq. (2)
σs	Denote the sizes of the window for extension of the neighborhood.
*g^-*	Obtained by Eq. (1)
σr	Denote the variation in intensities of the amplitude of an edge
GABF	Gaussian-adaptive bilateral filter
ROGABF	The Relativity-of-Gaussian-adaptive bilateral filter
ROF_GAB	The Relativity-of-Gaussian-adaptive bilateral filter
T	The smoothing image
$\nabla T_{x, y}$	The gradient of T in the x or y direction
$σ$	An adjustable parameter, which belongs to the local Gaussian spatial kernel of Eq. (2)
F_GAB	The Gaussian-adaptive bilateral filter
F	The input image in Eq. (6)
R	Local regularization
u	A parameter larger than zero
$∥ T - F ∥_{2}^{2}$	An L2-norm fidelity term
arg min f(x)	When f(x) is the minimum, the value of x
$F_{G A B}^{σ 1}$	The local Gaussian spatial kernel of Gaussian-adaptive bilateral filter is $σ 1$ ; $F_{G A B}^{σ 1}$ is defined analogously;
$\| \| f (x) {\| \|}_{1}$	Which is the L1-norm
ε	The value is set as 0.001
T	The vector forms of T; F is defined analogously;
*Wx,y*	Which contains a diagonal matrix of the weight W_x,y
*D_x and D_y*	They are discrete differential operators
*T^k*	The k-th iteration for vector T
E	Which is an identity matrix in Eq. (10)
*L^k*	Which is a sparse five-point Laplacian matrix [26] in Eq. (11)
a	An adjustable parameter in Eq. (12)
M₀(x,y)	Which represents the local variance of the image to be processed.
w₁	Which is the window size of the Gaussian low-pass filter in Eq. (13)
σ	Which is the variance of the Gaussian function
A, B and D	They are all adjustable parameters
V₀ (i, j)	Which is the local variance of the image
n_x	Which represents the number of pixel values that are also x
T	Which is a threshold in Eq. (14)
n_valid	Which represents the number of effective gray value
RMSC	The Root-Mean-Square Contrast index
$I (i, j)$	Which is the pixel values of the experiment image
$\bar{I}$	Which is the average intensity of all pixel values of the experiment image

Algorithms	Image 1	Image 2	Image 3	Image 4	Image 5
DYHE	0.6294	0.8701	0.6889	0.8659	0.7999
BF&DDE	0.833	0.8671	0.8961	0.8954	0.9139
GF&DDE	0.7526	0.7998	0.8745	0.8887	0.9135
Proposed	0.9089	0.9016	0.9124	0.9008	0.8206

Detail enhancement for infrared images based on Relativity of Gaussian-Adaptive Bilateral Filter

Abstract

1. Introduction

2. Principle of our research

2.1 Relativity-of-Gaussian-Adaptive Bilateral Filter

2.2 Weight coefficient of the detail layer

2.3 Histogram projection of the base layer

3. Experimental results

3.1 Dataset

3.2 Visual effect comparison

3.3 Objective evaluation index comparison

3.4 Ablation study

4. Conclusions

Funding

Disclosures

Data availability

References

Supplementary Material (2)

Data availability

Cited By

Figures (12)

Tables (5)

Equations (16)

OSA Continuum

Algorithm of ROF _GAB
Input: original image F, adjustable scale parameter $σ 1, σ 2$ , positive parameter u, and maximum iteration K. The parameter k_i, $σ r$ of GABF.
Output: The result of processed image T.
step 1:initialize $T^{0} \leftarrow F$
step 2:for k=1 to K do
step 3:compute the weights $w_{x, y}$ in Eq. (8)
step 4:update $T^{k}$ using Eq. (10)

Algorithm of ROF _GAB
Input: original image F, adjustable scale parameter $σ 1, σ 2$ , positive parameter u, and maximum iteration K. The parameter k_i, $σ r$ of GABF.
Output: The result of processed image T.
step 1:initialize $T^{0} \leftarrow F$
step 2:for k=1 to K do
step 3:compute the weights $w_{x, y}$ in Eq. (8)
step 4:update $T^{k}$ using Eq. (10)