Effective estimation of background light in underwater image dehazing

Miao Yang; Bo Yin; Zhiqiang Wei; Yixiang Du; Jintong Hu

doi:10.1364/OSAC.2.000767

1. Introduction

The optical imaging configurations commonly used in the underwater field include cameras as well as devices such as a range-gated and synchronous scanning laser, coherent modulation and demodulation devices, structured light devices, polarization-based light devices, and non-uniform light field underwater imaging devices [1,2]. Although these devices have improved the range and clarity of the underwater imaging, every imaging system has limitations or operational requirements, e.g., bulky load, low incidence power to the observation plane, large blind zone, narrow field of view, limited depth of operation, and complex operation process. At present, vision as a scientific exploration measure is becoming more and more indispensable in a marine survey. The absorption and scattering dominated by the inherent optical properties (IOPs) of the water disturb underwater imaging [3] and cause degradations when the light is traveling through the water. Namely, the forward scattering leads to blurring, and the backscattering generally reduces the image contrast, which results in an image covered by a foggy veil. Light attenuation causes the color disappeared gradually. Besides, the concentrations of the plankton, color dissolved organic matter (cDom), and total suspended matter (TSM), and distance of a target to the camera also influence the quality of underwater images [4]. Artificial lights can be used to extend the visual range, which often results in inhomogeneous lighting that manifests a bright spot in the image center surrounded by a dark area; even worse, the artificial lighting makes the scattering caused by the suspended particles aggravate. Moreover, waves, swirls, and silt also produce irregular blurring in underwater images. Therefore, the restoration or enhancement of the underwater images is necessary.

The enhancement of underwater images has been studied in several aspects such as the image contrast [5, 6], clarity [7–9] and color compensation [10–16]. For a single image, Iqbal et al. [11] proposed the integrated color model (ICM) and the unsupervised color-correction method (UCM) [17]. Fu et al. [18] built a variational retinex model to enhance the illumination of underwater images. Chiang and Chen [19] restored the underwater images by combining the dark channel prior (DCP) based dehazing with the wavelength compensation. Ancuti et al. [20] enhanced the visual quality of underwater images using the fusion principles; further, they proposed an improved white balance by compensating the loss of red channel before resorting to the conventional Gray-World method [21]. The Gamma correction and a high-pass filter were adopted in the fusing process to get the final output. Since deep learning has shown remarkable performance in high-level attribute representation by constructing multi-hidden layer model and massive training data, some techniques [22,23] were used to rectify the colour distortion in underwater images. Li et al. [23] proposed a weak supervised color transfer method based on CycleGAN [24] to correct the color distortion of underwater images, which learns a cross domain mapping function between underwater images and air images. For the generators of forward and backward networks, the adversarial loss, cycle consistency loss and structural similarity index measure (SSIM) loss functions are included. Besides the complexity of the calculations, the clarity of input image are unrestored.

He et al. [25] put forward the DCP and pointed out that pixels of at least one of the three color channels in the fog-free natural images have low values, and the increased grayscales of pixels in the dark channel of a hazy image can be attributed to the haze. The background light estimation is a crucial step in the DCP based image dehazing. Fog can be removed by subtracting the background light from the degraded image. In recent years, the underwater image restoration based on the DCP has received extensive attention [26–31]. Galdran et al. [26] proposed the red channel prior method and estimated the background light from the pixel with the highest intensity in the red channel. Carlevaris-Bianco et al. [27] computed the dark channel image by the difference between the maximal value in the red channel and the maximal value in the green-blue channel, and the background light was estimated as the highest intensity of the dark channel. Peng et al. [28] proposed a transmission estimation by using the information on image blurriness, selected a single background light from the blurry regions in the underwater images [29], and proposed a transmission map estimation by applying the illumination differential with a known background light [30]. Li et al. [31] used the quad-tree subdivision and graph-based segmentation to obtain a single value of the global background light and estimated the medium transmission map by the minimum information loss principle. The mentioned DCP based methods generalized the background light conception in the air directly to the water media, while the background light in the underwater image should be related to the backscattering. In addition, these dehazing methods take the brightest pixel in dark channel as a background light, which can result in an erroneous estimation when a scene contains white or bright targets.

The backscattering light in the water manifests a non-uniform characteristic due to the complicated light-particle interactions. The concentration and composition of suspended particles in the water with a different type, current and depth, and in different season and weather conditions, make the distortions in underwater images to manifest characteristics of localization. Besides, the background light exhibits directionality characteristic for different imaging angle. A non-uniform background light has been considered by several underwater dehazing methods [20,32], but all these methods consider the maxima of the dark channel as a background light.

In this paper, the theoretical analysis of the distribution and the scale selection of background light in the underwater images are provided. The contributions of this work are as follows: (i) a new standpoint, which explains that assuming the 0.1% maxima of the dark channel as a background light (as it is commonly used in the dehazing methods) does not correspond to the real background light in underwater image, is introduced; (ii) according to the statistical analysis of the sub-image in the collected underwater images, the local scale in the underwater background is mainly below 20% of the image size, and concentrate around 5%; (iii) a machine learning based background light estimation (MLBE) and reconstruction method is proposed based on artificial neural network (ANN), and it is applied to predict whether the maximum region of the dark channel belongs to the background area of the underwater image or not. The estimation accuracy of the background light of 91.0703% is achieved. Also, by using the proposed MLBE during the background light estimation process, the DCP based underwater image restoration can be improved in the dark area, sharpness and color restorations.

The rest of the paper is organized as follows. The underwater imaging model and related theory are presented in Section 2. In Section 3, the statistical analysis of underwater background light is introduced, and the contradiction of the traditional background light estimation methods in the DCP based underwater image restoration is pointed out. In Section 4, the framework of the proposed machine learning based background light estimation (MLBE) is described and the transmission map estimation is summarized. The experimental and comparative results are provided and discussed in Section 5. Lastly, the conclusions are given in Section 6.

2. Underwater imaging optical model

2.1 Jaffe-McGlamery model

According to the Jaffe-McGlamery underwater imaging model [33], the total radiance of an image that reaches an observer consists of three additive components: direct component E_d, forward-scattering component E_f, and backscattering component E_b, as shown in Fig. 1.

Fig. 1. The underwater imaging model.

Download Full Size | PDF

E_d represents the attenuated version of reflected light, and is expressed as:

(1)$${E_d} = {J_c}(x){e^{ - {p_\lambda }d(x)}}, $$

where J_c(x) is the radiance of the object, c={R, G, B} denotes the colour channel of an RGB image, d(x) is the distance between the observer and object, and ${p_\lambda }$ is the attenuation coefficient. In the underwater environment, ${p_\lambda }$ denotes the sum of the absorption coefficient ${a_\lambda }$ and scattering coefficient ${b_\lambda }$, of which both are wavelength $\lambda$ dependent, which can be expressed as:

(2)$${p_\lambda } = {a_\lambda } + {b_\lambda }.$$

The exponential term ${e^{ - {p_\lambda }d(x)}}$ is also known as the transmission map t_c(x) through the water medium:

(3)$${t_c}(x) = {e^{ - {p_\lambda }d(x)}}.$$

For offshore water (depth <20 m), E_b can be expressed as:

(4)$${E_b}(x) = B_\infty ^c(x)(1 - {e^{ - {p_\lambda }d(x)}}),$$

where $B_\infty ^c(x)$ is known as the backscattering light of c={R,G,B} channel or the background light at pixel x. In general, E_f is associated with a small fraction of the overall image degradation process, and the simplified underwater optical imaging model employed in most existing dehazing techniques becomes:

(5)$${I_c}(x) = {J_c}(x){t_c}(x) + (1 - {t_c}(x))B_\infty ^c(x).$$

By inserting a raw image I_c(x), $B_\infty ^c(x)$ and t_c(x) into Eq. (5), the restored underwater image J_c(x) can be obtained from:

(6)$${J_c}(x) = \frac{{{I_c}(x) - B_\infty ^c(x)}}{{{t_c}(x)}} + B_\infty ^c(x).$$

2.2 Dark channel prior

According to the DCP [25], the dark channel L_DC of an underwater color image I_c(x) can be derived from the minimization of a z×z local block centered at x, Ω(x) in the R, G and B channels, that is:

(7)$${L_{DC}} = {\min\nolimits_{y \in \Omega (x)}}({\min\nolimits_{c \in R,G,B}}{I_c}(y)),$$

Applying the dark channel prior [25] to the original image J_c(x), we have:

(8)$${\min\nolimits_{y \in \Omega (x)}}({\min\nolimits_{c \in R,G,B}}{J_c}(y)) = 0.$$

By applying the minimum operators given by Eqs. (7) and (8) to Eq. (5), the dark channel L_DC of an underwater color image I_c(x) becomes:

(9)$${L_{DC}} = \tilde{B}_\infty ^c(x)(1 - {\tilde{t}_c}(x)),$$

where $\widetilde B_\infty ^c(x)$ and ${\tilde{t}_c}(x)$ are the estimated background light and transmission map respectively, as already specified.

2.3 Background light estimation

He et al. proposed to use the 0.1% brightest pixels of the dark channel as the global background light [25]. Although the dark channels used in the underwater image restoration or denoising have been constructed in various ways [25–31], in the majority of them the maximum of L_DC is still used as an estimated background lighting, e.g., Li et al. [31] proposed a graph-based segmentation to obtain a global background light and estimated the medium transmission map by the minimum information loss principle. The score of each region was computed by the absolute difference between the regional average and standard deviation, and a flat region B_s with the highest score was extracted, and then segmented to remove the influence of the target. Finally, the pixel with the maximum blue-red difference in the 0.1% of the brightest pixels of the flat region B_s was selected as a background light pixel.

(10)$$\widetilde B_\infty ^c(x) = {I_c}(\mathop {\arg \max }\limits_{_{x \in {B_s}_{0.1\%}}} |{I_R}(x) -{I_B}(x)|).$$

Galdran et al. [26] proposed the red channel hypothesis in which the background light $\widetilde B_\infty ^c$ was estimated by choosing the brightest pixel in the red channel as follows:

(11)$$\tilde{B}_\infty ^c(x) = ({I_{^R}}({x_0}),{I_{^G}}({x_0}),{I_{^B}}({x_0}))\quad {I_{^R}}({x_0}) \ge {I_{^R}}(x)\,\forall x.$$

The method proposed by Carlevaris-Bianco et al. [27] estimates the transmission map ${\tilde{t}_c}(x)$ from the maximum difference between the blue-green and red channels and uses the minimum of the transmission map as a background light, which is given by:

(12)$$\tilde{B}_\infty ^c(x) = {I_{^c}}(\mathop {\arg \min }\limits_x {\tilde{t}_c}(x)).$$

Peng et al. [28] estimated the global background light image by averaging the 0.1% brightest pixels in L_DC:

(13)$$\tilde{B}_\infty ^c(x) = \frac{1}{{|{{S_{0.1}}} |}}\sum\limits_{x \in {S_{0.1}}} {{I_{^c}}(x)},$$

where ${S_{0.1}}$ represents the 0.1% brightest pixels in L_DC. Ancuti et al. [20] applied the local maxima in the dark channel and estimated the background light by:

(14)$$\widetilde B_\infty ^c(x) = \mathop {\max }\limits_{y \in M_{DC}^{\Psi (x)}} \left( {\mathop {\min }\limits_{z \in \Omega (y)} {I_{^c}}(z)} \right),$$

where $M_{DC}^{\Psi (x)}$ is the pixel in $\Psi (x)$, and $\Psi (x)$ is the local window in L_DC centered at x, the maximum of $\Psi (x)$ is y, and z represents the pixel in $\Omega (y)$.

Whereas, the underwater images are degraded by the comprehensive light attenuation due to the IOPs of pure water, the solute in seawater (molecular scattering and absorption), and suspended particles in the water in general (particle scattering and absorption). As the imaging effects also depend on the distance of the objects, the degradations are local and cannot be corrected by global operations. The background light is generated by scattering of the ambient light from a large range of angles. The related research has shown that the light rays traveling in the underwater environment encountered the beam-particle interaction with different random times [34]. A single background light value over the entire image fails to explain the real interactions of light rays and particles in the water medium. On the other hand, since the 3-D space should be sliced into planes to calculate the backscattering irradiance of every small plane in different directions and distances relative to the camera, the background light can be regarded as a superposition of many point-sources of the light in the space, which produces a non-uniform image intensity.

3. Contradiction with classical DCP

3.1 Directionality of backgound light in offshore underwater images

Often, it is assumed that the background light is uniformly distributed to the target at the same distance from infinity in natural image defogging. However, not only the natural lighting from the water surface but also the artificial lighting is commonly used in underwater imaging. The background light in the underwater images shows directional characteristics due to the angles of photography and irradiance distributions of the artificial light.

In the offshore area, the concentrations of cDOM and TSM are higher than in the far-shore area because of the impact of terrigenous input and sediment resuspension [35–36]. Therefore, the light propagation incurs strong interaction with random particles, which means that the backscattering is one of the most important factors in degradation of offshore underwater images.

Considered that the current dehazing methods mainly aim to the underwater images taken under shallow water, we collected 300 daytime offshore (depth < 20 m) underwater images including the sea farming, sea creature investigation, and coastal underwater engineering, and some of them are given in Section 5. The water (body) background areas were interactively selected to constitute a collection of the offshore underwater background images. To illustrate the directionality of the background light in the offshore underwater images, the linear fitting angles of column cumulative and average brightness distributions of the background images were analyzed, as shown in Fig. 2. In Fig. 2, it can be seen that the average brightness distribution of underwater background images had obvious directionality. The fitted angles of the underwater background light were all between -45° and 45°, which was correlated with the viewing angle and the relative position of lighting sources. Therefore, assuming a single global background light value as some of the DCP based underwater restorations was not in accordance with the actual situation.

Fig. 2. Fitted directional angles of the brightness of the background image: (a) column average intensity, (b) column cumulative intensity.

Download Full Size | PDF

3.2 Relationship between background light and dark channel for underwater images

The early dehazing methods for natural images take the brightest pixel in a scene as a background light [37,38]. However, such an approach can result in an erroneous estimation when a scene contains white or bright targets. Some hierarchical veiling light estimation methods for natural image dehazing [39–43] demonstrated the improved performance in the case of terrestrial scenes containing bright objects.

In order to illustrate the relationship between the pixel values of background light and dark channel in underwater environments, the following experiments were conducted. For 300 offshore underwater color images, the typical background pixels in each image were manually selected, and compared with the pixels in the maximum areas of the dark channel. The distribution of the ratios of dark channel maximum area belonging to the true background in 300 underwater images is shown in Fig. 3(a), and it can be seen that the extreme areas of the dark channel mostly belonged to the background lights of the underwater images. In Fig. 3(b), the histogram of the pixel values in the maximum area of the dark channel divided by the global maximum of the dark channel in 300 underwater images is presented. We noticed that the pixels in the maximum areas were not equal to the brightest value of the dark channel, as presented in Fig. 3(b); their ratios to the global maxima of the dark channel were distributed between 0 and 1, instead of the first 0.1% or 10% maxima of the dark channel images or the minima of the transmission maps, which was adopted in the classical dark channel prior theory. Since each underwater image was different in content, the proportions of pixels in the maximum areas of the dark channel to the true background were also different.

Fig. 3. The relationship between the background pixels and dark channel maximum area of the offshore underwater images. (a) The histogram of the ratios of the maximum area in the dark channel belonging to the true background light in 300 underwater images. (b) The histogram of the pixel values in the maximum area of the dark channel divided by the global maxima of the dark channel in 300 underwater images.

Download Full Size | PDF

3.3 Background intensity localization

To verify the real distribution of underwater environment lighting, the intensity of ambient light obtained of different areas on a flat target surface (the checkerboard with the size of 110 cm × 182 cm, containing 10 × 7 patches) placed in a tank was measured as shown in Fig. 4. The tank had the length of 2 m, the width of 1.2 m, and the height of 1 m. Every test was repeated for five times under the conditions of clear, median muddy, and turbid water [44], respectively. The average illuminance of each block was computed, and the results are shown in Fig. 5. The centers of the target board in Figs. 5(a)-(c) were 55 cm, 53 cm, and 53 cm from the water surface, respectively. The intensity of illuminance on the water surface in Figs. 5(a)-(c) was 344-390 lx, 177-196 lx, and 63-78 lx, respectively. As Fig. 5 illustrates, the underwater lighting illuminance were not uniform along the image plane but related to the position of the light source and water body.

Fig. 4. Illuminance measurement.

Download Full Size | PDF

Fig. 5. Illuminance distribution of the target panel under natural lighting conditions.

Download Full Size | PDF

Furthermore, the brightness distribution of the background areas in real underwater images is shown in Fig. 6. In Figs. 6(a)-(c), the background patches of three underwater images are presented. The distributions of R component of these patches in 3D and plan views are fitted in Figs. 6(d)-(i). In addition to the non-uniformity brightness presented in Figs. 6(d)-(i), due to the sea snow, the bright light caused by particle scattering existed in the background images and exhibited the local Gaussian-like pattern, as shown in Figs. 6(d)-(f).

Fig. 6. The R-channel luminance distribution of the background areas in underwater images. (a)-(c) The original underwater background images, (d)-(f) the fitted R-channel luminance in the 3D view, (g)-(i) the fitted R-channel luminance in the plan view.

Download Full Size | PDF

Therefore, by estimating the background light locally in the restoration of underwater images, the under- or over-saturation could be avoided while recovering the underwater images with better clarity. The statistics on the scale of local distribution of the middle row for 300 offshore underwater background images was also carried out, and the results are shown in Fig. 7. Based on the results presented in Fig. 7, it can be concluded that local scales of the underwater background image intensity were mainly below 20% of the image size and it was about 5%.

Fig. 7. Luminance distributions of the middle row in R, G, and B channels of the offshore underwater background images.

Download Full Size | PDF

4. Machine learning based background light estimation

4.1 ANN based background light estimation

In this work, we propose an ANN-based model to distinguish the maximum area of the dark image channel which originates from a real water background, as shown in Fig. 8. The proposed model was trained with 300 real offshore underwater images. In order to obtain the best performance, a three-layer back propagation (BP) neural network with the tangent-sigmoid activation function in all hidden layers was used, and both input and output values were normalized on the scale of 0 to 1. The background light was estimated by the Gaussian guided morphological reconstruction based on the nearest neighbor interpolation. The training data was obtained by labeling the true background pixels in the underwater images interactively.

Fig. 8. The structure of the ANN model for the background light recognition.

Download Full Size | PDF

Let M_DCi be the maximum area i in the dark channel image L_DC of an underwater image. Thus, when the pixels included in M_DCi were close to the true water background, the label y(i) of M_DCi was equal to 1. As the input used in training, six different features from each maximum area M_DCi were extracted:

(15)$$x_{DC}^{(i)} = \{ R_{DC}^{(i)},G_{DC}^{(i)},B_{DC}^{(i)},V_{RGB}^{(i)},V_{DC}^{(i)},L_{DC}^{(i)}\}. $$

In Eq. (15), $R_{DC}^{(i)},G_{DC}^{(i)},B_{DC}^{(i)},V_{RGB}^{(i)}$ denote the normalized color means and the standard color deviation of pixels in M_DCi, $V_{DC}^{(i)}$ is the standard deviation of M_DCi, and $L_{DC}^{(i)}$ is the normalized pixel value in M_DCi.

The background light pixels were obtained from the nearest interpolation. The proposed machine learning based background light estimation (MLBE) for an underwater image I_c(x) can be defined as:

(16)$$\left\{ {\begin{array}{{c}} \begin{array}{l} \tilde{B}_\infty ^c(x) = {I_{^c}}(x),\;\;\quad \quad x \in M_{DC}^{{y^{(i)}} = 1}\\ \end{array}\\ {\mathop {\tilde{B}_\infty ^c(x)}\limits_{x \in N_{M_{DC}^{{y^{(i) = 1}}}}^i} = \mathop {Mean}\limits_{z \in M_{DC}^{{y^{\left( i \right)}} = 1}} \;({I_c}(z)),\quad otherwise} \end{array}} \right. ,$$

where $M_{DC}^{{y^{(i)}} = 1}$ denotes the maximum area i of the dark channel with the label value of ${y^{(i)}} = 1$, i = 1, 2, …, n, where n is the total number of maximum areas of the dark channel with the label of 1; $N_{M_{DC}^{{y^{(i)}} = 1}}^i$ denotes the pixels surrounding $M_{DC}^{{y^{(i)}} = 1}$. The Gaussian filtered R, G, and B images with a scale of 2% of the image size were applied as the guided image to reconstruct the interpolated background light image. An example of the proposed MLBE is shown in Fig. 9. The outputs of the other background light estimations are shown in Figs. 9(b)-(e). For the mist-like underwater images shown in Fig. 9(a), the MLBE achieved the background light images having the object scattering light distribution which was more similar to the real one than those obtained by the other methods were.

Fig. 9. Examples of the background light estimation. (a) The original images, (b) the images obtained by the method proposed by Galdran et al. [26], (c) the images obtained by the method proposed by Peng et al. [28], (d) the images obtained by the method proposed by Li et al. [31] and (e) the images obtained by using our MLBE method.

Download Full Size | PDF

4.2 Transmission map estimation

Assuming the background light was known, Ancuti et al. [20] used the DCP to estimate the transmission map by:

(17)$${\tilde{t}_c}(x) = 1 - \mathop {\min }\limits_{y \in \varOmega (x)} \left( {\mathop {\min }\limits_{c \in \{ R,G,B\} } \frac{{{I_c}(y)}}{{\widetilde B_\infty^c(y)}}} \right).$$

Carlevaris-Bianco et al. [27] calculated the dark channel using the difference D(x) between the maximum in the red channel and the other channels (green and blue):

(18)$$D(x) = \mathop {\max }\limits_{x \in \Omega ,c \in R} {I_c}(x) - \mathop {\max }\limits_{x \in \Omega ,c \in \{ G,B\} } {I_c}(x).$$

The transmission map was estimated by adjusting the difference until it became equal to 1, which is given by:

(19)$${\tilde{t}_c}(x) = D(x) + (1 - \mathop {\max }\limits_x D(x)).$$

The red channel prior proposed by Galdran et al. [26] exploited the reciprocal of global background light and minimization on the R, G and B channel to obtain the transmission map of the red channel:

(20)$${\tilde{t}_{^c}}(x) = 1 - \min \left( {\frac{{\mathop {\min }\limits_{y \in \Omega (x)} (1 - {I_{^R}}(y))}}{{1 - \tilde{B}_\infty^R}},\frac{{\mathop {\min }\limits_{y \in \Omega (x)} {I_{^G}}(y)}}{{\tilde{B}_\infty^G}},\frac{{\mathop {\min }\limits_{y \in \Omega (x)} {I_B}(y)}}{{\tilde{B}_\infty^B}}} \right).$$

Let I(x) be the grayscale version of I_c(x) and ${G^{k,\sigma }}$ be the input image filtered by k × k Gaussian filter with the variance ${\sigma ^2}$; Peng et al. [28] proposed a fuzzy map B(x) which is given by:

(21)$$B(x) = \frac{1}{n}\sum\limits_{i = 1}^n {({|{I(x) - {G^{{r_i},{r_i}}}(x)} |} )}. $$

In Eq. (21), ${r_i} = {2^i}n + 1$, and n = 4. The transmission map was estimated by using the local maximum of the fuzzy map:

(22)$${\tilde{t}_c}(x) = \mathop {\max }\limits_{y \in \Omega (x)} B(y).$$

Given the global background light of the red channel $\tilde{B}_\infty ^R$, Li et al. [31] estimated the transmission map of the red channel utilizing the principle of minimum information loss:

(23)$${\tilde{t}_{^R}}(x) \ge \max [\mathop {\min }\limits_{y \in \Omega (x)} \left( {\frac{{{I_{^R}}(y) - \tilde{B}_\infty^R}}{{ - \tilde{B}_\infty^R}}} \right),\mathop {\max }\limits_{y \in {B_s}} \left( {\frac{{{I_{^R}}(y) - \tilde{B}_\infty^R}}{{255 - \tilde{B}_\infty^R}}} \right)]. $$

Then, the transmission map of G and B channels was obtained by using the attenuation coefficient priors:

(24)$${\tilde{t}_{^c}}(x) = {({t^R}(x))^{{p^c}/{p^R}}},\;\;c \in \;\{ G,B\},$$

(25)$$\frac{{p{}^c}}{{{p^R}}} = \frac{{{b^c}\tilde{B}_\infty ^R}}{{{b^R}\tilde{B}_\infty ^c}}, $$

(26)$${b^c} = ( - 0.00113{\lambda _c} + 1.62517)b(555nm).$$

Generally, in the water, ${\lambda _R}$, ${\lambda _G}$, and ${\lambda _B}$ are respectively 620 nm, 540 nm, and 450 nm, and b(555 nm) is the scattering coefficient at a particular wavelength of light [45]. However, when the theoretical maximum of a background light was used as a denominator in the computing of transmission map, the over saturation would cause the artifacts in the background area due to low transmission values.

5. Experiments and discussion

The proposed approach was validated on hundreds of underwater images. The data included 300 images collected online which associated with the offshore investigation and sea farming and had the size from 194×259 to 3968×2976 pixels. Matlab ANN fitting tool was used to study the features and the corresponding labels of the maximum areas of the dark channel. The three-layer BP neural network was adopted to the classification of 131,544 maximum areas of the dark channels for 300 underwater images. The proportion of the data samples used in training, verification, and test was 70%, 15%, and 15%, respectively. The transfer functions of hidden layers were all the tangent-sigmoid function, and linear function (pureline) was used as the activation function of the output layer. The model achieved the accuracy of 91.0703% on test data.

The restoration methods were evaluated by comparing the obtained improvements when the MLBE was used. In the experiments, the block size of the dark channel and Gaussian filter was 15×15. Some of these experimental results are shown in Fig. 10. The existing techniques including that proposed by Peng et al. [28] and Li et al. [31]. It can be seen from the results that the tested methods produced better sharpness and color restoration effects by applying the proposed MLBE during the background light estimation process (the transmission map estimation remains unchanged), as shown in Fig. 10. From the visual analysis, some areas in the results recovered by [28] suffered from under-saturation, as shown in the first three pictures in Fig. 10(b). By using the MLBE, more uniform contrasts were boosted than when the original method of Peng et al. [28] was used, as shown in Fig. 10(c). As shown in Fig. 10(d), the method proposed by Li et al. [31] recovered the color casting of the underwater images but had blurred details; but, combining this method with the proposed MLBE the dark area was improved and clear detail of the objects in the testing images was obtained.

Fig. 10. Comparisons of underwater image restoration results, from the up to the down are: original images, results of cited methods and improved results with MLBE used.

Download Full Size | PDF

Two underwater images with bright targets or corrugated light existed and the restoration results of them with or without applying MLBE are given in Fig. 11. The existing dehazing methods, such as methods proposed by Peng et al. [28] and Li et al. [31], usually take the brightest pixel in the dark channel as a background light, which result in over saturation in the corner of the restored image but flare patches at the reflect light spots on the metals as shown in the second pictures of Figs. 11(a)-(b). For the corrugated light commonly visible in shallow underwater image, single background image pixel caused fake contours since the background lights supposed to be uniform in the fifth images of Figs. 11(a)-(b). However, by using the proposed MLBE in the restoration, not only the veiling effects covered the original images were relieved, but also the global contrast was improved, as shown in the pictures in the last column of Figs. 11(a)-(b).

Fig. 11. Comparisons of underwater image restoration results, from the left to the right are: original images, results of cited methods and improved results with MLBE used.

Download Full Size | PDF

The Chroma variance, contrast of brightness, and saturation of the underwater color images were proved to concur well with the observers’ perceptions [46]. These indexes, constituted an underwater color image quality evaluation (UCIQE) [46], which has been widely used to evaluate the quality of underwater color images [29,31,47]. The quantitative measurements of the images which are shown in Fig. 10 are listed in Table 1. As Table 1 shows, the UCIQE values were improved when the proposed MLBE method was applied to the background light estimation for underwater images restorations. For the second images in Fig. 10(a), the image restored by the method of Peng et al. [28] with the MLBE is shown in Fig. 10(c), and it can be seen that the highest UCIQE values was achieved. For the third picture in Fig. 10(a), the high-lighting areas in the restored images produced by the original method of Peng et al. [28] (shown in the third picture in Fig. 10(b)) resulted in the higher chroma variance and further influenced the final UCIQE values, as shown in Table 1. It can also be seen in Table 1 that the results when the proposed MLBE and the transmission map estimation based on the minimum information loss of Li et al. [31] were adopted the higher contrast gains and the best synthetic objective and subjective performances for the first and fourth images in Fig. 10(a) were achieved.

Table 1. Quantitative Measurement Results

View Table | View all tables in this article

Imatest [48] has been the most professional software for qualifying the camera imaging by comparing the difference between the real images of charts and the testing ones. To further verify the restoration capability of the proposed MLBE regarding the contrast, clarify and colourfulness objectively, the restoration effects were compared with the real color images and analyzed by presenting the quality of the restored Imatest 4.3 SFR board and ColorChecker 24 X-Rite Chart (21.59×27.94 cm) taken in a tank. The tank was 2.53 m long, 1.02 m wide, and 1.03 m high, as shown in Fig. 12. The chart images were taken with OTI-UWC-325/P/E color camera. The images (with the size of 960×576) were obtained in the water with the transparency of 325 cm [44] under natural lighting at the distance of 70 cm and 110 cm from the camera with the LED lighting, respectively. The restored images are illustrated in Figs. 13 and 14.

Fig. 12. Tank and targets.

Download Full Size | PDF

Fig. 13. Comparisons of underwater ColorChecker chart restorations results, from the left to the right are: original images, results of original methods and improved results with MLBE used.

Download Full Size | PDF

Fig. 14. Comparisons of underwater Imatest SFRplus restorations results, from the left to the right are: original images, results of original methods and improved results with MLBE used.

Download Full Size | PDF

The relative contrast at a given spatial frequency (output contrast/input contrast) is called the modulation transfer function (MTF). The spatial frequencies where MTF is 50% of its low frequency value (MTF50) are empirically used as the best indicators of image sharpness, and the ration of unit line width to the picture height (LW/PH) is commonly adopted for digital sensors [48]. The output data of Imatest 4.3 is given in Table 2. The data show that when the MLBE was used, the methods achieved better clarity. The capabilities of color restoration when the MLBE was applied were indicated by the color error in the device-independent CIELAB color space. The CIELAB was designed to be perceptually uniform, meaning that the perceived difference between the colors was approximately proportional to the Euclidean distance between them; ΔE*_ab (included luminance L*) and ΔC_ab* (included only the color, omitting L*), which are listed in Table 3, were computed by:

(27)$$\Delta {E^ \ast }_{\textrm{ab}}\; = {({({L_2}^\ast - {L_1}^\ast )^\textrm{2}} + {({a_2}^\ast - {a_1}^\ast )^\textrm{2}} + {({b_2}^\ast - {b_1}^\ast )^\textrm{2}})^{\textrm{1}/\textrm{2}}},$$

(28)$$\Delta {\textrm{C}_{\textrm{ab}}}^ \ast = {({({a_2}^ \ast - {a_1}^ \ast )^\textrm{2}} + {({b_2}^ \ast - {b_1}^ \ast )^\textrm{2}})^{\textrm{1}/\textrm{2}}},$$

where Δ C_ab* or ΔE*_ab of about 1 corresponded roughly to the just noticeable difference (JND) between colors. All measures showed that when the MLBE was utilized in the DCP based underwater restoration, lower ΔC_ab* and ΔE*_ab, and higher MTF values were achieved compared with the referred original methods.

Table 2. Imatest 4.3 SFR Analysis

View Table | View all tables in this article

Table 3. Imatest 4.3 ColorChecker Analysis

View Table | View all tables in this article

6. Conclusion

By analyzing the background patches in the underwater image set, a contradiction of the background light estimation in underwater image restoration based on the dark channel prior is found. The scale and directivity of the background light are analyzed in detail. A background light estimation method that applies the ANN to the maximum areas of the dark channel is proposed. The unphysical saturation and blurring caused by using the maximum of the dark channel as the global background light are avoided. The experiments were conducted on the tank and near-shore underwater images using different background light estimation methods, and the results proved that when the proposed MLBE was used, the distribution of light in turbid environments was more realistic and the blurring was removed more efficiently.

Funding

National Natural Science Foundation of China (NSFC) (61601194); “Six talent peaks” project in Jiangsu Province (No. DZXX-030); Lianyungang “521” funded project; Lianyungang “Haiyan” funded project.

References

1. J. S. Jaffe, “Underwater optical imaging: The past, the present, and the prospects,” IEEE J. Oceanic Eng. 40(3), 683–700 (2015). [CrossRef]

2. B. Zheng, H. Zhang, H. Zheng, and T. A. Gulliver, “Underwater Imaging Based on Inhomogeneous Illumination,” 2011 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PacRim), 873–876 (2011).

3. G. Johnsen, Z. Volent, E. Sakshaug, F. Sigernes, and L. H. Pettersson, “Remote sensing in the Barents Sea,” in Ecosystem Barents sea, E. Sakshaug, G. Johnsen, and K. Kovacs, (eds.), (Tapir Academic Press, 2009), pp. 139–166.

4. K. Ingrid, “Underwater Imaging and the effect of inherent optical properties on image quality,” Norwegian University of Science and Technology, Master thesis (2014).

5. R. Rai, P. Gour, and B. Singh, “Underwater Image Segmentation using CLAHE Enhancement and Thresholding,” Int. J. Emerg. Technol. Adv. Eng. 2(1), 118–123 (2012).

6. D. Jia and Y. Ge, “Underwater Image De-Noising Algorithm Based On Nonsubsampled Contourlet Transform And Total Variation,” 2012 International Conference on Computer Science and Information Processing (CSIP), 76–80 (2012).

7. A. Arnold-Bos, J. P. Malkasse, and G. Kerven, “A preprocessing framework for automatic underwater images denoising,” in Proceedings of the European Conference on Propagation and Systems, Brest, France (2005).

8. Y. Nie and Z. Y. He, “Underwater imaging and real-time optical image processing under illumination by light sources with different wavelengths,” Acta Opt. Sin. 34(7), 0710002 (2014). [CrossRef]

9. S. Bazeille, I. Quidu, L. Jaulin, and J. P. Malkasse, “Automatic underwater image pre-processing,” in Proceedings of the Characterizations du Milieu Marin (CMM ‘06), 16–19 (2006).

10. M. Chambah, D. Semani, A. Renouf, P. Courtellemont, and A. Rizzi, “Underwater color constancy: enhancement of automatic live fish recognition,” in Color Imaging IX: Processing, Hardcopy, and Applications, Proceedings of SPIE, 5293, 157–168 (2003).

11. K. Iqbal, R. Abdul Salam, A. Osman, and A. Zawawi Talib, “Underwater image enhancement using an integrated colour model,” Int. J. Comp. Sci. 34, 2 (2007).

12. G. Dudek, “Color correction of underwater images for aquatic robot inspection,” International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition Springer, Berlin, Heidelberg, 60–73 (2005).

13. J. Åhlén, D. Sundgren, and E. Bengtsson, “Application of underwater hyper spectral data for color correction purposes,” Pattern Recognit. Image Anal. 17(1), 170–173 (2007). [CrossRef]

14. F. Petit, A.-S. Capelle-Laize, and P. Carre, “Underwater image enhancement by attenuation inversion with quaternions,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP ‘09), 1177–1180, (2009).

15. A. T. Çelebi and S. Ertürk, “Visual enhancement of underwater images using Empirical Mode Decomposition,” Expert Syst. Appl. 39(1), 800–805 (2012). [CrossRef]

16. C. O. Ancuti, C. Ancuti, T. Haber, and P. Bekaert, “Enhancing underwater image and videos by fusion,” In Proc. CVPR, 81–88 (2012).

17. K. Iqbal, M. Odetayo, A. James, and R. Abdul Salam, “Enhancing the low quality images using unsupervised colour correction method,’’ in IEEE International Conference on Systems Man and Cybernetics (SMC) (2010).

18. X. Fu, P. Zhuang, Y. Huang, et al., “A retinex-based enhancing approach for single underwater image, “ Image Processing (ICIP), 2014 IEEE International Conference on., IEEE, 4572–4576 (2014).

19. J. Y. Chiang and Y. C. Chen, “Underwater image enhancement by wavelength compensation and dehazing,” IEEE Trans. Image Process. 21(4), 1756–1769 (2012). [CrossRef]

20. C. Ancuti, C. O. Ancuti, T. Haber, and P. Bekaert, “Enhancing underwater images and videos by fusion,” in Processing of IEEE Conference on Computer Vision and Pattern Recognition, 81–88 (2012).

21. C. O. Ancuti, C. Ancuti, C. DeVleeschouwer, and P. Bekaert, “Color balance and fusion for underwater image enhancement,” IEEE Trans. Image Process. 27(1), 379–393 (2018). [CrossRef]

22. J. Li, K. A. Skinner, R. M. Eustice, et al., “WaterGAN: Unsupervised generative network to enable real-time color correction of monocular underwater images,” IEEE Robot. Autom. Lett. 3(1), 387–394 (2018). [CrossRef]

23. C. Li, J. Guo, and C. Guo, “Emerging from water: Underwater image color correction based on weakly supervised color transfer,” IEEE Signal Process. Lett. 25(3), 323–327 (2018). [CrossRef]

24. J. Y. Zhu, T. Park, and P. Isola, et al., “Unpaired image-to-image translation using cycle-consistent adversarial networks,” arXiv preprint, (2017).

25. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2011). [CrossRef]

26. A. Galdran, D. Pardo, A. Picón, and A. Alvarez-Gila, “Automatic Red-Channel underwater image restoration,” J. Vis. Commun. Image Rep. 26, 132–145 (2015). [CrossRef]

27. N. Carlevaris-Bianco, A. Mohan, and R. M. Eustice, “Initial results in underwater single image dehazing,” in Proc. of IEEE OCEANS (2010).

28. Y. T. Peng, X. Zhao, and P. C. Cosman, “Single underwater image enhancement using depth estimation based on blurriness,” Image Processing (ICIP), 2015 IEEE International Conference on, IEEE (2015).

29. Y. T. Peng and P. C. Cosman, “Underwater image restoration based on image blurriness and light absorption,” IEEE Trans. Image Process. 26(4), 1579–1594 (2017). [CrossRef]

30. Y. T. Peng and P. C. Cosman, “Single image restoration using scene ambient light differential,” IEEE International Conference on Image Processing, ICIP, 1953–1957 (2016).

31. C. Li, J. Guo, S. Chen, Y. Tang, Y. Pang, and J. Wang, “Underwater image restoration based on minimum information loss principle and optical properties of underwater imaging,” IEEE International Conference on Image Processing , ICIP, 1993–1997 (2016).

32. S. Emberton, L. Chittka, and A. Cavallaro, “Hierarchical rank-based veiling light estimation for underwater dehazing,” BMVC, 125.1-125.12 (2015).

33. B. L. McGlamery, “A computer model for underwater camera systems,” Ocean Optics 0208, 221–231 (1980). [CrossRef]

34. S. Ghosh, R. Ray, S. R. K. Vadali, and S. N. Shome, “Light-Particle interaction in underwater: a modified PSF,” Communications and Signal Processing (ICCSP), 2014 International Conference on. IEEE (2014).

35. T. Johnsen, H. A. Bouman, S. Sathyendranath, and E. Devred, “Regional-scale changes in diatom distribution in the Humboldt upwelling system as revealed by remote sensing: implications for fisheries,” ICES J. Mar. Sci. 68(4), 729–736 (2011). [CrossRef]

36. C. L. Gallegos, D. L. Correll, and J. W. Pierce, “Modeling spectral diffuse attenuation, absorption, and scattering coefficients in a turbid estuary,” Limnol. Oceanogr. 35(7), 1486–1502 (1990). [CrossRef]

37. R. Fattal, “Single image dehazing,” ACM Trans. Graphics 27(3), 1 (2008). [CrossRef]

38. R. T. Tan, “Visibility in bad weather from a single image,” in Proc. of IEEE CVPR, 1–8 (2008).

39. K. B. Gibson, D. T. Vo, and T. Q. Nguyen, “An investigation of dehazing effects on image and video coding,” IEEE Trans. Image Process. 21(2), 662–673 (2012). [CrossRef]

40. J. Yu, C. Xiao, and D. Li, “Physics-based fast single image fog removal,” Proc. IEEE ICSP 37(2), 1048–1052 (2011).

41. C. H. Yeh, L. W. Kang, C. Y. Lin, and C. Y. Lin, “Efﬁcient image/video dehazing through haze density analysis based on pixel-based dark channel prior,” Proc. IEEE ISIC 238, 14–16 (2012).

42. F. C. Cheng, C. C. Cheng, P. H. Lin, and S. C. Huang, “A hierarchical airlight estimation method for image fog removal,” Eng. Appl. Artif. Intell. 43, 27–34 (2015). [CrossRef]

43. K. Wang, E. Dunn, J. Tighe, and J. M. Frahm, “Combining semantic scene priors and haze removal for single image depth estimation,” in Proc. of IEEE WACV, 800–807 (2014).

44. R. J. Davies-Colley, “Measuring water clarity with a black disk,” Limnol. Oceanogr. 33(4), 616–623 (1988). [CrossRef]

45. R. Gould, R. Arnone, and P. Martinolich, “Spectral dependence of the scattering coefficient in case 1 and case 2 waters,” Appl. Opt. 38(12), 2377–2383 (1999). [CrossRef]

46. M. Yang and A. Sowmya, “An underwater color image quality evaluation metric,” IEEE Trans. Image Process. 24(12), 6062–6071 (2015). [CrossRef]

47. S. Zhang, T. Wang, J. Dong, et al., “Underwater image enhancement via extended multi-scale Retinex,” Neurocomputing 245, 1–9 (2017). [CrossRef]

48. Imatest, Accessed: Jan. 3, 2015. [Online]. Available: http://www.imatest.com/.

Algorithm	Raw image	Peng et al. [28]		Li et al. [31]
Algorithm	Raw image	Original	MLBE	Original	MLBE
Contrast	0.6863	0.9843	0.8235	0.9922	0.9882
	0.8275	0.8196	0.9333	0.8275	0.9882
	0.7176	0.9725	0.7804	0.9922	0.9922
	0.8118	0.9843	0.9176	0.9804	0.9882
Average saturation	0.8484	0.8161	0.8485	0.8310	0.8300
	0.8369	0.7516	0.8385	0.8369	0.8322
	0.7904	0.8092	0.7909	0.8313	0.8309
	0.7681	0.8387	0.7692	0.8298	0.8293
Variance of chroma	0.3201	0.3806	0.3472	0.2888	0.3164
	0.2842	0.3145	0.3130	0.2842	0.2407
	0.3287	0.3764	0.3325	0.3315	0.3161
	0.4237	0.5164	0.4256	0.3704	0.3879
UCIQE	0.5567	0.6585	0.6071	0.6216	0.6332
	0.5757	0.5658	0.6187	0.5757	0.5983
	0.5544	0.6516	0.5735	0.6416	0.6343
	0.6190	0.7279	0.6492	0.6562	0.6665

SFR/analysis	MTF50 /(LW/PH) original	MTF50 /(LW/PH) with MLBE
Ancuti et al. [20]	13.14	22.49
Peng et al. [28]	14.34	48.31
Li et al. [31]	62.8	87.22

Colour/analysis	$Δ {E^{}}_{ab}$ _/ $Δ {C_{ab}}^{}$ original	$Δ {E^{}}_{ab}$ _/ $Δ {C_{ab}}^{}$ with MLBE
Ancuti et al. [20]	36/24.6	30.8/24.2
Peng et al. [28]	58.8/43.8	29.1/23.2
Li et al. [31]	52.5/41.2	52.4/40.8

Algorithm	Raw image	Peng et al. [28]		Li et al. [31]
Algorithm	Raw image	Original	MLBE	Original	MLBE
Contrast	0.6863	0.9843	0.8235	0.9922	0.9882
	0.8275	0.8196	0.9333	0.8275	0.9882
	0.7176	0.9725	0.7804	0.9922	0.9922
	0.8118	0.9843	0.9176	0.9804	0.9882
Average saturation	0.8484	0.8161	0.8485	0.8310	0.8300
	0.8369	0.7516	0.8385	0.8369	0.8322
	0.7904	0.8092	0.7909	0.8313	0.8309
	0.7681	0.8387	0.7692	0.8298	0.8293
Variance of chroma	0.3201	0.3806	0.3472	0.2888	0.3164
	0.2842	0.3145	0.3130	0.2842	0.2407
	0.3287	0.3764	0.3325	0.3315	0.3161
	0.4237	0.5164	0.4256	0.3704	0.3879
UCIQE	0.5567	0.6585	0.6071	0.6216	0.6332
	0.5757	0.5658	0.6187	0.5757	0.5983
	0.5544	0.6516	0.5735	0.6416	0.6343
	0.6190	0.7279	0.6492	0.6562	0.6665

SFR/analysis	MTF50 /(LW/PH) original	MTF50 /(LW/PH) with MLBE
Ancuti et al. [20]	13.14	22.49
Peng et al. [28]	14.34	48.31
Li et al. [31]	62.8	87.22

Effective estimation of background light in underwater image dehazing

Abstract

1. Introduction

2. Underwater imaging optical model

2.1 Jaffe-McGlamery model

2.2 Dark channel prior

2.3 Background light estimation

3. Contradiction with classical DCP

3.1 Directionality of backgound light in offshore underwater images

3.2 Relationship between background light and dark channel for underwater images

3.3 Background intensity localization

4. Machine learning based background light estimation

4.1 ANN based background light estimation

4.2 Transmission map estimation

5. Experiments and discussion

6. Conclusion

Funding

References

Cited By

Figures (14)

Tables (3)

Equations (28)

OSA Continuum