1. Introduction

Underwater signal detection in turbid water and occluded conditions is of great importance for many applications such as target detection, information transmission, and marine exploration [1–5]. Optical sensing using conventional image sensors in these environments is difficult as the captured images may provide reduced visibility and a low signal-to-noise ratio (SNR). Various approaches have been proposed for signal detection in a turbid medium, such as a correlation filter-based detector [2,3], polarimetric-based approaches [3], single-pixel detector [6], etc. Deep learning-based approaches have recently gained attraction for classification and recognition tasks due to their generalized learning capabilities and higher accuracy [7–9]. However, optical signal detection under degraded conditions remains a challenge, especially in cases where signals are partially occluded or embedded in the scattering medium.

In the previously reported correlation-based approach with multidimensional integral imaging for the detection of temporally encoded optical signals in the turbid medium [1–3], 3D integral imaging was shown to outperform conventional 2D imaging for signal detection capabilities. However, correlation-based filters have limitations, such as reduced generalization capability as compared with CNNs for classification and recognition tasks.

In recent years, different types of deep neural networks, namely, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory (LSTM), have been proposed to detect the signal in atmospheric turbulence and scattering medium [10–13], which can achieve better detection accuracy than the conventional methods. In this paper, temporally encoded optical signals are transmitted through a turbid and occluded underwater environment. This signal is captured using a CMOS image sensor array, and the recorded 2D elemental video frames are reconstructed using 3D integral imaging to improve the visibility of the optical signal in degraded conditions [14]. The CNN-BiLSTM based detector is used to detect the source signal in the reconstructed 3D video sequence. The proposed deep neural network is trained with clear water video sequences to perform the classification on the binary transmitted signal. To improve the performance and the generalization capabilities of the neural network model, we have used data augmentation to increase the size of the training dataset and to simulate noisy conditions. Since we use a 7-bit encoding scheme for the transmitted data sequence, a sliding window-based is used to extract each 7-frame video sequence from the recorded data. Each video sequence is fed to a pretrained CNN to extract the spatial feature vectors representing each frame. These features are then fed to a Bi-LSTM network which learns the temporal pattern hidden inside the encoded data. We demonstrate through our experiments that the conventional signal detection algorithms may fail under turbid and occluded underwater environments, whereas the proposed deep learning approach enhances the signal detection performance under such conditions. The performance of the proposed CNN-BiLSTM based detector is measured using performance metrics such as receiver operating characteristic (ROC) curves, area under the curve (AUC), and the number of bit errors. The signal detection results are compared for various turbidity conditions to demonstrate the improved performance of the proposed approach.

The rest of the paper is organized as follows: integral imaging and CNN-BiLSTM based detection are described in Section 2, followed by the details of the optical imaging system and underwater data collection procedure as described in Section 3. The experimental results and discussions are given in Section 4, and finally, the conclusions are presented in Section 5.

2. Methodology

2.1 3D integral imaging

Integral imaging (InIm) is a three-dimensional (3D) imaging technique, first proposed by Lippmann [15], which consists of the camera pickup and computational reconstruction process. In the pickup or capture process, multiple 2D elemental images are captured, and both intensity and directional information of optical rays are recorded [16–19]. These elemental images can be recorded by a lenslet array, an array of cameras, or a moving camera, as shown in Fig. 1(a). In the reconstruction process, a pinhole model is assumed, and the rays are back-propagated through their corresponding virtual pinholes at a particular depth plane to provide depth information about the 3D scene [20–22]. Mathematically, the reconstruction process can be described as follows:

 

Fig. 1. 3D integral imaging process: (a) Pickup stage of the integral imaging system and (b) computational volumetric reconstruction process for integral imaging.

Download Full Size | PDF

2.2 CNN-BiLSTM based optical signal detection

Figure 2 shows the detailed framework of CNN-BiLSTM network used for optical signal detection. First, a 3D reconstructed video is obtained using the integral imaging-based computational reconstruction algorithm. During the training phase, reconstructed 7-frame video sequences are used to train an end-to-end CNN-BiLSTM network. A pretrained GoogLeNet [23] trained on the ImageNet dataset [24] is used to extract the spatial features at each frame of the captured video sequence, followed by a bidirectional recurrent neural network to extract the temporal information content of the transmitted video sequence for each class [9]. The feature vectors are taken from the last pooling layer (“pool5-7 × 7_s1”) of the GoogLeNet and used as the feature vector representing the spatial information content of each video frame. The bi-directional long short-term memory (BiLSTM) network is a variant of the long short-term memory (LSTM) network, where two separate LSTM networks are used to extract the temporal information along the forward and backward direction [25–28]. The details of the LSTM cell structure are described in Appendix A. Assuming an M-bit encoding scheme is utilized, the information encoded in an M-frame sequence can be used to produce a k dimensional feature vector ${{\textbf x}_i}$, i=1,2,3,…M for each frame ${I_n}$, n=1,2,3,…M. Thus, for each M bit frame sequence, we get the feature matrix ${\textbf X}$ by concatenating the feature vectors ${{\textbf x}_i}$. The feature matrix ${\textbf X}$ for each video is then fed into a Bi-LSTM network, and its output is fed to a fully connected layer followed by a softmax layer for classification. During the testing phase, a sliding window-based approach is used to extract 7-frame video sequences from the recorded data sequence which is fed to the CNN-BiLSTM model. We have used BiLSTM layer with 100 hidden units to learn the temporal dependency of the feature vectors. The Bi-LSTM classification network was trained for 100 epochs with a batch size of 10 at each iteration. The network weights and biases were optimized using Adam optimizer with a learning rate of 0.0001 for the first 50 epochs and 0.00001 for the final 50 epochs. The Adam is a first order gradient-based optimization algorithm for stochastic objective functions. It is based on the adaptive estimates of lower-order moments. It is suitable for non-stationary objectives, noisy/spare gradients and have been proved to be robust and well suited for wide class of non-convex machine learning problems [29]. We have used the dropout layer with a dropout rate of 0.3. All the above parameters were chosen by tuning the hyperparameters of the network using 10% of the training dataset as validation data.

 

Fig. 2. Block diagram for Integral imaging-based optical signal detection using proposed deep learning architecture: InIm: Integral imaging, CNN: convolutional neural network, LSTM: Long Short-Term Memory Network. Red dotted block represents the bidirectional LSTM architecture and the arrows inside the block indicate the connection to forward and backward LSTMs.

Download Full Size | PDF

3. Experimental methods

This section describes the experimental setup, underwater data collection, and method for the proposed multidimensional integral imaging-based system in a turbid and occluded environment. The experimental setup is shown in Fig. 3. The light source used is a light-emitting diode (LED), operating at 630 nm, and is allowed to pass through a turbid water tank having dimensions 500(W) × 250(L) × 250(H) mm. The turbidity of the water is controlled through the addition of antacid, and experiments were carried out at different turbidity levels to assess the performance of the proposed system at various turbidities. The optical signals were coded using the gold code sequence [30]. The signal is transmitted through a turbid underwater medium with occlusion at a speed of 20 bits per second. When the transmitted data is 1, the 7-bit gold code is transmitted, and when the transmitted data is 0, the 7-bit flipped gold code is transmitted. Thus, the temporal structural difference between the videos representing class “1” and “0” could be utilized by the LSTM network to create a decision boundary between the two classes. For optical signal transmission, we choose an 8-bit data sequence as [1, 0, 0, 1, 1, 0, 1, 0], wherein each bit of the original data is coded with a 7-bit gold code ([1,1,0,0,1,0,1]) yielding a 56-bit encoded signal. An occluded underwater environment is created by placing an artificial plant in front of the light source inside the water tank. To quantify the turbidity level, we use Beer's coefficient, derived from the Beer-Lambert's law, which states that I = I₀e^-αd, where I₀ is initial intensity, I is the intensity after propagating a distance d in a turbid medium, and $\alpha $ is Beer’s coefficient. For the measurement of α a sample of water was taken from each turbidity condition, and the intensity of light I₀ and I was measured at two locations with d=10 mm, where d is distance between the I₀ and I. The intensities I₀ and I are measured using the optical power meter and detector (Newport 818-SL/DB Silicon) at a wavelength of 630 nm with a 10 mm aperture. In this experiment, α ranges from 0.0047 to 0.0318 mm^-1. The transmitted optical signals were captured using a 3 × 3 camera array consisting of G-192 GigE cameras and C-mount zoom lenses with a focal length of 20 mm. The pitch between cameras is 80 mm in both the horizontal and vertical directions. The spatial resolution of the camera sensor is 1600(H) × 1200(V) with a pixel size of 4.5 µm × 4.5 µm, and the camera array is synchronized to record video data set at a frame rate of 20 fps. In our experiments, the signal transmission and the receiving cameras are synchronized such that each recorded frame corresponds to the transmission of exactly 1 bit of the transmitted signal. This synchronization was done in order to transmit signals at the maximum speed allowable by the recording camera's framerate. Recent developments in camera technology allows much faster frame rates of 1 million frame per second.

 

Fig. 3. Water tank with turbid water and underwater plant to mimic underwater occlusion, a 3 × 3 camera array for 3D integral imaging pickup. Optical signal sent by the red LED light source. The external white light LED is used to mimic ambient light for shallow water scenes.

Download Full Size | PDF

The sliding window approach (depicted in Fig. 4) is used to slice each possible 7-frame video sequence from the transmitted data sequence. Thus, a series of 7 frame video sequences, U_i (i, i+1,…, i+6), are obtained from the transmitted signal using the sliding window approach, where i=1,2,3,…N-6 and N is the total number of frames in the encoded transmitted video sequence. There are 36 unique possible 7-frame video sequences, of these 36 possibilities, we can categorize them into three different cases: case 1) where the 7-frame video sequence exactly overlaps with the gold code sequence (class ‘1’), case 2) where the 7-frame video sequence exactly overlaps with the flipped gold code sequence (class ‘0’), and case 3) the remaining 34 possibilities wherein the sliced 7-frame video sequence neither matches exactly the gold code nor the flipped gold code sequences (‘idle’ class). Thus, to train our proposed CNN-BiLSTM optical signal detection network, we created three different class labels, ‘1’, ‘0’, and ‘idle’. In particular, our interest is to detect the classes ‘1’ and ‘0’. However, in our experiments, the inclusion of the ‘idle’ class in the training dataset results in better discrimination between the two classes of interest. For training the network, the transmitted signals are recorded in clear water. We recorded data with two different F numbers, F1.8 and F8. Thus, in the training dataset, before augmentation, we have two (one corresponding to each F number) videos each for class ‘1’ and class ‘0’, and 68 ‘idle’ class videos. To reduce network instability or bias towards the ‘idle’ class, the data sets of classes ‘0’ and ‘1’ are repeated 34 times in order to obtain 68 videos for each class.

 

Fig. 4. Proposed sliding window with CNN-BiLSTM-based classification approach.

Download Full Size | PDF

In total, we have 204 videos for training, 68 video sequences corresponding to each class. To improve the performance and generalization capability of the CNN-BiLSTM network in degraded environments, we have performed data augmentation on the training dataset. In a turbid water environment, we encounter optical signal attenuation and blurring. Therefore, we adopted two data augmentation models: (1) attenuation plus additive Gaussian noise, and (2) blurring plus additive Gaussian noise. Attenuation plus additive Gaussian noise model is modeled as I_degraded=β× I_original +n, where I_degraded is the degraded captured image (data), β is the attenuation factor, I_original is the original image (data), and n is the additive noise modeled as Gaussian with mean µ and variance σ². Similarly, the blurring plus additive Gaussian noise model can be written as I_degraded=h Ⓧ I_original +n, where Ⓧ represents the convolution operator and h is a 2D Gaussian kernel given as h=1/(2π s²)exp(-(x²+y²)) with zero mean (0,0) and variance s. To generate the augmented dataset, the augmentation models are applied to each of the training video sequences. For estimating the parameters of the augmentation model, a dataset of signal ‘on’ and ‘off’ images were recorded at various turbidity levels ranging from α=0.005 to 0.033 mm^-1 without occlusion. Parameters µ and σ² are estimated from the fitted normal distribution on the histogram of ‘on’ and ‘off’ images using maximum likelihood estimation at various turbidities. The additive Gaussian noise distribution mean and variance vary in the range of 0.015 to 0.032 and 0.004 to 0.01, respectively. The parameter s is assumed to be chosen from a uniformly distributed random variable ranging from 1 to 10. The parameter β is chosen from uniformly distributed random values ranging from 0 to 1. Since the test data contains the clear water image (β = 1) and turbid water images (β <1), it is reasonable to assume a range of β from 0 to 1.

The CNN-BiLSTM classifier outputs the classification scores for the respective classes, the maximum value between the classification scores of ‘0’ and ‘1’ is selected; if the selected value corresponds to class ‘0’, then it is multiplied by -1 else, if it corresponds to class ‘1’, then the selected value remains unchanged as shown in Fig. 4. The above score selection method is applied on the each U_i (i, i+1,…,i+6) video to generate a transformed classification score sequence S(i). For testing the performance of the proposed approach, we recorded data with occlusion and turbidity; a sample 2D elemental image is recorded using a central camera with an aperture size of F1.8 is shown in Fig. 5(a). The integral imaging-based computational reconstruction algorithm, as outlined in Eq.1, enhances the visibility of the optical signal in a turbid and occluded environment, as illustrated in Fig. 5(b) at α = 0.0047 mm⁻¹.

 

Fig. 5. (a) Images of the underwater occluded signal viewed from the central camera perspective, taken at α = 0.0047 mm⁻¹, and (b) reconstructed image using 3D integral imaging with occlusion at α = 0.0047 mm⁻¹.

Download Full Size | PDF

The transformed classification scores S(i) of the transmitted video sequence should have high and low peaks corresponding to the transmission of binary sequences 1 and 0, respectively. Given the 8-bit original data and 7-bit coding scheme, the transformed classification score S(i) is expected to have 8 prominent locations of either local minima or maxima values, each separated by 7-bits. By summation of the prominence of these peaks separated by 7-bits, we can find the correct start frame for the signal transmission as the frame which gives the maximum sum of the prominence across the recorded signal [2]. The final classification of the transmitted binary data sequences can be done using thresholding of the transformed classification score S(i). For our results, we set the threshold to be ‘0’. For each turbidity level, the encoded 56-bit signal transmission experiment is repeated eight times corresponding to the 64-bits of original data to be decoded to compute the performance metrics. Thus, in total for testing, we have 2880 7-frame videos, 576 videos for each turbidity level. The overall flow chart for optical signal transmission and signal detection is shown in Fig. 6.

 

Fig. 6. Flow chart of the proposed system for (a) optical signal transmission and (b) deep learning-based detection in underwater communication. InIm denotes Integral Imaging.

Download Full Size | PDF

4. Results and discussion

The performance of the proposed underwater signal detection system for the experimental data under varying turbidity conditions is evaluated using the receiver operating characteristic (ROC) curves, the area under the curve (AUC), and the number of detection errors. The ROC analysis for underwater signal detection is shown in Fig. 7(a) for lower turbidity (α = 0.0114 mm⁻¹) and in Fig. 7(b) for higher turbidity (α = 0.0318 mm⁻¹) levels. The ROC for the 3D integral imaging system with CNN-BiLSTM (blue line) method achieves an AUC of 1 at α = 0.0114 mm⁻¹ and 0.8965 at α = 0.0318 mm⁻¹, which outperforms the other tested methods. The conventional 2D imaging system with CNN-BiLSTM (black line) gives AUC values of 0.5684, and 0.4805, respectively. The 3D integral imaging system with nonlinear correlation (green line) gives AUC values of 0.899, and 0.4561, respectively. Finally, the conventional 2D imaging with nonlinear correlation (red line) [1,31] give AUC values of 0.5117, and 0.4453, respectively, at turbidity levels of α = 0.0114 mm⁻¹ and α = 0.0318 mm⁻¹.

 

Fig. 7. ROC (Receiver operating characteristic) curves for underwater signal detection at a turbidity level (α=0.0114 mm^-1 and α=0.0318 mm^-1). Results are compared between the 3D integral imaging reconstructed video data with CNN-BiLSTM (blue line), a conventional 2D imaging system with CNN-BiLSTM (black line), a 3D integral imaging system with nonlinear correlation (green line), and conventional 2D imaging with nonlinear correlation (red line).

Download Full Size | PDF

The AUC and number of detection errors are shown in Fig. 8 as a function of Beer's coefficient. Figure 8(a) shows the AUC versus the Beer's law coefficient. The AUC of the proposed method maintains a higher value than that of all other tested methods. Also, from Fig. 8(b), the number of detection errors increases for all methods as the turbidity increases; however, the number of errors using the proposed CNN-BiLSTM approach is lower than that of all other tested methods.

 

Fig. 8. (a) Area under curves and (b) number of detection errors for underwater signal detection at various turbidity levels. Results are compared between the 3D integral imaging reconstructed video data with CNN-BiLSTM (blue line), a conventional 2D imaging system with CNN-BiLSTM (black line), a 3D integral imaging system with nonlinear correlation (green line), and conventional 2D imaging with nonlinear correlation (red line).

Download Full Size | PDF

From Fig. 7 and Fig. 8, the experimental comparison shows that the proposed CNN-BiLSTM approach is more effective than the previously reported correlation-based approach [2,3] in challenging experimental conditions, including occlusion and turbidity. Compared with the deep neural network-based approaches, the correlation-based approaches are less generalizable and particularly useful when limited training datasets are available. Even though, the 3D imaging system has increased computational load as well as calibration requirements as compared to a single camera system, in degraded environments and applications where the accuracy is of prime concern, the 3D integral imaging-based system proves to be useful. Thus, the proposed 3D integral imaging-based CNN- BiLSTM approach is promising for underwater signal detection in occluded and turbid environments. The estimated time (in seconds) of the proposed system is 9.2 and 6.10 respectively for the 3D integral imaging reconstruction and the sliding window-based CNN-BiLSTM classification. The computation time has been calculated for 72 frames video (0.21 seconds/frame). For 2D imaging, the computation time for sliding window approach and classification remains the same as 3D imaging. Thus, the total computation time for 2D imaging is 6.10 seconds (0.085 seconds/frame). However, the computation time for 3D integral imaging reconstruction can be further reduced by using GPU based stream-processing [32] and much faster cameras at mega frames per second which is a focus of our future work.

5. Conclusion

In summary, we have presented an underwater signal detection approach in turbid and occluded medium based on multidimensional integral imaging and deep neural networks. We have compared the performance of the proposed method with conventional 2D imaging and correlation-based approaches. The results show that multidimensional integral imaging substantially improves the performance of optical signal detection in comparison to other imaging modalities under degraded environments such as partial occlusion and turbidity. Future experiments may be extended in more challenging environments such as underwater turbulence [33], much faster cameras, and exploring other deep learning strategies which greatly reduce the computation as well as calibration requirements for enhanced optical signal detection under degraded conditions [34].