Expand this Topic clickable element to expand a topic
Skip to content
Optica Publishing Group

Improved double-threshold denoising method based on the wavelet transform

Open Access Open Access

Abstract

Preprocessing of spectral data is a key part of infrared spectroscopy and is an important basis for building robust models. Therefore, the measured signal needs to be preprocessed to achieve accurate and reliable measurement results. After sketching out the basic principles and basic methods of the wavelet transform, a new modified double-threshold denoising method combined with the proposed threshold method is presented in the paper. Two sets of comparative simulation experiments are also done to demonstrate the performance of the new denoising method. Block signals with a signal length of 2000 and the sinusoidal signal with a signal length of 1000 and the measured spectra are used for denoising with traditional schemes and the proposed method. The results of simulation data have demonstrated that the proposed method outperforms the traditional thresholding schemes for increasing the signal-to-noise ratio (SNR) without distorting the signal.

© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

1. Introduction

In the process of obtaining multi-component pollutant gas spectrum by open path FTIR (OP-FTIR), it is easily affected by light, temperature, sensor characteristics and other factors, which will make the spectrum infected with noise and cause the submergence problem of weak characteristic peaks in the spectrum [1,2]. Volatile organic compounds (VOCs) are the main components of air pollution [3]. The spectral signal containing noise will be de-noised to restore the characteristic spectrum of VOCs as much as possible, and to eliminate or weaken the interference factors in subsequent spectral identification and concentration inversion. Thereby improving the accuracy of spectral analysis [4].

In addition to useful chemical information, the spectrum also contains a large amount of interference information such as background noise and irrelevant information. Therefore, spectral preprocessing is required before the multivariate calibration model is established. The purpose of spectral preprocessing is to weaken the influence of various non-target factors on the target spectrum, retain effective information, improve spectral resolution, reduce the complexity of the model, and improve the robustness of the model [5,6].

In the traditional de-noising method, Fourier transform based on global transformation can only be analyzed from the time domain or the frequency domain, and non-stationary signal and singular signal cannot be processed [7,8].

Short-time Fourier transform and wavelet transform are all generated by the fact that the traditional Fourier transform cannot meet the requirements of signal processing, but in essence, since it uses a fixed short-term window function, the short-time Fourier transform still has insurmountable defects in signal analysis [9,10].

At present, the better methods in the infrared spectral denoising method include wavelet transform and wavelet packet transform denoising, empirical mode decomposition (EMD) and ensemble empirical mode decomposition (EEMD) adaptive denoising. However, these methods have certain deficiencies. Wavelet packet transform denoising is complicated, and the wavelet coefficients near the threshold are discontinuous, which will reduce the accuracy of the reconstructed signal. EMD de-noises the frequency band where the signal and noise overlap to produce modal aliasing. In EEMD method, the valid information contained in the high frequency part will be removed as noise [11,12].

Based on the Fourier transform method, wavelet transform is proposed and developed, however, the construction of wavelet basis and result analysis all also depend on Fourier transform [13]. Wavelet transform overcomes locality and can perform time-frequency analysis. It is suitable for pre-processing non-stationary signals, such as spectral signals [14,15].

At present, the denoising methods in the field of wavelet transform can be divided into three categories: wavelet modulus maximum value denoising method, filtering algorithm based on wavelet coefficient region correlation and wavelet threshold denoising method. Among them, the wavelet threshold shrinkage method which was proposed by Donoho and Johnstone in 1992 proves that the estimated risk generated by the estimate is closest to the theoretical minimum risk [16]. The threshold determination and the selection of threshold values are two key factors of wavelet threshold denoising method. Soft and hard threshold denoising methods are two traditional methods of wavelet threshold denoising, but these two threshold function methods have corresponding disadvantages [17,18]. Some scholars have made great contributions to the improvement of wavelet threshold denoising methods, however, the problem that the hard threshold is discontinuous and the soft threshold method has a fixed deviation still exists [19,20]. Recently, some investigations have been contributed to proposing a wavelet de-noising method under a new double threshold function [21]. The de-noising effect is greatly improved compared with the traditional soft and hard threshold methods, but there is still a problem of fixed deviation. Moreover, some adjustments have been made to the construction of threshold function for this problem, but there is still room for improvement [22]. Aiming at the problem of fixed deviation, a lot of research has been done on the determination of the threshold and some improvements have been made to the threshold function, so as to ensure the continuity of the function and remove the fixed deviation [23]. Section II presents the principle of wavelet transform and introduces the theory of the proposed method. In section III, some comparative experiments based on the traditional method and the proposed method are carried out and the experimental results are analyzed. The conclusion is presented in section IV.

2. Principle of wavelet transform and the proposed method

2.1 Principle of the wavelet transform

The wavelet transform has good time-frequency localization characteristics. It can analyze the local features of the signal and its window size is variable. After the discrete wavelet transform, the wavelet coefficient of the noisy signal is composed of two parts, one part is the wavelet coefficient of the original signal, and the other part is the wavelet coefficient of the noise. However, the useful signal and noise have different coefficient characteristics. It can be considered that the wavelet coefficient with a relatively large amplitude is dominated by a signal, and the coefficient with a small amplitude is largely noise. Therefore, an appropriate threshold can be selected and the part larger than the threshold can be retained as a signal, or the part smaller than the threshold can be discarded as noise with a fixed quantity shrinking to zero, thus achieving the effect of denoising.

Assuming that the original signal is f(t) and the contaminated noise signal is s(t), the basic noise model can be expressed as:

$$\textrm{s}(\textrm{t} )= \textrm{f}(\textrm{t} )+ {\sigma \textrm{e}}(\textrm{t} )$$
where e(t) is the noise and ${\sigma }$ is the noise intensity. e(t) is generally assumed as Gaussian white noise. The results of discrete wavelet transform for noisy signals are shown below [24]:
$${\textrm{W}_\textrm{s}}({\textrm{j},\textrm{k}} )= {\textrm{W}_\textrm{f}}({\textrm{j},\textrm{k}} )+ {\textrm{W}_\textrm{e}}({\textrm{j},\textrm{k}} )$$
j = 0,1,2,…,J; k = 0,1,2,…,N.

In this equation, ${\textrm{W}_\textrm{s}}({\textrm{j},\textrm{k}} )$, ${\textrm{W}_\textrm{f}}({\textrm{j},\textrm{k}} )$, and ${\textrm{W}_\textrm{e}}({\textrm{j},\textrm{k}} )$ are the wavelet coefficients of noisy signal, original signal and noise signal respectively, which are denoted as ${\textrm{w}_{\textrm{j},\textrm{k}}}$, ${{\mu }_{\textrm{j},\textrm{k}}}$,${\textrm{e}_{\textrm{j},\textrm{k}}}$.

The parameter J represents the maximum hierarchical classification of wavelet transform, while the parameter N indicates the length of the signal.

It can be seen from the above formula that the wavelet coefficients can be composed of the original signal wavelet coefficient ${{\mu }_{\textrm{j},\textrm{k}}}$ and the noise wavelet coefficient ${\textrm{e}_{\textrm{j},\textrm{k}}}$ after wavelet transform. Therefore, the wavelet coefficient ${\textrm{w}_{\textrm{j},\textrm{k}}}$ is processed by selecting an appropriate threshold ${\lambda }$, and the wavelet coefficient is estimated as ${{\hat{\textrm w}}_{\textrm{j},\textrm{k}}}$ so as to make the value of the formula $||{{\hat{\textrm w}}_{\textrm{j},\textrm{k}}} - {{\mu }_{\textrm{j},\textrm{k}}}||$ as small as possible. The estimated signal ${{\hat{\textrm s}}_{\textrm{j},\textrm{k}}}$ obtained from the wavelet reconstruction by the estimated wavelet coefficients, however, the estimated signal ${{\hat{\textrm s}}_{\textrm{j},\textrm{k}}}$ is supposed as the signal after denoising.

2.2 Criteria of threshold selection

2.2.1 Traditional threshold

The determination of the threshold plays a decisive role in the denoising effect of the wavelet threshold function. When the threshold value is too low, it will result in unremoved noise, and when the threshold value is too high, the useful components of the signal will be lost [25].

At present, the most commonly used is the fixed threshold, which is determined according to the threshold estimation risk theorem proposed by Donoho et al. The expression of the fixed threshold is presented as follows:

$${\textrm{T}_\textrm{i}} = {\sigma }\sqrt {2\textrm{logN}} $$
The parameter ${\sigma }$ indicates the standard deviation of a noise signal. In the actual environment, the parameter ${\sigma }$ is often unknown. Therefore, when calculating the threshold value, mathematical estimation method should be used to determine the standard deviation of the noise signal. Among them, the most common estimation method is as follows:
$${\sigma } = \frac{{\textrm{median}({|{{\textrm{w}_{\textrm{j},\textrm{k}}}} |} )}}{{0.6745}}$$
The parameter ${\textrm{w}_{\textrm{j},\textrm{k}}}$ reflects the detail coefficients of all wavelet after multi-scale decomposition.

2.2.2 Proposed threshold

Donoho proved that when the threshold value is ${\textrm{T}_\textrm{i}} = {\sigma }\sqrt {2\textrm{logN}} $, the estimated risk generated by the estimate is closest to the theoretical minimum risk. In practical application, considering that the propagation properties of signal and noise wavelet coefficients are different in different scales. In general, as the decomposition scale increases, the modulus of the signal increases, and that of the random noise decreases rapidly.

Take $\textrm{w}({\textrm{u},\textrm{v}} )$ to represent the binary wavelet transform value of spectral signal at ${\mu }$ scale, take the transform value of adjacent scale for correlation calculation, define the formula $\textrm{corr}({\textrm{u},\textrm{v}} )= \textrm{w}({\textrm{u},\textrm{v}} ){\ast \textrm w}({\textrm{u} + 1,\textrm{v}} )$ when calculate $\textrm{corr}({\textrm{u},\textrm{v}} )$ of each scale and adjacent scale, and then normalize it [26].

Formulas are as follows:

$$\textrm{k}({\textrm{u},\textrm{v}} )= \textrm{corr}({\textrm{u},\textrm{v}} ){\ast }\frac{{\sqrt {{\textrm{P}_\textrm{w}}(\textrm{u} )/{\textrm{P}_{\textrm{corr}}}(\textrm{v} )} }}{{\textrm{w}({\textrm{u},\textrm{v}} )}}$$
$${\textrm P}_{\textrm w}\left( {\textrm u} \right) = \sum\nolimits_{\textrm v} {{\textrm w}{\left( {{\textrm u},{\textrm v}} \right)}^2} $$
where
$${\textrm P}_{{\textrm {corr}}\left( {\textrm u} \right)} = \sum\nolimits_{\textrm v} {{\textrm {corr}}{\left( {{\textrm u},{\textrm v}} \right)}^2} $$
In the equation, * represents convolution.

At different scales, depending on the different characteristics of wavelet coefficients of signal and random noise, the modulus values are greater than one or less than one, respectively. Define the new parameter h (u, v) as the follow:

$$\textrm{h}({\textrm{u},\textrm{v}} )= 1 - \textrm{ln}|{\textrm{k}({\textrm{u},\textrm{v}} )} |$$
Hence, the new threshold in the proposed method is set as:
$$\textrm{T}({\textrm{u},\textrm{v}} )= \frac{{{\sigma }\sqrt {2\textrm{logN}} }}{{\textrm{h}({\textrm{u},\textrm{v}} )}}$$
The selected new threshold overcomes the disadvantages of fixed threshold and achieves good results.

2.3 Threshold function

2.3.1 Traditional threshold function

In the traditional threshold function, the soft and hard threshold denoising methods have been widely used due to its simple structure and good denoising effect.

The expression of hard threshold function is:

$${\hat{w}_{j,k}} = \left\{ \begin{array}{ll} {w_{j,k}} &|{{w_{j,k}}} |\ge \lambda \\ 0 &|{{w_{j,k}} |< \lambda } \end{array} \right.$$
The expression of soft threshold function is:
$${\hat{w}_{j,k}} = \left\{ \begin{array}{ll} sign({{w_{j,k}}} )\ast ({|{{w_{j,k}}} |- \lambda } )& |{{w_{j,k}}} |\ge \lambda \\ 0& |{{w_{j,k}}} |< \lambda \end{array} \right.$$
Although soft and hard threshold methods are certified as simple and feasible, they still have some potential weakness. The estimated signal ${{\hat{\textrm w}}_{\textrm{j},\textrm{k}}}$ obtained by the hard threshold function is discontinuous at $\pm {\lambda }$, which will cause the Pseudo-Gibbs phenomenon. The parameter ${{\hat{\textrm w}}_{\textrm{j},\textrm{k}}}$ obtained by the soft threshold function has better continuity, and the signal after denoising is relatively smooth, but a permanent deviation between ${{\hat{\textrm w}}_{\textrm{j},\textrm{k}}}$ and ${\textrm{w}_{\textrm{j},\textrm{k}}}$ always exits, which leads to the signal being too smooth at some sharp points and thus losing some features of the original signal.

2.3.2 Improved threshold function

In order to overcome the disadvantages described above with respect to the soft and hard thresholds, a large number of scholars have proposed improvements or proposed new threshold functions. Liu et al. proposed a new double threshold methods as shown in the following expression [21]:

$${\hat{w}_{j,k}} = \left\{ {\begin{array}{ll} {sgn({{w_{j,k}}} )\ast [{|{{w_{j,k}}} |- {\lambda_1}{e^{({{\lambda_1} - {\lambda_2}} )}}} ]}&{|{{w_{j,k}}} |\ge {\lambda_2}}\\ {sgn({{w_{j,k}}} )\ast [{|{{w_{j,k}}} |- {\lambda_1}{e^{({{\lambda_1} - |{{w_{j,k}}} |} )}}} ]}&{{\lambda_1} < |{{w_{j,k}}} |< {\lambda_2}}\\ {0\; }&{|{{w_{j,k}}} |< {\lambda_1}} \end{array}} \right.$$
where ${{\lambda }_2}$ is the upper threshold and ${{\lambda }_1}$ is the lower threshold, satisfying ${{\lambda }_1} = \textrm{k}{{\lambda }_2}$ and $0 < k < 1$ the function is continuous at both threshold points, and in the case of ${{\lambda }_1} < |{{\textrm{w}_{\textrm{j},\textrm{k}}}} |< {{\lambda }_2}$, as ${\textrm{w}_{\textrm{j},\textrm{k}}}$ increases, ${\textrm{e}^{({{{\lambda }_1} - |{{\textrm{w}_{\textrm{j},\textrm{k}}}} |} )}}$ decreases continuously, in accordance with the index attenuation characteristics, but in the case of $|{{\textrm{w}_{\textrm{j},\textrm{k}}}} |\ge {{\lambda }_2}$, ${\textrm{e}^{({{{\lambda }_1} - {{\lambda }_2}} )}}$ is fixed, there is still a problem of constant difference, so for the problems of the above functions, this paper proposes a new and improved double threshold function denoising method, which is introduced as follows:
$${\hat{w}_{j,k}} = \left\{ {\begin{array}{ll} {sgn({{w_{j,k}}} )\ast [{|{{w_{j,k}}} |- {\lambda_1}{e^{({{\lambda_1} - {\lambda_2} - |{{w_{j,k}}} |} )}}} ],}&{|{{w_{j,k}}} |\ge {\lambda_2}}\\ {sgn({{w_{j,k}}} )\ast [{|{{w_{j,k}}} |- {\lambda_1}{e^{({{\lambda_1} - |{{w_{j,k}}} |} )}}} ],}&{{\lambda_1} < |{{w_{j,k}}} |< {\lambda_2}}\\ {0,\; }&{\; |{{w_{j,k}}} |< {\lambda_1}} \end{array}} \right.$$
From the analysis in the second part, it can be known that the upper threshold satisfies ${{\lambda }_2} = \frac{{{\sigma }\sqrt {2\textrm{logN}} }}{{\textrm{h}({\textrm{u},\textrm{v}} )}}$, and the lower threshold satisfies ${{\lambda }_1} = \textrm{c}{{\lambda }_2}$, where $0 < c < 1$. This function is continuous at both threshold points. At ${{\lambda }_1} < |{{\textrm{w}_{\textrm{j},\textrm{k}}}} |< {{\lambda }_2}$, as $|{{\textrm{w}_{\textrm{j},\textrm{k}}}} |$ continues to increase, ${\textrm{e}^{({{{\lambda }_1} - |{{\textrm{w}_{\textrm{j},\textrm{k}}}} |} )}}$ decreases continuously, which is consistent with the exponential attenuation property. Moreover, in the case of ${{\hat{\textrm w}}_{\textrm{j},\textrm{k}}} \to \infty $, the function approximates ${{\hat{\textrm w}}_{\textrm{j},\textrm{k}}} = {\textrm{w}_{\textrm{j},\textrm{k}}}$, thus overcoming the problem of constant deviation.

As shown in Fig. 1, the hard threshold function is discontinuous at the threshold point ${\textrm{w}_{\textrm{j},\textrm{k}}} = {\lambda }$, and the reconstructed signal is prone to cause the pseudo-Gibbs phenomenon. The soft threshold function has good continuity, but there is a constant deviation in the case of $|{{\textrm{w}_{\textrm{j},\textrm{k}}}} |\ge {\lambda }$. The function of this paper is continuous, and the transition region at the threshold point remains smooth. In addition, as ${\textrm{w}_{\textrm{j},\textrm{k}}}$ becomes larger, ${{\hat{\textrm w}}_{\textrm{j},\textrm{k}}}$ continuously approaches ${{\hat{\textrm w}}_{\textrm{j},\textrm{k}}} = {\textrm{w}_{\textrm{j},\textrm{k}}}$, and the function reaches approximately coincidence, so that the pseudo phenomenon of the reconstructed signal at the mutation point is obtained. Suppresses, and controls the problem of excessive smoothing at sharp points, effectively controlling signal mutation and loss.

 figure: Fig. 1.

Fig. 1. Comparison of soft and hard thresholds and proposed thresholds.

Download Full Size | PDF

3. Simulation and analysis of the proposed method

To de-noise a signal using wavelets, the first step is to identify which part of the signal contains noise, and then discard these parts for reconstruction. As more and more high-frequency information is filtered out of the signal, the corresponding low-frequency part is getting cleaner and the noise is getting smaller and smaller.

For the wavelet decomposition of one-dimensional signals, a wavelet basis function is chosen at first and a decomposition level N is determined, and then N-layer decomposition is performed on the signal. It can be seen from the formula (5) to (9) in section 2.2.2 that the wavelet basis function is only beneficial to the decomposition of the signal, and does not have any influence on the threshold of the proposed method. Generally, the wavelet basis function is selected from the aspects of support length, vanishing moment, symmetry, regularity and other factors. Adequate experiments prove that db wavelet function has good regularity, which has better effect on signal decomposition than other wavelet basis functions.

In wavelet decomposition, the choice of the number of decomposition layers is a very important step. The larger the number of layers, the more obvious the different characteristics of noise and signal, and the better the separation of the two. On the other hand, the larger the number of decomposition layers, the larger the distortion of the signal obtained by the reconstruction. So, it is very important to choose a suitable decomposition scale. Take superimposed sinusoidal signal as an example and decompose it in multiple layers, the experimental result is shown in the Fig. 2 below.

 figure: Fig. 2.

Fig. 2. Original signal s and its low and high frequency parts.

Download Full Size | PDF

As can be seen from the decomposition results shown in Fig. 2, the fourth layer of the low frequency clearly separates the composition of the lowest frequency in the sinusoidal signal. In the Fig. 2, a1 to a4 respectively represent the low frequency parts of the layers after signal decomposition, and d1 to d4 respectively represent the high frequency parts of the layers after signal decomposition.

To evaluate the de-noising effect of the new threshold function, there are generally two methods: subjective comment and objective comment. This paper introduces two indicators of signal-to-noise ratio (SNR) and mean square error (MSE) to evaluate and verify the denoising effect. The SNR and MSE algorithms are shown below:

$$\textrm{SNR} = 10\textrm{lg}\left[ {\frac{{\mathop \sum \nolimits_{n = 1}^N {{\bar{X}}^2}(n )}}{{\frac{1}{N}\mathop \sum \nolimits_{n = 1}^N {{[{S(n )- X(n )} ]}^2}}}} \right]$$
$${\textrm {RMSE}} = \sqrt {\displaystyle{1 \over N}} \sum\nolimits_{n = 1}^N {{\left[ {X\left( n \right)-\bar{X}\left( n \right)} \right]}^2}$$
S(n) is the original signal, X(n) is the signal after denoising. SNR of the signal after denoising is larger, indicating that the denoising effect is better, and the MSE is smaller, the better the signal can reproduce the original signal.

3.1 Simulated signal denoising experiment

3.1.1 Block signal denoising experiment

At first, a block signal with a length of 2000 is generated, and then a Gaussian white noise with a signal-to-noise ratio (SNR) of 12 is added. The noised blocks signal is decomposed by a single-layer discrete wavelet, and then reconstructed. Then, it is decomposed into four layers by db wavelet basis function, and the low-frequency coefficients and high-frequency coefficients are extracted and reconstructed. Finally, the fourth layer of low frequency coefficients is retained to remove high frequency noise.

Figure 3 shows the fourth layer low frequency coefficients and the high frequency coefficient after the discrete wavelet transform. Figure 4 presents a front-back comparison of the high frequency information.

 figure: Fig. 3.

Fig. 3. (a) Fourth layer low frequency coefficient; (b) First layer high frequency coefficient; (c) Second layer high frequency coefficient; (d) Third layer high frequency coefficient; (e) Fourth layer high frequency coefficient.

Download Full Size | PDF

 figure: Fig. 4.

Fig. 4. (a) Original Signal; (b) Level 4 approximation coefficient.

Download Full Size | PDF

A comparative experiment is conducted on blocks signal and the result spectrum is shown in Fig. 5. The noisy signal is de-noised by traditional threshold method and threshold method proposed by liu et al [21]. and improved threshold method respectively. The results are shown in diagram(c), (d), (e) and (f) respectively, and the corresponding difference from the original signal spectrum are also shown in Fig. 6. As can be seen from (c) in Fig. 5, the hard threshold function method is prone to significant oscillating points due to the discontinuity of the hard threshold function. It can be seen from (d) in Fig. 5 that the signal as whole is very smooth and too smooth at the sharp point due to the fixed deviation of the soft threshold function. It can be seen from (f) in Fig. 5 that the proposed method is superior to the traditional threshold function method and threshold method in literature [21].

 figure: Fig. 5.

Fig. 5. (a) Original pure signal; (b) Noisy signal; (c) Hard threshold de-noised signal; (d) Soft threshold de-noised signal; (e) Literature [21] threshold de-noised signal; (f) Improved threshold de-noised signal.

Download Full Size | PDF

 figure: Fig. 6.

Fig. 6. (A) The differential spectrum between(a) and (c); (B) The differential spectrum between (a) and (d); (C) The differential spectrum between (a) and (e); (D) The differential spectrum between (a) and (f).Diagram (a), (c), (d), (e)and (f) are based on Fig. 3.

Download Full Size | PDF

In the range of 0 to 1000 bands, it can be clearly seen that due to the constant deviation problem of the threshold method proposed in the literature [21], the effective information part of the original signal is lost, and the improved method can better retain the effective information in the original signal.

Figure 6 shows the residuals of the de-noised spectrum and the original spectral signal spectrum after denoising using different threshold methods. It can be clearly seen from the four comparison diagrams in Fig. 6 that the improved method has a significant optimization in the denoising effect compared with the conventional threshold method, and the characteristic spectral information is not weakened while eliminating the superimposed noise.

Table 1 compares the signal-to-noise ratio (SNR) and mean square error (MSE) of different threshold denoising method. According to the theory that the bigger the value of SNR, the better the denoising effect, and the MSE is smaller, the noisy signal can reconstruct the original signal, the improved denoising method has obvious denoising effect on the infrared spectrum of Gaussian noise pollution and it is superior to the traditional discrete wavelet soft threshold denoising effect by analyzing the data in Table 1. Hence, the proposed method is better than the traditional wavelet-based method in terms of denoising effect.

Tables Icon

Table 1. The SNR and MSE values after denoising of different threshold method based on block signal

3.1.2 Set of bumps signal denoising experiment

Take the sinusoidal signal as an example. At first, the sinusoidal signal with a length of 1000 is generated, and then a Gaussian white noise with a signal-to-noise ratio (SNR) of 30 is added. A comparative experiment is conducted on sinusoidal signal and the result spectrum is shown in Fig. 7. The noisy signal is de-noised by traditional threshold method and threshold method proposed by liu et al [21]. and improved threshold method respectively. The results are shown in diagram(c), (d), (e) and (f) respectively.

 figure: Fig. 7.

Fig. 7. (a) Original pure signal; (b) Noisy signal; (c) Hard threshold de-noised signal; (d) Soft threshold de-noised signal; (e) Literature [21] threshold de-noised signal; (f) Improved threshold de-noised signal.

Download Full Size | PDF

Through the comparison of the denoising performance shown in Fig. 7, the same conclusions as the block signal denoising experiment in section 3.1.1 can be obtained. It can be seen from (e) in Fig. 7 that although the method in literature [21] obtains a significant denoising effect on the basis of the soft and hard threshold method, the signal is not smooth at the unevenness of the signal, which may result in loss of useful information of the signal. The improved method is smoother at successive points of the signal, thereby facilitating the retention of more valid information.

The signal-to-noise ratio (SNR) and mean square error (MSE) of different threshold denoising method are compared in Table 2. The proposed method has obvious denoising effect on the signal of Gaussian noise pollution and its calculation speed is significantly improved compared the method in literature [21].

Tables Icon

Table 2. The SNR and MSE values after denoising of different threshold method based on sinusoidal signal

3.2 Actual signal denoising experiment

Volatile organic compounds are the main compounds of air pollution in China. The sources of volatile organic compounds in this experiment are mainly from urban traffic. In this experiment, the OP-FTIR online measurement system is selected. The system directly measures the infrared absorption spectrum of the atmosphere in a certain period of time through the open optical path.

After obtaining the original interferogram, the data is adjusted and the interferogram is apodized. The apodization function adopts the Hamming window and the phase is corrected. Finally, the infrared spectrum is restored by the FFT function. The spectrum of the measured signal is shown in Fig. 8.

 figure: Fig. 8.

Fig. 8. (a) Fourth layer low frequency coefficient; (b) First layer high frequency coefficient; (c) Second layer high frequency coefficient; (d) Third layer high frequency coefficient; (e) Fourth layer high frequency coefficient.

Download Full Size | PDF

In a similar way, the fourth layer low frequency coefficients and the high frequency coefficients after the discrete wavelet transform are shown in Fig. 8. Figure 9 presents a front-back comparison of the high frequency information.

 figure: Fig. 9.

Fig. 9. (a) Original Signal; (b) Level 4 approximation coefficient.

Download Full Size | PDF

The denoising contrast experiment is carried out on the measured spectral signal, as shown in Fig. 10. The wavelet basis function db is used to de-noising the spectral signal with four layers of hard and soft thresholds, and the improved threshold method is also used to de-noising. The obtained spectral results are shown in figure(b), (c), (d) and (e). Similarly, the corresponding spectral difference with the original signal is shown in Fig. 11.

 figure: Fig. 10.

Fig. 10. (a) Measured signal; (b) Hard threshold de-noised signal; (c) Soft threshold de-noised signal; (d) Literature [21] threshold de-noised signal; (e) Improved threshold de-noised signal.

Download Full Size | PDF

 figure: Fig. 11.

Fig. 11. (A) The differential spectrum between(a) and (b); (B) The differential spectrum between (a) and (c); (C) The differential spectrum between (a) and (d); (D) The differential spectrum between (a) and (e). Diagram (b), (c), (d) and(e) are based on Fig. 7.

Download Full Size | PDF

Through a simple signal difference operation and the difference spectrum variation shown in Fig. 11, it can be found that the improved threshold algorithm retains more spectral features than the conventional threshold algorithm, while retaining the continuity of the method proposed in literature [21], the constant deviation problem is also solved.

4. Conclusions

This paper presents a system framework for the double threshold denoising improvement function based on the wavelet transform. Aiming at the influence of the determination of threshold value and the establishment of the threshold function on the de-noising effect, this paper proposes an optimization and improvement scheme based on the double threshold method proposed in recent years, and verifies the denoising effect from the mathematical point of view. Through experiments based on simulated signal and the measured signal, it can be seen that the modified double threshold function proposed in this paper effectively suppresses Pseudo-Gibbs phenomenon and keep the signal as smooth as possible. By observing the above figures and combining the data in Table 1, it can be clearly concluded that the improved method is significantly better than the traditional threshold method and the method proposed by liu et al. [21] in terms of denoising effect and running time.

Funding

Ministry of Science and Technology of the People's Republic of China (MOST) (JZ2015KJZZ0254).

Acknowledgements

Zhang (MS student) has put forward the main idea, designed the experiment and wrote the original manuscript. Lu (PhD professor) has revised and checked the paper. Liu (Professor) has reviewed and edited the original draft.

Disclosures

The authors declare that there are no conflicts of interest related to this article.

References

1. J. Zhu, W. Liu, Y. Lu, M. Gao, X. Zhao, T. Zhang, and L. Xu, “Research on analyzing atmospheric trasmittance based on infrared radiation measurements,” Proc. SPIE 5832, 83–90 (2005). [CrossRef]  

2. R.J. Berry, B.K. Hart, R.L. Richardson, and P.R. Griffiths, “A low-resolution spectrometer for open-path Fourier-transform infrared spectrometry,” Field Anal. Chem. Technol. 3(2), 131–138 (1999). [CrossRef]  

3. J.L. Hopkins, Introduction to Spectroscopy (Saunders College, 2001), pp. 1–19.

4. W.K. Surewicz, H.H. Mantsch, and D. Chapman, “Determination of protein secondary structure by Fourier transform infrared spectroscopy: a critical assessment,” Biochemistry 32(2), 389–394 (1993). [CrossRef]  

5. P.S. Snowball, “Spectral analysis of signals,” Leber Magen Darm 13(2), 57–63 (2005).

6. S.L Marple, “Digital spectral analysis: with applications,” J. Acoust. Soc. Am. 86(5), 2043 (1989). [CrossRef]  

7. Y. Xu, J.B. Weaver, D.M. Healy, and J. Lu, “Wavelet transform domain filters: a spatially selective noise filtration technique,” IEEE Trans. on Image Process. 3(6), 747–758 (1994). [CrossRef]  

8. P. Li, Y. Zheng, C. Han, L.J. Song, and B.F. Cao, “Wavelet threshold methods used in lightning transient electrical signals denoising,” in International Conference on Fuzzy Systems and Knowledge Discovery. 2012.

9. D. Griffin and J.S. Lim, “Signal estimation from modified short-time Fourier transform,” IEEE Trans. Acoust., Speech, Signal Process. 32(2), 236–243 (1984). [CrossRef]  

10. S. H. Nawab, T. F. Quatieri, and J. S. Lim, “Signal reconstruction from short-time Fourier transform magnitude,” IEEE Trans. Acoust., Speech, Signal Process. 31(4), 986–998 (1983). [CrossRef]  

11. P. Bassan, A. Kohler, H. Martens, J. Lee, H.J. Byrne, P. Dumas, E. Gazi, M. Brown, N. Clarke, and P. Gardner, “Resonant Mie scattering (RMieS) correction of infrared spectra from highly scattering biological samples,” Analyst 135(2), 268–277 (2010). [CrossRef]  

12. J. Bertoin and A. R. Watson, “A probabilistic approach to spectral analysis of growth-fragmentation equations,” J. Funct. Anal. 274(8), 2163–2204 (2018). [CrossRef]  

13. A. R. F. da Silva, “Wavelet denoising with evolutionary algorithms,” IEEE Trans. Circuits Syst. 15(4), 382–399 (2005). [CrossRef]  

14. I. K. Fodor and C. Kamath, “Denoising through wavelet shrinkage: An empirical study,” J. Electron. Imaging 12(1), 151–160 (2003). [CrossRef]  

15. M. Safta, P. Svasta, and M.O. Dima, “Wavelet signal denoising applied on electromagnetic traces,” in Design and Technology in Electronic Packaging. 2018.

16. D.L Donoho, “De-noising by soft-thresholding,” IEEE Trans. Inf. Theory 41(3), 613–627 (1995). [CrossRef]  

17. G. Deng, D. B. H. Tay, and S. Marusic, “A signal denoising algorithm based on overcomplete wavelet representations and Gaussian models,” Signal Processing 87(5), 866–876 (2007). [CrossRef]  

18. H. Zang, Z. Wang, and Y. Zheng, “Analysis of signal de-noising method based on an improved wavelet thresholding,” in International Conference on Electronic Measurement & Instruments. 2009.

19. J.F. Wang, “A wavelet denoising method based on the improved threshold function,” in International Conference on Wavelet Analysis and Pattern Recognition. 2014.

20. H. Cui, R. Zhao, and Y. Hou, “Improved Threshold Denoising Method Based on Wavelet Transform,” in International Conference on Intelligent Information Technology Application. 2016.

21. H. Liu, W. Wang, C. Xiang, L. Han, and H. Nie, “A de-noising method using the improved wavelet threshold function based on noise variance estimation,” Mech. Syst. Signal Process. 99, 30–46 (2018). [CrossRef]  

22. G. Chen, Y. Zeng, and L.I. Zhe, “Research of self-adaptive double threshold method for ECG signal detection,” Journal of Jinan University, 2018.

23. J. Zhang, Z. Qiang, J. Zhang, S. Lin, and J. Wang, “A novel algorithm for threshold image denoising based on wavelet construction,” Cluster Computing, 2018: 1–8.

24. Z. Di, J. Zhang, and C. Jia, “An Improved Wavelet Threshold Denoising Algorithm,” in International Conference on Intelligent System Design & Engineering Applications. 2013.

25. M. Srivastava, C.L. Anderson, and J.H. Freed, “A New Wavelet Denoising Method for Selecting Decomposition Levels and Noise Thresholds,” IEEE Access 4, 3862–3877 (2016). [CrossRef]  

26. G. Y. Chen and T. D. Bui, “Multiwavelets denoising using neighboring coefficients,” IEEE Signal Process. Lett. 10(7), 211–214 (2003). [CrossRef]  

Cited By

Optica participates in Crossref's Cited-By Linking service. Citing articles from Optica Publishing Group journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (11)

Fig. 1.
Fig. 1. Comparison of soft and hard thresholds and proposed thresholds.
Fig. 2.
Fig. 2. Original signal s and its low and high frequency parts.
Fig. 3.
Fig. 3. (a) Fourth layer low frequency coefficient; (b) First layer high frequency coefficient; (c) Second layer high frequency coefficient; (d) Third layer high frequency coefficient; (e) Fourth layer high frequency coefficient.
Fig. 4.
Fig. 4. (a) Original Signal; (b) Level 4 approximation coefficient.
Fig. 5.
Fig. 5. (a) Original pure signal; (b) Noisy signal; (c) Hard threshold de-noised signal; (d) Soft threshold de-noised signal; (e) Literature [21] threshold de-noised signal; (f) Improved threshold de-noised signal.
Fig. 6.
Fig. 6. (A) The differential spectrum between(a) and (c); (B) The differential spectrum between (a) and (d); (C) The differential spectrum between (a) and (e); (D) The differential spectrum between (a) and (f).Diagram (a), (c), (d), (e)and (f) are based on Fig. 3.
Fig. 7.
Fig. 7. (a) Original pure signal; (b) Noisy signal; (c) Hard threshold de-noised signal; (d) Soft threshold de-noised signal; (e) Literature [21] threshold de-noised signal; (f) Improved threshold de-noised signal.
Fig. 8.
Fig. 8. (a) Fourth layer low frequency coefficient; (b) First layer high frequency coefficient; (c) Second layer high frequency coefficient; (d) Third layer high frequency coefficient; (e) Fourth layer high frequency coefficient.
Fig. 9.
Fig. 9. (a) Original Signal; (b) Level 4 approximation coefficient.
Fig. 10.
Fig. 10. (a) Measured signal; (b) Hard threshold de-noised signal; (c) Soft threshold de-noised signal; (d) Literature [21] threshold de-noised signal; (e) Improved threshold de-noised signal.
Fig. 11.
Fig. 11. (A) The differential spectrum between(a) and (b); (B) The differential spectrum between (a) and (c); (C) The differential spectrum between (a) and (d); (D) The differential spectrum between (a) and (e). Diagram (b), (c), (d) and(e) are based on Fig. 7.

Tables (2)

Tables Icon

Table 1. The SNR and MSE values after denoising of different threshold method based on block signal

Tables Icon

Table 2. The SNR and MSE values after denoising of different threshold method based on sinusoidal signal

Equations (15)

Equations on this page are rendered with MathJax. Learn more.

s ( t ) = f ( t ) + σ e ( t )
W s ( j , k ) = W f ( j , k ) + W e ( j , k )
T i = σ 2 logN
σ = median ( | w j , k | ) 0.6745
k ( u , v ) = corr ( u , v ) P w ( u ) / P corr ( v ) w ( u , v )
P w ( u ) = v w ( u , v ) 2
P corr ( u ) = v corr ( u , v ) 2
h ( u , v ) = 1 ln | k ( u , v ) |
T ( u , v ) = σ 2 logN h ( u , v )
w ^ j , k = { w j , k | w j , k | λ 0 | w j , k | < λ
w ^ j , k = { s i g n ( w j , k ) ( | w j , k | λ ) | w j , k | λ 0 | w j , k | < λ
w ^ j , k = { s g n ( w j , k ) [ | w j , k | λ 1 e ( λ 1 λ 2 ) ] | w j , k | λ 2 s g n ( w j , k ) [ | w j , k | λ 1 e ( λ 1 | w j , k | ) ] λ 1 < | w j , k | < λ 2 0 | w j , k | < λ 1
w ^ j , k = { s g n ( w j , k ) [ | w j , k | λ 1 e ( λ 1 λ 2 | w j , k | ) ] , | w j , k | λ 2 s g n ( w j , k ) [ | w j , k | λ 1 e ( λ 1 | w j , k | ) ] , λ 1 < | w j , k | < λ 2 0 , | w j , k | < λ 1
SNR = 10 lg [ n = 1 N X ¯ 2 ( n ) 1 N n = 1 N [ S ( n ) X ( n ) ] 2 ]
RMSE = 1 N n = 1 N [ X ( n ) X ¯ ( n ) ] 2
Select as filters


Select Topics Cancel
© Copyright 2024 | Optica Publishing Group. All rights reserved, including rights for text and data mining and training of artificial technologies or similar technologies.