1. Introduction

Wide-field optical microscopy is a robust and efficient imaging method but has no depth sectioning capability. The 3D optical transfer function of wide field imaging shows that high spatial frequency structures are always in-focus while low frequency (especially zero frequency) components are always transmitted from every depth (the “missing-cone” problem) [1]. In contrast, scanning microscopy methods, e.g., confocal microscopy and two-photon microscopy, has 3D capability but their frame rates are often limited by the pixel residence time required to generate images with sufficient signal-to-noise ratio (SNR). Powerful methods for 3D resolved imaging based on structured light illumination have been developed recently by several groups allowing wide-field visualization of the focal plane while rejecting out-of-focus background “haze” [2–6]. When a single spatial frequency grid pattern is projected onto the sample, the microscope efficiently image only those part of the grid patterns that lies within its focal depth. Various demodulation approaches have been used for the reconstruction of optical sectioned image. There have been frequent attempts to achieve higher resolution in both transverse and axial dimensions using structured illumination in optical microscopy (bright-field and fluorescence). Based on the recorded images corresponding to the different phases of the projected grid, Wilson and associates have demonstrated 3D optical sectioning [3] and Gustafsson and associates have shown spatial resolution enhancement [4]. Essentially, these methods require recording of minimum three structured images corresponding to precise phase shifts of the grid. Most recently, Mertz and associates presented a new approach, called the HiLo microscopy [5], that combines the in-focus high frequency content extracted from a uniformly illuminated image and the in-focus low frequency content extracted from a structured light illuminated image to generate an optically sectioned image. While these methods are able to improve image contrast, the loss of out-of-focal plane photons limits the final image signal-to-noise ratio. Additionally, if the contrast of the projected grid is low this reduces the modulation and provides the weaker sectioning capability and as a result reduces the intensity of the optically sectioned image [6]. Therefore, the applicability of these methods is limited for the imaging thick, turbid biological tissues where the number of photons is already limited because of scattering. In this paper, we proposed a method, extending structured light 3D imaging methods, such as HiLo microscopy, which can assign photons generated from off-focal plane excitation improving final image SNR. Our method seeks to better utilize these “lost” photons by using the ‘prior knowledge’ about the optical transfer function of the structured light illumination. While this method may be applied for a variety of contrast mechanisms, we focused here on fluorescence based imaging. Utilizing a maximum likelihood approach, we identify the most likely fluorophores distribution in 3D that will produce the observed image stacks under structured and uniform illuminations using an iterative maximization algorithm. We further demonstrate that this approach provides comparable background rejection as existing structured light imaging methods, such as HiLo microscopy, while improving final image SNR.

2. Mathematical formulation of algorithm

Figure 1 shows the schematic of structured light imaging based wide field fluorescence microscope system. Consider the coherent light, wavelength ${\lambda }_1$ and wave number $k_1$, which is transmitted by a one dimensional sinusoidal grating (spatial frequency $k_g$) is used for fluorescence excitation under structured illumination. Let $\lambda (x,y,z)$be the fluorophores distribution function in the specimen. For specimen positioned at axial displacement location $\overline{z}$, the emission intensities corresponding to uniform and structured illuminations are defined as ${\lambda }_U(x,y,z,\overline{z})=\lambda (x,y,z,\overline{z})\times U(x,y,z)$ and ${\lambda }_S(x,y,z,\overline{z})=\lambda (x,y,z,\overline{z})\times S(x,y,z)$, respectively, where $U(x,y,z)=|E_U(x,y,z){|}^2$ and $S(x,y,z)=|E_S(x,y,z){|}^2$ are the projected uniform and structured light intensity distributions at the object space respectively. The 3D distribution of electric fields, assuming sinusoid grid focused at object plane $(\xi ,\eta )$, corresponding to uniform and structured excitations can be calculated, using the Fresnel equation [7], as

 

Fig. 1 Schematic of structured light imaging based wide field fluorescence microscope system

Download Full Size | PDF

For uniform illumination, the intensity function of the detected fluorescence photons at the CCD plane can be written as [8]

Here $h_{3D}$ is the 3D point spread function (PSF) of the system. Similarly, for structured illumination the intensity function of the detected fluorescence photons can be written as

Uniform and structured excitation patterns are focused at multiple z-planes, by moving the object in axial direction, to generate the full 3D image data.

Let $N_U(x\text{'},y\text{'},\overline{z})$ be the number of photon detected at the pixel coordinate $(x\text{'},y\text{'})$ with uniform illumination focused at axial displacement $\overline{z}$ of the object. Since it follows the Poisson distribution, the log-likelihood function can be written as [9]

Similarly, let $N_S(x\text{'},y\text{'},\overline{z})$ be the number of photon detected at pixel $(x\text{'},y\text{'})$ with structural excitation pattern focusing at depth $\overline{z}$. Since it also follows the Poisson distribution, the log-likelihood function is

The combined log-likelihood function for uniform and structured illuminations can be written, by using Eq. (4) and Eq. (5), as

Images represented by $f_U(x^{\text{'}},y^{\text{'}},\overline{z})$ do not provide depth sectioning because of the presence of out-of-focus blur and thus not suitable for 3D imaging applications. As discussed before, optical sectioned image can be generated by using demodulation method that requires the uniform $f_U(x\text{'},y\text{'},\overline{z})$ and structured $f_S(x\text{'},y\text{'},\overline{z})$ images [5], but the loss of out-of-focal plane fluorescent photons limits SNR.

Our methodology provides maximization of log-likelihood function to identify the most likely $\lambda (x,y,z)$ by utilizing an iterative maximization algorithm. The maximization process provides the reassignment of the photons originally distributed in the wide field uniform image and lost in process of optical section image generation. Since the fluorophores distribution in 3D can already be known from using existing structured light reconstruction algorithms, this known distribution is used as the ‘first guess’ for the maximization algorithm that iteratively improves the SNR of this distribution. In such case, maximization can be performed by any fast numerical optimization method. We used Newton’s method used for minimizing least square criteria or for maximizing likelihood function and the iteration step can be written as [10],

The solutions of the first and second order derivatives of the log-likelihood function, defined in Eq. (7), are given as

The maximization process updates the new estimate ${\widehat{\lambda }}^{(k+1)}(x,y,z)$of the object function for iterative step k. The accuracy of the reconstruction partly depends on the smoothness of initial estimate chosen and the constrained parameters, i.e., positivity of the fluorophores, for convergence of the algorithm. Total log-likelihood function is calculated for each iteration process and it is used for the convergence behavior. The values of log-likelihood function increase with iteration number and then its convergence decides the numbers of iteration needed for maximization. Figure 2 provides the summary of the proposed photon reassignment algorithm.

 

Fig. 2 Algorithm of photon reassignment process.

Download Full Size | PDF

3. Results and analysis

3.1. Imaging of fluorescent beads

The performance of the proposed method was evaluated with a z-stack of wide-field fluorescence images. The sample was prepared using $6\mu m$ size fluorescent, with an excitation wavelength of $480nm$ and an emission wavelength of $530nm$. We prepared $180\mu m$thick agarose gel (2%) specimen with embedded beads. A stock solution of beads, with initial concentration of $2.1\times {10}^8$particles/ml, was diluted into DI water and the concentration of beads was adjusted to $200:1$ that produced sufficient bead density for imaging multiple beads in the observed volume.

Fluorescence excitation was performed using a diode laser of wavelength $488nm$(Sapphire 488HP, Coherent Inc., Santa Clara, CA, USA). For structured illumination, a Ronchi ruling was used. After passing through the excitation optics, the resultant spatial frequency of the grid at the object plane was $320linepairs/mm$. The imaging is performed using a water immersion microscope objective, 20x, NA 1.0 (Zesis). Long pass filter (BLP01-488R-25 from Semrock, USA) used for fluorescence signals imaging. Effective pixel size at the sample plane was$0.202\mu m$. Axial positioning of the object was controlled by using the piezo-actuated stage (MAX314-3-Axis nano Max Stage, Thor Labs) and uniform and structured images are recorded corresponding to the $90\mu m$ depth range with $3\mu m$ step size using the CCD camera (MEGAPLUS, Precision Instruments, Trenton, NJ, USA) and stored on a computer. Ten sets of uniform and structured images were recorded at each axial position throughout the $90\mu m$ depth range.

The uniform and structured image for three axial positions of the specimen at $24\mu m$, $48\mu m$ and $79\mu m$ are shown in Figs. 3(a)–(c) and 3(d)–(f), respectively. The HiLo images are shown in Figs. 3(g)–(i). We considered the $150\times 150\times 150$sample points in the volume which provide object size approximately $40\mu m$ in x and y dimensions and $90\mu m$ in z dimension. To perform the maximization process, we first numerically simulate the excitation intensities corresponding to uniform and structured patterns using MATLAB^TM. These excitation patterns were simulated with same number of sample points corresponding to each plane, i.e., there were a total of 30 3D excitation matrices (the size of each matrix is $150\times 150\times 150$) for structured excitation patterns and same for uniform. 3D PSF for the microscope objective was also simulated [11]. First, we calculated the HiLo images corresponding to each axial plane and then used these images as the first prior for the object function of the corresponding plane ${\widehat{\lambda }}^{(1)}(x,y,z)$. The maximization is performed, as described in Fig. 2, and the reconstructed images are shown in Figs. 3(j)-(l) for the three aforementioned planes. Figures 3(m)-(o) are the deconvolution images of the same axial planes. Custom maximum likelihood deconvolution is carried out using information from uniform illumination images with positivity constraint. The obtained results are comparable with commercial Imaris deconvolution package (Bitplane Inc., Zurich Switzerland) although commercial package appears to include certain smoothness constraint. To observe the maximization process for our approach, values of log-likelihood function was calculated as a function of iteration cycle and we observe the convergence starts after around 30 iterations. In contrast, the deconvolution method took around 200 iterations (non-blind deconvolution). From the reconstructed results, it is apparent that the proposed method provides improvement in image quality by effectively increasing the in-focus signal while removing the out-of-focus background.

 

Fig. 3 (a)–(c) Uniform and (d)–(f) structured images of fluorescence beads at object axial positions 24 μm, 48 μm, and 79 μm, respectively; (g)–(i) are the corresponding HiLo microscopy images, (j)–(l) are the reconstructed images using proposed model, and (m)–(o) are the deconvolution images.

Download Full Size | PDF

The performance of reconstruction results are quantitatively compared with the images acquired using HiLo method by plotting the line profile, as shown dotted line in Fig. 3(c). The plot of intensity values is shown in Fig. 4(a) . It can be clearly observed that the HiLo method effectively rejects the background photons of the uniform image improving image contrast. However, the proposed approach increases the number of the in-focus photons by assigning photons generated from out-of-plane excitation to appropriate locations and simultaneously rejects the background photons and hence improves the SNR and signal-to-background ratio (SBR) of the resultant image.

 

Fig. 4 (a) Line intensity plot showing the performance of the proposed method, (b) Analysis of the in-focus and background signals, (c) SNR and SBR plots.

Download Full Size | PDF

Analysis of SNR and SBR was performed for the proposed method. We recorded 10 sets of uniform and structured images at each axial position for 90µm depth range with 3µm step size. Maximization was performed independently for each set using the proposed approach. Plot of in-focus signal (in-focus beads region) and background for an axial plane is shown in Fig. 4(b). It can be clearly observed that the maximization provides significant improvement for the in-focus signal while keeping the background level similar as for HiLo image. The improvement in the SBR for the reconstruction can be clearly observed and it is shown in Fig. 4(c).

The image SNR of the fluorescence beads is calculated as

where $\overline{I}$ and ${\sigma }_I$ represent the mean and standard deviation of the in-focus intensities of the bead image. It is important to note that recorded beads image under uniform illumination includes background noise and this background need to subtract for calculation of SNR. However, both HiLo and maximized images are subjected to background suppression. Table 1 shows the comparison of SNR, SBR and image contrast for an axial plane.

Table 1. Evaluation of image parameters

View Table

Several conclusions may be drawn from this table. First, the Maximized method improves SNR and SBR metrics compared to the HiLo method, while contrast remains mainly unaffected; this is because in the Maximized method the signal and noise statistics are taken into account explicitly, and therefore the available photons are utilized more effectively. Compared to uniform illumination (bright field), the contrast improvement is similar for HiLo and Maximized, while the SBR improvement is even more drastic for the Maximized case. However, in terms of SNR HiLo underperform compared to uniform illumination; this could be because of the filters applied to HiLo effectively removes photons from the final image and, hence, the final image is subject to noise fluctuation. For the Maximized method, the recovery of in-focus photons increases the maximized image SNR and is superior to both uniform illumination and HiLo method. The analysis at other axial planes and a comparison of the results with deconvolution method is provided in the supplementary material.

3.2. Imaging of thick tissue from zebra fish

We have used fluorescently stained zebrafish intestine tissue sample for imaging in our wide-field fluorescence microscope. Adult zebrafish was dissected to extract the intestine sample. The sample was cleaned, fixed with 4% paraformaldehyde (PFA) and permeabilized with 0.2% tritonX-100. Tissue sample was stained with Phalloidin with Alexa-488 as a fluorescent marker for labeling actin. For structured illumination, the spatial frequency of the grid at the object plane was $320linepairs/mm$. The imaging was performed using water immersion microscope objective, 20X, NA 1.0 (Zesis). Images were captured with a CCD camera and stored on a computer. Uniform and structured image for 30 axial planes were recorded with depth steps of $3\mu m$.

Figures 5(a) and 5(d) illustrates the wide field uniform fluorescence images taken for the axial positions of the specimen at $36\mu m$ and $51\mu m$, respectively. As discussed before, it is important to note, due to high scattering effect, the contrast of projected grid reduces with an increase in sample depth. This reduces the modulation and provides the weaker sectioning capability for HiLo processing and result as reduce in number of photons used to reconstruct the HiLo image. The reassignment of the photons using the proposed approach is thus particularly significant in such situation. Figures 5(b) and 5(e) show the reconstructed images at $36\mu m$ and $51\mu m$ respectively. In order to determine the efficiency of proposed approach, the results are compared with the HiLo images by plotting the intensity line profile (red dotted lines in Figs. 5(a) and (c)), which is shown in Figs. 5(c) and 5(f), respectively. The improvement in SNR of the reconstructed images compared to the HiLo images is quite apparent. By assigning out-of-plane signal to the correct locations, the in-focus image is strengthened while the background is reduced. The utilization of the ‘prior knowledge’ about optical sectioning associated with the structured illumination provides the accuracy of photon reassignment for the correct 3D spatial location and prevents noise amplification during iteration process.

 

Fig. 5 (a) and (d) Wide field fluorescence uniform images of zebrafish intestine; (b) and (e) are the corresponding reconstructed images, for tissue axial positions 36 μm and 51 μm, respectively; (c) and (f) show the line profile of intensity values

Download Full Size | PDF

4. Conclusions and outlook

We reported a method for 3D visualization of the structured light wide-field fluorescence microscopic imaging system, which provides better SNR and SBR by reassigning fluorescence photons generated from off-focal plane excitation. Novel methods for 3D resolved imaging based on structured light illumination have been developed by several groups [2–6] allowing wide-field visualization of the focal plane while rejecting out-of-focus background “haze”. While these methods improve image contrast, the loss of out-of-focal plane fluorescent photons limits image SNR. Our proposed method seeks to better utilize these “lost” photons by using the ‘prior knowledge’ about the optical transfer function of the structured light illumination. Since we observed improvement in both SNR and SBR we speculate that the proposed method should be useful to image deeper into biological tissues. It should be noted that this approach is different from typical deconvolution methods. The fluorophores distribution in 3D is already known from using existing structured light reconstruction algorithms although with limited SNR. This known distribution is used as the prior for the maximization algorithm that iteratively improves the SNR of this distribution. We demonstrate that the proposed method numerically maximizes with an order of magnitude fewer iterations than standard deconvolution [12]. More importantly, the proposed method should converge to the actual 3D structure with better fidelity since axial information is not lost in the “missing-cone” and will be evaluated in future studies. This new technique effectively accounts for out-of-focus excitation due to the lack of depth sectioning capability of wide-field microscopes. However, this technique does not take into account of out-of-focus signal due to tissue scattering of the excitation and the emission photons that may be incorporated in future algorithms. Other promising future research directions include the incorporation of this photon reassignment method into other wide field tissue imaging techniques such as various light-sheet based microscopes. For the presented results the processing time was around 1 hour for maximization of entire volume (150x150x150 pixels) by using the Intel(R) Core(TM) i5 CPU 3.33GHz processer with 4GB RAM, while the known 3D excitation patterns matrix and 3D PSF were calculated prior to maximization. Our future works will be focused on improvement of computational speed by utilizing the parallelized computing.