Open Access
Add to BibSonomy
Abstract
Multifocal structured illumination microscopy (MSIM) can rapidly retrieve 3D structures of thick samples by using multi-spot excitation and detection. Although numerous super-resolution (SR) and optical sectioning (OS) methods have been introduced in this field, the existing OS-SR method in MSIM still has the difficulty in rejecting deep defocused light, which may lead to strong background signal in the retrieved results. To this end, an enhanced OS-SR method is proposed to simultaneously achieve the desired OS capability and significant resolution improvement in MSIM. The enhanced OS-SR image is obtained by combining the standard deviation image with the conventional OS-SR image in the frequency domain. The validity of the proposed method is demonstrated by simulation and experimental results.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Confocal laser scanning microscopy (CLSM) [1,2], a useful tool in bio-imaging community, has the ability to extract optical sectioning (OS) images from whole fluorescent organelles or tissues. In CLSM, a sample is raster-scanned with a single laser focal spot, and the defocus signal is effectively removed by using a pinhole. The point by point scanning in CLSM may restrict the imaging speed and make it difficult to image the temporal changes in a living specimen. Comparatively, much faster imaging speed can be achieved in multifocal structured illumination microscopy (MSIM) [3–5]. MSIM is the parallelized version of CLSM that is created by using a sparse lattice of excitation foci, and uses a detector array replacing the single-point detector. By pinholing, rescaling, and summing these multi-spot images, MSIM can obtain OS images with improved spatial resolution ($\sqrt 2 $-fold better than the diffraction limit, twofold better after deconvolution) at ∼1 Hz frame rates [3]. In conventional MSIM systems, a pinhole array is used to reject the scattered and defocused light, but is incapable of removing the deep defocused light and may lead to obvious background signal in the reconstructed images [6]. The differential multipoint-scanning confocal imaging technology [7] was presented by Dussaux et al. to decrease the defocused light, which subtracts the conventional wide-field image from the multipoint-scanning confocal image. Nevertheless, the focus signal and the signal-to-noise ratio (SNR) may also be reduced when using this method. In the standard deviation (SD) method [6], the defocused light can be greatly suppressed by calculating the values of SD of MSIM raw data.
Apart from the OS capability, much effort has been devoted to acquiring higher spatial resolution in this field. In image scanning microscopy (ISM) proposed by Muller et al., to form an ISM image, one can use the pixel-reassignment (PR) method [3,4,8,9]. All the scanned images are added together after each image is shifted by a vector that in the ideal case depicts the relative position of the corresponding detector element, properly scaled by a PR factor. The final ISM image has a point spread function (PSF) equivalent to (or smaller than) that of the ‘ideal’ (infinitely small pinhole) CLSM image, but it maintains a higher SNR. Through frequency modulation, Kuang et al. also introduced a super-resolution (SR) method, called virtual k-space modulation optical microscopy (Vikmom) [10,11], to improve the lateral resolution of the confocal microscope by a factor of 2. By processing the MSIM raw data with a sparse Bayesian learning (MSBL) algorithm, Wu et al. were able to obtain a higher-than-twofold resolution enhancement [12]. In Feng’s method, multi-spot images were used to build the stripe modulated images, and a standard construct reconstruction procedure of structured illumination microscopy (SIM) was performed to obtain the final image with enhanced resolution [13]. York et al. reported an analog implementation of structured illumination microscopy at acquisition speeds up to 100 Hz by using optical instead of digital image-processing operations [5]. While, for the inability of a pinhole array to reject deep defocused light, an enhanced OS-SR method is required in MSIM to obtain clear and fine structures of thick samples. Hexagonal line confocal SIM (XL-SIM) employ optical means to suppress out-of-focus light before its noise can spoil SIM mathematics. This not only increases tissue penetration considerably, but also provides a better S/N performance [14]. Although many OS and SR methods have been presented, only one approach exists in MSIM to reconstruct optically sectioned images with lateral resolution enhancement (OS-SR). The existing OS-SR approach in MSIM is accomplished by combining the ISM method with a pinhole array [3,4].
Herein, an enhanced OS-SR method is proposed that combines the ISM method with the SD method to obtain an image with desired OS capability and significant lateral resolution improvement. The enhanced OS-SR method has much better performance in reducing out-of-focus light than a pinhole array and has excellent lateral resolution improvement compared with the SD method [6]. Different from the existing OS-SR method in MSIM, the enhanced OS-SR image is retrieved by combining the two images (the SD and conventional OS-SR images) in the frequency domain. After summing the low-frequency parts of the SD image and the high-frequency parts of the conventional OS-SR image, the two advantages (desired OS capability and significant resolution improvement) can be found in the enhanced OS-SR image. For exciting samples with sparse multi-spot patterns, MSIM may achieve much higher imaging depth than SIM [5,15]. Thus, the proposed method is more suitable to retrieve images with clear structures of thick samples.
2. Principle
2.1 Lateral resolution improvement
The ISM method is based on the fact that the position of an emitter is at halfway between the nominal center of the excitation spot and detection position, which is also adopted to achieve lateral resolution improvement. Therefore, improved lateral resolution can be obtained by using pixel reassignment [8]. The lateral optical transfer function (OTF) of an ISM system can be simply described as
where q is the radial Fourier coordinate perpendicular to the optical axis and $\tilde{D}$ denotes the OTF of the wide-field microscope.As seen in Eq. (1), the support of the OTF is approximately twice as large as that of the wide-field OTF, which signifies a doubled lateral resolution. However, as the ideal OTF for a microscope with doubled resolution would be $\tilde{D}$(q/2), the amplitudes of high-frequency components are under-weighted in the OTF of an ISM system. The low amplitudes of high-frequency components may limit the spatial resolution of an ISM image. To further improve the spatial resolution of the ISM method, the direct solution would be to convert he OTF of an obtained ISM image into the ideal OTF with doubled resolution. As the OTF of an ISM system can be seen as the product of two ideal OTFs, the spatial resolution of the ISM method can thus be improved by re-weighting a Fourier transform with a weight function [4], as follow
where ɛ is a small normalizing parameter, which should be smaller than the maximum amplitude of |$\tilde{D}$(q)| by 1 order or more. This prevents amplification of image noise at high frequencies in Fourier space.2.2 OS method
Because the parallelized excitation and detection in MSIM may lead to an obvious image background while a thick sample is being scanned, a SD method was proposed in our previous research [6], that can significantly suppress deep defocused light by taking full advantage of the statistic property of MSIM raw data. With the shifted multi-spot illumination patterns pi, the captured MSIM images Ii can be expressed as follows
Usually, the emitted light of a defocused fluorophore will occupy a wider area on the focus plane and with weaker intensity than that of the in-focus one. Thus, even with the same set of shifted illumination patterns, the focused fluorophore may yield larger intensity fluctuation on the focal plane than the defocused fluorophore. Therefore, the defocused signal Iout can be greatly suppressed by computing the variance of MSIM raw images as follows
2.3 Enhanced OS-SR method
In the existing OS-SR method of MSIM, the pinhole array is used to reject the defocused light, and the OS-SR image is finally achieved by processing these pinholed MSIM images with the ISM approach [3]. As the pinhole array has difficulty in suppressing deep defocused light, obvious backgrounds can be seen while imaging thick samples with this OS-SR method. To overcome the issue in MSIM, an enhanced OS-SR method is proposed here that simultaneously has the advantages of the SD method in OS capability and of the ISM approach in resolution improvement.
In the enhanced OS-SR method, the OS and SR images can be calculated respectively by processing the same set of MSIM raw data with the SD method and ISM approach. As the intensity of background signal changes slowly, it will yield low-frequency components in the Fourier domain. The OS methods can improve the image contrast by suppressing background signals. Thus, compared with wide-field image, the advantage of an OS image lies in the low-frequency part of the frequency spectrum. Moreover, lateral resolution can be improved by extending the frequency range, as in the SIM technique. As mentioned above, the SD method can significantly suppress deep defocused light, and the frequency range of ISM is approximately twice as large as that of the wide-field image. Therefore, the enhanced OS-SR image in MSIM can be obtained by merging the low-frequency components of the SD image and the high-frequency components of the SR image in the frequency domain [16,17]. The computational approach to combine the frequency information of the OS image (${\tilde{I}_{OS}}$) and SR image (${\tilde{I}_{SR}}$) is given as
where L and H are the low and high-pass filter, respectively, in the frequency domain, and can be defined asThe detailed procedures of the enhanced OS-SR method are illustrated in Fig. 1.
3. Simulation
The OS capability and lateral resolution improvement of the enhanced OS-SR method were first evaluated via numerical simulations. In the simulations, a thin spoke-like sample and a uniform sample were designed using MatLab software (MathWorks, USA) and placed at the objective focal plane and defocused plane, respectively. The fluorescence density of the spoke-like sample can be expressed as
where (r, θ) are the polar coordinates in the sample plane.As shown in Fig. 2(a), the defocused plane was 3 µm from the focal plane in object space. The normal PSF for the thin spoke-like sample placed at focal plane can be determined with simulation parameters. For example, the numerical aperture (NA) of the objective lens was 1.45 (NA=1.45), and the wavelength of the fluorescence emission light was 510 nm (λem=510 nm). The image pixel size in the focal plane was set to 43 nm (the minimal pixel size that the microscope system employed can achieve). The defocused PSF (for the uniform sample placed at the defocused plane) can also be obtained from these parameters and the defocus distance (3 µm). When the two samples were first stimulated with uniform illumination and then the two stimulated samples separately convoluted with a normal PSF and defocused PSF, the wide-field image could finally be obtained by adding the two convolution results, as shown in Fig. 2(b). The simulated MSIM raw images are also wide-field images, but are illuminated with multi-spot patterns. By processing the simulated MSIM raw images with different algorithms, the SD image, conventional OS-SR image, and enhanced OS-SR images can all be retrieved separately, as illustrated in Figs. 2(c)-(e). As the recommended value [16], the filter order of the enhanced OS-SR method is set as 3. Moreover, noise was not introduced to evaluate the performance under the ideal conditions.
Fig. 2. Comparison of simulation results with defocused light. (a) Designed two-layer sample of spoke-like sample. (b)-(e) Wide-field, SD, conventional OS-SR and enhanced OS-SR images, respectively. (f) Contrast curves of wide-field, SD, conventional OS-SR and enhanced OS-SR images, respectively. Scale bars represent a distance of 2 µm.
The obvious background in the wide-field image can be seen with the defocused light (Fig. 2(b)). Comparatively, much higher image contrast can be seen in the SD image (Fig. 2(b)), which means the effective rejection of out-of-focus light. By comparing the radius of blue circles and magnified views in Figs. 2(b)-(e), it is clear that the significant resolution improvement is achieved in both the conventional and enhanced OS-SR images. However, the enhanced OS-SR method has much better OS capability than that of the conventional OS-SR method. To assess the performance quantitatively, the contrast curves C(p), which are related to the period in the wavelength, were calculated in this simulation,
where, fp represents the intensities corresponding to the period p, and max and min are maximum and minimum intensity of the whole image, respectively.From the diameters of the blue circles in Figs. 2(b)-(e), the lateral resolutions (wide-field, SD, conventional and enhanced OS-SR images) are determined and illustrated in Fig. 2(f). As the background signal is introduced in this simulation, the contrast of the wide-field image is greatly reduced. For this reason, the contrast of the wide-field image is less than 0.2 even when the stripe interval is wider than 400 nm. Owing to the difference in OS capacity, the contrasts of the SD, conventional OS-SR and enhanced OS-SR images are enhanced to different degrees. Thus, it is difficult to determine the image resolution in this simulation with the same image contrast value. Thus, the diameters of blue circles d in Figs. 2(b)-(e) are determined by continually decreasing the diameters until no periodic fluctuation can be seen on the circumference. The lateral resolution l can be then computed (l=πd/48).
It should be noted that, due to the strong background signal, the lateral resolution of the wide-field image is reduced from the theoretical value (215 nm, 0.61×λem /NA) to the computed value (226 nm). Comparatively, the resolution improvement by a factor of 1.66 is achieved both in the conventional and enhanced OS-SR images (136 nm). In addition, a slight resolution improvement also can be obtained in the SD image (187 nm).
Although the contrast curves of the wide-field, SD, and conventional OS-SR images increase linearly when the stripe interval becomes wider. Regarding the combination result of the SD image with the conventional OS-SR image, the contrast curve of the enhanced OS-SR image is nearly equal to that of the conventional OS-SR image for the narrow stripe intervals (less than 170 nm), and becomes closer to the contrast curve of the SD image with the larger stripe intervals.
To assess the noise sensitivity of the enhanced OS-SR method, the simulation fluorescence raw images (Figs. 3(a)-(c)) were separately contaminated by Poisson noise with different SNRs. As shown in Figs. 3(a)-(c), significant Poisson noise and background signal exist in raw images. Compared with the reconstructed results of the SD, conventional OS-SR, and enhanced OS-SR methods, the defocused light in raw images also has an obvious influence on the wide-field and conventional OS-SR images, but a tiny influence on the SD and enhanced OS-SR images.
Fig. 3. Comparison of simulation results with different SNRs. (a)-(c) Reconstructed results of wide-field, SD, conventional OS-SR and enhanced OS-SR images using raw images SNR values of 31, 23, and 11 dB, respectively. (d)-(g) Contrast curves corresponding to (a)-(c), respectively.
From the diameters of the blue circles in Figs. 3(a)-(c), the contrast curves (wide-field, SD, and conventional and enhanced OS-SR images) were determined and illustrated in Figs. 3(d)-(g). Compared with the contrast curves of conventional and enhanced OS-SR images with different SNR values (SNR=31, 23 and 11), the obvious contrast damage is shown in wide-field and the conventional OS-SR images in low SNR conditions, while the contrast of SD and enhanced OS-SR images are less affected. It is verified the enhanced OS-SR method is less sensitive to noise than the conventional OS-SR method.
4. Experimental results
A solid-state laser source (Sapphire 488-200 CW CDRH Coherent, Inc., USA) was used to generate an excitation beam in the MSIM system as previously described [6], and the sparse multi-spot patterns were projected by a digital micromirror device (DMD). In this experiment, a total of 224 shifted multi-spot illumination patterns were used to scan the samples. Each DMD pixel size (10.8 µm×10.8µm) was de-magnified by a factor of 150× by the telescope system, which consisted of a lens (f=300 mm) and a high-NA objective lens (Nikon, Plan Apo λ, 100×, NA=1.45) to dimensions of 72 nm×72 nm in the specimen plane. The spacing between DMD “on” pixel was set as 16 pixels. Emitted fluorescent light was filtered using a band-pass filter (ZET488/640 m, Chroma, λem = 510 nm). The MSIM raw images were then captured by a SCMOS (ORCA Flash 4.0 V2, Hamamatsu Inc., Japan) with a pixel size of 65 nm (after demagnification by 1/100 from 6.5 µm in image acquisition). In the conventional OS-SR method, the digital pinhole has a full width at half maximum (FWHM) 1.5 times wider than that of the system PSF. For all imaging, the acquisition time was set to 20 ms/frame.
A thin commercial sample (FluoCells Prepared Slide#1, Invitrogen Corp., USA) with stained BPAE cells was imaged first. In this sample, F-actin was stained using green-fluorescent Alexa Fluor 488 phalloidin. The wide-field, SD, conventional OS-SR, and enhanced OS-SR images are shown in Figs. 4(a)-(d). By comparing these images and the magnified views (Fig. 4(e)), the desired OS capability can be seen in the SD image but with little resolution improvement, and significant resolution improvement can be seen in the conventional OS-SR image with obvious background signal. Comparatively, both the desired OS capability and significant resolution improvement are achieved in the enhanced OS-SR image. The intensity profiles (Fig. 4(f)) show that the enhanced OS-SR method can distinguish two close F-actin with a distance of 144 nm between each other.
Fig. 4. Experimental results of thin fluorescent samples (F-actin in fixed cells). (a) Wide-field image. (b-d) Retrieved SD, and conventional and enhanced OS-SR images, respectively. (e) Magnified views of the regions outlined by green boxes in (a-d), respectively. (f) Normalized intensity profiles. Scale bar (in magnified views) represents a distance of 0.5 µm.
The performance in imaging thick samples was then tested with a thick commercial sample (FluoCells Prepared slide #3, Invitrogen Corp., USA, 16 µm cryostat section of mouse kidney stained with Alexa Fluor 488 WGA). The wide-field, SD, and conventional and enhanced OS-SR images of different imaging depths (z = 1, 8, and 16 µm, respectively) are shown in Figs. 5(a)–5(c). It is clear that, strong background signals exist in the wide-field images, which yields the difficulty in observing sample structures. By comparison, the desired OS capability can be obtained by the SD method. In addition, both significant resolution improvement and obvious background can be seen in the conventional OS-SR images. As expected, the enhanced OS-SR images demonstrated two advantages even with an imaging depth of 16 µm. It can also be also demonstrated with the intensity profiles (Fig. 5(e)), that more rapid intensity fluctuations can be seen in the conventional and enhanced OS-SR images, which means finer structures are being reconstructed. Moreover, the OS capability can be verified with the fluctuation range of the entire intensity profile, and the enhanced OS-SR and SD images have much higher fluctuation ranges (0.763) than those of the conventional OS-SR image (0.617) and wide-field image (0.379).
Fig. 5. Performance verification with thick sample (16 µm cryostat section of mouse kidney) and different imaging depths (z = 1, 8, and 16 µm). (a-c) Wide-field, SD, and conventional and enhanced OS-SR images, respectively. (d) Magnified views of box in (c). (e) Intensity profiles along the white lines in magnified views. Scale bar (in magnified views) represents a distance of 2 µm.
To further demonstrate the feasibility of the enhanced OS-SR method for in vivo applications, a zebrafish larva (10 dpf) stained using a 2 µg/mL Live Cell Tracking Kit (Green Fluorescence, Abbkine, KTA1002) for 4 h at room temperature in the dark was imaged. Here, only a 24 µm region at the brain near the zebrafish eye was selected for imaging with different depths. The wide-field, SD, and conventional and enhanced OS-SR images of different imaging depths (z = 1, 16, and 32 µm, respectively) are shown in Figs. 6(a)-(c). At the depth of 32 µm, the intensity profiles (Fig. 6(c) Right) of the enhanced OS-SR image still show more rapid intensity fluctuation and a higher fluctuation range than the SD and OS-SR images, respectively. These results firmly demonstrate that the enhanced OS-SR method can simultaneously achieve the desired OS capability and significant resolution improvement in MSIM. In our experiments, the living zebrafish sample was imaged up to 32µm. Although the MSIM system can actually achieve a sample depth of up to 50 µm [4], its performance may be greatly degraded by the scattered and out-of-focus light in living samples. The enhanced OS-SR approach has good performance in reducing out-of-focus light, but scattered light in living zebrafish may degrade the depth of MSIM imaging.
Fig. 6. Performance verification of living zebrafish larva with different imaging depths (z = 1, 16 and 32 µm). (a-c) Wide-field, SD, and conventional and enhanced OS-SR images with imaging depths of 1, 16 and 32 µm, respectively. (d-f) Intensity profiles along white dotted lines in magnified views correspond to (a)-(c).
5. Conclusions
An enhanced OS-SR method in MSIM is proposed, which is accomplished by combining a SD image with a conventional OS-SR image in the frequency domain. Compared with the SD and conventional OS-SR methods, both the desired OS capability and significant lateral resolution improvement can be achieved by the proposed method. The performance of the enhanced OS-SR method has been demonstrated with simulation and experimental results. The proposed method is suitable to reconstruct the fine 3D structures of thick samples.
Funding
National Natural Science Foundation of China (61525503, 61620106016, 61775144, 61835009, 61975131); Project of Department of Education of Guangdong Province (2016KCXTD007); Natural Science Foundation of Guangdong Province (2018A030313362); Shenzhen Fundamental Research Program (JCYJ20170412105003520, JCYJ20170818141701667, JCYJ20170818144012025).
Disclosures
The authors declare no conflicts of interest.
References
1. T. Wilson, Confocal Microscopy (Academic, 1990).
2. C. J. R. Sheppard, “Super-resolution in confocal imaging,” Optik 80(2), 53–54 (1988).
3. A. G. York, S. H. Parekh, D. D. Nogare, R. S. Fischer, K. Temprine, M. Mione, A. B. Chitnis, C. A. Combs, and H. Shroff, “Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy,” Nat. Methods 9(7), 749–754 (2012). [CrossRef]
4. M. Ingaramo, A. G. York, P. Wawrzusin, O. Milberg, A. Hong, R. Weigert, H. Shroff, and G. H. Patterson, “Two-photon excitation improves multifocal structured illumination microscopy in thick scattering tissue,” Proc. Natl. Acad. Sci. U. S. A. 111(14), 5254–5259 (2014). [CrossRef]
5. A. G. York, P. Chandris, D. D. Nogare, J. Head, P. Wawrzusin, R. S. Fischer, A. Chitnis, and H. Shroff, “Instant super-resolution imaging in live cells and embryos via analog image processing,” Nat. Methods 10(11), 1122–1126 (2013). [CrossRef]
6. W. Zhang, B. Yu, D. Y. Lin, H. H. Yu, S. W. Li, and J. L. Qu, “Optimized approach for optical sectioning enhancement in multifocal structured illumination microscopy,” Opt. Express 28(8), 10919–10927 (2020). [CrossRef]
7. C. Dussaux, V. Szabo, Y. Chastagnier, J. Fodor, J. F. Léger, L. Bourdieu, J. Perroy, and C. Ventalon, “Fast confocal fluorescence imaging in freely behaving mice,” Sci. Rep. 8(1), 16262–16276 (2018). [CrossRef]
8. C. B. Müller and J. Enderlein, “Image scanning microscopy,” Phys. Rev. Lett. 104(19), 198101 (2010). [CrossRef]
9. O. Schulz, C. Pieper, M. Clever, J. Pfaff, A. Ruhlandt, R. H. Kehlenbach, F. S. Wouter, J. Grobhan, G. Bunt, and J. Enderlein, “Resolution doubling in fluorescence microscopy with confocal spinning-disk image scanning microscopy,” Proc. Natl. Acad. Sci. U. S. A. 110(52), 21000–21005 (2013). [CrossRef]
10. C. F. Kuang, Y. Ma, R. J. Zhou, G. A. Zheng, Y. Fang, Y. K. Xu, X. Liu, and T. C. S. Peter, “Virtual k-space modulation optical microscopy,” Phys. Rev. Lett. 117(2), 028102 (2016). [CrossRef]
11. R. Cao, C. F. Kuang, Y. Li, and X. Liu, “Super-resolution via saturated virtual modulation microscopy,” Opt. Express 25(26), 32364–32379 (2017). [CrossRef]
12. J. J. Wu, S. W. Li, H. Q. Cao, D. Y. Lin, B. Yu, and J. L. Qu, “Resolution improvement of multifocal structured illumination microscopy with sparse Bayesian learning algorithm,” Opt. Express 26(24), 31430–31438 (2018). [CrossRef]
13. L. Feng, L. F. Zhou, X. L. Sun, Q. Xu, P. Chen, X. L. Wang, and W. W. Liu, “A method for the reconstruction of multifocal structured illumination microscopy data with high efficiency,” Sci. Rep. 9(1), 13378–13386 (2019). [CrossRef]
14. M. Schropp, C. Seebacher, and R. Uhl, “XL-SIM: Extending super resolution into deeper layers,” Photonics 4(4), 33–50 (2017). [CrossRef]
15. Y. Wu and H. Shroff, “Faster, sharper, and deeper: structured illumination microscopy for biological imaging,” Nat. Methods 15(12), 1011–1019 (2018). [CrossRef]
16. H. Shabani, A. Doblas, G. Saavedra, E. S. Ortiga, and C. Preza, “Improvement of two-dimensional structured illumination microscopy with an incoherent illumination pattern of tunable frequency,” Appl. Opt. 57(7), B92–B101 (2018). [CrossRef]
17. M. G. L. Gustafsson, L. Shao, P. M. Carlton, C. J. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, and J. W. Sedat, “Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination,” Biophys. J. 94(12), 4957–4970 (2008). [CrossRef]
