1. Introduction

Fluorescence microscopy and phase microscopy are two distinct imaging techniques that leverage different contrast mechanisms: fluorescence microscopy images specific structures that are labeled by fluorescent tags in a biological sample; phase microscopy, on the other hand, images the refractive index (RI) of a sample and can be used for visualizing label-free structures, while lacking molecular specificity. Multimodal microscopy methods that can image both fluorescence-labeled and label-free structures enables correlating the two. The phase images reconstruct the RI of the sample, which may give structural information about the sample, or can be used to correct sample-induced scattering effects computationally in the fluorescence.

Previous work either focused on reconstructing fluorescence signals through tissue scattering by wavefront shaping [1–3], ultrasound-assisted optical imaging [4–7], measurement of transmission [8,9] or reflection matrix [10,11], and computational optimization [12–16]; or focused on measuring RI with optical diffraction tomography [17–19], computational phase retrieval [20–24], or optical coherence refractive tomography [25]. A few existing methods recover both fluorescence and phase information [26–29], but they require sequential experiments or independent measurements, which makes it difficult to register the images spatially. No previous method reconstructs both RI and fluorescence from a single dataset captured by one camera. One of the reasons is that fluorescence microscopy is usually in epi-mode, while phase microscopy is usually in transmission mode, which limits the application for in vivo imaging. Here, we introduce an epi-mode multimodal microscopy system that merges the function of fluorescence microscopy and epi-mode optical diffraction tomography (ODT), which could become a powerful tool for bioimaging.

We provide a proof-of-principle demonstration for 3D bi-functional refractive index and fluorescence microscopy (BRIEF), which reconstructs both 3D fluorescence and 3D RI from fluorescence images captured in epi-mode. We focus the microscope at the top of the sample and collect dozens of images, each having different fluorophores within the volume ‘on’. The fluorophores deeper in the sample illuminate different parts of the phase objects near the surface from different angles. We can estimate the fluorophore 3D position from the fluorescence images and also reconstruct the 3D RI, as long as the set of captured images contains diverse illumination angles for each lateral position. To recover the 3D information, we solve an inverse problem with a physics-based multi-slice model that accounts for multiple scattering. Because both modalities are reconstructed from the same dataset captured by a single camera, the fluorescence and RI signals are strictly registered in space and time.

2. Results

The experimental setup of BRIEF is shown in Fig. 1. Our test samples are 3D phase objects (beads or cells) seeded with fluorophores that are sufficiently deep in the sample (hundreds of microns) to illuminate the phase objects from below. In general, BRIEF could be used to image fluorescence-labeled biological tissue which has either sparsely blinking fluorophores in scattering tissue (e.g., the dyes used in super-resolution localization microscopy [30,31], or the fluorophores can be selectively illuminated by a photo-stimulation setup. For the latter case, one can use a widefield fluorescence stack to identify fluorophores’ locations first, followed by stimulating them sequentially; or one can scan an illumination ‘point source’ through the sample volume and capture a 2D image at each scanning location, then only use the images in which a fluorophore is emitting. We choose the latter of these approaches for convenience. We modulate collimated laser light at 473nm wavelength with a Digital Micromirror Device (DMD) located at the relayed image plane (Fig. 1(a)). The pixels on the DMD are binned into $20 \times 20$ pixel patches (corresponding to $8 \times 8\mathrm{\mu }{\textrm{m}^2}$ at the sample), termed “super-pixels” and each super-pixel is turned on one-by-one to scan the volume laterally. Given weak scattering and sparse fluorophore distribution, either a single fluorophore or no fluorophore is excited at each scanning location; we select the images containing fluorophores as our measurements (Fig. 1(c)). The fluorophores act like point light sources inside the sample, creating a circular bright area in the measurement, inside which we see fine structures due to the phase objects (live CHO cells). The size of the illuminated area is proportional to the depth of the fluorophore: deeper fluorophores will give larger illuminated areas. Fluorophores located at different positions will illuminate different parts of the phase object from different angles. If the phase object is near or beneath the fluorophore, it is not measured by this fluorophore but measured by other fluorophores whose illuminated areas cover the phase object. If the fluorophores are on the top of sample and above the phase object, they will not be used as light sources. We collect dozens fluorescence images, each illuminated by a different fluorophore, as raw measurements, such that each part of the phase object sees a diversity of illumination angles over the set of captured images.

 

Fig. 1. 3D bi-functional refractive index and fluorescence microscopy (BRIEF). (a) In order to excite different fluorophores at different times, a collimated illumination beam is modulated by a digital micromirror device (DMD) to selectively excite each fluorophore sequentially. Emitted fluorescence light scatters through the label-free tissue above and is imaged onto a camera sensor, which is focused at the top of the sample. (b) The sample here consists of fluorescence-labeled structures on the bottom and non-labeled cells on the top. A multi-slice scattering algorithm models how fluorescence light propagates through each depth slice of the sample, scattering according to its 3D refractive index (RI). (c) Two examples of raw images captured by the camera, corresponding to fluorophore #1 and #2 in (b) being turned ‘on’. The size of the circular region is determined by the depth of the fluorophore and the numerical aperture (NA) of the system; fluorophore #1 is located deeper than fluorophore #2, so it illuminates a larger volume. Fine structures inside of the circular area carry phase information about the cells and can be considered as intensity images taken with different defocus and/or illumination angles.

Download Full Size | PDF

The 3D information in BRIEF comes from illuminating the phase objects from different angles via the spherical waves generated by the fluorophores at various depths. This is somewhat analogous to a fan-beam version of ODT, which illuminates phase objects from different angles in transmission mode. Therefore, BRIEF can incorporate similar forward models as ODT for 3D RI reconstruction. Here, we use one with a multi-slice scattering model [21,23,32] and treat the fluorescence sources inside the tissue, denoted by ${o_f}(\boldsymbol{r} )$, as spherical emitters, where each fluorophore’s light is spatially coherent (with itself) but incoherent with other fluorophores. The multi-slice model approximates the bulk tissue as a series of thin layers, where the RI of the kth layer is denoted by ${n_k}(\boldsymbol{r} )$. Light propagation through the tissue is modeled via sequential layer-to-layer propagation of the electric field. The intensity of the exit electric field at the image plane z, I_l(r;z), accounting for the accumulation of diffraction and multiple scattering, is recorded by a camera as the lth measurement. Both the 3D fluorescence distribution, ${o_f}(\boldsymbol{r} )$, and the 3D RI, n(r), are unknown in the model. However, the approximate position of fluorescence objects can be estimated from the measurements if the scattering is weak and isotropic, such that the intensity image has a clear circle area illuminated. In the lateral direction, the fluorophore position is approximately the center of gravity of the circle, which is the initial value of ${o_f}(\boldsymbol{r} )$. In the axial direction, the fluorophore position is estimated by fitting to the point-spread-function (PSF) at each axial depth. Therefore, we first estimate the 3D fluorescence distribution ${o_f}(\boldsymbol{r} )$ from multiple 2D fluorescence images and then use the expected fluorescence positions to estimate the 3D RI, $n(\boldsymbol{r} )$, by solving an optimization problem with Tikhonov regularization, denoted by $R$:

We solve this optimization problem with the fast iterative shrinkage-thresholding algorithm (FISTA), which is a first-order gradient descent algorithm. Details about the forward model and reconstruction process are in the supplementary materials.

We first demonstrate that our method can reconstruct 3D RI from fluorescence images by using a calibrated sample with ground truth information of its 3D RI. The calibration sample consists of two layers: the bottom layer of PDMS (RI 1.43) is about 150 µm thick with 0.71µm red fluorescence beads inside as the light sources (positions are unknown); the top layer of PDMS is about 100µm thick with glass beads of known size and RI (RI 1.50, size 5-50µm) inside as weak scattering phase objects (Fig. 2(a)). We first cure the bottom layer of PDMS with fluorescent beads, and then paste the mixture of PDMS and glass beads on top of the bottom layer and cure the entire sample. There is no restriction on the thickness of the two layers because we can easily adjust the number of slices and the separation between slices in the model to fit the sample. To find the positions of the glass beads for our ground truth information, we capture transmission widefield images at several axial planes with infrared LED illumination (Fig. 2(e-f)). For our BRIEF reconstruction, we capture 21 fluorescence measurements with different fluorophores on. The images are captured at 10 frames per second. A representative measurement is shown in Fig. 2(b), where we can see a defocused fluorescence light source illuminating some of the glass beads. After processed by our algorithm, the reconstructed 3D RI clearly distinguishes each glass bead in a cluster with good optical sectioning ability (Fig. 2(c-d)), and the 3D position of the glass beads matches with the ground truth widefield focus stack. The reconstructed RI of glass beads is 1.43-1.50 as shown in Fig. 2(c-d), matching well with the ground truth value. This experimental result demonstrates our ability to reconstruct 3D RI from fluorescence images.

 

Fig. 2. BRIEF for reconstructing 3D RI of glass beads from a single dataset of experimentally measured fluorescence images taken with different fluorescent beads emitting. (a) The sample consists of fluorescent beads on the bottom layer of the PDMS to act as light sources and glass beads on the top layer of the PDMS to act as non-fluorescence phase objects to be reconstructed. (b) One of the 21 raw measurements used for reconstruction, with the defocused fluorescence signal scattered by the glass beads. (c, d) Reconstructed RI of glass beads at $\Delta z = 80\mu m$ and $\Delta z = 56\mu m$ below the image plane, respectively. The 3D view of the reconstructed RI shows our technique achieves a good z-sectioning ability. (e, f) Widefield images of the sample under transmitted illumination to show the position of the glass beads as a ‘ground truth’ to compare with the reconstruction results.

Download Full Size | PDF

Next, we demonstrate BRIEF for reconstructing both fluorescence and RI from the same raw measurements of live biological cells. The test sample in this case is made by two steps: first, we fixed red fluorescent beads in PDMS on top of a coverslip; next, we coated the top surface of the PDMS with poly-lysine and cultured a thin layer of non-labeled CHO cells on it, in order to obtain a realistic biological phase object. During the experiment, we placed the sample in phosphate-buffered saline solution (RI 1.33) and collected 23 fluorescence images. An example of the fluorescence images for one excited fluorescent bead is shown in Fig. 3(b). We treat the previous experiment of glass beads as a calibration reference and used the same parameters in the optimization algorithm for reconstruction. The reconstructed RI of CHO cells at $\Delta z = 40\mu m$ below the image plane is shown in Fig. 3(c), and Fig. 3(d) shows the position of the cells using the transmission widefield image with infrared LED illumination (note that these are different contrast measurements so the images should not be compared directly).

 

Fig. 3. Experimental results with a sample consisting of fluorescent beads beneath a thin layer of alive CHO cells. (a) 3D view of fluorescent signals from fluorescent beads (magenta) and RI from CHO cells (green). (b) A representative raw measurement with one fluorescent bead ‘on’. (c) Reconstructed 3D RI of CHO cells at $\Delta z = 40\mu m$ below the image plane from 23 forward measurements. (d) Widefield intensity image of the sample under transmitted infrared illumination to show the ground truth of the cells’ lateral positions. (e) Overlap of the maximum intensity projection (MIP) of the widefield image stack of fluorescence beads excited by 473nm laser (ground truth, red) and the reconstructed 3D distribution of the fluorescence beads’ location (green) from 148 forward measurements like (b). Note this 3D fluorescence image stack is not used in the reconstruction of fluorescence distribution.

Download Full Size | PDF

In addition, our method also reconstructs the 3D fluorophore locations in the sample from the same raw data. In our case, each fluorescence image only contains one fluorophore, and the 23 reconstructed fluorophores are fairly bright and located relatively deep in the sample. The raw data carries more information about the phase objects because they have higher SNR and the fluorophores in these images illuminate a larger volume compared to the raw data containing dim fluorophores. To find a balance between reconstruction quality and computing burden, we only used these raw data for RI reconstruction. For fluorescence reconstruction, to also reconstruct dim fluorophores and fluorophores that are near the image plane, we select another 116 images (139 in total, with the 23 images used in the RI reconstruction) from the raw data that we collected in the same scanning process as described above. The reconstruction results (Fig. 3(e)) indicate the location of fluorophores, with intensity being proportional to the sum of fluorescence intensity in the measurement. For validation of our method, we also take a 3D axial-scanned image stack of fluorophores under widefield illumination with a 473nm laser (Fig. 3(e), red). This image stack is used as ground truth for fluorophore positions, and not used in the reconstruction of localization of fluorophores (Fig. 3(e), green). Given the scattering from the CHO cells is weak, the image stack is close to the ground truth of the fluorophores. Compared to localization of fluorophores from the 3D image stack, our method can reconstruct the location of most bright fluorophores accurately at high spatial precision (SSIM = 0.9786). Since the RI and fluorescence are collected in the same experiment with a single camera, they are automatically registered in the 3D volume (Fig. 3(a)), unlike previous methods [29] that collect fluorescence and RI information with two different cameras.

Knowing both the fluorescence and the 3D RI information not only provides a multimodal reconstruction of the sample’s structural and functional maps, but also can be used to digitally correct multiple-scattering effects in the fluorescence images. To demonstrate, we use a negative fluorescent USAF target as the fluorescence object (instead of fluorescent beads), placed below a visually opaque glass-bead sample as the scattering phantom, with immersion oil (RI = 1.515) in between (Fig. 4(a)). We first focus on the USAF target through the scattering glass-bead sample and take a widefield fluorescence image in focus (Fig. 4(d)). Then we move the focal plane above the glass-bead sample and collect scattered fluorescence images as the measurements for RI reconstruction, like in the previous experiments. Even though the fluorescent USAF target is not a point source, the excited area selected by the “super-pixel” of the DMD is so small that it can be treated as a point source. We reconstruct the 3D RI of the glass-bead sample from 10 measurements (Fig. 4(b)). One representative plane of the reconstructed 3D RI is shown in Fig. 4(c). This result also demonstrates this technique works for continuous fluorescent structures other than spherical fluorescent beads.

 

Fig. 4. Multi-modal microscopy for digital correction of multiple scattering in fluorescence images. (a) A negative fluorescence USAF target is used as the fluorescent sample and a highly-scattering glass-beads phantom on top of it acts as the scattering media. (b) A representative image of the raw measurements. The glass beads are illuminated by fluorescence from the USAF target instead of fluorescence beads in the previous experiments. (c) Orthogonal slice views of the reconstructed 3D RI. (d) Raw fluorescence image of the USAF target under wide-field blue laser illumination. This image is taken in focus, whereas (b) is captured with the system focused at the top surface of the sample. (e) Zoom-in view of the area in the blue box in (d), containing line pairs of element 6, group 7. (f) Reconstructed image after correcting multiple scattering with the multi-slice model. (g) Zoom-in view of the area in the red box in (f). (h) Normalized intensity profiles along the horizontal direction (H) and the vertical direction (V) in (e, blue) and (g, red).

Download Full Size | PDF

To digitally correct multiple-scattering effects in the fluorescence image (Fig. 4(d-e)), we take two steps: we first calculate the scattered PSF by the convolution of a 2D delta function and the reconstructed 3D RI of the scattering phantom (Fig. 4(c)); we next perform Richardson-Lucy deconvolution on the scattered image (Fig. 4(d)) with the scattered PSF. The reconstructed image can clearly distinguish the finest line pairs (Element 6 of Group 7, 228.1line pairs/mm) on the USAF target (Fig. 4(f-g)), whereas the scattered image cannot. Fig. 4(h) quantitatively compares the normalized intensity profiles of the line pairs in the scattered image (Fig. 4(e)) and in the reconstructed image (Fig. 4(g)) in both horizontal (H) and vertical (V) directions. Hence, we have shown that the reconstructed RI can be used to digitally correct multiple-scattering effects and improve image SNR of fluorescence imaging. Our advantage over the previous work [1–16] on imaging through scattering is that we can potentially correct scattering effect for fluorescence objects at any location in 3D inside of the scattering tissue without additional experimental measurements, since we already reconstructed the 3D RI of the whole volume.

3. Discussion

In the study, we demonstrated and validated BRIEF, a new imaging method that operates in epi mode and reconstructs 3D fluorescence and 3D RI from only fluorescence images by solving the inverse problem of multiple scattering based on a multi-slice model. We experimentally demonstrate the 3D reconstructed RI of glass beads and alive CHO cells registered with fluorescence beads in the same sample. We also demonstrate an application of BRIEF with the reconstructed RI by digitally correcting multiple scattering effects and improving the SNR of fluorescence images that are taken through a visually opaque phantom.

Our technique not only works for sparse fluorophores but also for dense fluorophores. There are two strategies to use BRIEF with dense fluorophores. First, BRIEF can leverage multiphoton absorption to only excite a diffraction-limited small region in a densely labeled fluorescent sample. For example, the state-of-the-art fluorescence microscopy has been demonstrated for fluorescence imaging at 1mm depth in the mouse brain [33]. Multiphoton excitation has better z-section ability than one-photon excitation, avoiding accidentally excitation of other untargeted fluorophores on the superficial layer. Second, in the above experiments, we excited only one fluorophore for each measurement, given that the fluorophores are sparsely distributed. For dense fluorophores, we probably cannot excite a single fluorophore at a time but excite multiple fluorophores (say N fluorophores) simultaneously at each measurement (Fig. S1). In this case, since the measured fluorescence image is the sum of the intensity of the scattered light illuminated by each fluorophore, each fluorophore needs to be measured at least N times combined with different fluorophores to acquire mutually independent measurements. As an example, we perform a simulation that contains 10 fluorophores illuminating a 3D phantom (ΔRI = 0.01) from below (Fig. S1a). The first result is reconstructed from 10 measurements and each measurement is illuminated by a single fluorophore (Fig. S1b); the second result is also reconstructed from 10 measurements, but each measurement is illuminated by 5 fluorophores simultaneously (Fig. S1c). The result from single fluorophore illumination is better than the result from multiple fluorophores illumination given the same number of measurements (SSIMsingle = 0.9999, SSIMmultiple = 0.9998). Therefore, we choose to illuminate a single fluorophore in each measurement in all the experiments above to achieve better reconstruction results, but our technique also works if illuminating multiple fluorophores in each measurement.

The resolution of our method ultimately is limited by the finite numerical aperture, so the axial resolution is poorer than the lateral resolution. Like ODT, the more measurements we take, the higher axial resolution we can achieve. However, more measurements and finer axial grid in the multi-slice model cause heavier computational burden. Current reconstruction takes over 1 hour to reconstruct the RI of $1200 \times 1200 \times 60$ pixels with a GPU (GeForce GTX 1080 Ti, NVIDIA). With prior knowledge of the sparse phase objects, we are able to resolve individual glass beads and cells with dozens of intensity measurements. We could potentially achieve higher axial resolution with more measurements and more slices in z.

When imaging through highly scattering tissue, our current method may have poor performance due to model mismatching. We assume weak and isotropic scattering in the current model, so we can accurately estimate the 3D position of fluorophores and then use this information to reconstruct 3D RI. However, under strong scattering, 3D localization of fluorophores from the raw measurements will be challenging. Also, highly scattering tissue contains more heterogeneous structures. To reconstruct RI at higher resolution in order to distinguish these structures, our method will require finer grid in the multi-slice model and more raw measurements, which will add more computational burden as we discussed above. The computational burden could be solved by using more powerful graphic cards. In addition, as the imaging depth increases, the SNR of fluorescence raw measurements will drop, which is common for fluorescence microscopy. We can try to push the depth limit by leveraging high-sensitivity cameras, large cross-section fluorophores, and high excitation power.

After the arXiv version of our manuscript came out [34], a similar idea was published [35] but it is different because: (1) it is considered for a different application of individual emission of fluorophores in localization microscopy; (2) it requires measurements from two defocused image planes, but we use only one; (3) it is simulation only, whereas we provide an experimental demonstration.

In conclusion, we provide a proof-of-concept experiment of BRIEF to reconstruct RI information from fluorescence images, which is beyond the conventional applications of fluorescence microscopy. We experimentally demonstrated the feasibility of the new method with in vitro samples, which shines light to in vivo experiments in the future. BRIEF is a versatile technique and is compatible with both one-photon and multiphoton microscopy, which will facilitate a wide range of applications in biology.