1. Introduction

Quantitative Phase Imaging (QPI) modalities have become increasingly valuable for the biological research community [1,2]. These techniques recover the sample’s morphological features without dyes or fluorescent labels using scatter-based measurements and inverse scattering physical models [1]. Tomographic QPI [2] approaches are particularly advantageous because they recover cellular and subcellular 3D morphometric features relevant for immuno-oncology [3], cytopathology [4], stem cell research [5], and numerous other fields [6,7]. Recovering this structural information without labels is ideal for live sample imaging where the object of interest can be studied in more native environments. Because live samples are typically dynamic, tomographic QPI systems require fast acquisition rates to prevent motion artifacts from deteriorating image quality. Interferometric-based techniques have achieved high-speed acquisition [4,5,8–10] but require expensive optical hardware limiting their widespread adoption in biological research. Alternatively, intensity-based QPI methods have shown excellent 3D QPI [11–13] using easily-implementable optical setups [13]. These techniques typically implement sample scanning [11,14–16], sample rotation [17], or diverse illumination [12,13] for tomographic imaging at the cost of temporal resolution. One such technique, Intensity Diffraction Tomography (IDT), implements linear tomographic QPI reconstruction in a scan-free setup using diverse illumination from hundreds of LEDs on a rectangular LED array [13]. This large dataset requires tens of seconds for a single measurement preventing this simple optical setup from evaluating living biological specimens. Here, we propose multiplexed intensity diffraction tomography (mIDT) combining multiple illuminations following model-based design choices to achieve high volume-rate recovery of living biological sample morphology.

One intensity-based technique already providing high-speed QPI measurements is differential phase contrast microscopy (DPC) [18]. This technique uses asymmetric illumination pairs to recover the object’s phase gradient for quantitative phase recovery [18–20]. This technique only requires two to four images [18,21], enabling QPI over large fields-of-view at high speeds. Due to the DPC’s low coherence illumination designs, however, this approach provides poor quality 3D phase recovery unless additional images are taken with through-focus object scanning [11]. Furthermore, these methods still sacrifice temporal resolution for volumetric recovery. By multiplexing illuminations following the linear IDT model in [13], our illumination architecture preserves the object’s 3D information under low spatial coherence conditions and reduces the image number and system exposure time for high volume-rate IDT.

Designed illumination schemes for QPI have been previously explored in numerous modalities including Fourier Ptychography [22–24] and DPC [21,25]. Existing techniques utilize nonlinear optimization [22,23], learned illumination designs [21,24,25], and multi-spectral systems [26,27] to pattern the illumination for maximized object information in reduced dataset sizes. Significant work has also incorporated model-based design in DPC to enhance object Fourier coverage through non-uniform illumination patterning [18,28–30]. These various approaches optimize 2D object information recovery instead of volumetric objects and may not translate to IDT and other tomographic QPI approaches. Our approach uses illumination design constraints based on IDT’s physical model for achieving high-speed IDT with minimal reconstruction artifacts. This method maintains the system’s resolution, complex object recovery, and volumetric reconstruction capabilities in a design adoptable for any microscope with a programmable source array.

Our work presents the multiplexed Intensity Diffraction Tomography (mIDT) illumination scheme for real-time biological sample imaging (Fig. 1). We designed our mIDT model from the linear IDT model in [13] and optimize the illumination by balancing the multiplexed system’s Fourier weight distribution, akin to minimizing the condition number of the underlying system. We utilize illumination downsampling, Poisson disk random sampling [31], and geometric constraints to achieve hardware-limited 4Hz acquisition rates with minimal reductions in reconstruction quality (Fig. 1(b)–(d)). For directly comparing our multiplexed designs with conventional IDT without introducing significant human error, we derive a semi-automated recipe for selecting the Tikhonov regularization parameter based on the mIDT scheme’s image and multiplexed illumination numbers. Using low multiplexing, highly downsampled mIDT schemes, we recover quantitative volumetric reconstructions of living Caenorhabditis elegans (C. elegans) worms (Fig. 1(e)), C. elegans embryos, and epithelial buccal cells. Our mIDT scheme provides an illumination design framework applicable to any diverse illumination source enabling high-speed acquisition with minimal loss to reconstruction quality for dynamic sample imaging.

 

Fig. 1. (a) mIDT imaging system composed of an inverted microscope equipped with an LED array. (b) mIDT reduces both acquisition speeds and image numbers. (c) Example mIDT ($N_m=6$,$L=16$) intensity images (top) and spectra (bottom) for a live C. elegans worm . (d) Example mIDT real and imaginary TFs across multiple depths. (e) Real and imaginary refractive index reconstructions and depth-coded projections of live C. elegans worm volumetric reconstructions, demonstrating minimal motion artifacts across a 1-minute acquisition period.

Download Full Size | PDF

2. Theory

2.1 mIDT forward model

We consider the first Born approximation derived IDT linear model [13] when designing our illumination patterns. This model recovers the object’s 3D complex permittivity contrast as a series of independent 2D slices from intensity images taken under oblique illumination. Following image normalization and background-subtraction, the intensity spectra $\widehat {I}$ for a single image under oblique plane wave illumination with lateral spatial frequency $\boldsymbol {u}_i$ has the form

 

Fig. 2. (a) The in-focus weight distribution $W[0]$ of conventional IDT, Annular illumination IDT, and downsampled annular illumination TFs without multiplexing. Removing LEDs from the grid provides equivalent Fourier coverage while reducing the number of images required for IDT. (b) The real and imaginary TF behavior for multiplexed symmetric (top) and non-symmetric (bottom) illuminations. The loss of phase information for symmetric illumination necessitates geometric illumination constraints to maximize the object’s recovered phase. (c) The weight distribution and VMSE comparison of mIDT designs using pseudorandom and poisson disk random sampling for LED selection. Poisson disk sampling provides equivalent or lower VMSE to pseudorandom sampling because it reduces TF overlap by spatially separating multiplexed illuminations.

Download Full Size | PDF

2.2 mIDT illumination scheme design

Our multiplexing scheme considers two categories for illumination design: 1) the multiplexed TF, and 2) the single-LED TF characteristics. From the multiplexed TF, we develop a custom metric evaluating the Fourier space weighting within the 3D bandwidth to ensure uniform coverage. The metric is derived by applying the singular value decomposition (SVD) to the multiplexed system to obtain a weight distribution $W[q]$ for every axial slice, as detailed in Appendix B. An example of this distribution for the standard IDT is shown in Fig. 2(a). We preserve this weight distribution when selecting multiplexed illumination designs using the metric

With this random search, we implement model-based design constraints on the available illuminations for each multiplexed image. First, we remove low angle illuminations with NA$\leq 0.3$ (Fig. 2(a)) because they provide minimal phase information. Second, we downsample the total available LED grid (Fig. 2(a)) to remove redundant Fourier coverage. Next, we geometrically restrict the available LEDs in each image to one quadrant of the Fourier space to prevent symmetric illumination multiplexing from cancelling out object phase information [21,23]. This behavior is highlighted in Fig. 2(b). Finally, we implement Poisson disk random sampling to enforce spatial separation between the remaining multiplexed LEDs [31]. This constraint reduces TF overlap and helps preserve the propagation phase for higher quality volumetric object recovery. We show this improvement in Fig. 2(c), where mIDT designs using poisson disk sampling show lower Volumetric Mean-Squared Error (VMSE) object reconstructions compared to conventional IDT than designs using pseudorandom LED sampling. More details on the multiplexing design are discussed in Appendix B.

2.3 Regularization with mIDT

For direct comparison of mIDT designs with conventional IDT, we evaluate their reconstruction quality based on Tikhonov regularization:

To determine the optimal $\tau$, we evaluate the signal SNR behavior under multiplexed illumination and multi-image conditions following the Wiener deconvolution analysis [32]. In Appendix C, we derive the relation for calculating the mIDT regularization given the conventional IDT regularization $\gamma$:

3. Results

3.1 Optimal multiplexed illumination

We now evaluate mIDT designs for the optimal combination of illumination multiplexing, image number, and acquisition speed for high volume-rate mIDT. We consider the visual volumetric object reconstruction, the VMSE compared to conventional IDT, and the theoretical acquisition speed for finding the optimal design. Each mIDT design is simulated using conventional IDT measurements of a $\sim 110 \times 120 \times 40 \mu m^3$ diatom biological structure fixed in glycerin ($n=1.47$). For the reconstruction, we use the regularization from Eq. (6) allowing for direct comparison across mIDT designs.

Figure 3 highlights the key factors of our evaluation. For non-multiplexed, downsampled illumination schemes, we observe the real permittivity contrast depth-coded projection (Fig. 3(a), rows) and VMSEs (Fig. 3(b)) provide low error compared to conventional IDT. Reconstruction artifacts become significant, however, with the introduction of multiplexing (Fig. 3(a), columns). We attribute these artifacts to the system point-spread function (PSF) under multiplexed conditions. Our multiplexed illumination designs attempt to generate uniform weight distribution over the recovered object bandwidth, but the TF overlap between different illuminations creates uneven distributions in the system TF (Fig. 2(b)–(c)). The resulting non-uniform system PSF from this result creates object-dependent structural artifacts and corrupts the reconstruction quality. This is evident in the blurry mIDT reconstructions shown in Fig. 3.

 

Fig. 3. (a) Depth-coded projections of conventional IDT (Upper Left) reconstructions compared with various mIDT designs. Each row is fixed with a specific multiplexing value and each column has a fixed downsampled LED grid. Downsampling without multiplexing preserves the reconstruction quality while multiplexing illuminations increases the reconstruction artifacts. (b) Volumetric mean-square errors (VMSEs) of mIDT designs using different downsampling and multiplexing conditions and their corresponding theoretical acquisition speed. Each mIDT case is compared to the conventional IDT reconstruction. The results show multiplexing and downsampling are necessary to achieve a theoretical 10Hz acquisition rate with our hardware setup.

Download Full Size | PDF

This degradation from the PSF, however, still provides lower VMSE when multiplexing without downsampling (Fig. 3(b)). With more images, the patterned system TF becomes smoothed out and reduces these artifacts. The most significant degradation occurs when both downsampling and multiplexing are implemented in mIDT. These results suggest the best mIDT reconstructions result providing the fastest acquisition speeds result from measurements using sparse illumination with minimal multiplexing.

3.2 Multiplexed vs. conventional IDT

Based on our simulations, we evaluated whether multiplexing itself is necessary. The large illumination quantity and long exposure time of 30-40ms required for conventional IDT in our setup initially motivated the use of multiplexed illumination. The artifacts introduced through mIDT, however, may provide larger VMSE than conventional IDT measurements using fewer LEDs at a lower SNR. To investigate this case, we compared IDT and mIDT under equivalent theoretical acquisition speeds using downsampled illumination grids with shorter exposure times.

For these measurements, we used our experimental setup (Fig. 1(a)) consisting of a Nikon TE 2000-U microscope equipped with a custom programmable 632nm LED array [23]; a 0.65NA, 40x objective (Nikon, MRL00402), and an sCMOS camera (PCO.Edge 5.5). In all cases, we evaluated the same diatom sample discussed in the prior section. For conventional IDT, we used the previously evaluated L=96 illumination case and an annular illumination design using high NA illuminations (NA=0.575) inspired by the work of Li et al. [33]. These choices consider whether many low SNR measurements or a few high SNR, high NA illuminations provide better conventional IDT reconstructions, respectively. We acquired multiple IDT measurements under different exposure times to match the theoretical acquisition speeds of the $N_m=3, L=32$, $N_m=6, L=32$, and $N_m=6, L=16$ mIDT measurements at 2Hz, 4.7Hz, and 9.5Hz, respectively. We also acquired L=384 conventional IDT measurements for our reference object and mIDT measurements to compare the visual reconstruction quality and VMSE.

The results from this experiment are presented in Fig. 4. We show the depth-coded projections (Left) from each reconstruction on a fixed scale and the real VMSE (Right) as a function of the measurement SNR. We estimate the SNR as the average standard deviation ratio between the signal and background ($\sigma _{\textrm {sig}}/\sigma _{\textrm {bk}}$) over all illuminations, where $\sigma _{\textrm {sig}}=\sigma _{\textrm {image}} - \sigma _{\textrm {bk}}$ denotes the standard deviation difference between the entire image and a blank image region. In agreement with our simulation, the $L=96$ conventional IDT case shows the lowest VMSE under standard exposure times Fig. 4, Right) but loses reconstruction quality with decreasing SNR. This case matches our expectations for noise-limited IDT measurements where the quality reduces with additional system noise. The mIDT measurements exhibit the next lowest error, followed by a counter-intuitive VMSE increase with longer exposure times for the $L=16$ case. This behavior can be better understood from the depth-coded projections (Fig. 4, Left). Both the $L=16$ and mIDT measurements generate structural artifacts in their reconstruction due to their sparse illuminations creating patterned weight distributions for the system TF (Fig. 2). Because these cases maintain higher SNR with longer exposure times, they recover higher contrast object features better matching the $L=384$ conventional IDT case but also amplify these artifacts. This is particularly evident for the $L=16$ case’s star-shaped reconstructions and VMSE trend. The end result is a trade-off between slow, noise-limited conventional IDT measurements and fast measurements with object-dependent structural noise. For imaging dynamic samples, these results indicate the best solution is a high-speed illumination source without significant downsampling or multiplexing.

 

Fig. 4. Depth-Coded Projections (Left) and Volumetric Mean-Square errors (Right) for conventional IDT and mIDT measurements under different SNR conditions. The $N_m=1, L=96$ case shows noise-limited reconstruction quality with increasingly underestimated object permittivity while both the mIDT and $N_m=1, L=16$ show higher contrast, better permittivity recovery with object-dependent structural artifacts. These artifacts cause the VMSE to increase with longer exposure times.

Download Full Size | PDF

Physical system constraints unfortunately make this optimal condition difficult. The system utilized here is fundamentally limited by the LED Array’s 60Hz refresh rate. Regardless of exposure time and camera acquisition speed, this system can only acquire images with six different illumination patterns for achieving 10Hz live sample imaging. This large reduction in image quantity would result in significant VMSE from the system’s uneven weight distribution. To achieve high-speed imaging with the lowest available VMSE, we are thus limited by our system constraints to mIDT measurements with large $N_m$ and small $L$. For the following experiments, we use an mIDT design with $N_m=6$ and $L=16$ for an acquisition rate of 4Hz. Despite larger error and slower acquisition speeds, we show quantitative recovery of bacterial, cellular, and tissue 3D structure on living organisms using mIDT.

3.3 Static object reconstructions with mIDT

We first compare mIDT with phase contrast microscopy (PhC) and conventional IDT on fixed biological samples. This step experimentally validates whether our mIDT design provides adequate volumetric object reconstructions without introducing severe artifacts. To provide ground-truth phase information in the sample volume, we acquire a stack of axially-scanned PhC images on epithelial buccal cells in aqueous media. We subsequently capture mIDT and conventional IDT measurements at a fixed axial plane and reconstruct the object to compare with PhC over the volume. The PhC objective used here matched the magnification and NA of our IDT objective. These results are shown in Fig. 5. Both IDT and mIDT recover the same features as PhC across the defocus planes. Due to the specific PhC objective used for this measurement, the corresponding phase features are inverted compared to IDT and mIDT. As expected from simulation, mIDT provides reduced quality reconstructions compared to conventional IDT from the use of multiplexed illumination. However, mIDT has a much faster acquisition speeds of 0.2s compared to conventional IDT at 12s and the axially-scanning PhC measurements at 40min. This trade-off between speed and reconstruction quality makes mIDT advantageous for live sample imaging where IDT’s slow acquisition speed will generate significant motion artifacts. These artifacts increase the reconstruction error in conventional IDT beyond that seen in mIDT. We show this improvement with mIDT on living C. elegans worms and embryos as well as epithelial buccal cell specimens.

 

Fig. 5. Comparison of Phase Contrast (Top), conventional IDT (Middle), and mIDT (Bottom) measurements on two epithelial buccal cells. The phase contrast measurements show inverted phase information compared to IDT. mIDT recovers identical features to PhC and conventional IDT across different depths but includes slightly more artifacts as discussed in the main text.

Download Full Size | PDF

3.4 Dynamic object reconstructions with mIDT

We show the high-volume rate imaging capabilities of mIDT on a C. elegans worm in Fig. 1(e), 6, and in Visualization 1. mIDT’s large Field-of-View (FOV) simultaneously recovers the worm’s pharynx, pharyngeal bulbs, and intestine (Fig. 6(a)) as well as high-resolution tissue features across multiple depths (Fig. 6(b)). In particular, mIDT captures fine-details including granular structures (Fig. 6, red arrows), the worm’s grinder (Fig. 6(b), white arrow), and the pharyngeal-intestinal valve (Fig. 6, white box). The worm’s wall muscles (Fig. 6, white bracket) and intestinal tract (Fig. 6, red bracket) are also visible across multiple depths. With mIDT’s high-speed acquisition, we can monitor these 3D features in time (Fig. 6(c), Visualization 1) and observe external organisms, such as bacteria (Fig. 6(c), blue arrows), interact with the worm. mIDT’s larger reconstruction error does appear more prominently with increasing defocus (Fig. 6(b)–(c)), but lipid droplets and other features are still apparent at these depths. These results highlight the potential utility of mIDT as an easily implementable tool for evaluating 3D morphology and multicellular organism response to its environment and external variables.

 

Fig. 6. (a) Full-field refractive index reconstruction of a live C. elegans worm at the in-focus plane at time $T=0$s. The full video of the reconstruction is provided in Visualization 1. (b) Outsets at $T=0$s of the live worm across multiple depths. The markers highlight the following structures: lipid droplets and granular structures (red arrows), the grinder (white arrow), The pharyngeal-intestinal valve (white box), the intestinal tract (red bar), and wall muscle (white bar). mIDT reconstruction artifacts are more prominent at defocused slice reconstructions, but some structures are still recoverable. (c) Time lapse images of the C. elegans worm moving through outset regions at $Z=0\mu$m (Top), $Z=6\mu$m (Middle), and depth projections (Bottom) through the object volume. Lipid droplets (red arrows) and external native bacteria (blue arrows) are highlighted showing finely detailed features are captured with mIDT. The variouss in the depth projection show tissues and bacteria are recovered across the reconstructed volume.

Download Full Size | PDF

We note here that motion artifacts still occur with our mIDT design under periods of rapid C. elegans motion in Visualization 1. Our system is hardware-limited by the LED array’s 60Hz refresh rate to 4Hz acquisition rates and thus is too slow for rapidly moving living samples. This problem solvable with the use of faster LED arrays, which we hope to investigate in future work.

We further show the utility of mIDT for embryogenesis using C. elegans embryos (Fig. 7, Visualization 2). Using our mIDT design, we recover the volumetric morphology of two embryos in the three-fold (red arrow) and quickening (orange arrow) stages of development. mIDT easily resolves developing tissues including the worm’s buccal cavity (Fig. 7, white box) and evaluates a cross-section of the worm’s intestine (Fig. 7, blue box). A native bacteria (Fig. 7, blue arrow) is also capture at a defocus plane with mIDT. Our mIDT architecture’s high-speed acquisition rate enables longitudinal monitoring of the embryo development, which shows significant promise for this technique in developmental biology applications.

 

Fig. 7. (Top) In-focus refractive index reconstruction of C. elegans embryo temporal measurement and (Bottom) depth-coded projections of volumetric reconstruction. The full video of the reconstruction is provided in Visualization 2. mIDT’s reconstruction quality enables the identification of the embryos in the three-fold (red arrow) and quickening (orange arrow) development stages. Individual developing tissues including the buccal cavity (white box), intestine (blue box), and native bacteria (blue arrow) are clearly recovered with mIDT.

Download Full Size | PDF

mIDT also shows promise for bacteria-cell interactions as shown in Fig. 8 and Visualization 3. Live epithelial buccal cells are evaluated in saliva and temporal projections show 3D native bacteria motion throughout the measurement (Fig. 8(a)). We observe diplococci bacteria, likely native Escherichia coli, interacting near the cells (Fig. 8(b)) as well as an unknown bacterial cluster moving within a membrane (Fig. 8(b)). A feature within this cluster is highlighted with a red arrow. Furthermore, we can track bacterial movement in 3D as shown in the maximum intensity projections of our temporally coded reconstructions (Fig. 8(c)). mIDT provides quantitative volumetric information on both cells and bacteria that could be used for tracking, cell-bacteria interaction studies, and numerous other applications. These results show mIDT is highly promising for cytopathology and immunological research fields where these interactions are critical to understanding disease propagation and infection.

 

Fig. 8. (a) Temporally-coded in-focus reconstruction of epithelial buccal cells and native bacteria. The volumetric reconstruction cross-sections capture moving bacteria across multiple depths. The full video of the reconstruction is provided in Visualization 3. (b) The refractive index reconstructions of diplococci bacteria (left) and a native bacterial cluster (right) across a one minute acquisition period. Both outsets show bacteria motion is quantitatively captured without artifacts using mIDT. The red arrow highlights a dynamic feature of the native bacterial cluster. (c) Maximum intensity projections of temporally encoded refractive index volume reconstructions of a single bacteria. The cross-sections recover 3D particle motion across multiple axial planes during the measurement highlighting mIDT’s potential for particle tracking.

Download Full Size | PDF

4. Discussion

Quantitative phase image modalities provide a unique platform for evaluating the morphology of biological specimens in their natural state. Acquiring data in native environments quickly becomes difficult because high-speed, large FOV volumetric imaging with adequate resolution is required to capture dynamic biological samples. These parameters typically requires expensive setups or sample fixation to evaluate the object without motion artifacts. mIDT removes this limitations in an easily-implementable, scan-free imaging system. Our model-based illumination design maximized the recovered object bandwidth within each image and minimized redundant sample information and phase information loss. From these designs, we achieved reconstructions with minimal quality loss at near real-time acquisition rates. We validated the reconstruction quality against both conventional IDT and PhC systems and showed its utility on dynamic samples including C. elegans worms, embryos, and epithelial buccal cell samples. This modality provides a straight-forward, easily accessible tool for wide-spread biological research applications.

Our mIDT design could be significantly improved with specializing the illumination design. Our evaluation of mIDT designs suggests the optimal reconstruction quality is achieved from highly downsampled illumination grids with no multiplexing. This result suggests that specializing the illumination hardware to use a few high power, high NA illuminations to capture the object’s information at maximum object bandwidth would improve the reconstruction quality and provide high-speed imaging. We showed this approach is successful in a separate high volume-rate IDT paper [33]. This work utilized a specialized annular LED arrays providing angles matching the objective NA. This choice enhanced the object bandwidth and reduced TF overlap to provide high-quality reconstructions from 8 illuminations with even weighting distributions for the system TF. Specializing the illumination hardware is not always advantageous, however, when multi-modal illuminations are required. Using generic square LED arrays enables other microscopy techniques, such as Darkfield [34], Fourier Ptychography [22,23,35] and Differential Phase Contrast [18], that can be advantageous for numerous research applications. Our mIDT design only modifies the illumination pattern and thus provides a flexible alternative approach to achieve high volume-rate IDT.

Another factor of consideration in our design is the procedure used for LED design selection. Our SVD-based metric is non-differentiable and thus requires search-based procedures to find optimal LED combinations. As specific illumination designs will be system-dependent, the optimal illumination choice requires significant computation time. To conserve time, we limit this search to 100 realizations and provide adequate reconstruction quality (Fig. 3) despite not achieving a true optimal mIDT design. To further improve the results, learning based illumination designs [21,24] and phase recovery [36–39] may be a fruitful area of future research.

Finally, mIDT’s reconstruction quality and robustness to large biological structures could be improved by considering multiple-scattering. mIDT’s underlying physical model relies on the first Born approximation [13] and thus is limited to weakly scattering structures. This limitation creates a trade-off between the object’s refractive index contrast and overall height to provide an accurate reconstruction. Both model-based [12,40–44] and machine learning-based [45–48] approaches have shown excellent results in extracting useful information from multiple scattering. We will expand mIDT to consider multiple-scattering in future work following these methods.

Appendix A: mIDT forward model in matrix form

For a given axial slice $q$, the real and imaginary TFs for a single-LED plane wave illumination have the following form (adapted from [13])

(7a)$$ H_{\mathrm{re}}(\boldsymbol{u},q|\boldsymbol{u}_i) = j\frac{k_0^2}{2}S(\boldsymbol{u}_i)P(\boldsymbol{u}_i)\Big\{P(\boldsymbol{u}-\boldsymbol{u}_i)\frac{e^{{-}j[\eta(\boldsymbol{u}-\boldsymbol{u}_i)-\eta(\boldsymbol{u}_i)]q\Delta z}}{\eta(\boldsymbol{u}-\boldsymbol{u}_i)} - P(\boldsymbol{u}+\boldsymbol{u}_i)\frac{e^{j[\eta(\boldsymbol{u}+\boldsymbol{u}_i)-\eta(\boldsymbol{u}_i)]q\Delta z}}{\eta(\boldsymbol{u}+\boldsymbol{u}_i)}\Big\},$$

(7b)$$ H_{\mathrm{im}}(\boldsymbol{u},q|\boldsymbol{u}_i) ={-}\frac{k_0^2}{2}S(\boldsymbol{u}_i)P(\boldsymbol{u}_i)\Big\{P(\boldsymbol{u}-\boldsymbol{u}_i)\frac{e^{{-}j[\eta(\boldsymbol{u}-\boldsymbol{u}_i)-\eta(\boldsymbol{u}_i)]q\Delta z}}{\eta(\boldsymbol{u}-\boldsymbol{u}_i)} + P(\boldsymbol{u}+\boldsymbol{u}_i)\frac{e^{j[\eta(\boldsymbol{u}+\boldsymbol{u}_i)-\eta(\boldsymbol{u}_i)]q\Delta z}}{\eta(\boldsymbol{u}+\boldsymbol{u}_i)}\Big\},$$

where $k_0$ is the wavenumber, $S$ the source function, $P$ is the real and symmetric pupil function, and $\eta$ is the axial wavevector. Derivation of these TFs may be found in [13].

We consider the full mIDT system using matrix notation. For an $N\times N\times N_z$ object recovered using $L$ $N\times N$-pixel images with $N_m$ unique multiplexed LEDs for each image, the forward model is represented as

(8)$$\boldsymbol{\mathbf{\widehat{I}}} = \boldsymbol{\mathbf{H}}\boldsymbol{\widehat{\mathbf{\Delta\epsilon}}}$$

where $\boldsymbol {\mathrm {\widehat {I}}}$ is an $LN^2\times 1$ vector containing all $L$ intensity spectra, $\boldsymbol {\widehat {\mathbf {\Delta \epsilon }}}$ is a $2N_zN^2\times 1$ vector of the object’s complex permittivity contrast, and $\boldsymbol {\mathrm {H}}$ is the system TF composed of an $LN^2 \times 2N_zN^2$ matrix with all real and imaginary multiplexed TFs, respectively. This system TF has the form

(9)$$\boldsymbol{\mathrm{H}}= \begin{bmatrix} H_{\mathrm{re}}[1,N_m^1] & \ldots & H_{\mathrm{re}}[N_z,N_m^1] & H_{\mathrm{im}}[1,N_m^1] & \ldots & H_{\mathrm{im}}[N_z,N_m^1]\\ \vdots & \ddots & \vdots & \vdots & \ddots & \vdots\\ H_{\mathrm{re}}[1,N_m^L] & \ldots & H_{\mathrm{re}}[N_z,N_m^L] & H_{\mathrm{im}}[1,N_m^L] & \ldots & H_{\mathrm{im}}[N_z,N_m^L] \end{bmatrix}$$

where $H_{\mathrm {re}}[q,N^l_m]$ and $H_{\mathrm {im}}[q,N^l_m]$ represent the TFs for the $l^{\mathrm {th}}$ multiplexed intensity at the $q^{\mathrm {th}}$ axial slice, written in the form of $N^2\times N^2$ diagonal matrices. Because our IDT model is derived from the first Born approximation [13], resulting in independent scattering contribution from each axial slice, the system TF can be simplified for recovering the $q^{\mathrm {th}}$ slice

(10)$$\boldsymbol{\mathrm{H}}_q= \begin{bmatrix} H_{\mathrm{re}}[q,N_m^1] & H_{\mathrm{im}}[q,N_m^1]\\ \vdots & \vdots\\ H_{\mathrm{re}}[q,N_m^L] & H_{\mathrm{im}}[q,N_m^L] \end{bmatrix}.$$

and allow object recovery slice-by-slice. We apply SVD analysis to this forward model to optimize illumination patterns for achieving high volume-rate mIDT.

Appendix B: mIDT illumination scheme

Our metric is derived by analyzing the multiplexed TF via SVD. The resulting diagonal matrix $W[q]$ represents the Fourier weighting for the system TF at slice $q$ for a given mIDT design

(11)$$ \begin{aligned}W[q]&= \left(\sum_{\ell=1}^L{\big{|}}H_{\mathrm{re}}[q,N_m^\ell]{\big{|}}^2\right)\cdot \left(\sum_{\ell=1}^L{\big{|}}H_{\mathrm{im}}[q,N_m^\ell]{\big{|}}^2\right) \\ & \quad - \left(\sum_{\ell=1}^LH^*_{\mathrm{re}}[q,N_m^\ell]H_{\mathrm{im}}[q,N_m^\ell]\right)\cdot \left(\sum_{\ell=1}^LH_{\mathrm{re}}[q,N_m^\ell]H^*_{\mathrm{im}}[q,N_m^\ell]\right),\end{aligned}$$

where $\cdot$ denotes element-wise multiplication. Notably, this term matches the denominator for Tikhonov-based reconstruction (in Eq. (5a) and (5b)) and controls the system conditioning for a single object slice. By optimizing this term for equal Fourier weighting across the available bandwidth, we improve the inverse problem’s overall stability without sacrificing resolution.

With the constraints and the metric, our final mIDT design procedure follows the approach described below:

• 1. Select desired number of illuminations and LED downsampling,
• 2. Define axial range for evaluating TF weight distributions,
• 3. Implement poisson disk sampling with model-based constraints,
• 4. Calculate metric with Eq. (4),
• 5. Repeat M times and select design with largest value of D.

Appendix C: Derivation and validation of regularization choice for mIDT

To derive our regularization term in Eq. (6), we consider the cost functional for a phase-only object $\widehat {\Delta \epsilon }_{\mathrm {re}}$ as a random field in the presence of additive white gaussian noise $\widehat {w}$

(12)$$D = \underset{G_{\ell}}{\mathrm{argmin}} \mathop{\mathbb{E}}{\bigg{\{}}\Big{\|}\widehat{\Delta \epsilon}_{\mathrm{re}} - \sum_{\ell}^{L}G_{\ell}\widehat{I}_{\ell}\Big{\|}^2{\bigg{\}}}$$

where $G_{\ell }$ denotes each multiplexed intensity’s transfer function and $I_{\ell } = H_{\mathrm {re}}[q,N_m^{\ell }]\widehat {\Delta \epsilon }_{\mathrm {re}}~+~\frac {\hat {w}}{N_m}$. We assume unit magnitude illumination from each multiplexed plane wave on the sample such that the reference field intensity has value $N_m$. Since we normalize the intensity image and remove this background signal, the noise is reduced by $N_m$. Applying the gradient over $G_{\ell }$ and set it to zero, we obtain the following optimal choice of $G_{\ell }$:

(13)$$G_{\ell} = \frac{H_{\mathrm{re}}^*[q,N_m^{\ell}]}{\sum_{\ell}^{L}|H_{\mathrm{re}}[q,N_m^{\ell}]|^2+\frac{\gamma}{N_m}}$$

where $\gamma =W/S$ is the SNR of each single LED illuminated intensity image, given the power spectral densities of the noise $W$ and permittivity contrast $S$, respectively. The value of $\gamma$ will vary between each intensity image due to fluctuations in the image-specific SNR, but this global approximate enables each mIDT scheme’s regularization value to be automatically predicted from the conventional IDT regularization. This enables semi-automatic regularization choices and fair comparison across reconstructions using different multiplexing designs.

We validate this predictive $\tau$ term through comparison with the manually found $\tau$ across different mIDT conditions shown in Fig. 9. We evaluate $\tau$ with fixed multiplexed illumination $N_m$ and variable image counts $L$ (Fig. 9(a)), variable $N_m$ with fixed $L$ (Fig. 9(b)), and regularization across different axial reconstructions (Fig. 9(c)). To quantify the reconstruction quality, we compare the VMSEs for predicted and optimal $\tau$ reconstructions with conventional IDT (Fig. 9(d)). For these predictions, we use $\gamma$ from manual tuning for the conventional IDT reconstruction.

 

Fig. 9. Predicted and manually determined Tikhonov regularization values for (a) fixed $N_m$ and variable $L$, (b) fixed $L$ and variable $N_m$, (c) fixed $L$ and $N_m$ with variable defocus, and (d) VMSEs comparing mIDT and conventional IDT using predicted and manually determined Tikhonov values. We observe linearly increasing $\tau$ with $L$ and linearly decreasing $\tau$ with $N_m$ as predicted from our derivations. Our VMSE is increased with our predictions but are still considered small for finding optimal mIDT designs.

Download Full Size | PDF

The manually found $\tau$ follows directly proportional and inversely proportional relationships with $L$ and $N_m$, respectively, as predicted with our derived $\tau$ relationship (Fig. 9(a)–(b)). Our prediction does not exactly match the manually found $\tau$, but this is expected since both the predicted and manually found regularization values are inherently based on user-selected values. We do observe that our predictions are within one order of magnitude of the manual $\tau$ which is acceptable for Tikhonov regularization [32]. We also observe the manual $\tau$ value varies within an order of magnitude across all axial reconstructions, indicating that a fixed regularization parameter is viable across all reconstructed slices. Furthermore, the VMSEs show a small increase in error using the predicted $\tau$ for both the real and imaginary permittivity contrast reconstructions. This error is still an order of magnitude smaller than the error introduced with mIDT and is considered insignificant. These results indicate our predictive $\tau$ choice gives us a semi-automated approach to uniformly regularizing all mIDT conditions for evaluating our available mIDT designs.

Funding

National Science Foundation (1840990, 1846784).

Acknowledgments

Alex Matlock acknowledges the National Science Foundation Graduate Research Fellowship (DGE-1840990).

Disclosures

The authors declare that there are no conflicts of interest related to this article.

References

1. T. Kim, R. Zhou, L. L. Goddard, and G. Popescu, “Solving inverse scattering problems in biological samples by quantitative phase imaging,” Laser Photonics Rev. 10(1), 13–39 (2016). [CrossRef]  

2. D. Jin, R. Zhou, Z. Yaqoob, and P. So, “Tomographic phase microscopy: Principles and applications in bioimaging,” J. Opt. Soc. Am. B 34(5), B64–B77 (2017). [CrossRef]  

3. V. Nandakumar, L. Kelbauskas, K. F. Hernandez, K. M. Lintecum, P. Senechal, K. J. Bussey, P. C. Davies, R. H. Johnson, and D. R. Meldrum, “Isotropic 3D nuclear morphometry of normal, fibrocystic and malignant breast epithelial cells reveals new structural alterations,” PLoS One 7(1), e29230 (2012). [CrossRef]  

4. Y. Park, M. Diez-Silva, G. Popescu, G. Lykotrafitis, W. Choi, M. S. Feld, and S. Suresh, “Refractive index maps and membrane dynamics of human red blood cells parasitized by plasmodium falciparum,” Proc. Natl. Acad. Sci. 105(37), 13730–13735 (2008). [CrossRef]  

5. P. A. Sandoz, C. Tremblay, S. Equis, S. Pop, L. Pollaro, Y. Cotte, G. F. Van Der Goot, and M. Frechin, “Label free 3D analysis of organelles in living cells by refractive index shows pre-mitotic organelle spinning in mammalian stem cells,” BioRxiv p. 407239 (2018).

6. H. Majeed, S. Sridharan, M. Mir, L. Ma, E. Min, W. Jung, and G. Popescu, “Quantitative phase imaging for medical diagnosis,” J. Biophotonics 10(2), 177–205 (2017). [CrossRef]  

7. Y. Park, C. Depeursinge, and G. Popescu, “Quantitative phase imaging in biomedicine,” Nat. Photonics 12(10), 578–589 (2018). [CrossRef]  

8. T. Kim, R. Zhou, M. Mir, S. D. Babacan, P. S. Carney, L. L. Goddard, and G. Popescu, “White-light diffraction tomography of unlabelled live cells,” Nat. Photonics 8(3), 256–263 (2014). [CrossRef]  

9. T. H. Nguyen, M. E. Kandel, M. Rubessa, M. B. Wheeler, and G. Popescu, “Gradient light interference microscopy for 3D imaging of unlabeled specimens,” Nat. Commun. 8(1), 210 (2017). [CrossRef]  

10. G. Kim, M. Lee, S. Youn, E. Lee, D. Kwon, J. Shin, S. Lee, Y. S. Lee, and Y. Park, “Measurements of three-dimensional refractive index tomography and membrane deformability of live erythrocytes from pelophylax nigromaculatus,” Sci. Rep. 8(1), 9192 (2018). [CrossRef]  

11. M. Chen, L. Tian, and L. Waller, “3D differential phase contrast microscopy,” Biomed. Opt. Express 7(10), 3940–3950 (2016). [CrossRef]  

12. L. Tian and L. Waller, “3D intensity and phase imaging from light field measurements in an led array microscope,” Optica 2(2), 104–111 (2015). [CrossRef]  

13. R. Ling, W. Tahir, H.-Y. Lin, H. Lee, and L. Tian, “High-throughput intensity diffraction tomography with a computational microscope,” Biomed. Opt. Express 9(5), 2130 (2018). [CrossRef]  

14. N. Streibl, “Phase imaging by the transport equation of intensity,” Opt. Commun. 49(1), 6–10 (1984). [CrossRef]  

15. J. Li, Q. Chen, J. Sun, J. Zhang, J. Ding, and C. Zuo, “Three-dimensional tomographic microscopy technique with multi-frequency combination with partially coherent illuminations,” Biomed. Opt. Express 9(6), 2526–2542 (2018). [CrossRef]  

16. J. M. Soto, J. A. Rodrigo, and T. Alieva, “Optical diffraction tomography with fully and partially coherent illumination in high numerical aperture label-free microscopy,” Appl. Opt. 57(1), A205–A214 (2018). [CrossRef]  

17. E. Wolf, “Three-dimensional structure determination of semi-transparent objects from holographic data,” Opt. Commun. 1(4), 153–156 (1969). [CrossRef]  

18. L. Tian and L. Waller, “Quantitative differential phase contrast imaging in an LED array microscope,” Opt. Express 23(9), 11394–11403 (2015). [CrossRef]  

19. D. Hamilton and C. Sheppard, “Differential phase contrast in scanning optical microscopy,” J. Microsc. 133(1), 27–39 (1984). [CrossRef]  

20. S. Mehta and C. Sheppard, “Quantitative phase-gradient imaging at high resolution with asymmetric illumination-based differential phase contrast,” Opt. Lett. 34(13), 1924–1926 (2009). [CrossRef]  

21. M. Kellman, E. Bostan, N. Repina, and L. Waller, “Physics-based learned design: Optimized coded-illumination for quantitative phase imaging,” IEEE Trans. Comput. Imaging 5(3), 344–353 (2019). [CrossRef]  

22. L. Tian, X. Li, K. Ramchandran, and L. Waller, “Multiplexed coded illumination for Fourier ptychography with an LED array microscope,” Biomed. Opt. Express 5(7), 2376–2389 (2014). [CrossRef]  

23. L. Tian, Z. Liu, L.-H. Yeh, M. Chen, J. Zhong, and L. Waller, “Computational illumination for high-speed in vitro Fourier ptychographic microscopy,” Optica 2(10), 904–911 (2015). [CrossRef]  

24. M. Kellman, E. Bostan, M. Chen, and L. Waller, “Data-driven design for fourier ptychographic microscopy,” in 2019 IEEE International Conference on Computational Photography (ICCP), (IEEE, 2019), pp. 1–8.

25. B. Diederich, R. Wartmann, H. Schadwinkel, and R. Heintzmann, “Using machine-learning to optimize phase contrast in a low-cost cellphone microscope,” PLoS One 13(3), e0192937 (2018). [CrossRef]  

26. Z. F. Phillips, M. Chen, and L. Waller, “Single-shot quantitative phase microscopy with color-multiplexed differential phase contrast (cdpc),” PLoS One 12(2), e0171228 (2017). [CrossRef]  

27. W. Lee, D. Jung, S. Ryu, and C. Joo, “Single-exposure quantitative phase imaging in color-coded led microscopy,” Opt. Express 25(7), 8398–8411 (2017). [CrossRef]  

28. J. Li, Q. Chen, J. Zhang, Y. Zhang, L. Lu, and C. Zuo, “Efficient quantitative phase microscopy using programmable annular led illumination,” Biomed. Opt. Express 8(10), 4687–4705 (2017). [CrossRef]  

29. H.-H. Chen, Y.-Z. Lin, and Y. Luo, “Isotropic differential phase contrast microscopy for quantitative phase bio-imaging,” J. Biophotonics 11(8), e201700364 (2018). [CrossRef]  

30. Y. Fan, J. Sun, Q. Chen, X. Pan, L. Tian, and C. Zuo, “Optimal illumination scheme for isotropic quantitative differential phase contrast microscopy,” Photonics Res. 7(8), 890 (2019). [CrossRef]  

31. R. Bridson, “Fast poisson disk sampling in arbitrary dimensions,” in SIGGRAPH sketches, (2007), p. 22.

32. M. Bertero and P. Boccacci, Introduction to Inverse Problems in Imaging (Taylor & Francis, 1998).

33. J. Li, A. Matlock, Y. Li, Q. Chen, C. Zuo, and L. Tian, “High-speed in vitro intensity diffraction tomography,” arXiv preprint arXiv:1904.06004 (2019).

34. Z. Liu, L. Tian, S. Liu, and L. Waller, “Real-time brightfield, darkfield, and phase contrast imaging in a light-emitting diode array microscope,” J. Biomed. Opt. 19(10), 106002 (2014). [CrossRef]  

35. G. Zheng, R. Horstmeyer, and C. Yang, “Wide-field, high-resolution Fourier Ptychographic microscopy,” Nat. Photonics 7(9), 739–745 (2013). [CrossRef]  

36. A. Sinha, J. Lee, S. Li, and G. Barbastathis, “Lensless computational imaging through deep learning,” Optica 4(9), 1117–1125 (2017). [CrossRef]  

37. Y. Rivenson, Y. Zhang, H. Günaydın, D. Teng, and A. Ozcan, “Phase recovery and holographic image reconstruction using deep learning in neural networks,” Light: Sci. Appl. 7(2), 17141 (2018). [CrossRef]  

38. T. Nguyen, Y. Xue, Y. Li, L. Tian, and G. Nehmetallah, “Deep learning approach for fourier ptychography microscopy,” Opt. Express 26(20), 26470–26484 (2018). [CrossRef]  

39. Y. Xue, S. Cheng, Y. Li, and L. Tian, “Reliable deep-learning-based phase imaging with uncertainty quantification,” Optica 6(5), 618–629 (2019). [CrossRef]  

40. U. S. Kamilov, I. N. Papadopoulos, M. H. Shoreh, A. Goy, C. Vonesch, M. Unser, and D. Psaltis, “Learning approach to optical tomography,” Optica 2(6), 517–522 (2015). [CrossRef]  

41. U. S. Kamilov, I. N. Papadopoulos, M. H. Shoreh, A. Goy, C. Vonesch, M. Unser, and D. Psaltis, “Optical tomographic image reconstruction based on beam propagation and sparse regularization,” IEEE Trans. Comput. Imaging 2(1), 59–70 (2016). [CrossRef]  

42. W. Tahir, U. S. Kamilov, and L. Tian, “Holographic particle localization under multiple scattering,” Adv. Photonics 1(03), 1 (2019). [CrossRef]  

43. S. Chowdhury, M. Chen, R. Eckert, D. Ren, F. Wu, N. Repina, and L. Waller, “High-resolution 3d refractive index microscopy of multiple-scattering samples from intensity images,” Optica 6(9), 1211–1219 (2019). [CrossRef]  

44. J. Lim, A. B. Ayoub, E. E. Antoine, and D. Psaltis, “High-fidelity optical diffraction tomography of multiple scattering samples,” Light: Sci. Appl. 8(1), 1–12 (2019). [CrossRef]  

45. Y. Sun, Z. Xia, and U. S. Kamilov, “Efficient and accurate inversion of multiple scattering with deep learning,” Opt. Express 26(11), 14678–14688 (2018). [CrossRef]  

46. S. Li, M. Deng, J. Lee, A. Sinha, and G. Barbastathis, “Imaging through glass diffusers using densely connected convolutional networks,” Optica 5(7), 803–813 (2018). [CrossRef]  

47. Y. Li, Y. Xue, and L. Tian, “Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media,” Optica 5(10), 1181–1190 (2018). [CrossRef]  

48. A. Goy, G. Rughoobur, S. Li, K. Arthur, A. I. Akinwande, and G. Barbastathis, “High-resolution limited-angle phase tomography of dense layered objects using deep neural networks,” Proc. Natl. Acad. Sci. 116(40), 19848–19856 (2019). [CrossRef]