1. Introduction

Structured illumination microscopy (SIM) is the widefield super-resolution technique that results in a resolution that is twice that of classical optical resolution by combining only a small number of raw images [1–3]. SIM involves illuminating the sample with a diffraction-limited sinusoidal pattern and acquiring images with different phase shifts and orientations of the illumination pattern. The acquired raw images are transformed in such a way that they represent the output of several optical transfer functions (OTFs) having frequency support beyond the classical diffraction limit; these extended OTFs are simply the shifted versions of the original diffraction limited OTF covering an isotropically extended volume in the Fourier space. Typically the transformed data are put together via Wiener filtering to obtain a properly-weighted super-resolved image. The nonlinear extension of SIM known as saturated SIM achieves a resolution that is more than twice the classical resolution limit by using nonlinear response of the fluorophores to laser irradiation [4]. Example applications that led to new biological findings by the use of 3D SIM include study of nuclear envelope [5], spindle structure [6], chromosome structure during cell division [7], and plasmodesmata [8], and so on.

One of the major factors leading to the loss of resolution in any widefield-based technique is the depth dependent spherical aberration caused by mismatch between the mean refractive index of imaging specimen and the refractive index of the immersion medium. As with any other widefield system, 3D SIM suffers loss of resolution due to spherical aberration while imaging thick samples. In conventional widefield microscopy, the effect of depth dependent spherical aberration has been well studied [9,10] and optical [11,12] and computational [13–15] methods to correct these aberrations have been proposed. In SIM, the effects of depth dependent spherical aberrations have so far only been considered for systems that use 2D illumination patterns for z-sectioning [16], but never for the systems that achieve three-dimensional super-resolution.

To treat the effect of depth dependent spherical aberration on three-dimensional SIM systems, we derive a compact model representing the object to image formation that accurately accounts for the effect of spherical aberration on both illumination and detection. From the model, we demonstrate that the aberration has different effects on the illumination and detection due to the fact that the detection works on the incoherent fluorescence signal, whereas the illumination pattern is generated using the mutual coherence of the illuminating wavefronts. Specifically, we show that the aberration effect on detection causes to a depth dependent attenuation leading to deviation of the actual transfer function from the standard convolution model. On the other hand, aberration of the illumination pattern causes only a deviation from the standard imaging model with no increased attenuation at high frequencies. Using the new 3D model, we provide guidelines for optical corrections of aberrations with different complexities, and explain how the proposed corrections simplify the forward model. In particular, we show that it is possible to correct the aberration for both illumination and detection using a deformable mirror only on the detection path of the microscope.

2. Derivation of the depth-variant model

In a widefield imaging system, there are two quantities representing axial distances: (i) the distance of the focal plane of objective lens from the coverslip, which is the user-controlled sectioning variable, z, and (ii) the distance of a given plane in the imaging sample from the coverslip, which is the object depth variable z′. Assuming a depth independent imaging response is equivalent to supposing that the transfer function depends only on the difference between these quantities; this leads to a 3D representation of the imaging transfer function. In reality, the transfer function depends on both the axial variables, and the term “depth variant response” signifies the dependence of the imaging response on the depth variable z′.

In the following, we will describe a schematic of a SIM system emphasizing the dependence of the transfer function on the above-mentioned axial variables. We will then describe the 4D amplitude transfer function that includes z and z′ as its independent variables which is responsible for making the SIM depth-variant. Next, from the amplitude transfer function, we formulate the depth dependent imaging transfer function for detection. Next we derive the expression for the illumination intensity pattern in terms of z and z′ and finally derive the overall imaging model of a SIM system. In this paper, only the forward model that represents the raw images in terms of the imaging specimen is developed. We do not consider the reconstruction problem.

2.1. Description of the imaging set-up

Figure 1(a) depicts the imaging set-up of a 3D SIM. The light exiting the fiber is collimated by the lens CL and directed to a linear transmission phase grating located at the plane P′_IM, which diffracts the beam into several orders. A beam block in an intermediate pupil plane (not shown) discards all diffraction orders except orders 0 and ±1. The three beams are refocused by the tube lens TL′ so that each forms an image of the fiber end-face in the back focal plane (P_BF) of the objective lens. The beams produced as diffraction orders +1 and −1 were focused near opposing edges of the P_BF aperture, and order 0 at its center. The line connecting the fiber images at the P_BF will be orthogonal to the grating lines. The objective lens recollimates the beams and makes them intersect with each other in the objective lens’s focal plane, where they interfere to form an illumination intensity pattern with both axial and lateral structure. The overall intensity variation is essentially two-dimensional with the principal directions of variation are the z-axis and the direction orthogonal to the pattern direction of the diffraction grating. In particular, the 2D intensity pattern at the focal plane of the objective lens (P_F) is a demagnified and lowpass-filtered image of the grating.

 

Fig. 1 3D structured illumination microscope. (a) optical set-up; (b) simplified schematic.

Download Full Size | PDF

The fluorescent object to be studied is mounted under a glass coverslip and placed below the objective lens. The space between the objective lens and the coverslip is filled with the immersion medium (oil) whose refractive index matches with that of glass. Hence the coverslip, even though its thickness is non-negligible, can be represented by a single plane, P_I. The refractive index of the object differs from that of the coverslip and oil, and is typically assumed to be uniform. Hence, the entire medium below the objective lens is composed of two regions of differing refractive indices separated at plane P_I. Fluorescence light emitted by the sample as a response to the above-mentioned illumination pattern is gathered by the same objective lens and deflected by a dichroic mirror to the tube lens, TL, which focusses the light onto the image sensor located at the image plane P_IM.

Figure 1(b) gives an alternative schematic of the SIM system that will facilitate the derivation of the depth dependent model in terms of the depth variable z and z′ defined at the beginning of this section. A 3D image acquisition involves a sequence of 2D acquisitions with different values of z such that P_F samples the entire object depth with an appropriate step-size. For each value of z, the acquired 2D image includes the sum of contributions from all the planes in the object corresponding to different values of z′. It is important to note that the image plane P_IM and the grating plane P′_IM are optically identical with the respect to the sample space, i.e., the transfer functions from these planes to the sample space are identical. It should also be emphasized that this function is mainly determined by transfer function from P_BF to the sample space, since the light propagation from the planes P_IM and P′_IM to P_BF is a simple Fourier transformation, and aberrations therein are negligible. This transfer function is a 4D function, which we will denote by T, and is composed of a set of 2D functions representing the transformation from a given sample plane P_S to P_BF for all relevant values of z and z′. In the section 2.2, we specify the above mentioned amplitude transfer function, T. From this T, in the section 2.3, we express the 4D intensity transfer function, h, that relates the measured image in terms of the emitted fluorescence intensity. The transfer function h has been well characterized [9, 10, 13–15]. We however re-derive in the section 2.3 in terms of present notations to facilitate the readability. Then in the section 2.4, we determine the effect of T on the illumination intensity. Finally, we derive the complete SIM transfer function.

2.2. The transfer function from a sample plane to the back-focal plane

In order express this transfer function from P_S to P_BF in Fourier space (see Fig. 1), we first consider the transfer function for z = 0 and z′ = 0. For an ideal objective lens (objective lens with perfect immersion correction), this is an aperture function in 2D Fourier space whose value is identity inside a circular region and zero outside. In practice however, this function will contain phase variations inside the circular region to represent the phase aberrations in the objective lens. Denoting the lateral Fourier Frequencies by X and Y (note the capitalization), let P(X,Y) = A(X,Y)exp(j2πQ(X,Y)) be this transfer function, where Q is the function representing the aberrations, and A(X,Y) is the aperture function. The function A(X,Y) is given by

T(X,Y,z,z′) represents the wavefront at P_BF when a point source is placed at P_S. By reciprocity, it is also the Fourier transform of the wavefront at P_S if the image plane P_IM is illuminated with a point source. Equivalently, it is the Fourier transform of the wavefront at P_S if the P_BF is illuminated by a plane wave. T(X,Y, z, z′) forms the basis for the derivation of the imaging model with or without structured illumination. Note that T is in a mixed representation, where lateral variables are in the Fourier space, and the axial variables are in real space.

2.3. The detection transfer function

Let S_a(x,y,z′) be the emitted fluorescence amplitude distribution and let S_a(X,Y, z′) be its section-wise Fourier transform. For a given value of z, the wavefront amplitude distribution at P_BF is given by

Under this notion, when the system is assumed to be depth independent, for example when n_imm = n_obj, the emission response is a function of only the difference z – z′. In other words, the depth invariance corresponds to following relation:

In order to derive the overall depth dependent response in a form that is comparable with the conventional depth invariant model, we first transform Eq. (2) such that it has highest resemblance to a standard convolution. To this end, we apply a z′ dependent shift on z, such that the value of latter satisfies the following condition: z = 0 represents the position of the focal plane P_F such that the 2D image of the object plane at z′ reaches the image plane P_I with minimum distortion. This shift is known as the focus shift [18]. It is known that this shift is proportional to z′ by a factor that depends on refractive indices involved. Let the transformed DV-PSF be given by

2.4. The effect of depth dependent spherical aberration on the illumination pattern

With reference to Fig. 1(b), let d_k be the vector in lateral plane representing the direction orthogonal to grating lines. The complex amplitudes of the selected plane waves exiting the grating (P′_IM) can be represented by ${\{\mathbf{u}}_{k0}\text{exp}\left(j2\pi {\mathbf{v}}_0^T\mathbf{x}+{\varphi }_{k0}\right)$, ${\mathbf{u}}_{k_{+}}\text{exp}\left(j2\pi {\mathbf{v}}_{k_{+}}^T\mathbf{x}+{\varphi }_{k_{+}}+{\varphi }_s\right)$, ${\mathbf{u}}_{k_{-}}\text{exp}\left(j2\pi {\mathbf{v}}_{k_{-}}^T\mathbf{x}+{\varphi }_{k_{-}}-{\varphi }_s\right)\}$, where x = [x,y,z] represents 3D spatial position. Here, {v₀, v_k+, v_k−} are the wave vectors. The vector v₀ is parallel to the z-axis and is of the form ${\mathbf{v}}_0={\left[\begin{array}{ccc}0& 0& 1/\lambda \end{array}\right]}^T$, where λ is the wavelength of illumination. The remaining vectors will be of the form

The objective lens performs an inverse transform of the wavefront at P_BF and projects into the sample space with a 3D intensity pattern that is dependent on both z′ and z. As mentioned before, the principal directions of intensity variation are the z-axis and the direction orthogonal to the pattern direction of the diffraction grating. For each direction k, five 3D stacks are acquired with different values of the phase ϕ_s, and this process is repeated for directions k = 0,1,2 such that the directional vectors {(X_k,Y_k), k = 0,1,2} are 120° apart in the x – y plane. Let E_k(X,Y, z, z′) represent the set of 2D Fourier transform of the illumination of the object plane located at z′ with focal plane positioned at z. This function is determined by the transfer function T(X,Y,z,z′) and is given by E_k(X,Y, z, z′) = E_BF,k(X,Y)T(X,Y, z, z′).

Finding the effect of depth dependent spherical aberration is equivalent to finding the illumination intensity as function of four variables, x,y,z, and z′; in contrast, as we will show, ignoring the depth dependent aberration is equivalent to assuming the illumination intensity to be a function of three variables only, i.e., a function of x, y, and z′ – z. Let L′(x,y,z,z′) be the illumination intensity function, and let L̂(X,Y,z,z′) be its partial Fourier transform taken along x and y directions. It is given by

In the appendix A, we show that for a sinusoidal grating (when only the 0th and ±th orders are projected on the back focal plane), L′(x,y,z,z′) can be expressed as

Now we consider the effect of the finite size of the fiber end-face. First we note that light from the laser source typically goes through a phase scrambler before passing through diffraction grating, and hence each pair of points at the fiber end-face are mutually incoherent. However, each photon exiting the fiber is split by the grating into the three orders, satisfying the need for mutual coherence required to create the interference pattern as given by Eq. (7). Further,for all such triplets, the relative distances and the phase differences among three points will be the same. Consequently, the illumination patterns generated by each point of the fiber end-phase will all be identical, and hence the present analysis equivalently accommodates the finite size of the fiber.

2.5. The complete imaging model

Under structured illumination, the emitted fluorescence intensity is the product of the object dye structure and the illumination intensity. As discussed above, due to the depth dependent spherical aberration, the illumination intensity is z-dependent. The fluorescence intensity is given by

Figure 2 gives the flow chart for the complete depth variant SIM model given in Eq. (10). The first term in Eq. (10) is represented by the branch A in the flow-chart and it expresses the widefield imaging operation with depth dependent aberration as developed in [15]. The fourth term is represented by the branch B and it is the widefield detection on the laterally modulated signal S_k2(x,y,z) = S(x,y,z) cos(2P(x, y)) with P(x, y) = 2π(X_kx + Y_ky) + φ_k – ϕ_s. Note that the modulation frequency here is twice that of the wavefront at the back-focal plane P_BF. This term is responsible for the doubling of the lateral resolution. The second and third terms correspond to the axial resolution extension. They are represented by the branches C.1 and C.2 and express the depth variant convolutions by functions h(x,y,z,z′)cos(2πf_zz – φ_kz) and h(x,y,z,z′)sin(2πf_zz – φ_kz), where multiplications by cos(2πf_zz – φ_kz) and sin(2πf_zz – φ_kz) provide the axial resolution extension. Inputs for these two terms are derived from S_k(x,y,z) = S(x,y,z)cos(P(x,y)) by multiplying with axial functions cos(P_z(z)) and sin(P_z(z)) that represent the effect of aberration on the illumination, where P_z(z) = 2πf_z(a – a_f)z. Note that the function S_k(x,y,z) is obtained by laterally modulating the original signal with the frequency of the illumination amplitude at the back-focal plane. Further, recall that a is the detection focus shift, and a_f is the illumination focus shift given by a_f = f′_z/f_z with the frequencies f_z and f′_z being given by Eqs. (8) and (9).

 

Fig. 2 Flow diagram of SIM imaging model with depth dependent spherical aberration.

Download Full Size | PDF

When the refractive indices match, i.e., when n_obj = n_imm, the DV-PSF h(x,y,z, z′) in Eq. (10) becomes independent of z′ and hence the depth variant convolution DVC becomes a standard convolution. Further, the detection focus shift becomes a = 1, and, with reference to Eqs. (8) and (9), the illumination focus shift also becomes a_f = 1. As a result, P_z(z) becomes zero and hence the branch C.2 disappears; further, the multiplicative factor in the branch C.1 becomes equal to 1. The equivalent imaging equation is given by

We will now qualitatively compare the effect of depth dependent aberration on the detection and illumination. To this end, we compare Eq. (10) with the depth invariant imaging model given in Eq. (11). The main difference is that the detection response is taken care by a simple convolution by h(x,y,z) in the depth-invariant model, and by the depth-variant convolution by h(x,y,z,z′) in the complete model. It is well known that this difference is more than a difference in the complexity. For z′ > 10 μm, the Fourier magnitude of the 3D transfer function falls-off steeper meaning that signal strength of the high frequency components is much lower than the case without spherical aberration [9]. Hence, spherical aberration will lead to the loss of signal even if the exact model is used in the reconstruction. Next, to analyze the effect of aberration on illumination alone, we first replace DVC by the convolution by h(x,y,z). We get

3. Imaging with adaptive optics

In this section, we analyze the effect of adaptive optics schemes proposed in the papers [11,12] on the imaging model, and show how the models become simplified under these schemes. An electronically controlled deformable mirror is placed in a complementary plane that is optically identical to P_BF, and mirror shape is controlled in a z-dependent way such that the spherical aberration is compensated. There are two type of schemes of compensating for depth dependent spherical aberration as explained below.

In the first scheme, the physical position of P_F is kept fixed at P_I (see Fig. 1), and z-sectioning is performed only by controlling the shape of the mirror. For each value of z, mirror shape is controlled to match the function DM₁(X,Y, z) = −zn_objN(X,Y, z, n_obj) [11]. The resultant transfer function is the product of T(X,Y,0,z′) and exp(j2πDM₁(X,Y, z)), which is given by

To derive the effect of adaptive optics on the detection, we first write the detection equation based on the new coherent transfer function T_ao(X,Y, z, z′):

Contrary to the general belief, it is possible to make the imaging model entirely free of depth dependent aberration by applying adaptive optics only on the detection path. To show this, we consider the model resulting from applying adaptive optics on the detection, which is given in Eq. (17). If we set the user-defined focus shift ã to be equal to the focus shift of illumination a_f, then P_z(z) becomes zero, and hence Eq. (17) becomes

Interestingly, it is possible to correct the aberration for illumination alone by using a single adaptive element. To this end, we construct a complementary back-focal plane for illumination and apply the following form of z-dependent function as an adaptive mirror:

By following the same step as in the derivation of Eq. (7), it can be shown that the resultant illumination pattern can by obtained by replacing f_z by f_z + f̂_z in Eq. (7). As a result, P_z(z) in Eq. (10), which represents aberration in the illumination pattern becomes

4. Conclusions

We have developed an imaging model for structured illumination microscopes that takes into account the full effect of depth dependent spherical aberration caused by refractive index mismatch between the immersion and mounting mediums. The model explicitly reveals the effect of aberration on the detection response and the illumination pattern distinctly, and will allow their effect in terms of signal loss to be compared by implementing computational correction independently for each effect. We demonstrated that depth dependent spherical aberration leads to loss of signal only due to its effect on the detection response of the system, and its effect on the illumination only amounts to an increased computational complexity in the forward model. Thus signal loss due to illumination aberration only occurs if an incorrect model is used in the reconstruction process. This contrasts with detection aberrations, which lead to loss of signal even if the exact depth variant model is used during data processing.