1. Introduction

While confocal fluorescence microscopy provides exquisite sub-cellular resolution, it remains limited essentially to surface imaging. To enable access to internalized in-vivo tissue structures in their natural environment, considerable research has been devoted to the implementation of confocal fluorescence microscopy in an endoscopy configuration. Indeed, confocal endomicroscopy promises to become an essential tool for biomedical imaging, both for basic research and in the clinic.

To date, several strategies have been adopted to perform confocal endomicroscopy, which may be separated into two basic categories [1]. In the first category, a single light conduit (typically an optical fiber) delivers illumination to the specimen, and confocal scanning is performed at the distal (specimen) end of the conduit using some kind micromechanical device [2, 3, 4, 5, 6, 7, 8]. Alternatively, in the second category, confocal scanning is performed at the proximal end of the light conduit. Previous implementations of this second category have made use of a rigid gradient-index (GRIN) lens [9, 10, 11] or a flexible imaging fiber bundle [12, 13, 14, 15, 16, 17]. The advantage of proximal scanning is that it obviates the need for precision moving parts at the distal end of the light conduit, facilitating miniaturization and robustness. The advantage of an imaging fiber bundle conduit is that it is flexible and can be quite long, making it particularly amenable to clinical endoscopy applications.

In this paper, we present an implementation of fluorescence endomicroscopy with a flexible fiber bundle that makes use of an imaging technique called structured illumination microscopy (SIM) [18], which is well known to provide confocal-like out-of-focus background rejection without the need for beam scanning. SIM is conceptually simple and can be readily implemented in any widefield imaging device. SIM has already been employed in a rigid Hopkins-type endoscope configuration [19], but to our knowledge a flexible fiber optic version has not yet been demonstrated. In this paper, we present a fiber-optic version of a SIM endomicroscope, as well as a discussion of imaging artifacts and a strategy to mitigate these artifacts. The purpose of this paper is to establish the potential of SIM endomicroscopy for future intravital imaging. To this end we demonstrate imaging of mouse colonic mucosa.

2. Endomicroscope setup

Structured illumination microscopy [18] is based on the illumination of a sample with a periodic light pattern, typically a one-dimensional grid pattern. Three fluorescence images of the sample are taken at three different grid positions, each laterally translated by a third of the grid period. A final optically-sectioned image I _SIM(x) is then generated by the simple algorithm

where I ₁(x), I ₂(x) and I ₃(x) are the three raw images, and x is a 2D coordinate in the imaging plane.

The basic idea in constructing a SIM endomicroscope is to make use of an imaging fiber bundle to serve as a relay that guides both the illumination grid pattern to the specimen and the resulting fluorescence distribution back to a CCD camera. A schematic of our setup is illustrated in Fig. 1.

 

Fig. 1. Endoscope setup. A grid pattern is produced by a spatial light modulator (SLM) and projected onto a sample via an imaging fiber bundle equipped with a miniaturized objective (O₂). The fluorescence generated from the sample is then imaged through this fiber bundle onto CCD camera. Notation: lenses (L), spatial light modulator (SLM), polarizing beam splitter (PBS), dichroic (D), objectives (O), sample (S), filter (F).

Download Full Size | PDF

Various strategies have been implemented for the application and translation of a grid pattern [21, 20, 22]. In our case, we generate a grid pattern with a liquid crystal spatial light modulator (SLM: Holoeye LC-R 768). The beam from a CW diode-pumped solid-state laser (Cobolt Calypso, λ=491nm, TEM ₀₀, coherence length > 10m) is expanded and projected onto the SLM and reflected through a polarizing beamsplitter, allowing the SLM to be operated in an amplitude-modulation mode. In this manner, an arbitrary amplitude pattern can be imparted on the beam profile by computer control of the SLM. In our case, this is a rectangular grid pattern of user-defined period and phase, controlled by Labview software. The pattern is then imaged onto the proximal end of the imaging fiber bundle by way of a lens and microscope objective (Olympus, 20×, NA=0.4) producing a net demagnification of 13×. The main advantage of an SLM is that it allows us to very conveniently vary our grid period and phase. It is limited in speed, however, providing a maximum effective refresh rate of about 10 Hz, and ultimately leading to a net SIM frame rate of about 2 Hz.

The imaging fiber bundle itself consists of 30,000 cores, each approximately 1.9µm in diameter separated by an average distance of 3.3µm (center to center). The total useful diameter of the fiber bundle is 600µm. A water-immersion miniaturized objective (Mauna Kea Technologies) was optically cemented onto the distal end of the fiber, projecting an image of this distal end into the sample with a demagnification of about 2.5×, yielding a field of view of about 240µm and an imaged core separation in the sample of 1.3µm. The working distance of the miniaturized objective is 60µm and the NA is about 0.8.

Finally, the fluorescence from the sample is guided back to the proximal end of the fiber bundle and then imaged onto a CCD camera (QImaging Retiga XR) with a magnification of 13×. A dichroic (Chroma Z488RDC) and emission filter (Chroma 525/50m) are used to spectrally isolate the fluorescence.

 

Fig. 2. (a) Raw image of a thin uniform fluorescent sample illuminated with a rectangular grid pattern. Cores are clearly separated showing that the resolution is limited by the core sampling, not imaging optics. (b) Corresponding Fourier transform of this raw image (log brightness scale). The outer ring of frequencies stems from the quasi-periodic distribution of the fiber cores. The inner dashed line is the effective cutoff frequency according to the Nyquist theorem. (c) Blowup of the Fourier transform about the origin. The characteristic harmonics of the grid pattern are apparent.

Download Full Size | PDF

Figure 2 depicts a raw image of a thin, uniform fluorescent plane illuminated with a grid pattern, as well as 2D Fourier transforms of this image. Two main features are of note in Fig. 2(b). The diffuse ring is the result of the quasi-periodic spacing of the cores in the fiber bundle and the inner feature centered about the origin (dc frequency) corresponds to the fluorescence pattern produced by the sample. A blow-up of this feature is illustrated in the Fig. 2(c) to highlight both a strong dc component and sidebands. The strongest sidebands immediately to the left and right of dc correspond to the grid fundamental frequency. The additional sidebands correspond to higher harmonics that arise from our use of a rectangular grid pattern.

Fluorescence imaging in our endomicroscope is performed in two steps, a first step from the object plane to the distal fiber bundle plane, and a second step from the proximal fiber bundle plane to the CCD camera. The first step is found to be the most limiting in terms of optical resolution, and from a calculation based on the NA of the distal mini-objective one might expect an optical resolution of about 0.3µm. However, such a resolution cannot be attained with our system because the fluorescence image is sampled by discrete fiber cores. The quasi-periodic sampling frequency of these cores is clearly apparent as a diffuse ring in Fig 2(b). In effect, this sampling frequency restricts the imaging bandwidth of our endomicroscope to an associated Nyquist frequency equal to half this sampling frequency, as depicted by the dashed circle in Fig 2(b). Frequency components in our raw images that lie beyond this Nyquist cutoff contain no information about the sample (at least none that can be readily exploited), and we systematically remove these by applying a Gaussian low-pass filter to all our raw images. The width of this filter is set to approximately the same bandwidth as the Nyquist cutoff. That is, we remove the apparent discretization of our images due to the fiber core spacing.

3. SIM imaging

 

Fig. 3. (a and b) Widefield and SIM image of lens-cleaning paper labeled with a drop of fluorescein solution. (c and d) Widefield and SIM image of an exteriorized rat colonic mucosa labeled with BCECF-AM dye. Artifacts at the grid period (21µm) are apparent in both SIM images.

Download Full Size | PDF

To test our SIM endomicroscope, we imaged a simple sample comprising lens paper labeled with a small drop of fluorescein solution. The resulting images are illustrated in Fig. 3(a). What is referred to as a widefield image is the average of the three raw images I ₁, I ₂ and I ₃, roughly corresponding to the non-sectioned fluorescence image that would be occasioned without the use of SIM. A reduction in fluorescence background is manifestly apparent when the images are processed by the SIM algorithm (Eq. 1). In acquiring these images, a grid period of 21µm at the sample was used. Based on the theoretical analysis provided in Ref. [18], the FWHM of our axial sectioning profile with this grid period is estimated to be about 15µm.

A second biologically relevant example is presented in Fig. 3. Mouse colonic mucosa was exteriorized, following mouse sedation and non-survival laparotomy. The colonic mucosa was loaded luminally with 10µM BCECF-AM dye (Invitrogen) for 10 minutes at physiological temperature and then stretched onto the sylgard gel surface. The thickness of the tissue was approximately 0.5mm and the light power delivered to the sample was no more then 1mW. Endomicroscope imaging was then performed without (Fig. 3(c)) and with (Fig. 3(d)) the use of SIM. In the latter case, the out-of-focus background is significantly reduced and tissue features including surface cells, goblet cells, and dye along the long axis of the crypts become readily apparent.

The grid period of 21µm at the sample plane was chosen largely as a compromise. While a smaller grid period (higher grid frequency) leads in theory to stronger axial sectioning, it also leads in practice to a reduced contrast of the grid pattern in the sample. Moreover. a difficulty specific to our setup stems from our use of an imaging fiber bundle. Because of the irregular distribution of fiber cores in the bundle, a grid pattern projected into the sample exhibits unavoidably jagged edges, provoking the introduction of extraneous pattern frequencies. While the systematic effects of an uneven fiber core distribution could in principle be corrected for, such a correction would be computationally intensive and was not attempted here. Our grid period of 21µm comprised roughly 16 fiber cores, or 8 fiber cores per bright or dark stripe, providing reasonable grid rendition and contrast (the contrast in the raw images used to obtain Fig. 3(d) was about 0.4).

Despite our use of a relatively low grid frequency (~12% of the Nyquist cutoff frequency), Figs. 3(b) and 3(d) reveal a common drawback of SIM, namely the persistence of a residual grid pattern that is clearly visible in the SIM images. In the next section we turn our attention to this artifact as well as some possible strategies to mitigate it.

4. Artifact reduction

The problem of residual grid patterns in SIM microscopy has been well appreciated [23, 24]. In particular, several causes have been identified for these artifacts, including non-sinusoidal pattern illumination and inaccurate phase shifting. Other causes can be temporal variations in the illumination power or in the fluorescence response (e.g. as a result of photobleaching). In either case, spurious variations can occur between the raw images that are not related to the grid pattern. These are not properly rejected by Eq. 1 and ultimately lead to artifacts in the final SIM image. A partial solution to this problem of temporal power variations was proposed in Ref. [23] which consists in normalizing each raw image to its mean prior to processing with Eq. 1. An alternative approachwas introduced in Ref. [24] based on the minimization of artifacts by an optimization of several parameters including corrections to illumination power fluctuations and inaccurate phase shifting. This optimization approach, however, is computationally intensive and slow (at least in our hands). Moreover, we have found that it can lead to artifacts of its own.

We present yet another alternative strategy for removing residual grid patterns based on our observation that the residual patterns are more prevalent when imaging thick scattering samples. To understand this strategy, it is useful to revisit Eq. 1, however this time in the Fourier domain.

To begin, it is well known that Eq. 1 is mathematically equivalent to the algorithm [18]

This may be recast in the form

where FT^-1 corresponds to an inverse Fourier transform and

where 𝓘₁(k), 𝓘₂(k) and 𝓘₃(k) correspond to the Fourier transforms of I ₁(x), I ₂(x) and I ₃(x) respectively. We refer to Eq. 4 as a phase stepping algorithm. Note the absence of an absolute value in Eq. 4 indicating that 𝓘_φ(k) is complex in general.

Let us first imagine a fluorescent sample that is thin and perfectly in focus. When imaging such a sample with SIM, then I ₁(x), I ₂(x) and I ₃(x) are raw fluorescence images that each exhibit an intensity modulation at the grid-pattern frequency, though phase-shifted between images. Because the sample here is defined to be in focus, the contrast of this modulation can be assumed to be reasonably high (provided the grid frequency k_g is much smaller than the cutoff frequency of the system optical transfer function (OTF)). As a result, if the grid pattern is sinusoidal, then 𝓘₁(k), 𝓘₂(k) and 𝓘₃(k) each contain three components. The first component is the Fourier transform of the unmodulated sample structure centered about k=0 (dc). The second and third components are the same Fourier transform of the sample structure but centered about the sideband frequencies k=±k_g and somewhat attenuated by the OTF. The principle of SIM relies on isolating the sample structure in a single sideband. This isolation cannot be achieved by simple one-sided filtering because of the overlap of the sidebands with themselves and with the unmodulated sample structure. However this isolation can be achieved by phase stepping. That is, the effect of the phase stepping algorithm (Eq. 4) is to remove both the unmodulated (dc) component and a single sideband component, while preserving the other sideband component.

Let us now imagine that this same thin fluorescent sample is displaced far out of focus. Because the OTF severely attenuates high spatial frequencies, then 𝓘₁(k), 𝓘₂(k) and 𝓘₃(k) exhibit only narrow frequency components centered about k=0, regardless of the grid pattern modulation. That is, the grid pattern is so blurred as to play no role. In this case, the effect of applying a phase stepping algorithm, since it removes unmodulated sample structure, is to force 𝓘_φ(k) to vanish. This is the well-known principle of SIM that leads to out-of-focus background rejection. It should be noted, however, that if the phase stepping is inaccurate in any way, then the unmodulated sample structure is not fully rejected.

In practice, when imaging a thick sample, then 𝓘₁(k), 𝓘₂(k) and 𝓘₃(k) contain both infocus and out-of-focus frequency components. These are schematically illustrated before (Fig. 4(a)) and after (Fig. 4(b)) the application of grid illumination, where, again, only the in-focus component is observed to be spatially modulated in the latter case. Figure 4(c) illustrates the result of phase stepping, where we have introduced a slight inaccuracy that leads to an imperfect rejection of the out-of-focus background, and hence to a spurious peak in |𝓘_ϕ(k)| in the vicinity of k=0. Our hypothesis of inaccurate phase stepping is borne out in experiment (Fig. 4(d)) where |𝓘₁(k)| is illustrated before phase stepping and|𝓘_ϕ(k)| is illustrated after phase stepping. A spurious peak about k=0 is readily apparent in |𝓘_ϕ(k)|. As a result of this spurious peak, the application of Eq. 3 to 𝓘_ϕ(k) leads to a residual grid pattern in the final SIM image (as is manifested in Figs. 3(b) and 3(d)).

Having isolated a cause of the residual grid pattern, a strategy to remove this pattern becomes clear. In particular, a high-pass filter can be applied to 𝓘_ϕ(k) prior to the application of Eq. 3. The cutoff frequency of this high-pass filter should be high enough to eliminate most of the spurious peak about k=0 while being low enough to perturb the remainder of 𝓘_ϕ(k) as little as possible. It should be noted that the strategy presented in [23] of normalizing each raw image I ₁(x), I ₂(x) and I ₃(x) to its respective mean achieves a similar result, though it removes only the component of 𝓘_ϕ(k) at exactly k=0 while retaining residual components in the vicinity of k=0. In other words, the removal of the spurious peak by normalization is incomplete. Examples of the results of narrow, sharp high-pass filtering where only the 𝓘_ϕ(k=0) component is removed versus broader high-pass filtering adjusted to the width of the spurious peak about k≈0 are illustrated in Fig. 5. The residual grid pattern is still apparent in Fig. 5(a) (although improved with respect to Fig. 3(d)), while it is manifestly eliminated from Fig. 5(b). Moreover, the contrast, background rejection, and overall appearance of Fig. 5(b) have been visibly improved.

 

Fig. 4. (a) Schematic illustration of in-focus (red) and out-of-focus (blue dashed) contributions to the intensity spectrum of a widefield image. (b) In-focus and out-of-focus contributions to the intensity of a raw image with grid pattern illumination (|𝓘₁(k)|). Note: only the in-focus contribution becomes modulated. (c) Phase stepping (|𝓘_ϕ(k)|) suppresses both the unmodulated contributions (in-focus and out-of-focus) as well as a single sideband of the modulated contributions (here, left sideband). Imperfect phase stepping leaves behind a residual peak about k=0 (blue dashed) that arises dominantly from the out-of-focus background. (d) Experimental results derived from the sample in Fig. 3(d). The intensity spectrum (magnitude) of a single raw image with grid pattern illumination (blue dashed) is plotted alongside the intensity spectrum after phase stepping (red). In the latter case, a residual peak about k=0 is readily apparent. This peak can be rejected with the use of a high pass filter (black dotted).

Download Full Size | PDF

Some comments are in order. First, the question arises as to why phase stepping applied to out-of-focus background might be inaccurate in the first place. Many possibilities can account for this. For example, phase stepping is inaccurate if the out-of-focus backgrounds in images I ₁(x), I ₂(x) and I ₃(x) exhibit spatially local inhomogeneities in fluorescence brightness that vary between raw images. This can be caused by grid-phase-dependent variations in the local illumination power delivered to the out-of-focus caused by irregularities in the fiber core density (see discussion above) or by sample-induced scattering or aberrations. Given a preponderance of out-of-focus background when imaging thick tissue samples, such variations need only be slight to provoke visible artifacts. Moreover, because our illumination source is coherent, we further expect that the in-focus grid pattern becomes deteriorated out of focus and more closely resembles speckle [25, 26, 27], in turn exacerbating the problem of random illumination inhomogeneities. In principle, the problem of out-of-focus speckle can be mitigated with the use of a rotating diffuser in the illumination path, however this can lead to difficulties in coupling the laser illumination into the fiber bundle. It should also be be mentioned that a portion of the spurious peak at 𝓘_ϕ(k≈0) can also arise from inaccurate phase stepping of the in-focus light (as opposed to the out-of-focus light), again possibly due to inaccurate grid periodicity caused by inhomogeneities in the fiber-bundle core distributions or sample-induced aberrations. However given the dominance of out-of-focus background over in-focus signal in the raw images apparent in Fig. 4(d), this last possibility is likely to play a lesser role. In any event, the key point of applying a high-pass filter to 𝓘_ϕ(k) is to remove the effect of inaccurate phase-stepping regardless of its source.

As a second comment, the application of a high-pass filter to 𝓘_ϕ(k) is tantamount to correcting for intensity variations between the raw images locally rather than globally. When this high-pass filter is combined with the low-pass filter described above to eliminate spatial frequency components beyond the Nyquist cutoff, the net result is a bandpass filter very similar to what was referred to as wavelet prefiltering, which was previously applied to dynamic speckle illumination microscopy [28].

Finally, it should be noted that it is possible to remove grid artifacts by simply filtering out any components of I _SIM(x) at the grid frequency k_g (i.e. after the application of Eq. 3). However this would also eradicate any sample structure at the grid frequency. An advantage of applying a high-pass filter to 𝓘_ϕ(k) prior to the application of Eq. 3 is that it partially preserves sample structure at the grid frequency. Nevertheless, our strategy should be used with caution. It is particularly effective when applied to thick tissue imaging that exhibits significant out-of-focus background, as illustrated in Fig. 5(b). It becomes less effective, however, when applied to thin samples, particularly samples that exhibit spatial frequencies much higher than the grid frequency. In this last case, our strategy runs the risk of introducing spurious ringing in the proximity of sharp edges in the sample. In practice, we have found that it is best to adjust the cutoff frequency of our high-pass filter depending on the sample in question.

 

Fig. 5. (a) SIM image of rat colonic mucosa (same sample as in Fig. 3(c–d)) when each raw image is normalized to its respective mean. Note: this technique only partially suppresses the residual grid pattern in Fig. 3(d). (b) SIM image using the technique of high-pass filtering of 𝓘_ϕ(k≈0) to minimize the effects of imperfect phase stepping. Image quality is manifestly improved.

Download Full Size | PDF

5. Conclusion

In conclusion, we have demonstrated the implementation of SIM with a flexible imaging fiber bundle appropriate for fluorescence endomicroscopy. Our field of view is about 240µm and our final image resolution is on the order of 2.6µm, limited by the Nyquist frequency associated with the quasi-periodic core separation in the fiber bundle (as imaged in the sample). In addition, we have described a strategy to largely suppress artifacts that result from inaccurate phase stepping. This strategy is found to be particularly beneficial when imaging thick samples that exhibit significant out-of-focus background. Such samples are of the type likely to be encountered when performing endomicroscopy in vivo.

Our application of SIM to endomicroscopy is specifically intended to reduce out-of-focus background without enhancing resolution. Alternative applications of SIM not based on Eq. 1 could, in principle, provide super-resolution [29], however this was not the goal of our present work. To our knowledge, our results represent the first demonstration of endomicrosopy with SIM, which we hope will have a broad impact in the biomedical community.