1. Introduction

With the popularization of super-resolution methods [1,2], fluorescence microscopy research is experiencing somewhat of a renaissance. A plethora of new optical microscopy techniques have been introduced, some able to provide a roughly two-fold resolution improvement beyond the diffraction barrier [3,4] and others, usually referred to as nanoscopy or diffraction-unlimited techniques, able to break such a barrier altogether [5–7]. The two-fold super-resolution microscopy methods – structured illumination microscopy (SIM) [8], confocal laser scanning microscopy (CLSM) [9] and image scanning microscopy (ISM) [10] – have become go-to imaging tools in pre-clinical research. These techniques are reliable, simple, familiar, highly compatible with all kinds of fluorescence labels and work well with many types of samples – whereas current nanoscopes fall short on at least some of these characteristics.

Structured illumination microscopy encompasses a collection of super-resolution implementations that make use of structured excitation light [8] to improve resolution. In contrast to nanoscopy, SIM does not require any special sample preparation [11]. In its original form [3], SIM was implemented in a wide-field microscope, by producing a striped illumination pattern onto the sample with a line spacing close to the diffraction limit, thereby shifting high-frequency information of the sample into the pass-band of the optical system. Because the excitation pattern is diffraction limited, the maximum resolution gain obtainable with linear SIM techniques is a factor of two (i.e. the cut-off frequency doubles) with respect to conventional microscopy [3].

Similarly to SIM, CLSM requires no special sample preparation, and the same resolution gain can be obtained by closing the confocal pinhole [12]. However, such an enhancement is only theoretical, since it comes at the cost of extremely low signal-to-noise ratio (SNR). The limited SNR problem can be overcome by image-scanning microscopy (ISM) [4,13]. There are essentially two types of ISM implementations that we denote computational and optomechanical ISM. In computational ISM, the single-point detector of a regular point-scanning CLSM is substituted with a detector array, which collects a small image at each sampling position. Because the detector array has a field-of-view of 1-1.5 AU, no separate pinhole is required to reject out-of-focus light [14], and thus the important optical-sectioning ability of CLSM is maintained. The ISM result is obtained computationally post-acquisition, by pixel reassignment or deconvolution. This approach was theoretically proposed in the 80s [4,15], but straightforward implementations have only recently become feasible, thanks to the development of fast detector arrays (bandwidth $\gg$ kHz, i.e. faster than the pixel dwell-time of a typical scanning microscope), such as the Airyscan [16] and our single-photon-avalanche-diode (SPAD) array module [17]. Also computational ISM methods involving traditional cameras and multi-spot excitation schemes have been proposed [18,19]. Optomechanical ISM [20,21] on the other hand is based on the use of a traditional camera in a (re-scanned) wide-field configuration; the ISM image reconstruction is achieved without computation, by mechanically and/or optically enlarging the final image by a fixed factor – usually two – with respect to the laser scanning grid. Multi-spot excitation schemes have also been proposed in the context of optomechanical ISM [22,23]. The optomechanical ISM implementations are sometimes classified as SIM techniques, since they use structured excitation/illumination (spots instead of stripes), wide-field architectures and conventional cameras. Similarly, computational ISM is sometimes called spot-scanning SIM [24]. This reveals the close interrelation of the SIM and ISM concepts. However, because of the complexity in generating patterned illumination deep into a sample, traditional SIM imaging is usually constrained to a few tens of micrometers in depth [2], even when combined with adaptive optics approaches [25]. On the other hand, thanks to the point-scanning architecture, the computational ISM is more compatible with thick samples.

Initially, all the different flavors of ISM were implemented with one-photon excitation (1PE), which limits the practically attainable imaging depth in thick biological samples, due to extensive scattering of the illumination light. More recently, to address this issue, both optomechanical and computational ISM implementations employing two-photon excitation (2PE) [26–29] have been proposed [30–34]. Here, the scattering illumination problem is reduced by using infrared excitation light, which is also helpful in lowering photo-toxicity, as biological samples do not commonly absorb it. However, since the working principle of ISM is based on the assumption of ballistic fluorescence photons, the benefits of 2PE-ISM may vanish when the scattered fluorescence photons overcome ballistic ones. This latter situation is where non-descanned microscopy architecture – with no pinhole and large single-element detector – provides the best results. On the other hand, non-descanned detection fails in the case of high near-surface (out-of-focus) fluorescence background signal, since there is no pinhole to reject it [35,36]. In any case, as long as there are some ballistic photons, 2PE-ISM can provide a significant resolution increase. In addition, when a small array detector is used (1 - 1.5 AU), the background rejection capability of the system is improved – similar to using a small pinhole, but without the associated signal loss.

The current 2PE-ISM implementations require significant changes to the regular point-scanning microscopy architecture, and/or do not allow the use of a pinhole to reject out-of-focus light, and/or do not take into account the sub-optimal and variable imaging conditions that can be considered omnipresent in typical samples in which 2PE is necessary. Computational ISM gives access to important information about the imaging conditions/distortions, as recently demonstrated by us [17] – with an appropriate image reconstruction approach, it can be leveraged to ensure high-resolution and high-SNR imaging results, even when working under sub-optimal and variable imaging conditions, such as in deep imaging. It should be noted here that a similar approach is not possible in optomechanical ISM, because the reassignment is defined by the optical and mechanical configuration, and cannot be easily changed during imaging. An approach to compensate for imaging distortions in multiple-spots ISM has been proposed [37], but it requires tedious pre-calibration of the system, prior to every imaging session. Furthermore, such calibration is error-prone, as the imaging conditions in the actual sample, especially in depth, cannot be expected to be similar to the calibration sample.

Here, we present a calibration-free, robust and straightforward 2PE-ISM implementation, which can be implemented on any regular point-scanning microscope. Similarly to our one-photon excitation ISM implementation [17], we automatically record all the raw scanned images – without speed concerns and large data overhead that hampered the early computational ISM implementations. We propose a fully automatic and adaptive image reconstruction method, that is shown to ensure high quality imaging results – with resolution typically close to the theoretical limit – even in non-ideal imaging conditions. The ISM reconstruction requires no calibration or prior knowledge of the sample or microscope configuration. In our view, this ease-of-use is a rather important feature of our method, as it makes the 2PE-ISM more accommodating towards the microscope users.

2. Methods

2.1 Custom microscope setup

Our 2PE-ISM system was implemented as a modification to a confocal microscope. As shown in Fig. 1, the femtosecond two-photon excitation laser (Chameleon Ti:Sapphire, Coherent) was coupled on the confocal microscope’s common optical path via dichroic mirror DM*1 (720 nm SP, Semrock, USA). The fluorescence signal was directed to the SPAD array with a second dichroic mirror DM*2 (720 nm LP, Semrock, USA). Both DM*1 and DM*2 are removable/exchangeable, which allows using the same microscope for single-photon ISM and regular confocal imaging. The lens pair SL-L5 and L3-L4 conjugate the SPAD array [17] with the object plane and adjust the magnification ($\sim 500 \times$), to give the SPAD array $\sim 1.5$ Airy unit field-of-view. The lenses L3-L4 are not strictly necessary, as the required magnification can already be reached at the focal plane of L5 – in our case the extra relay was added due to spatial constraints on the optical table. A (512/40 nm, Semrock, USA) emission filter was installed in front of the SPAD array to block ambient light. The point-scanning was implemented using a galvanometric mirror XY scanner (6215H, Cambridge Technology, USA), coupled with $f=50$ mm Leica scan lens and a $f=200$ mm Leica tube lens. Axial scanning, as well as sample focusing, was implemented with a piezo stage (NanoMax MAX302, Thorlabs, USA). A Leica HCX PL APO 100$\times$/1.4-0.7 Oil CS (Leica Microsystems, Germany) objective lens was used in all experiments, except for the brain images shown in Figs. 6, and 7, which were acquired with a Leica HC PL APO 20$\times$/0.70 IMM CORR SC2 objective lens. In order to obtain a suitable SPAD array field-of-view with the new objective lens, lenses L4-L5 were removed and a single f = 600 mm lens was used in place of L3. For imaging with the non-descanned detection, a long pass dichroic beam splitter DM*3 (560 nm LP, Semrock, USA) was placed between the objective lens and the tube lens, to reflect the fluorescence emission onto a photomultiplier tube (H10723-01, Hamamatsu Photonics, Japan), after passing through a BP525/50 fluorescence emission filter.

 

Fig. 1. Schematic of the 2PE-ISM setup. Our 2PE-ISM setup is basically a regular confocal/2PE system, with the addition of a SPAD array detector. (a) Only three additional lenses (L3-L5) and a dichroic mirror (DM*2) are needed to convert a confocal/2PE system into a ISM super-resolution microscope. The dichroic mirrors DM*1 and DM*2 can be reconfigured to enable ISM imaging with the visible excitation lines. (b) The DM*3 can be added to enable imaging with the non-descanned photomultiplier detector (NDD). This made it possible to directly compare the results obtained with the SPAD and NDD on the same system.

Download Full Size | PDF

The image acquisition was performed with our Carma microscope control software [38,39], which takes care of the real-time hardware control tasks (scanning, laser control etc.) as well as fluorescence signal recording from the 25 SPAD array pixels; the software also has a PC user interface for controlling the various functions of the microscope system and to preview and process the imaging results.

2.2 Point-spread functions simulation

The point-spread function (PSF) simulations were implemented with the LightPipes optical beam propagation toolbox in Python. In our simulation, a Gaussian beam is propagated through a lens to form a PSF in the focus. For the excitation and emission wavelength, $\lambda _{exc}$=950 nm and $\lambda _{em}$=520 nm were used, respectively. The PSF $h_i$ associated to the $i$-th element of the detector array was calculated following the most conventional model for confocal microscopy:

2.3 Static ISM pixel reassignment

According to the pixel reassignment theory [40] and using the image of the central element as a reference, the shift-vector $\textbf {s}_i = (x,y)$ associated with the detector element $i$ is based on simple geometrical considerations: (i) the shift-vector direction equals the line joining the centers of the considered element and the central one; (ii) the length is proportional to the pixel reassignment factor $\alpha$, the magnification of the system $M$, and the physical distance between the element and the central one $d_i$, namely $\overline {\textbf {s}}_i = d_i*\alpha /M$; (iii) assuming identical excitation and emission wavelength, the pixel reassignment factor is equal to 0.5. In this paper we show static ISM reconstruction (PR-ISM) with both the traditional $\alpha =0.5$ factor, and $\alpha =0.67$, which according to both our previous work [41] and the results shown in this paper, is more suitable for 2PE-ISM. We also show that it is possible to calibrate the static ISM shift-vectors with our adaptive ISM reconstruction, which can be preferable, e.g. if computational resources are limited.

2.4 Adaptive pixel reassignment and blind deconvolution

Adaptive pixel reassignment (APR-ISM) consists of two tasks: image registration and image fusion. Image registration is needed to derive the shift-vectors, aligning the 25 scanned images/sub-images (Fig. 2(b)) into a common coordinate system; image fusion is used to combine the registered sub-images into a single result. Notably, if the registration step is bypassed, the sum of all the scanned images results in the conventional open pinhole (1.5 AU in our case) 2PE image (Fig. 2(a)).

 

Fig. 2. ISM image reconstruction methods at a glance. (a) At every scan, 25 images, somewhat shifted with respect to each other, are obtained. By summing the 25 images, the regular 2PE image (in our case, with $\sim$1.5 AU pinhole) is obtained. (b) For pixel reassignment ISM image reconstruction, one first needs to determine the image shifts. Shifts s for the static reassignment (PR-ISM) are obtained by scaling the 5 $\times$ 5 meshgrid gr representing the SPAD array with the pixel pitch $p_{p}$ = 75 $\mu m$, the magnification of the microscope $M$ = 500, and the pixel-reassignment factor $\alpha$: $\textbf {s}=\textbf {gr}*p_{p}*\alpha /M$. Shifts for the adaptive pixel reassignment (APR-ISM) are obtained with an iterative image registration method, in which all the 25 images are aligned with the image from the central detector element (12). The ISM reconstruction is obtained by shifting and summing the 25 scanned images. (c) After pixel reassignment, blind deconvolution, either Wiener (APR-ISM$^{\mathrm {bW}}$) or RL ((APR-ISM$^{\mathrm {bRL}}$) is applied to the reassignment result; the FWHM falue for the PSF is obtained from the ISM reassignment result image with FRC. The background value for APR-ISM$^{\mathrm {bRL}}$ is estimated as the mean intensity value of sparse (few details) image regions.

Download Full Size | PDF

Previously, several phase-correlation based methods have been proposed, to estimate the shift vectors [17,42,43]. Here, we introduce a new iterative image registration method, based on software originally developed for three-dimensional tomographic STED microscopy [44]. In image registration one tries to find a spatial transformation that aligns the details in two images as closely as possible. In iterative image registration, as illustrated in (Fig. 2(b)) the search for the optimal spatial transformation is considered an optimisation problem: one image, called fixed image, is used as a reference, while a second image, called moving image, is translated until the details in the two images match. The images are considered to match when a chosen similarity metric reaches its maximum value. In our ISM registration implementation, the central detector image is always used as the fixed image and the 24 other images are sequentially registered with it. After the shift-vectors for all the 24 scanned images have been calculated, the images are shifted and then added together to produce the adaptive pixel reassignment (APR-ISM) result.

The ISM image registration methods in our MIPLIB library [45] leverage the Insight Toolkit (ITK) [46]. All registration and image transformation tasks in ITK/MIPLIB are performed in physical units ($\mu m$), which means that the images can be re-sampled to a different pixel size without changing the registration result. In ISM this is particularly useful if the original data is sampled too sparsely to support the two-fold resolution improvement. One can also shrink the images before registration to increase speed and optimize memory consumption. The registration is also always sub-pixel: for the results shown in this work, the minimum optimizer step size was set to 0.5 nm.

Furthermore, both in ISM and SIM, some form of deconvolution or frequency domain filtering (re-weighting) is commonly used in order to maximize the effective resolution (e.g. [3,22,30,31,42]). To this end, as illustrated in (Fig. 2(c)), we apply our blind Fourier ring correlation (FRC) based deconvolution approach [47] to the APR-ISM image; no prior knowledge of the PSF is needed, but it is estimated directly from the APR-ISM result image. In essence, the FRC analysis is applied on the APR-ISM image to extract the resolution, from which we derive the FWHM for a Gaussian PSF. In this work two different deconvolution algorithms are employed: a linear Wiener filter and an iterative Richardson Lucy (RL) algorithm. The Wiener deconvolution algorithm is based on inverse filtering, taking advantage of the fact that a convolution operation in the spatial domain becomes a multiplication in the frequency domain – and thus deconvolution can be considered a simple division in the frequency domain,

The iterative Richardson-Lucy (RL) algorithm can be described by

We chose to show results with both of these algorithms, because each has its advantages. Wiener filter is extremely simple to calculate, maintains pixel intensity ratios (linear and shift invariant) and in our experience, obtains a very high resolution with images that are not excessively noisy. It is also commonly used in literature (SIM, ISM) as well as commercial devices, such as Zeiss Airyscan. However, it is ill-equipped to working with low-SNR data (like the brain images shown here), as it is quite sensitive to noise background. Stronger regularisation can reduce noise, but it often also blurs useful details. The RL algorithm, especially with background correction as is done here, is clearly better suited for such images.

We denote the combination of APR-ISM with blind Wiener filtering as APR–ISM$^{\mathrm {bW}}$, and RL deconvolution APR-ISM$^{\mathrm {bRL}}$ (b for blind). The image shifting and Wiener filtering could be alternatively combined in a single step, as is done in [3,42,49,50]. Here we keep the two tasks separated, because the FRC approach that we used to estimate the PSF is sensitive to noise: the estimation of the PSF for the APR-ISM image is more robust than the estimation of the PSFs for each single scanned image.

2.5 Resolution measurement

The resolution measurements shown in this work were calculated with our Fourier ring correlation (FRC) assay [51]. When two identical images (i.e. images of the very same structures, but with different noise realisations) were not available, we used our one-image version of the FRC, which is described in detail in [47].

2.6 Test samples

We demonstrated the enhancement on spatial resolution of 2PE-ISM via imaging of fluorescent beads, tubulin filaments and mouse brain sections.

Fluorescent beads In this study we used Yellow/Green fluorescent nanoparticles with a diameter of 100 nm (FluoSpheres, ThermoFisher Scientific, USA).

Tubulin filaments in fixed cells Human HeLa cells were fixed with ice-cold methanol for 20 min at -20$^{\circ }$C and then washed three times for 15 min in PBS. After 1 h at room temperature, the cells were treated in a solution of 3% bovine serum albumin (BSA) and 0.1% Triton in PBS (blocking buffer). The cells were then incubated with monoclonal mouse anti-$\alpha$-tubulin antiserum (Sigma-Aldrich) diluted in blocking buffer (1:800) for 1 h at room temperature. The $\alpha$-tubulin antibody was revealed by Abberior STAR488 goat anti-mouse (Abberior) for the custom microscope. The cells were rinsed three times in PBS for 5 min.

Mouse brain slices Thy1-EYFP-H transgenic mice were used in this study. Preparation of the cleared fixed brain slices (1 mm thickness) has been described in details in [52]. In the context of this work two slightly different clearing protocols were adopted. For Fig. 5 the brain slice was cleared with LUCID [53]; for Fig. 6 and 7 the brain slice was further modified with RapiClear 1.52. In both cases the brain slice was mounted on top of a microscope cover glass. Both samples were provided by Nikon Corporation.

3. Results

3.1 Simulating computational 2PE-ISM

In computational ISM, the most straightforward method to fuse the scanned images is to define theoretical values for the ISM shift-vectors based on the microscope’s optical configuration and the detector geometry. The shift-vectors are used to translate the images from each array detector element into a common coordinate system, after which all the images are summed up to produce the ISM result image. The shift-vectors are fixed a-priori, and hence we denote this type of pixel reassignment static.

We performed simulations [42] to quantify the resolution enhancement achieved by implementing 2PE-ISM with our $5\times 5$ -element SPAD array detector and with the static pixel reassignment (PR-ISM) approach (Fig. 3(b-d)). In our simulation, we obtained the shift-vectors $\textbf {s}_i$ by co-aligning the peak intensity points of all detector elements’ PSFs. Notably, this simulation is closer to the APR-ISM than to the PR-ISM, since it automatically considers the wavelength Stokes shifts, and it does not impose a common pixel reassignment factor for the different detector elements. As expected, since no distortions or aberrations are included in the simulations, our approach generates shift-vectors similar to the (theoretical) static values [41] for elements close ($\leq$ 1 AU) to the central reference [14].

 

Fig. 3. Point-Spread-Function simulation for 2PE-ISM. (a) Schematic representation of the simulation to calculate the PSFs of ISM based on the pixel-reassignment method. The effective detector size depends on the magnification of the microscope system and is usually expressed in Airy units (1 AU = 0.61$\times \lambda _{em}/NA$, where $\lambda _{em}$ is the wavelength of the fluorescence light and NA is the objective numerical aperture). In this simulation we used: NA = 1.4, $\lambda _{em}$ = 525 nm. Each element of the detector array acts as a shifted pinhole, thus 25 PSFs have been calculated using a rigorous model for the focused emission/excitation light intensity distribution ($\lambda _{exc-2PE}$ = 950 nm, $\lambda _{exc-1PE}$ = 488 nm, oil objective lens). The shift-vectors for the simulation are calculated by assuming that an optimal shift should co-align all PSFs at their maximum intensity. Finally, all the registered PSFs are summed to produce the PSF of ISM (PR image). (b, top) Optical resolution (i.e, FWHM of the PSF) comparison between confocal (i.e.,the PSFs are summed without any shift) and ISM as a function of the detector size, for both the 1PE and 2PE case. (b, middle). Gain in SNR (i.e., the ratio between the peak intensities of the ISM and the confocal PSFs) as a function of the detector size, for both the 1PE and 2PE case. (b, bottom) Fraction of fluorescence recorded – with respect to the total emission – as a function of the detector size; as expected, there is no difference between 1PE and 2PE, neither confocal nor ISM. (c) Side-by-side comparison of the PSFs for the different imaging configurations. Here, 1 AU is assumed as detector size. Both intensity normalized and un-normalized versions are reported. (d) Radial intensity profiles for the PSF shown in c). Scale bars: 1 $\mu m$

Download Full Size | PDF

Since the number of elements of our SPAD array is limited, an optimal magnification $M$ of the system needs to be found, as it is desirable to maximize the photon collection efficiency, but in such a way that out-of-focus background rejection and ISM reassignment performance are not compromised [14]. With this in mind, we calculated the PSF for both 2PE-ISM and 2PE as a function of the effective SPAD array size (in Airy unit of the focused fluorescence light). Figure 3(b, top, bottom) shows the optical resolution (FWHM of the PSF) and the total fluorescence intensity, respectively. Large detector sizes ($>$ 1.5 AU) compromise the resolution enhancement expected from ISM, and small detector sizes ($<$ 1 AU) discard a substantial part of the fluorescence signal. Within the range 1-1.5 AU, the resolution is improved by a factor of 1.6 (with respect to the conventional pinhole-less architecture), only a negligible 10% of the fluorescence signal is lost (with respect to the conventional pinhole-less architecture), and out-of-focus background is rejected. Thus, we decided to give the SPAD array a 1.5 AU field-of-view.

Notably, the resolution improvement obtained by pixel reassignment in 2PE is stronger than the one-photon excitation case, Figure 3(c,d): (i) the resolution gain for the 2PE case is $\sim$ 1.6 versus $\sim$ 1.4 for one-photon excitation; (ii) the super-brightness effect (i.e., the peak intensity increase between the ISM and the pinhole-less PSFs) is $\sim$ 2.2 for 2PE and $\sim$ 1.8 for one-photon excitation (Figure 3(b, middle)). These results are in line with previous theoretical work [41].

3.2 Benchmark 2PE-ISM measurements using simple test samples

In order to get an idea of the performance of our 2PE-ISM system, we took some benchmark measurements under quasi aberration-free imaging conditions (at the cover-slip interface), with two very common microscope test samples: i) a sample of 100 nm yellow-green carboxylate modified fluorescent nanospheres, and ii) a sample of fixed HeLa cells stained with an alpha–tubulin antibody and with a Star488 secondary antibody.

In (Fig. 4(a)) we compare four images of the nanoparticle sample: the open pinhole two-photon image (2PE), the closed pinhole two-photon image (2PE ph.), the ISM image after our adaptive pixel reassignment procedure (APR-ISM) and the Wiener filtered APR-ISM image (APR-ISM$^{\mathrm {bW}}$). The 2PE image was obtained by summing the signal from all SPAD array elements, ignoring the registration step, whereas the 2PE ph. image corresponds to the image registered by the central element (pixel no 12 in Fig. 2(b)); the pixel itself works as a pinhole of approximately 0.3 Airy units in size. For the APR-ISM image, the 25 sub-images were first registered and then summed. As shown with the FRC analysis (Fig. 4(c)), the APR-ISM method improves the resolution by a factor of $\sim 1.7$, which is further improved to a factor of $\sim 3$ with a blind Wiener filter (APR-ISM$^{\mathrm {bW}}$). The numerical resolution values obtained with FRC measurements on the nanoparticle sample (2PE: 295 nm, APR-ISM: 179 nm and APR-ISM$^{bW}$: 79 nm) correspond quite well with the theoretical FWHM values ($\textrm {FWHM}_{2PE}$ = 263 nm and $\textrm {FWHM}_{ 2PEph}$ = $\textrm {FWHM}_{ 2PE-ISM}$ = 147 nm from our PSF simulation (see Fig. 3)). In d) line profile measurements are shown, across the two nanoparticles that are highlighted in the magnified APR-ISM$^{\mathrm {bW}}$ image in a) – at approximately 130 nm apart, they are clearly resolved in the APR-ISM$^{\mathrm {bW}}$ image, but completely invisible in all the others.

 

Fig. 4. 2PE-ISM imaging with simple test samples Side-by-side comparison of open pinhole 2PE, closed pinhole 2PE (2PE ph.), APR-ISM, and APR-ISM$^{\mathrm {bW}}$) images of 100 nm sized fluorescent nanoparticles (a) and HeLa cells (b). Insets show the magnified views of the regions inside the white dashed boxes. Peak intensities for the fluorescent nanoparticles highlighted in the white circle are reported. (c) The FRC measures show a nearly three-fold resolution improvement in APR-ISM$^{\mathrm {bW}}$ with respect to 2PE. (d) Line profile measurements across an aggregate of two beads, shown in (a) support the FRC cut-off values. (e) The shift-vectors are very similar for experiments, and form a somewhat tilted grid. The shifts are a bit smaller for the cell image, suggesting a larger PSF size, which is consistent with the FRC resolution measurements. All images and insets are normalized to the respective maximum intensity values. Scale bars: 2 $\mu m$ (a); 4 $\mu m$ (b)

Download Full Size | PDF

Considering the fixed cell sample, as shown in (Fig. 4(b)), the resolution of the APR-ISM image is improved by a factor of $\sim 1.6$ with respect to the regular 2PE image counterpart (Fig. 4(c)), and it is further enhanced to a factor of $\sim 2.5$ thanks to Wiener filtering. However, it is worth noticing that the retrieved resolution values are different than the ones obtained with the nanoparticles sample (2PE: 373 nm, APR-ISM: 240 nm and APR-ISM$^{bW}$: 110 nm). The resolution with the cell sample is close to the theoretical FWHM value for conventional microscopy ($\textrm {FWHM}_{2PE} = 0.51\lambda _{2PE}/ \textrm {NA}=346~nm$, for $\lambda _{2PE}$ = 950 nm, NA = 1.4 [54]), and the APR-ISM resolution values are in good agreement with previous reports on non-linear ISM [30,31].

For the sake of clarity, we highlight that the FRC-based resolution corresponds to the maximum spatial frequency found in an image that surpasses the specified 1/7 noise threshold. Thus, when the FRC is applied to reconstructed images, such as the one obtained from deconvolution, the results must be taken with a grain of salt, as artefacts – due to noise amplification – may affect the result. That is the reason why we report line profile measures as well. In the case of well-known and sparse structures, such as fluorescent beads or a tubulin network, the introduction of artefacts is less likely and easy to detect. Also, there is no background to speak of. For these reasons, and thanks to the high SNR of the raw data, the Wiener deconvolution can be pushed to its limits. This explains the $\sim$ 100 nm resolution obtained for APR-ISM$^{\mathrm {bW}}$. With noisier raw data, much more modest results (and more artefacts) should be expected.

The ISM shift-vectors obtained with image registration for both the nanoparticle and the cell sample, are shown in (x,y) scatter plots in (Fig. 4 e). Interestingly, while the shift-vectors form a grid-like structure, even with a basic sample like beads the grid is somewhat rotated. This is actually a property of our specific microscope system, as it is produced by a tilt in the XY scan mirrors (with respect to the SPAD array). It would be possible to compensate for the tilt by physically rotating the detector, but it is not necessary here, as the APR-ISM image reconstruction takes it into account. The pixel no. 24 in the SPAD array used for the 2PE experiments is more noisy (i.e., higher dark-counts) than the others, which results into the single outlier shown in the scatter plot.

3.3 Diving into a hazy brain slice

A multi-photon microscope arguably is most at home with thick samples that are hard or impossible to image with regular fluorescence microscopy methods, and thus we decided to test our 2PE-ISM in such samples. An optically translucent mouse brain that expresses YFP in neurons was imaged at different penetration depths, up-to the maximum working distance of a high numerical aperture oil immersion microscope objective lens. In the brain sample, we used RL deconvolution instead of Wiener filtering, to enable the automatic background correction, which is shown to dramatically reduce the amount of scattered fluorescence background signal and dark-counts noise. The APR-ISM$^{\mathrm {bRL}}$ is able to maintain rich details and good contrast all the way up to the maximum working distance (Fig. 5(a)). The APR-ISM$^{\mathrm {bRL}}$ images have FRC-based spatial resolution values of $\sim$ 200 nm, whereas the resolution in 2PE images degrades from 480 nm to 530 nm as a function of depth, as can be expected (see FRC measurements in Fig. 5 d). The resolution improvement obtained with the regular pixel reassignment (APR-ISM) is a factor of $\sim 1.5$, and stays nearly constant at the different imaging depths. Line profile measurement in c) shows that both APR-ISM and APR-ISM$^{\mathrm {bRL}}$ are able to resolve the two neurite filaments approximately 350 nm apart – although in APR-ISM the structure is very faint, as it happens to be just at the resolution limit measured with FRC. The numerical values are in good agreement with those obtained in [30,31]. In Fig. 6 additional images obtained with a longer working distance objective lens are shown with penetration depths up-to 500$~\mu$m. The results are not directly comparable with Fig. 5 as the sample used in Fig. 6 is much brighter, but the APR-ISM$^{\mathrm {bRL}}$ produces equally improved results in both cases.

 

Fig. 5. 2PE-ISM imaging of a mouse brain. In a) images are shown at different penetration depths (45 $\mu m$ – 140m $\mu m$). In b) a magnified section is shown of the results at 45 $\mu m$ depth. In c) a line profile plot across the structure highlighted in b) is shown. In d) FRC measures for the different types of images are shown as a function of imaging depth. The PR-ISM ($\alpha$=0.5) results were generated with purely theoretical shift-vectors. The APR-ISM$^{\mathrm {bRL}}$ results are shown with slight gamma correction ($\gamma = 0.8$), for easier viewing of the shadow details, and to further highlight the quality of the background removal. Scale bars: 2 $\mu m$

Download Full Size | PDF

 

Fig. 6. Deep brain imaging with 2PE-ISM. Examples are shown of 2PE-ISM imaging deep inside a mouse brain slice. The APR-ISM$^{\mathrm {bRL}}$ results are shown with slight gamma correction ($\gamma = 0.8$), for easier viewing of the shadow details. Scale bar 4 $\mu m$.

Download Full Size | PDF

 

Fig. 7. Comparing 2PE-ISM and with a non-descanned detector. Images recorded with the SPAD array and a regular non-descanned photomultiplier (NDD) are compared. The images were taken at a depth of 500$~\mu m$ in the same mouse brain slice that was used in Fig. 6. All the array detector images (2PE, APR-ISM and APR-ISM$^{\mathrm {bRL}}$) have dramatically superior contrast and details, with respect to the NDD. Scale bar 4 $\mu m$.

Download Full Size | PDF

In Fig. 7 images recorded with the SPAD array and a regular non-descanned (NDD) photomultiplier are compared. The purpose of such a experiment is to obtain a direct comparison between our descanned ISM system, and a NDD – the de-facto detector configuration of 2PE microscopy. The images were acquired at 500$~\mu m$ depth in the mouse brain sample, in a single field of view. In order to obtain the non-descanned image, a dichroic mirror (DM*3, Fig. 1) was placed on the detection path to guide the fluorescence to the NDD instead of the SPAD array. All the SPAD array recordings, with and without ISM reconstruction show clearly superior contrast and details, which underlines the strength of an optical pinhole also in 2PE microscopy – even a large one (the SPAD array field-of-view is $\sim$ 1.5 AU).

3.4 Calibrating the static ISM shift vectors

In the FRC measurements in Fig. 5 d) we also show numerical results obtained with the static ISM reconstruction with a pixel reassignment factor $\alpha = 0.5$. The results are shown here for reference, as the static reassignment with $\alpha = 0.5$ is the most common method for ISM pixel reassignment found in the literature. Although it appears to work rather well, we decided to see whether we could make some improvements. According to our theoretical calculations [14], the static and adaptive approaches, in absence of major optical distortions, should provide identical results. We therefore came up with a method to leverage the APR-ISM shift-vectors to calibrate the PR-ISM. As shown in (Fig. 8) this is essentially a two-step process. First. we determined the effective rotation angle of the APR-ISM shift-vector grid and then applied the same rotation on the theoretical PR-ISM shift-vectors. Second, we found the ISM reassignment factor value, at which the APR-ISM and PR-ISM shift-vectors align on the scatter plot – the value $\alpha = 2/3$ also matches with our theoretical calculations [41]. After such corrections, the APR-ISM and PR-ISM at least in relatively regular imaging conditions, provide practically identical results. Based on results shown in Fig. 5 and Fig. 8 it should thus be clear that careful calibration/system alignment is required in order to ensure that PR-ISM works optimally. APR-ISM on the other hand automatically adapts to changing imaging conditions.

 

Fig. 8. Calibrating static ISM. The process to calibrate the PR-ISM shift-vectors with the APR-ISM results from the HeLa cell (Fig. 4). After correcting for the observed rotation and changing into optimized reassignment factor ($\alpha =2/3$), PR-ISM and APR-ISM produce identical results, which confirms our previous theoretical calculations [14].

Download Full Size | PDF

4. Conclusion

In this paper, we introduced a computational 2PE-ISM system based on a SPAD array detector that thanks to the fully automatic and adaptive image reconstruction method, is very similar to use as a regular 2PE system. No specific alignment, calibration measurements or other additional steps are required to achieve the improved resolution and SNR.

We demonstrated how our adaptive blind ISM image reconstruction is able to generate nearly constant quality, high resolution images, with different samples, and at various penetration depths. In addition to the accurate image alignment, deconvolution – linear (APR-ISM$^{\mathrm {bW}}$) or iterative (APR-ISM$^{\mathrm {bRL}}$) – appears to be an important step in computational ISM image reconstruction. Using iterative deconvolution, such as RL, in addition to optimizing the effective resolution, makes it possible to also correct for a strong background fluorescence signal. Linear Wiener filtering on the other hand provides a simple and straightforward approach that usually works extremely well. In our view, both of the two methods have their strengths, and therefore we decided to show results with both. While there currently is no magic recipe for selecting one method over the other, as deconvolution is a post-processing task, one could think of using several algorithms in parallel and then picking the most appropriate result for analysis/visualization. This selection could potentially be done automatically, e.g. by a quantitative method such as [55]. FRC provides a robust quantitative measure of effective resolution, but it cannot alone differentiate between details and noise artefacts. Also, in order to take full advantage of the array detector, we expect that multi-image deconvolution [17,42,50], with appropriately modelled PSFs for the 25 sub-images, will provide an even larger improvement.

The data recorded with the SPAD array may contain a key for finding an improved model for the PSFs for deconvolution, as e.g., the varying image shifts observed in the adaptive image reconstruction are indicative of system and/or sample based aberrations. Because 2PE signal can be directly modelled as a guide star (point source) [56], changes in the fluorescence distribution on the SPAD array are indicative of changes in the shape of the excitation/emission wave-fronts. This information, in addition to enabling more realistic PSF modelling, when properly quantified, may in the future enable new kinds of adaptive optics based aberration correction schemes. Realising such a system is the next major step in our 2PE-ISM project.

While in this paper we focused on 2PE fluorescence imaging, the proposed architecture may also work in other non-linear (label-free) microscopy methods [30,43,57] and other imaging techniques, such as ophthalmoscopy [58]. Furthermore, the single-photon ability of the SPAD array can be used to correlate 2PE-ISM with fluorescence lifetime imaging [17] or to combine 2PE with quantum ISM [59], further improving the spatial resolution.

Our 2PE-ISM implementation was based on a descanned architecture, mainly for the sake of simplicity, sample compatibility, and high-frame rate characteristic. Unlike in optomechanical ISM, in which the re-scanning is an integral part of the ISM architecture, technically it would be possible to implement a non-descanned architecture in our point-scanning system, by simply employing stage scanning instead of laser scanning. Placing the SPAD array closer to the objective would mainly increase the detection of the scattered fluorescence photons. However, one has to consider that the whole working principle of ISM pixel reassignment is based on simple geometry, and the assumption of ballistic photons. Scattered photons do not follow those assumptions, and thus a new type of an image reconstruction algorithm would need to be developed, in order to take advantage of them. Devising such an approach remains a subject of a future study. In any case, since the ratio between non-scattered/ballistic fluorescence photons and scattered fluorescence photons reduces for increasing depth, there will be a sample-dependent depth at which non-descanned 2PE imaging provides an effective resolution higher than our 2PE-ISM implementation. However, up to this limit non-descanned 2PE implementations can not perform better than our descanned 2PE-ISM, since the optical resolution in 2PE-ISM is always better by a factor $\sim$ 1.6. Improvements on the photon collection efficiency of SPAD arrays [60–62] will move this limit deeper and deeper. Moreover, one should not underestimate the benefit of having an effective optical pinhole (that is not detrimental to the SNR) in a 2PE system, as the effective working distance of a typical 2PE system is ultimately limited by near-surface (out-of-focus) fluorescence signal that the wide area detectors cannot reject [35,36]. Our simple comparison in Fig. 7 showed that already with relatively bright and accessible samples, the pinhole effect is rather dramatic.

One important application of 2PE microscopy is in-vivo imaging, which necessitates high frame rate imaging. Typically resonant scanners and fast photo-multiplier detectors are used. In this context it is important to highlight that the parallel nature of the SPAD array detector increases the imaging dynamic range [63]. Despite the relatively long hold-off/dead time of each single SPAD array element (200 ns in the device used here, 20 ns in more recent models [60]), because the fluorescent photons are spread across the many elements of the SPAD array, its effective dynamic range is rather high. In fact we have already used the SPAD array with resonant scanners, as was shown in [17]. One could thus consider using our SPAD array based ISM in in-vivo imaging applications as well.

The final thoughts, as well as several examples in this paper, help to underline the importance of computation in modern microscopy. In wide-field microscopy, similar approaches have been in regular use already for a while – e.g., axial sectioning based on deconvolution, SIM, localization based super resolution – but confocal and 2PE microscopy have remained largely analog and microscope users often have to content with the raw data. We think that the huge potential of single-photon capable array detectors, such as the SPAD array and simple, ideally parameter-free image reconstruction/restoration algorithms – as highlighted by its application to 2PE-ISM here – can put an end to that.