Image scanning microscopy with a quadrant detector

Marco Castello; Colin J. R. Sheppard; Alberto Diaspro; Giuseppe Vicidomini

doi:10.1364/OL.40.005355

Confocal scanning microscopy (CSM) is, in principle, a super-resolution technique, allowing the diffraction barrier to be overcome by a factor of $\sqrt{2}$ [as defined by the full-width at half-maximum (FWHM) of the point spread function (PSF)] [1,2]. Unfortunately, for fluorescence microscopy, at least, this resolution enhancement is only theoretical; in practice, it can be obtained only by reducing the pinhole diameter. However, shrinking the pinhole size comes along with a heavy reduction in the detection efficiency and, therefore, with a reduction of the signal-to-noise ratio (SNR) of the resulting images [3]. The full $\sqrt{2}$ improvement is only obtained in the limit of a vanishingly small pinhole size. Thus, in practice, the lateral resolution is sacrificed to obtain a good SNR.

A nonconfocal alternative is represented by structured illumination microscopy (SIM), in which the sample is illuminated with a series of grating-like light patterns, which encode normally inaccessible high-frequency information into the resulting image [4,5]. In contrast to the confocal, the resolution enhancement can be obtained without reducing the signal and, therefore, the SNR, but several other disadvantages emerge in terms of technical complexity, acquisition time, and sensitivity to optical imperfections and aberrations, always found in biological samples. Almost 30 years ago, Sheppard proposed a way to overcome the trade-off between lateral resolution and the signal in CSM [6]. This method subsequently took the name image scanning microscopy (ISM) [7], although several other alternatives have been proposed in the literature [8–10]. In a few words, the single detector is replaced with an imaging detector (usually a camera) to image the emitted light from different excitation positions during the scanning of the sample. The final image is obtained by computationally combining the information contained in the whole dataset of the acquired images with operations either in the space or in the Fourier domain [7,10,11]. Alternatively, the images can also be optically recombined through dedicated (optical) hardware [8,9]. It can be observed that ISM constitutes a particular implementation of SIM microscopy, where the structured illumination pattern is just the focused laser spot. In fact, CSM itself can also be regarded as SIM with a single spot structured illumination pattern. Basically, all the different implementations of SIM (apart from the SIM implementations in which the emission rate of the sample responds nonlinearly to the illumination intensity [12]) result in the spatial-frequency bandwidth of conventional imaging being doubled. The similarity between CSM and SIM is particularly apparent when considering spinning disk confocal microscopy or multi-focal microscopy, where the illumination pattern is an array of spots, which can be viewed as a grating structure. In spinning disk confocal microscopy, the image is reconstructed optically using an array of pinholes, rather than a digital reconstruction normally employed in SIM. In fact, it is interesting to note that the original implementation of SIM also used optical reconstruction, using a grating. It is noteworthy that both elegant and straightforward combinations of spinning disk confocal microscopy [13] and multi-focal microscopy [14,15] with ISM have been recently proposed.

CSM is the most widely used modern optical microscopy technique, but as ISM can, in principle, result in a simultaneous improvement in both resolution and signal strength, incorporation of the ISM technique is highly desirable. Although several implementations have been proposed since ISM was first theorized, in our opinion, none of them is straightforward enough to replace the CSM as we know it; they all have some drawbacks in terms of flexibility, acquisition time, collection efficiency, hardware complexity, or computational cost. In this Letter, we propose a technique, based on ISM, to decouple practically lateral resolution from fluorescence signal level and detection efficiency. Optical integration methods need substantial modification of the apparatus and do not normally offer a flexible approach to image reconstruction. Alternatively, implementations with a camera are relatively straightforward, but the major drawback is still the slow imaging speed. The best solution, therefore, seems to be an array of detectors such as photo-multipliers or avalanche photodiodes. We show that good performance can be achieved with a small number of detector elements. This condition is paramount to an efficient implementation, since a large number of detector elements need closely integrated electronics, thus reducing the optical fill factor. Hence, we investigate replacing the single detector of CSM with a quadrant detector, giving a quadrant ISM (QISM).

Today, quadrant detector arrays with the same performance as the faster single detectors are available on the market, and this, in our opinion, can pave the way to an implementation of ISM fully compatible with CSM (no drawbacks, no change in the workflow, no new parameters to set). Avalanche photodiodes (APDs) exhibit high quantum efficiency. In addition, count-rate is a limitation of APDs, but the use of an APD array reduces the count-rate required by each element. We present simulations demonstrating that the resolution achievable by using a quadrant detector is comparable to that obtainable with a pixel matrix having a large number of pixels. That we can achieve such good results with just four pixels may seem surprising, but this can be explained on the grounds of information capacity (Lukosz [4]), as we can argue that doubling the spatial frequency bandwidth in two directions needs about four times as much information.

We start from the equation defining the PSF of a standard CSM [16]:

h_{CSM} (x, y, z) = h_{exc} (x, y, z) \cdot (h_{det} (x, y, z) * a (x, y)),

where

h_{exc}

and

h_{det}

are the intensities of the electrical field of excitation and detection light, respectively;

a

is a function modeling the pinhole shape (i.e., the quadrant, square, or circular aperture) and size; and the convolution operator acts only on the lateral coordinate

(x, y)

. The intensity PSF is thus given by the product of the illumination and collection PSFs, with the latter degraded by convolution with the detector distribution. It is possible to observe that (1) the smaller the pinhole, the narrower is the result of the convolution operation; (2) the smaller the pinhole, the lower the detection efficiency, noting that the integral of the convolution representing the detection on the whole space can be simply obtained as the product of the integrals of the two terms taken separately (Fubini’s theorem); (3) when the pinhole radius tends to infinity, the PSF is just given by the excitation term (as in a nonconfocal scanning microscope); (4) when the pinhole radius tends to zero, the pinhole can be modeled as a Dirac delta function, and the resulting PSF is the product of the excitation and detection intensity profiles. The last limiting case provides a way to approximate easily the theoretical lower limit of the FWHM of the PSF for a CSM. In fact, if we consider a Gaussian profile for both the excitation and the detection, and we approximate

σ = σ_{exc} \approx σ_{det}

where

σ = λ / (2 \cdot N A)

we obtain

h_{CSM} = α \cdot h_{exc} \cdot h_{det} \approx α \cdot G (p, σ_{exc}) \cdot G (p, σ_{det}) \approx α \cdot G^{2} (p, σ) \approx α \cdot G (p, σ / \sqrt{2}),

where

G (p, σ)

denotes a Gaussian function centered in

p

and with width

σ

. Unfortunately, as a consequence of the second statement, the

α

constant decreases with the decreasing of the pinhole aperture, and tends to zero in the ideal case. Image scanning microscopy is a strategy to approach the ideal case without discarding light. This may lead to a FWHM reduced by a factor of

\sqrt{2}

in the space domain, and a doubled cutoff spatial frequency, compared with a conventional microscope, in the Fourier domain. Our simulator was created starting from Eq. (1). The intensity of the excitation electrical field,

h_{exc}

and detection light

h_{det}

on the focal point was estimated using the algorithm presented in [17]. First, we simulated the PSF of a standard CSM (

λ_{exc} = 640 nm

,

λ_{det} = 650 nm

,

N A = 1.4

) for several pinhole apertures in the range between 0.1 and 3.0 AU [Fig. 1(a)]. We normalized everywhere the intensity to the confocal having pinhole size 1 AU. Then we studied the PSFs of the four elements of the quadrant detector taken separately under consideration. As expected, we observed that the PSF of the first element (top-right) is anisotropic (and asymmetric), and its maximum value is shifted from the center of the focus upward and to the right [Fig. 1(b)]. However, as each quadrant is smaller than the complete circle, the width of the PSF is smaller for the quadrant with the same outer radius. It is easy to observe that given the geometry of the detector (quadrant), the four PSFs are symmetric, and the center of symmetry is the center of focus. We estimate the shift by measuring the distance of the maximum value to the focal center [Fig. 1(b)]. This shift from the focus increases in a quasi-linear way as the detector radius increases and tends to a plateau for large quadrants. Figure 1(d) shows a comparison between standard confocal and single-quadrant (top-right) imaging. Given the four raw images, the final QISM image can be obtained by shifting back and adding up the raw images. This process is called pixel reassignment. Since the pixel reassignment operation is linear, we can model the QISM in the fluorescence mode as a linear imaging system characterized by its own PSF and optical transfer function (OTF). The PSF and the OTF are linked by a Fourier transform [Figs. 2(a) and 2(b)]. The OTF is given by the sum of the OTFs of the four detector elements, each properly phase shifted (a change in the phase in the Fourier domain corresponds to a shift in the space domain [Fig. 1(b), insert]. Thus, it is possible to benchmark and compare a standard CSM having a pinhole aperture of 1 AU with a QISM having the same size for the total detector radius. The resulting PSF of the QISM [Fig. 2(a)] proves to be narrower than that of the respective confocal system. This fact comes along with another phenomenon, which we call super-brightness: since the detection efficiency is the same because the pinhole and the quadrant discards the same amount of light (same radius), if the PSF is narrower, a detected photon is more likely to come from the center of the focus. This feature can also be observed by considering the peak intensity as a function of radius for both the CSM and the QISM [Fig. 2(c)], and leads to a higher (peak) SNR and contrast.

Fig. 1. (a), (b) Line intensity profile of the PSF of a standard confocal (a), varying the pinhole size (from 0.1 to 3.0 AU), and of the first detector (top-right) of a quadrant (b) for different radii (from 0.1 to 3.0 AU). (b, insert) Plot of the shift (respect to the center of the focus) of the maximum value of the PSF related to the first detector (top-right) of a quadrant as a function of the total radius (AU). (c), (d) Starting from the tubulin phantom (c), the imaging process was simulated for a standard confocal (1 AU) and the first (top-right) detector of the quadrant. (d) Pinhole and quadrant radius are normalized to the Airy disk at the image plane. Scale bar: 1 μm.

Download Full Size | PDF

Fig. 2. (a)–(e) Comparisons between a standard confocal with 1 AU pinhole aperture (black line) and a QISM microscope (red line) based on pixel reassignment through different figures of merit: the projection of the PSF (intensity normalized on 1 AU confocal) (a) and of the OTF (e) on $x$ -axis; the projection of the PSF along the optical axis (b), the peak intensity (c), and the FWHM (d) as a function of the pinhole (or quadrant) radius. (f) The FWHM of the resulting PSF after pixel reassignment for a squared detector matrix as a function of the number of detectors (N) for different sizes of the detector half-side ( $r$ ).

Download Full Size | PDF

The theoretical FWHM ( $\approx 170 nm$ in our simulations) can be satisfactorily approached ( $\approx 180 nm$ ) with a quadrant having a radius equal to 1 AU [Fig. 2(d)]. As the rate of change of the peak intensity is zero at 1 AU, the optimum combination of resolution and signal level is achieved for a radius slightly smaller than 1 AU.

Figure 2(b) shows the OTF for CSM and ISM with a radius of 1 AU. The OTF for ISM is broader, as expected from the improved resolution. Further, it is seen that the OTF for CSM with a pinhole radius of 1 AU actually exhibits some regions of negative value, equivalent to an aberrating phase shift. Figure 2(e) shows the axial PSF for CSM and ISM with a radius of 1 AU. The widths of the curves are very similar, but the super-brightness is evident in the ISM case. Basically, the axial imaging performance, in both cases, stems from the presence of the confocal pinhole. A microscope with a large detector array exhibits no optical sectioning, so that the size of the array must be limited to retain the confocal sectioning property. An array of small size does not necessarily need to consist of a large number of elements because the signal would be oversampled; thus, four elements as in QISM is an appropriate choice.

We also ran simulations for another geometry of detector array, the classical square pixel matrix, varying the number of detector elements [Fig. 2(f)]. Interestingly, the square matrix of 2 by 2 pixels scores more or less the same FWHM as the quadrant, suggesting that the shape of the pinhole does not affect the PSF of the instrument. Through a large number of elements, the theoretical resolution ( $\approx 169 nm$ ) may be reached, but the gain may not be worth the technical effort required to condition and process the signal coming from a large number of elements. However, this last implementation may bring additional benefits (for instance, “virtual” adjustable pinhole or background estimation). Returning to the quadrant, in regard to the optical sectioning, although there is no significant improvement, the super-brightness is still evident in the peak of the axial PSF. It is important to note that these achievements are obtained without degrading the scanning speed of a standard CSM. In fact, each individual quadrant images a slightly different point of the sample, so that if the sample spacing in the detector plane is equal to the sample spacing of the object illumination, the reconstructed image exhibits double the sampling rate, which is appropriate for the increased spatial frequency bandwidth. Thus, the illumination can be sampled at a conventional Nyquist rate, rather than at a confocal Nyquist rate, giving a corresponding speed advantage over a conventional confocal. Another advantage of this implementation is that by summing up the images produced by the detectors, it is possible to obtain the image of a standard CSM having pinhole radius equal to the quadrant (backward compatibility).

The process of obtaining the final image in ISM can be seen as a problem of image reconstruction and multi-sensor (sometimes called diversity) data fusion. The literature reports several different strategies to combine the information content of image data sets. In particular, in QISM, there are four different images, characterized by (almost) the same SNR, scale, and rotation; just the field-of-view is shifted in space. As mentioned before, the simplest method to obtain the final sub-diffraction image is by pixel reassignment [Figs. 3(a)–3(c)]: despite its simplicity, it offers significant advantages such as linearity and real-time execution on almost any computing architecture. It is easy to see that this method does not need a complete a priori model of the system, but just an estimate of the shift of each component image with respect to the center of focus. (In the case of ideal alignment, the absolute value is the same for the four elements; just the direction changes.) In a practical implementation, the four images can be registered without this parameter; in fact, the shift can be estimated directly on the data by using phase correlation. Each view $i_{i}$ can be registered with another $i_{r}$ (reference) by calculating

(δ_{x}, δ_{y}) = \underset{x, y}{argmax} (F^{- 1} {\frac{I_{i} \cdot I_{r}}{| I_{i} \cdot I_{r} |}}),

where

F

denotes the Fourier transform,

I_{i} = F {i_{i}}

,

I_{r} = F {i_{r}}

, and the matrix product is in the Hadamard sense. More advanced approaches, extensively studied in the literature and successfully applied to microscopy, are based on multi-image deconvolution and, particularly, generalizations of Wiener filtering and of the Richardson–Lucy algorithm [18]. Although these methods require a more extensive knowledge of the system (model) and more computational efforts, their ability to “invert the system” usually leads to better results in terms of information content of the result (fidelity to the observed object). In this Letter, we employed Wiener filtering [Figs. 3(a), 3(c), and 3(d)] in the form

g = F^{- 1} {\frac{\sum_{i = 1}^{4} F^{*} {{PSF}_{i}} \cdot F {i_{i}}}{\sum_{i = 1}^{4} {| F {{PSF}_{i}} |}^{2} + η}},

where

{PSF}_{i}

is the PSF associated to the image

i

obtained by the

i

-th element of the detector and

η > 0

is the regularization parameter. This strategy is very often used in SIM [12]. Wiener deconvolution with a reasonable choice of the regularization parameter

η

is, in general, considered to be more conservative than the Richardson–Lucy algorithm; on the other hand, it tends to be more powerful in recovering high-frequency information content, but may introduce artifacts in the resulting images. Image deconvolution methods are more expensive from a computational point of view than pixel reassignment but, in recent years, graphics processing units (GPUs) have been proved to be valid and cheap instruments to perform this kind of computation in real time [19].

Fig. 3. (a), (b), (d) Result of the pixel reassignment for a QISM (1 AU) (b) is shown next to a standard confocal image having the same nominal resolution (0.5 AU) (a) for a phantom counting 50 peak photons. The information collected by the four detectors of the quadrant can be recombined using a multi-image extension of Wiener filtering ( $η = 0.01$ ) (d). (c) Signal-to-error ratio (SER) [18] between the phantom and the image of a standard confocal (1 AU) (cyan), a QISM microscope based on pixel reassignment (blue), a QISM using Wiener filtering (light green), and Wiener filtering on a standard confocal image (dark-green), for different SNR (photon counts) ( $η = 0.01$ ). Simulated noise model, shot noise; scale bar, 1 μm.

Download Full Size | PDF

Summarizing, the appearance on the market of quadrant detector arrays of avalanche photodiode (APD) can pave the way to the diffusion of ISM. This theoretical work demonstrates that a quadrant is able to obtain a resolution very close to the confocal limit without great technical complexity. The quadrant detector can be used as an add-on to an existing confocal microscope; it also can be used with the existing confocal pinhole to tune the overall balance among signal level, resolution, and optical sectioning.

Funding

Ministry of Education and Research (MIUR) (PRIN 2010JFYFY2_002).

Acknowledgment

The authors thank S. Saporito from Technische Universiteit Eindhoven and M. Buttafava, F. Villa, and A. Tosi from Politecnico di Milano for fruitful discussions. A. D. and G. V. were partially supported by MIUR.

REFERENCES

1. C. J. R. Sheppard and A. Choudhury, Opt. Acta 24, 1051 (1977). [CrossRef]

2. G. J. Brakenhoff, P. Blom, and P. Barends, J. Microsc. 117, 219 (1979). [CrossRef]

3. C. J. Sheppard, X. Gan, M. Gu, and M. Roy, Handbook of Biological Confocal Microscopy, J. B. Pawley, ed. (Springer, 1995), pp. 363–371.

4. W. Lukosz, J. Opt. Soc. Am. 57, 932 (1967). [CrossRef]

5. M. G. L. Gustafsson, J. Microsc. 198, 82 (2000). [CrossRef]

6. C. J. R. Sheppard, Optik 80, 53 (1988).

7. C. B. Müller and J. Enderlein, Phys. Rev. Lett. 104, 198101 (2010). [CrossRef]

8. G. M. De Luca, R. M. Breedijk, R. A. Brandt, C. H. Zeelenberg, B. E. de Jong, W. Timmermans, L. Nahidi Azar, R. A. Hoebe, S. Stallinga, and E. M. Manders, Biomed. Opt. Express 4, 2644 (2013). [CrossRef]

9. S. Roth, C. J. R. Sheppard, K. Wicker, and R. Heintzmann, Opt. Nanoscopy 2, 5 (2013). [CrossRef]

10. K. Weisshart, Tech. Rep., Zeiss (2014).

11. C. J. R. Sheppard, S. B. Mehta, and R. Heintzmann, Opt. Lett. 38, 2889 (2013). [CrossRef]

12. M. G. L. Gustafsson, L. Shao, P. M. Carlton, C. J. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, and J. W. Sedat, Biophys. J. 94, 4957 (2008). [CrossRef]

13. O. Schulz, C. Pieper, M. Clever, J. Pfaff, A. Ruhlandt, R. H. Kehlenbach, F. S. Wouters, J. Großhans, G. Bunt, and J. Enderlein, Proc. Natl. Acad. Sci. USA 110, 21000 (2013). [CrossRef]

14. A. G. York, S. H. Parekh, D. D. Nogare, R. S. Fischer, K. Temprine, M. Mione, A. B. Chitnis, C. A. Combs, and H. Shroff, Nat. Methods 9, 749 (2012). [CrossRef]

15. A. G. York, P. Chandris, D. D. Nogare, J. Head, P. Wawrzusin, R. S. Fischer, A. Chitnis, and H. Shroff, Nat. Methods 10, 1122 (2013). [CrossRef]

16. I. J. Cox, C. J. R. Sheppard, and T. Wilson, Optik 60, 391 (1982).

17. M. Leutenegger, R. Rao, R. A. Leitgeb, and T. Lasser, Opt. Express 14, 11277 (2006). [CrossRef]

18. M. Castello, A. Diaspro, and G. Vicidomini, Appl. Phys. Lett. 105, 234106 (2014). [CrossRef]

19. R. Zanella, G. Zanghirati, R. Cavicchioli, L. Zanni, P. Boccacci, M. Bertero, and G. Vicidomini, Sci. Rep. 3, 2523 (2013). [CrossRef]

Image scanning microscopy with a quadrant detector

Abstract

Funding

Acknowledgment

REFERENCES

Cited By

Figures (3)

Equations (4)

Optics Letters