1. Introduction

Super-resolution optical fluctuation imaging (SOFI) is an established approach for sub-diffraction fluorescence imaging. It delivers a theoretically-unlimited spatial resolution by sampling the stochastic emission fluctuations or ‘blinking’ events present in many types of fluorophores [1]. The technique has been applied to a wide variety of samples, with fluorophores including fluorescent proteins [2,3], organic dyes [4], genetically-encoded biosensors [5,6], and quantum dots [1]. Compared to other sub-diffraction approaches, SOFI imaging can work over a broad range of conditions including low emitter brightness, high emitter density, and/or high background, with a temporal resolution of seconds and typically an up to two- or three-fold improvement in spatial resolution [7–9]. Especially, when quantitative comparisons between samples need to be performed, much can be gained from the robust nature of the analysis when compared to localization microscopy [10]. The technique can be applied standalone or can even be combined with other modalities to deliver an enhanced imaging performance [11–13].

The full theoretical description of the image formation shows that the SOFI imaging process is reliable if the sample is stationary and the emitters show independent fluorescence fluctuations [14]. This analysis process can also be extended to separate emitters based on their blinking dynamics [15], or to correct for photodestruction or diffusion of the emitters [16,17]. Furthermore, the accuracy of the resulting images can be readily estimated using statistical resampling, which also allows the estimation of the sample quality [18], and, when combined with other metrics, can lead to quantitative and objective tools to evaluate the quality of SOFI images [19].

In practice, SOFI operates on stacks of wide-field images acquired from the same position on the sample, where a single stack containing hundreds or more fluorescence images is analyzed to obtain a single sub-diffraction SOFI image. The analysis calculates the cumulants of the intensity distributions seen in every detector pixel, where the order of the cumulants used is known as the order of the SOFI calculation. The values of these cumulants then become the pixel values in the resulting SOFI image. To avoid shot noise, one usually does not calculate autocumulants, but rather calculates cross-cumulants between different detector pixels (analogous to the distinction between variance and co-variance). These cross-cumulants then result in the creation of ‘virtual pixels’ [20], which makes it possible to exploit the increase in spatial resolution by increasing the sampling density of the calculated SOFI image. Alternatively, one can also use Fourier interpolation to increase the sampling density [21], though to our knowledge the calculation of virtual pixels remains the most common methodology.

A typical SOFI image thus consists of multiple different types of virtual pixels, where each type of virtual pixel results from a distinct combination of detector pixels at different distances from each other (Fig. 1(a)). However, since the signal of a blinking molecule correlates only with itself, the SOFI signal arising from a cross-cumulant will scale with the distances between the detector pixels relative to the size of the point spread function (PSF). This means that the SOFI image will in general show pixelation artifacts due to the different scale factors associated with each virtual pixel type, which must be corrected.

 

Fig. 1. Application of different SOFI correction schemes on simulation data of randomly distributed lines of two-state emitters. (a) Schematic illustration of the composition of pixel combinations for the calculation of virtual pixels. The virtual pixels (orange) are determined by the cross-cumulant between detector pixels (blue). The pixels are not only limited to the depicted nearest neighbour combinations, but also cumulants between pixels with longer distances can be taken into account. (b) Average image shows the randomly distributed lines of blinking emitters. The white box defines the zoom in area which was used to compare the different SOFI algorithms in (c)-(e). (c) The resulting image from a SOFI analysis without applying any correction approach shows pixelation artifacts. (d) The pixelation correction approach, adapted from Dertinger et al., based on a Gaussian PSF optimization approach [20]. In comparison to the uncorrected image (c), an improvement of the pixelation can be noticed. (e) Applying an average matching approach to the SOFI analysis also decreases the pixelation effects. Scale bars: $1 $ µm

Download Full Size | PDF

Previous work suggested the calculation of appropriate correction factors based on knowledge of the PSF [20]. Alternatively, if exact information on the PSF is unavailable, the authors suggested an iterative search for suitable correction factors by optimizing a Gaussian model for the unknown PSF. In practice, however, we found that this approach does not always lead to pixelation-free SOFI images on real datasets, even when the PSF can be considered to be Gaussian-like and is known, hinting to the presence of additional contributions that are not well captured by this method.

While the theoretical description of SOFI imaging assumes that signal arises only from the blinking of the fluorophores, in practice the SOFI signal may arise from a range of other sources, including photodestruction, probe diffusion, but also imperfections in the used detector. We used SOFI to judge the relative performance of industrial CMOS, scientific CMOS (sCMOS), and electron-multiplied CCD (EM-CCD) cameras [22], and observed that the industrial CMOS camera showed imperfections that contributed spurious SOFI signals. Such distortions not only affect the raw values of both, the conventional and SOFI images, but can also interfere with the required pixelation correction. We therefore wondered whether it is possible to identify alternative strategies that provide a more robust correction.

In this contribution, we describe a model-free and computationally efficient strategy for the correction of cross-cumulant pixelation in SOFI imaging. We show how this strategy is straightforward to implement and can also results in increased robustness against detector imperfections.

2. Methods

2.1 Simulation

The simulation data were obtained following the simulation procedure described in Ref. [16], consisting of randomly distributed lines of 50,000 emitters with a brightness of 45000 photons per second within 100 by 100 pixel images with a simulated optical pixel size of 100 nm. The emitters switch independently and randomly between two states. The on-time ratio is $14$ % with an on-time of $ 10 $ ms. The simulated movie consisted of 2000 fluorescence images with as simulated exposure time of 10 ms. All calculations were performed using Igor Pro 8 (WaveMetrics, Inc.) and the Localizer package [23].

2.2 Live cell data

The sample preparation and imaging of the live cells was detailed in Ref. [24]. A commercial cellTIRF microscope (Olympus) and an ImageEM camera were used for the imaging of HeLa cells labeled with Lyn-tagged rsGreen0.8, excited at $488 $ nm. The camera exposure time was set to $ 31 $ ms, and the EM gain was adjusted for an optimal dynamic range. $2,000$ frames were acquired for the dataset shown here.

A detailed description of the sample preparation and acquisition of live cell data with a Hamamatsu Fusion sCMOS camera was given in [22]. In brief, a Nikon Eclipse TI2 motorized microscope was used in combination with an Oxxius L6Cc Laser combiner box which provided $488 $ nm excitation light. A dataset of 500 frames of microtubules of Cos-7 cells was acquired with the sCMOS camera. The microtubules were stained with ffDronpa linked to microtubule-associated protein 4 (MAP4).

2.3 EM-CCD dark images

The EM-CCD dark current data without any incident light was acquired using the Andor iXon Ultra 897 with a temperature setting of $20$ °C. The pixel readout rate was set to $17 $ MHz and $1 $ MHz, depending on the measurement. The electron multiplier gain and the exposure time were set to 100 and 1 s, respectively. 50,000 frames were acquired for each dataset. We used an Andor camera for this acquisition instead of the Hamamatsu camera simply because the cooling can be switched off in this system, supported by our observation that all of the EM-CCD cameras that we have tested over our years of performing SOFI imaging show very similar distortions.

We excluded $ 0.005 $ % of the sensor pixels from the average and variance estimation since these appeared to have been affected by background radiation (cosmic rays or similar) at some point during the 13 hour measurements.

3. Results and discussion

Figure 1(a) illustrates how the virtual pixels in a SOFI image arise from the calculation of cross-cumulants between different detector pixels. Using this approach, a second-order SOFI image has effectively four times more pixels compared to the raw fluorescence images from which it was calculated.

We first examined a simulated dataset of randomly distributed lines of blinking emitters (Fig. 1(b)), which clearly showed extensive pixelation artifacts that must be corrected to result in accurate SOFI images (Fig. 1(c)). As was outlined in the introduction, Dertinger and colleagues [20] suggested the direct calculation of compensating scale factors from a Gaussian PSF model, either based on direct knowledge of the PSF, or by iteratively optimizing such a model to minimize the pixelation artifacts. This strategy indeed works well for simulated data (Fig. 1(d)), though it requires that the PSF can be well approximated by a Gaussian function and introduces the additional computational overhead and uncertainty inherent in using iterative nonlinear solvers.

Surprisingly, we observed that this strategy did not appear to work well on actual measured SOFI images, even in situations where the PSF was found to be well-approximated by a Gaussian function. Figure 2(a) and 2(b) reveal that pixelation still occurs despite the correction procedure, as is also confirmed by the presence of high spatial frequency components in the Fourier transforms of the uncorrected and corrected SOFI images. The iterative correction approach does succeed in reducing the high frequency components associated with the pixelation artifacts, though their effects remain apparent.

 

Fig. 2. Comparison of different SOFI correction approaches on HeLa cells labeled with Lyn-tagged rsGreen0.8 taken with an EM-CCD camera. (a) Uncorrected SOFI analysis results and an extract of the image used for comparison of the different correction algorithms and their effects on the pixelation artifacts. The Fourier transform has pronounced high frequency contributions, which are attributable to pixelation artifacts. (b) The optimization approach of Dertinger et al. leads to an improved image, however, pixelation effects are still present. The Fourier transform has less high frequency contributions. (c) Matching of the mean values of the four virtual pixel types also has a positive impact on the pixelation artifacts. The resulting image, however, has more high spatial frequencies than the PSF optimization approach. (d) The best results are reached by matching the average and variance of the four pixel types, also demonstrated with the Fourier transform image. Scale bars: $10$ µm (a (left)), $1 $ µm (a (right), b, c, d).

Download Full Size | PDF

This finding motivated us to devise a correction strategy that is both easier to calculate and imposes fewer assumptions on the system. Our key observation was that the pixelation of the detector is independent from the structure of the sample. For a sufficiently large imaged region, it is therefore reasonable to assume that the average signals of the different virtual pixels must be equal. Our initial correction approach was to calculate the average value of each type of SOFI virtual pixel, and then to apply a scale factor to all but one of these pixels such that they all have the same average signal. In principle, it does not matter which virtual pixel is chosen as the reference value since the linearity of the imaging with respect to the fluorophore concentration will still be preserved, and different SOFI images can still be compared directly as long as one consistently picks the same virtual pixel type as the reference.

We first tested whether this strategy worked successfully on simulated data, observing that the resulting images indeed did not show any pixelation artifacts (Fig. 1(e)). However, we did observe residual pixelation artifacts when applying this strategy to actual data recorded using an EM-CCD camera, similar to what we observed using the iterative correction approach (Fig. 2(c)).

While the origin of the discrepancy between the simulated and measured data remained unclear, we reasoned that the intensity distributions of the different virtual pixels should not only show the same average value, but also the same variance. We therefore extended our correction process by retaining both a scaling factor but also adding an offset to each virtual pixel value, such that both the mean and variance are matched for all of the virtual pixels. For each pixel with a value $x$, a corrected value $y = a_i x + b_i$ can be determined with the slope $a_i$ and y-intercept $b_i$ associated to the pixel’s virtual pixel type:

The index $i=2, 3, \ldots$ refers to the virtual pixel types, using the virtual pixel type denoted with $i=1$ as a reference which is not adjusted during the pixelation correction. The variance $\sigma _i^2$ and mean $\mu _i$ are calculated from all uncorrected pixel values of the pixel type $i$. Figure 2(d) shows an application of this correction algorithm, which indeed reveals a very good suppression of the high-frequency signals in the associated Fourier transform. The pixelation correction for this dataset increases the runtime by less than 500 ms on a commercially available laptop, corresponding to approximately $ 1.3 $ % of the SOFI calculation runtime. It therefore has a very small impact on the overall calculation time. Importantly, this correction algorithm is easy and fast to calculate, essentially eliminates all pixelation artifacts, and does not require any knowledge of the particulars of the imaging.

Though we have not described this approach before, the algorithm has been integrated in our Localizer software package [23]. For several years, it has been successfully and routinely applied on the many datasets that we have since acquired on a variety of different cameras, including also higher-order SOFI images. A similar strategy was independently reported in Ref. [25], though this included only the estimation of scale factors by equalizing the variances, entirely analogous to our original approach of equalizing the averages. Though it has been very useful, our expansion of the strategy to include both mean and variance equalization was largely empirical in nature, since the SOFI image formation model predicts that scaling the intensities to match the average values (or variances) should be sufficient to correct all pixelation. Indeed that is also confirmed by the simulations. So why did we need to also correct for the variance? We reasoned that the deviations observed in real datasets must originate from additional correlations in the measurement process, most likely from limitations in the camera used to record the images. We examined other datasets acquired using different kinds of EM-CCD cameras, and indeed found that the inclusion of both average and variance matching was required to achieve an adequate pixelation correction for all of these.

To better understand this observation, we examined the raw values for four different virtual pixels, calculated in the horizontal, vertical, and diagonal direction (Fig. 1(a)), on actual measured data. The diagonal and the vertical combinations have similar pixel values, but to our surprise the horizontal virtual pixel differed markedly (Fig. 3(a)), even though its intensity distribution should in principle be identical to those of the vertical pixel combination. This deviation was found in all inspected experimental datasets, and was consistently found associated with the horizontal pixel combination. This asymmetry was reminiscent of the readout process of an EM-CCD camera, where each of the pixel values is read out by a single amplifier. Pixels in a single line are consecutively shifted and read out at a rapid rate by this amplifier (‘serial’ direction), while the different lines are shifted towards the readout line in a much slower process (‘parallel’ direction). The deviating SOFI values would then indicate the presence of correlations in the camera readout process, where the readout value of a given pixel depends not only on the charge contained in that pixel, but also on the charge of the pixel(s) that preceded it in the readout process.

 

Fig. 3. Live cell data and dark images of EM-CCD cameras were examined. (a) For live cell data, the virtual pixel histogram shows a clear distinction of the pixel brightness of the horizontal pixel type from the other virtual pixel types. (b) The average image and the resulting image of a SOFI analysis of dark current images without any pixel correction (c) does not allow any predictions on non-idealities of the camera. (d) The histograms, however, also show a clear distinction between the horizontal virtual pixel in comparison to the other virtual pixels. (e) Matching mean and variance of all pixel types leads to an improved alignment of the histograms.

Download Full Size | PDF

While this explanation aligned with our observations, a clear analysis of this phenomenon was complicated because the observed signals included contributions both from the sample and from possible other factors such as detector imperfections. We therefore sought a more robust way of investigating these distortions, independent from the peculiarities of the sample. Homogeneous illumination of the detector would provide a solution to this issue, but can be surprisingly difficult to realize in practice. Instead, we reasoned that the dark current signal from spontaneously induced electrons could be used as a sample- and instrument-independent data source.

Figure 3(b) and 3(c) show an average and SOFI image of 50,000 dark images acquired on an EM-CCD camera for which the sensor cooling had been set to $20 $ °C. We do not fully understand why the average image displays a clear pattern, which was identified as an ‘annealing pattern’ by the camera manufacturer. Such patterns arise through the back-thinning manufacturing process, and should be unobservable in the absence of illumination. We assume that very small amounts of lights are generated within the system that become observable over the 13 hour integration. Repetitions of the dark current measurement under different conditions reinforced this assumption. The experiment was executed under different lab lighting conditions, during which the camera sensor was always completely shielded by a protective cover, and, hence, should not have been influenced by the outside conditions. The resulting average images showed the same signal strength and overall pattern. As before, the virtual pixel histograms reveal an aberrant distribution for the horizontal virtual pixel oriented along the serial direction (Fig. 3(d)), entirely similar to what was observed in the real measurement data. This distortion was reduced by lowering the readout rate from 17 MHz to 1 MHz, further supporting our assignment of this deviation to correlations in the readout process.

The observed offset between the pixel combinations readily explains why a simple scaling factor was inadequate to fully correct the pixelation artifacts. Matching both the mean and variance of the virtual pixels makes it possible to not only scale the SOFI pixel values, but also to shift these by an offset. The result of applying such a correction is shown in Fig. 3(e), demonstrating that readout-induced distortions inherent to the EM-CCD imaging can be adequately compensated using our correction scheme. While this correction does make the images more appealing, more fundamentally it also restores the quantitative nature of SOFI imaging with respect to concentration, which was disturbed by the additional offset signal.

We next examined to what extent the introduced pixelation correction approach can also be useful for sCMOS-based detectors. Figure 4 shows a dataset of ffDronpa-stained microtubules of Cos-7 cells acquired using a sCMOS camera. As expected, the uncorrected SOFI image showed clear pixelation artifacts (Fig. 4(a)). Applying the PSF optimization (Fig. 4(b)) and the average and variance matching (Fig. 4((c)) led to a similar reduction in pixelation artifacts, also confirmed by the Fourier transforms. These Fourier transforms show a non-symmetric behaviour around the center which stems from the sample structure and should, hence, not be removed by any pixelation correction. The larger numbers of pixels associated with sCMOS cameras also highlights the computational efficiency of our approach, requiring around two seconds to process these images in comparison to over an hour for the PSF optimization approach.

 

Fig. 4. Application of the average and variance matching on sCMOS data, acquired with a Hamamatsu Fusion camera. Microtubules of the Cos-7 cells, stained with ffDronpa, were imaged. (a) An image extract of the uncorrected SOFI analysis results show clear pixelation artefacts. The Fourier transform contains high frequency components, which are attributable to pixelation artefacts. (b) The PSF optimization approach decreases the pixelation effects, as also demonstrated in the Fourier transform image. (c) The average and variance matching delivers similar results as the PSF optimization approach. Scale bars: $1 $ µm.

Download Full Size | PDF

4. Conclusions

In this contribution, we have examined the correction of pixelation artifacts in cross-cumulant based SOFI imaging. We initially proposed a straightforward and model-free correction approach in which the values of the different pixel combinations are scaled such that all pixel combinations show the same average value. This approach is consistent with the theoretical model of SOFI imaging and works well on simulated data.

However, we found that residual pixelation remained observable in real data even when the average-matching correction strategy is applied. Detailed examination of the images and of thermal noise data showed that this residual distortion was due to the presence of imperfections in the camera readout process. These imperfections could be adequately suppressed by introducing an additional additive correction term such that the averages and variances of the different pixel combinations are matched.

Overall, our strategy provides a straightforward and robust method for the correction of cross-cumulant induced pixelation, which does not require any knowledge on the particulars of the imaging system or sample. Our results also confirm that cross-cumulant or cross-correlation based image assessment is a straightforward approach to characterize the non-idealities of imaging systems.