Towards real-time wide-field fluorescence lifetime imaging of 5-ALA labeled brain tumors with multi-tap CMOS cameras

David Reichert; David Reichert; David Reichert; Mikael T. Erkkilä; Mikael T. Erkkilä; Gerhard Holst; Nancy Hecker-Denschlag; Marco Wilzbach; Christoph Hauger; Wolfgang Drexler; Johanna Gesperger; Johanna Gesperger; Barbara Kiesel; Thomas Roetzer; Angelika Unterhuber; Georg Widhalm; Rainer A. Leitgeb; Rainer A. Leitgeb; Marco Andreana

doi:10.1364/BOE.382817

1. Introduction

Intraoperative fluorescence guidance using 5-aminolevulinic acid (5-ALA) induced protoporphyrin IX (PpIX) has made a strong impact in the field of neurosurgery since its first clinical use in 1998 [1]. Within the last two decades, 5-ALA guided surgery proved its efficacy in several trials [2–4] and was approved for use in both Europe (2007) as well as the United States (2017). While high-grade gliomas (HGG) emit strong fluorescence which can be seen by the surgeon, low-grade gliomas (LGG) as well as parts of infiltration zones of HGG do not exhibit sufficient visible fluorescence, thus limiting the range of applications [5,6]. To overcome this problem, quantification of PpIX concentration via spectroscopic methods has been proposed with promising results [7–9]. Hyperspectral wide-field fluorescence imaging has demonstrated the ability to quantify low PpIX emissions and can be considered state of the art for detecting LGG [10,11]. Also, this technology can readily be implemented with surgical instrumentation and has shown to achieve image acquisition rates of 0.5 to 1 Hz [11]. However, this method relies on the additional determination of the optical tissue properties which differ between tissues. Furthermore, recent results indicate that the fluorescence spectrum of PpIX is more complex than originally expected [12–14], making PpIX quantification even more challenging. Also, blood layers reduce the fluorescence signal excited at 405 nm [15], stressing the need for alternative visualization solutions which are less dependent on fluorescence intensity.

A potential alternative is fluorescence lifetime imaging (FLIM) [16–18] of endogenous fluorophores like nicotinamide adenine dinucleotide (NADH) and flavin adenine dinucleotide (FAD), which has proven useful for detecting tumor regions [17,19]. Most systems are based on time-domain lifetime sensing [20] which either requires direct sampling of the decay curve with high-speed digitizer cards [17] or time correlated single photon counting (TCSPC) which requires highly sensitive detectors [19]. However, the acquisition speed is either limited by the laser repetition rate for direct sampling or the dead time of the time analog converter used for TCSPC. In addition, the usual working distances are rather short with only a few millimeters [17]. Although this technology is viable for research applications, neurosurgeons utilize microscopes with long working distances above 200 mm, which limits the potential translation of these technologies for real life scenarios.

Recently, our group developed a frequency domain FLIM system [21] around a dual-tap complementary metal-oxide semiconductor (CMOS) camera [22,23] with similar working distance and field of view (FOV) as found in modern surgical microscopes. To reconstruct the fluorescence lifetime, the camera generates several images with a phase-shifted excitation for each frame to cover a full circle of 360 degrees. The lifetime map is then obtained by fitting a sine curve to the series of frames for each pixel. Using this system we found an increased detection sensitivity for weakly and non-fluorescent glioma samples compared to conventional fluorescence intensity imaging. However, this approach required the acquisition of 16 individual frames for reconstruction which took in total up to 3.2 seconds. Therefore, in this paper we show how the lifetime is affected when using less frames for reconstruction in order to reduce the overall acquisition time.

In a first step, we review the fundamentals of frequency domain fluorescence lifetime imaging with discrete sampling. We then derive how this translates to multi-tap CMOS camera based imaging. In addition, we examine the potential of lifetime weighted fluorescence maps, where the absolute value of the fluorescence lifetime is unknown. We demonstrate the performance of these different acquisition modes by imaging three representative human brain tissue samples: a HGG, an infiltration zone of a LGG and non-pathological brain parenchyma. Our findings are then used to find the optimal parameters for real-time visualization during a simulated incision. Finally, we discuss the remaining limitations for translation into the clinical setting and comment on the future of fluorescence lifetime guided surgery.

2. Theory

2.1 Discrete frequency domain fluorescence lifetime imaging

The fluorescence lifetime, $\tau$, is an ensemble measure describing the average delay between excitation of the fluorophore and emission of a photon. While techniques like TCSPC record a single photon decay event and generate a histogram by sequentially summing up each decayed photon, frequency domain FLIM (FD-FLIM) relies on the average decay of the photon population. In FD-FLIM the amplitude of the excitation laser is periodically modulated with the frequency $f_{\textrm {mod}}$. The emitted fluorescence $I_f$ will follow with the same frequency but with a time shift which is equivalent to an additional phase angle $\Phi _\tau$. The overall signal detected can then be written as

(1)$$I(t) = I_f \cdot \Bigl(\frac{1}{2} + \frac{1}{2}\cos{\left( 2\pi f_{\textrm{mod}} \cdot t + \Phi_\tau \right)\Bigr) + I_s}$$

where $I_s$ denotes any ambient light or noise. In this derivation we assume that the phase shift is solely due to the fluorescence decay and ignore time-of-flight effects which would add a static delay. The fluorescence lifetime $\tau$ is thereby related to the phase with [24]

(2)$$\tan\Phi_\tau = 2\pi f_{\textrm{mod}} \cdot \tau.$$

To measure the fluorescence lifetime, $I(t)$ is sampled over a certain time span $[t_1,t_2]$ and cross-correlated with the excitation signal to obtain the phase and then compute $\tau$ using Eq. (2). As the sampling is discrete the correlation integral becomes a sum. For simplicity we only integrate over a single period:

(3)$$\Phi_\tau = \arg \int_{t_1}^{t_2} I(t) \cdot e^{-it\cdot2\pi f_{\textrm{mod}}} \, dt \approx \arg \sum_{k = 0}^{N-1} I(k) \cdot e^{-i\cdot2\pi \cdot k/N}$$

where $N$ denotes the number of samples acquired. Inserting Eq. (1) leads to:

\begin{array}{c}\Phi_\tau \approx \arg \sum\limits_{k=0}^{N-1} \Bigl(I_f \cdot \Bigl( \frac{1}{2} + \frac{1}{2}\cos{\Bigl(2\pi \frac{k}{N} + \Phi_\tau \Bigr)\Bigr) + I_s} \Bigr) \cdot e^{-i\cdot2\pi k/N}\\ \approx \arg\; \Bigl(\; \sum\limits_{k=0}^{N-1} \Bigl( I_f \cdot \Bigl( \frac{1}{2} + \frac{1}{2}\cos{\Bigl( 2\pi \frac{k}{N} + \Phi_\tau \Bigr)} \Bigr) \cdot e^{-i\cdot2\pi k/N}\\ + \sum\limits_{k=0}^{N-1} I_s \cdot e^{-i\cdot2\pi k/N}\;\Bigr) \end{array}

As the sum will compensate for over- and undershooting values of uniform noise in $I_s$, the phase measured will stay more or less independent of the sample size $N$. However, as more samples are acquired, the sum will act like a weighted average. Thus the error of $\Phi _\tau$ will behave like the standard error of the mean by decreasing with $\sqrt {N}$. This is why a faster acquisition with less samples will increase the phase error and therefore lead to more inaccurate fluorescence lifetime estimations.

2.2 Multi-tap detector based fluorescence lifetime reconstruction

In a conventional CMOS camera photons are converted into photo-electrons which are then stored in a dedicated charge bin while the exposure is active. Based on this architecture, the number of electrons collected will be independent of the arrival time of the photons as long as it is within the set exposure time. On the contrary, multi-tap CMOS sensors use multiple charge bins to reconstruct the arrival time. Here we assume a dual-tap sensor architecture with two charge bins, namely tap A and tap B. While the excitation light is on, photon-electrons are collected by tap A. Similarly, while the excitation light is off, photo-electrons will be collected by tap B. Depending on the modulation frequency, the delayed fluorescence response will lead to a significant amount of photons collected during the dark period, thus encoding the phase shift into the ratio between both taps.

In our configuration, tap A integrates over the first half period and tap B over the second part. This is repeated for an exposure time $T$. The overall signal in each tap is given by:

(4)$$I_{\textrm{tap}} = \int_0^T I(t) \cdot \textrm{sign} \left[ \cos \left( 2\pi f_{\textrm{mod}}\cdot t + \alpha \right) \right]\, dt$$

where $\alpha$ is 0 and 180 degrees for tap A and B, respectively. We define the sign function as equal to $\textrm {sign}(x) = 1$ for $x \geq 0$ and zero for all negative values of x. This would be equivalent to a single sensor readout of the dual-tap sensor generating two individual phase frames (N = 2). Note that each phase frame corresponds to a single discrete sample in Eq. (3) as described in the previous section. To obtain more frames, one can either use a sensor with more charge bins or just repeat the measurement where the excitation laser is artificially delayed by an appropriate phase shift equally distributed over 360 degrees. For N = 4, this would mean a first measurement at $\alpha =$ 0 and 180 degrees for tap A and B, respectively, and a second one with 90 and 270 degrees (see Fig. 1).

Fig. 1. General working principle of the dual-tap fluorescence lifetime imaging method. While tap A integrates over the first half period of the fluorescence signal, tap B integrates over the complementary other half. This process is repeated until the set exposure time is reached. One sensor readout generates an image pair, corresponding to sampling points at 0$^\circ$ and 180$^\circ$. By introducing an additional phase shift of 90$^\circ$ to the integration windows, the fluorescence signal is sampled at 4 points.

Download Full Size | PDF

As shown in sect. 2.1, the number of phase frames will determine the accuracy of the lifetime measurement. Furthermore, a longer exposure time $T$ will generate an image with a better signal to noise ratio, as long as dark noise is negligible. When operating in the shot noise limit, the phase jitter $\Delta \Phi _\tau$ is inversely proportional to the square root of the signal-to-noise ratio (SNR) [25].

(5)$$\Delta\Phi_\tau \sim \sqrt{\frac{1}{SNR}} \sim \sqrt{\frac{1}{\frac{I_{Signal}}{\sqrt{2}}}}$$

The relation between the phase and the fluorescence lifetime is given by Eq. (2). We assume that $tan(\Phi _\tau ) \approx \Phi _\tau$ for small $\Phi _\tau$. The shot noise limited fluorescence lifetime standard deviation is then given by:

(6)$$\Delta\tau \approx \frac{\Delta\Phi_\tau}{2\pi f_{mod}} \sim \sqrt{\frac{1}{\frac{I_{Signal}}{\sqrt{2}}}} \cdot \frac{1}{2\pi f_{mod}}$$

This gives rise to an asymptotic behavior of the fluorescence lifetime standard deviation respective to the fluorescence signal intensity.

Note that the time delay measured is composed of the fluorescence lifetime $\tau$ of endogenous tissue autofluorescence (around 2 ns [26], PpIX native: 16 - 17 ns [27]) as well as the intrinsic time of flight of the excitation and fluorescence light. Based on the speed of light, this results in a time increase of roughly 1 ns for every 30 cm optical path traveled in air and needs to be taken into account if the distance to the sample is changed. We will, however, not further consider this issue as we assume that the microscope has a fixed working distance and any static phase has been subtracted.

2.3 Normalized difference imaging

As described in sect. 2.2, reducing the number of frames allows for faster visualization of the lifetime maps. By only acquiring a single readout of the dual-tap sensor, the number of frames is minimized to N = 2. While the Nyquist criterion only requires two samples per period to reconstruct the frequency of the observed signal, the recovered phase will not be unique. Therefore, it is not trivial to obtain the lifetime map with N = 2. However, it is possible to generate lifetime weighted intensity images. Thus, time delays of the excited fluorescence can be contrasted without considering the absolute value of the fluorescence lifetime. Similar to the work of Ballew and Demas [28], the normalized difference $\eta$ of the signal integrated in tap A and tap B is calculated:

(7)$$\eta = \frac{I_A - I_B}{I_A + I_B} = \frac{I_{\textrm{tap}}(\alpha = \beta) - I_{\textrm{tap}}(\alpha = \beta + \pi)}{I_{\textrm{tap}}(\alpha = \beta) + I_{\textrm{tap}}(\alpha = \beta + \pi)}.$$

Here, $\beta$ is a user-selectable phase delay and $I_A$ and $I_B$ represent the frames obtained from tap A and B, respectively. By inserting Eq. (1) and (4) into the normalized difference (see Appendix A.) we receive a fluorescence map that is weighted by the sine of the phase delay $\Phi _\tau$:

(8)$$\eta \sim \frac{I_f}{I_f + 2I_s} \cdot \sin \left( \Phi_\tau + \beta \right).$$

The lifetime $\tau$ is related to $\Phi _\tau$ according to Eq. (2). Assume a fluorophore with a lifetime $\tau _{\textrm {fl}}$ is imaged with the normalized difference method. As the phase delay induced by $\tau _{\textrm {fl}}$ is weighted by a sine, lifetime contrast can be maximized or completely suppressed by setting $\beta$ accordingly. Maximum lifetime contrast is achieved when the range [0 ns, $\tau _{\textrm {fl}}$] is centered symmetrically around the point of inflection of the sine. In other words, the argument of the sine has to be zero for the lifetime ($\tau _{\textrm {fl}}$ / 2). Thus, $\beta$ has to be set according to

(9)$$\beta = - \arctan (\pi f_{\textrm{mod}} \cdot \tau_{fl}) - k\cdot \pi,\;k \in \mathbb{Z}.$$

The normalized difference $\eta$ then becomes negative for all $\tau < (\tau _{\textrm {fl}} / 2)$ and positive for $\tau\;>\;(\tau _{\textrm {fl}} / 2)$. As a result, the normalized difference for the lifetime range [0 ns, $\tau _{\textrm {fl}}$] is centered around 0, leading to maximum lifetime weighted contrast. Contrarily, the lifetime weighting is minimized when $\beta$ is additionally shifted by $\pi /2$, thus centering [0 ns, $\tau _{\textrm {fl}}$] around the maximum of the sine. In this case, the normalized difference has the same value for 0 ns and $\tau _{\textrm {fl}}$ and lifetime contrast is completely blurred. Note that the normalized difference will only be independent of $I_f$ for $I_f$ $\gg$ $I_s$. For fluorescence signals in the magnitude of $I_s$, lifetime weighted contrast will be additionally weighted by the intensity of the florescence signal.

3. Material and methods

3.1 Fluorescence lifetime imaging system

Imaging was performed with the setup described in our previous publication [21] using a dual-tap CMOS camera (pco.flim, PCO AG, Germany) [22,23]. In brief, a 405 nm continuous wave laser is modulated sinusoidally at 10 MHz and illuminates the sample with 50 $mW/cm^2$. This is well below the ANSI limit for the maximum permissible exposure of 200 $mW/cm^2$ on skin. The emitted fluorescence is then collected through a macro photographic lens and a bandpass filter (665/150 BrightLine HC, Semrock, USA) before being detected by the camera. The resulting images show a macroscopic square FOV of 11.0 x 11.0 $\textrm {mm}^2$ with 1008 by 1008 pixels resolution. Lifetime maps were acquired by using 16, 8 and 4 individual phase frames equally spaced over 360 degrees to reconstruct the sinusoidal fluorescence emission. Prior to any measurement, the system was calibrated with a solid reference target in which PpIX is embedded in acrylic glass (custom made by Starna Scientific Ltd, Ilford, UK). Thus, intensity dependencies of the phase angle and the delay introduced through the time of flight could be corrected. This referencing step was performed using 16 phases. The computation of fluorescence lifetime maps is performed in real-time as the data is collected. Normalized difference images were computed using only two phase frames. As the camera records two taps simultaneously, the two frames are obtained using only a single sensor readout. The phase $\beta$ was then adjusted to yield maximum lifetime contrast.

3.2 Brain tissue samples

To validate our method we performed ex vivo measurements on three fresh human glioma samples obtained from two patients undergoing brain tumor surgery as part of an ongoing study (ethics approval number EK419/2008 - Amendment 04/2018). The first patient suffered from a grade IV glioblastoma which is known to exhibit strong fluorescence [2]. The second patient was selected due to a suspected LGG which was later found to be a grade II oligodendroglioma. These LGG in most cases do not accumulate sufficient PpIX to be seen under a conventional fluorescence surgical microscope [6]. The study protocol followed the same principles as described in [21]. Patients were administered 5-ALA (20 mg/kg body weight) 3 hours prior to the surgery. The surgeon safely removed the tissue and the samples were transferred in artificial cerebrospinal fluid (Landesapotheke Salzburg, 19C11S02) to the imaging lab within one hour after resection. The isotonic nature of this fluid (sodium chloride, potassium chloride, calcium chloride, magnesium chloride, glucose) mimics the physiological environment of the brain and thereby assures that the tissue is kept alive. After imaging the samples were handed over to the neuropathology department which determined the tumor type.

3.3 Simulated surgery

In order to evaluate the feasibility of real-time fluorescence lifetime guided surgery, we performed a simulated surgery on a piece of parboiled pork sausage. PpIX dimethyl ester (CAS: 5522-66-7, Sigma-Aldrich, St. Louis, Missouri, USA) was dissolved in dimethyl sulfoxide (DMSO) to a solution of 1 µg$/$ml and injected under the surface of the sample. An incision under real-time fluorescence lifetime guidance was then performed to expose the tissue where the injection was placed.

4. Results

4.1 Imaging rate limitations for PpIX fluorescence lifetime imaging

PpIX fluorescence lifetime imaging of 5-ALA labeled glioma samples has shown the potential to differentiate tumorous from non-tumorous tissue. [21,26]. To explore whether real-time FLIM is feasible for such an application, we determined the maximal visualization rates that could be achieved with our setup. We therefore analyzed the imaging rates as a function of exposure time for reducing the sampling density from 16 to 8, 4, and 2 phase frames. Furthermore, we investigated how the accuracy of fluorescence lifetime estimations depends on the sampling density and the fluorescence intensity emitted by a sample, as the latter limits the exposure time. To do so, we inspected a solution of 1 µg$/$ml PpIX dissolved in DMSO and acquired images for exposure times from 1 ms to 110 ms while sampling with 4, 8 and 16 phase frames. This concentration is comparable to values found in HGG [6]. The series of exposure times covered a range of relative fluorescence intensity from 0.016 to 0.491 where 0 and 1 correspond to the lowest and highest pixel value possible for the 14 bit range of the cameras A/D converter, respectively. However, the intra-scene dynamic range of the camera is 10 bit. Other cameras might differ in terms of dynamic range, dark noise and readout noise. The system characterization concerning lifetime standard deviation, relative fluorescence intensity, exposure time, and imaging rate is therefore only valid for our setup. Standard deviations were obtained over an area centered on the cuvette containing the PpIX solution and viewed as a function of the mean relative intensity of the emitted fluorescence signal. Note that a direct link between exposure time and relative fluorescence intensity would only be valid for samples emitting an equal radiant flux. To account for samples with differing radiant flux, the lifetime standard deviation is plotted as a function of the mean relative fluorescence intensity. Hence, the relation between the fluorescence lifetime standard deviation and the relative fluorescence intensity is to a good approximation independent of the sample concentration. While we measured a concentration of 1 µg$/$ml, the findings can be generalized for samples with differing PpIX concentrations when considering the relative fluorescence intensity only.

On average, decreasing the number of sensor readouts increased the imaging rate by a factor of $(1.96\pm 0.10)$, $(3.92\pm 0.26)$ and $(7.87\pm 0.39)$, respectively for 8 phases, 4 phases and the normalized difference method, all with regard to the acquisition of 16 phase frames (see Fig. 2(a)).

Fig. 2. (a) Maximum imaging rates of our setup in dependence of the exposure time for the normalized difference method and the acquisition of 4, 8 and 16 phase frames. (b) Standard deviation of the fluorescence lifetime in dependence of the relative fluorescence intensity for sampling with 4, 8 and 16 phase frames. Imaging was performed on a cuvette containing 1 $\mu$g$/$ml PpIX solution in DMSO. The shot noise limited fluorescence lifetime standard deviation was simulated according to Eq. (6).

Download Full Size | PDF

As to the question of the accuracy of fluorescence lifetime estimations in dependence of fluorescence intensity, we could identify two regimes (see Fig. 2(b)): First, a nonlinear regime for relative fluorescence intensities $<0.1$ with a strong increase of the fluorescence lifetime standard deviation towards lower intensities and, second, a rather linear regime for fluorescence intensities $>\;0.1$, with a slight decrease of the standard deviation towards higher intensities. While this applied for all three sampling modes, standard deviation increased when reducing sampling density. In detail, standard deviations for 4 and 8 phase frames were $(1.68\pm 0.10)$ and $(1.28\pm 0.04)$ times higher than for 16 phase frames, where all values with a relative intensity $<0.1$ were considered. For the linear regime $>\;0.1$, the influence of reducing sampling density decreased, with standard deviations of 4 and 8 phase frames being $(1.43\pm 0.06)$ and $(1.23\pm 0.01)$ times higher than standard deviation of 16 phase frames, respectively. The average fluorescence lifetime of the solution was measured to be $16.4\pm 1.0$ ns, considering an exposure time of 5 ms and when sampling with 16 phase frames.

Furthermore, we simulated the shot noise limited lifetime standard deviation as a function of the relative signal intensity according to the theoretical considerations of Eq. (6) (Fig. 2(b)). While the standard deviation for 16 phase frames at around 0.1 relative intensity was 0.64 ns, the simulated shot noise limited standard deviation was 0.24 ns. Sampling with more phase frames might further reduce the lifetime standard deviation but was limited to 16 phase frames in our setup. Furthermore, other sources of noise like electronic jitter might contribute to this difference. Nonetheless, the measured lifetime standard deviations corresponded to a good approximation to the theoretical considerations.

Note that we observed an increasingly nonlinear relation between the exposure time and the frame rate towards lower exposure times (Fig. 2(a)). This is due to the fact that the contribution of processing time in respect to the overall imaging time increases for lower exposure times. While 4 phase frames at 100 ms exposure time yield a frame rate of 4 fps, the exposure time for 8 phase frames would need to be reduced to 20 ms for achieving the same frame rate. As the lifetime standard deviation strongly increases for underexposed images (Fig. 2(b)), the exposure time has to be set to achieve a minimum of $10\;\%$ relative intensity compared to saturation to avoid under-exposed frames. The fastest fluorescence lifetime visualization for that specific exposure time is then achieved for sampling with 4 phase frames. For unknown samples this could be automated by tuning the exposure time until, for example, > $80\;\%$ of the area under the histogram are beyond 0.1 relative intensity.

In summary, we found that reducing the number of phase frames necessary for reconstruction of the fluorescence lifetime constitutes a compromise between higher visualization rates and the accuracy of lifetime estimations. We therefore wanted to investigate how this reduction of sensor readouts affected lifetime estimations in human tumorous tissue of different grades and the visualization rates that could be achieved.

4.2 Dark noise analysis

Dark noise was measured as the relative intensity averaged over all pixels for exposure times from 10 to 400 ms (Fig. 3). Images were acquired in the dark and the objective lens was covered by a lens cap. At an exposure time of 400 ms, the dark noise contribution stayed below $3.0\;\pm \;1.4\;\%$ relative intensity compared to sensor saturation. Thus, when imaging human tissue samples at a minimum of 10 % relative intensity dark noise is negligible and the SNR is shot noise limited. Note that as the exposure time increases, the variability of the dark noise over all pixels increases as well.

Fig. 3. Dark noise (mean relative intensity) as a function of the exposure time averaged over all pixels.

Download Full Size | PDF

4.3 PpIX fluorescence lifetime imaging of 5-ALA labeled high-grade glioma

Following the experiments that characterized the imaging rate limitations of our FLIM system, we examined how this translates to ex vivo imaging of a human HGG sample. Images were acquired using 4, 8 and 16 phase frames and exposure times from 5 ms to 50 ms. The average fluorescence lifetime and the corresponding standard deviation were evaluated in a region where strong PpIX fluorescence could be observed by the naked eye (ROI A) and a second region, where the visually perceived fluorescence was lower (ROI B). For an exposure time of 20 ms, the upper part of the sample emitted fluorescence with a relative intensity $>\;0.1$ (Fig. 4(a)). It is worth mentioning that the sample was not planar. Thus, the lower part of the sample (see ROI C) was slightly out of focus, resulting in relative fluorescence intensities $<0.1$ that were barely visible. Contrarily, fluorescence lifetime was still sensitive enough to contrast the lower part of the sample. Table 1 provides an overview of the fluorescence lifetimes averaged over ROI A and B. ROI A showed lifetimes in the range of 11-12 ns while ROI B entailed lifetimes of approximately 4 ns. Standard deviations stayed below 1.5 ns when imaging with 4 phase frames. This error, however, partly is induced by inhomogeneities of the sample within the ROI. When increasing the exposure time to values $>$20 ms we found a slight decrease of the standard deviation, while exposure times below 20 ms entailed a strong increase (Fig. 4(e-f)). To sum up, fluorescence lifetime visualization of the HGG was feasible at 20 ms exposure time. When sampling with 4 phase frames this corresponds to visualization rates of 12 Hz.

Fig. 4. (a) Relative fluorescence intensity of a HGG exhibiting strong PpIX fluorescence. (b) - (d) Fluorescence lifetime maps acquired with 16, 8 and 4 phase frames. Exposure time was set to 20 ms. Standard deviation of the lifetime increased towards lower sampling densities, but stayed $< 1.5$ ns. The lower part of the sample (ROI C) was slightly out of focus, leading to very low relative fluorescence intensities. Yet, fluorescence lifetime was still sensitive enough to contrast this part of the sample. (e) - (f) Reducing exposure time increased lifetime standard deviation for ROI A and B, respectively. For higher exposure times, a slight decrease of the standard deviation could be observed.

Download Full Size | PDF

Table 1. Overview of the fluorescence lifetimes $\tau$ for the HGG, LGG and non-pathological sample. Imaging was performed using 16, 8 and 4 phase frames. Lifetimes were averaged over the ROI A and B of the respective samples (see Fig. 4 to 6).

View Table | View all tables in this article

To compare those elevated PpIX fluorescence lifetime values to the lifetime of tissue autofluorescence found in non-pathological brain parenchyma, we imaged a sample resected on the access route to the tumor of the same patient (see Fig. 5). Exposure time was set to 200 ms to achieve a minimum of 0.1 relative intensity over the sample.

Fig. 5. (a) Relative intensity of the autofluorescence of a sample which was confirmed to be reactive brain parenchyma. (b) - (d) Fluorescence lifetime maps acquired with 16, 8 and 4 phase frames. Exposure time was set to 200 ms. Mean fluorescence lifetimes of the sample were in the range of 2 ns and below.

Download Full Size | PDF

The average fluorescence lifetime and the corresponding standard deviation were evaluated in a representative region of the sample (ROI A) and a region with slightly increased autofluorescence (ROI B). Table 1 provides an overview of the fluorescence lifetimes averaged over ROI A and B. Lifetimes were found to be in the range of 2 ns and smaller, corresponding well to reported values for the autofluorescence of brain parenchyma [26]. Histopathology confirmed the sample to be reactive brain parenchyma with minor single tumor cell infiltrations.

To sum up, the HGG showed regions with lifetimes that were about 10 ns higher than the lifetimes found in the reactive parenchyma. This differences also were seen when sampling with 4 phases and could thus be visualized with 12 Hz. Next we examined whether elevated lifetimes could also be observed in a LGG.

4.4 PpIX fluorescence lifetime imaging of 5-ALA labeled low-grade glioma

We then investigated if our system was sufficiently sensitive to lower concentrations of PpIX, which fluorescence is not visible to the surgeon in the operating theatre, and how fast we could visualize these dye accumulations while maintaining the necessary sensitivity. For this purpose, a third sample from a patient with a LGG was imaged (see Fig. 6). Exposure time had to be set to 100 ms to achieve a minimum of 0.1 relative intensity over the sample. Table 1 provides an overview of the fluorescence lifetimes averaged over ROI A and B. While the average lifetime in ROI B was slightly above the lifetime in the non-pathological brain parenchyma sample, ROI A entailed a clearly increased average lifetime around 4 ns. Regions of elevated PpIX lifetime corresponded to histopathological findings of diffusely infiltrating LGG tissue in the sample. It is worth mentioning that for both ROIs the average lifetime decreased when reducing the number of phase frames. This most likely was due to bleaching of PpIX, as imaging of 16, 8 and 4 phase frames was performed subsequently in the same order. We therefore measured a cuvette containing 1 $\mu$g$/$ml PpIX and inverted the order of phase frames when acquiring images (first 4, then 8, then 16). Fluorescence lifetime was $16.5\;\pm \;1.4$ ns, $15.9\;\pm \;1.0$ ns and $15.9\;\pm \;0.9$ ns, for 4, 8 and 16 phase frames, respectively. Again, bleaching reduced the average lifetime by 0.6 ns. To sum up, we found that fluorescence lifetime imaging of 5-ALA labeled gliomas was sensitive enough to delineate small accumulations of PpIX, invisible to the naked eye. Regions of elevated lifetime were visualized with 4 Hz by acquiring 4 phase frames.

Fig. 6. (a) Relative fluorescence intensity of a LGG sample exhibiting weak PpIX fluorescence. (b) - (d) Fluorescence lifetime maps acquired with 16, 8 and 4 phase frames. Exposure time was set to 100 ms. Areas with increased fluorescence lifetime were found, where no fluorescence could be observed visually.

Download Full Size | PDF

4.5 Normalized difference imaging of 5-ALA labeled glioma samples

To investigate whether the normalized difference method can provide lifetime weighted contrast, the three samples were imaged acquiring one sensor readout only. Acquisition was performed with exposure times of 50 ms for the HGG, 100 ms for the LGG, and 200 ms for the non-pathological sample. This translates to imaging rates of 14 Hz, 8 Hz, and 4.5 Hz, respectively.

Table 2 provides an overview of the normalized difference averaged over ROI A and B of the respective samples. At the optimal working point (see Fig. 7(a) and (c)), the normalized difference clearly highlights the same regions as observed in the corresponding absolute lifetime maps shown in the prior subsections. The range of normalized difference values was centered around zero, which is in accordance with the theoretical considerations described in sect. 2.3. While the normalized difference was around $\pm$ 0.20 for the HGG, lifetime differences in the LGG were more difficult to contrast with values around $\pm$ 0.05. The introduced relative phase shift was 210$^{\circ }$ for the HGG and 190$^{\circ }$ for the LGG.

Fig. 7. Normalized difference imaging of the HGG, LGG, and the non-pathological sample. The samples correspond to Fig. 4, Fig. 6, and Fig. 5 respectively. (a,c,e) Optimal working point for normalized difference imaging. Lifetime contrast is maximized. (b,d,f) 90$^{\circ }$ shifted least favorable working point. Lifetime contrast is blurred.

Download Full Size | PDF

Table 2. Overview of the normalized difference for the HGG, LGG and the non-pathological sample at the optimal and the least favorable working point (WP). Imaging was performed using one sensor readout. The normalized difference was averaged over ROI A and B of the respective samples (see Fig. 7).

View Table | View all tables in this article

On the contrary, at the 90$^{\circ }$ shifted least favorable working point (see Fig. 7(b) and (d)), areas with lower and higher lifetimes could not be contrasted by the normalized difference. The non-pathological sample was imaged with the same relative phase shift as the LGG sample with an exposure time of 200 ms (Fig. 7(e,f)). Clearly, the normalized difference method is limited when contrasting lower fluorescence lifetime differences. Also, defining a threshold between non-pathological and glioma tissue can be challenging, as no absolute lifetime values are obtained. Nevertheless, when correctly setting the relative phase shift, the normalized difference method could provide lifetime weighted contrast, both for the HGG and the LGG, reconstructed out of a single sensor readout.

4.6 Illustration of lifetime visualization with 12 Hz

As we found that FLIM imaging has the potential for fast and sensitive fluorescence lifetime visualization, the logical next step was to illustrate the perception of 12 Hz during a simulated surgical intervention. The exposure time was set to 20 ms and the fluorescence lifetime was sampled with 4 phase frames. As human glioma samples were spared for histopathological evaluation, the procedure was performed on a piece of parboiled pork sausage. A PpIX solution diluted to 1 µg$/$ml was injected under the surface of the sample and was not visible in the beginning of the video. Incision of the surface revealed elevated lifetimes of 8 ns at the injection site (see. Fig. 8). In contrast, it is hard to see any differences between PpIX and the tissue of the meat sample in the fluorescence intensity image on the left hand side. Note that the relative intensity of the parboiled pork sausage autofluorescence in some spots is higher than the relative PpIX fluorescence intensity. As some spots already are visible before incision of the surface, these spots most likely are due to autofluorescence. Yet, fluorescence lifetime imaging is sensitive enough to keep lifetime contrast. This supports the findings of the HGG sample (Fig. 4, ROI C), where the fluorescence lifetime managed to keep contrast even for low fluorescence signals.

Fig. 8. Snapshot taken from a video simulating cytoreductive surgery on a piece of parboiled pork sausage. While the elevated lifetime clearly delineates PpIX on the right-hand side, the intensity image on the left-hand side doesn’t show any contrast between PpIX and surrounding tissue. Imaging was performed at 12 Hz (see Visualization 1).

Download Full Size | PDF

5. Discussion

Although PpIX guided glioma surgery has shown to improve patient outcome [2], visual assessment of the fluorescence emission by the surgeon does not exploit the potential for the visualization of lower PpIX concentrations and more advanced methods lack the necessary real-time capability [8,29,30]. Here, we show for the first time that fluorescence lifetime of a 5-ALA labeled HGG could be visualized with 12 Hz using the full active area of the camera (1008 x 1008 pixels / 11.0 mm FOV) by reducing the number of acquired phase frames. Also, we found that fluorescence lifetime imaging is sensitive enough to visualize smaller accumulations of PpIX in LGGs, as regions of the investigated tissue sample showed elevated lifetimes. Here, visualization rates of 4 Hz were feasible. In our previous work, a single acquisition took up to 3.2 seconds [21]. Acquisition times of 20 seconds [31], 10 to 90 seconds [19], 2 minutes [32] and 29 minutes [33] are reported for macroscopic FLIM systems based on time-domain measurements. Finally, we could show that a single sensor readout can provide lifetime weighted contrast comparable to images acquired with higher sampling densities. Our findings suggest that multiple-tap CMOS cameras are potentially fast enough for real-time fluorescence lifetime imaging, which is a prerequisite for clinical applicability.

Fluorescence lifetime of native PpIX is known to be approximately 16 ns [27]. As a result, malignant 5-ALA labeled glioma tissue can be delinated from non-pathological parenchyma by elevated PpIX lifetimes [21,26]. While conventional fluorescence intensity imaging in our measurements on a HGG was not able to show low fluorescent areas, PpIX lifetime clearly displayed the borders of the sample. Furthermore, regions with strongly increased fluorescence lifetime were found compared to the lifetime measured in reactive brain parenchyma. Elevated fluorescence lifetimes were also found in a LGG, which is promising as most LGG cannot be visualized with the current state of the art in cytoreductive neurosurgery [34]. Lifetime values for the reactive brain parenchyma sample were similar to values reported for non-pathological brain parenchyma [26]. Note that the HGG sample (Fig. 4) was quite heterogenous, with higher lifetimes in ROI A than in ROI B. We hypothesize that the PpIX lifetime measured in tissue is not entirely independent of the PpIX concentration, as the influence of tissue autofluorescence becomes more dominant for low PpIX concentrations. For this HGG sample, the lower lifetimes in ROI B and the gradient towards ROI A could be indicative for an infiltration zone. The fact that ROI C showed higher lifetimes than ROI B, while entailing lower relative fluorescence intensities, might seem confusing at first sight. However, the HGG sample was not planar and ROI C was slightly out of focus, leading to very low relative fluorescence intensities (Fig. 4(a)). Nevertheless, we opted to keep the exposure time at 20 ms, as this sample nicely illustrated that the fluorescence lifetime could keep the contrast of the sample border (ROI C) even for low relative fluorescence intensities. Hence, fluorescence lifetime is less sensitive to intensity variations due to geometrical factors. Caution however has to be taken when it comes to intensity variations due to blood absorption. A selective spectral absorption of the signal contributions of PpIX and tissue autofluorescence might impact the measured fluorescence lifetime. This could be mitigated by the use of a narrower collection band, at the expense of a lower signal. To gain more information about the spectral components of the fluorescence observed, lifetime measurements may be combined with spectroscopic measurements, which was however outside the scope of this study. Furthermore, we could have obtained even more robust results if the referencing would have been performed individually for 4, 8 and 16 phases and not only for 16 phases as shown in this study. While performing all referencing measurements sequentially might have solved this issue, the reference target would have been exposed for an increased period of time. The increased exposure, however, leads to rapid bleaching. Under the current settings we observed a bleaching induced lifetime change of 2 ns over 100 measurement days. Therefore, we keep the exposure time of our target to the lowest possible and regularly compare it to a second reference standard for calibration. Note that we also observed a bleaching induced lifetime reduction of around 1 ns in our human tissue samples. This further limits the detection of low PpIX accumulations as found in LGG. Hence, it may be worth to reduce the laser power when more sensitive cameras come to the market.

Albeit first results are promising, detailed studies including different histopathological subtypes and an increased sample batch are needed for a better understanding of the benefits of PpIX lifetime imaging for surgical resection. Nevertheless, in this study we rather wanted to emphasize the potential real-time capability of FLIM imaging, which would facilitate integration into the surgical workflow. In the context of the perceptual and psychomotor tasks associated with the resection of tumors, lifetime visualization of HGGs with 12 Hz should be fast enough to prevent unwanted impact on surgical performance through imaging latency [35]. However, real-time visualization of smaller PpIX accumulations is not yet possible. In the case of the LGG imaged in this study, visualization rate was limited to near real-time at 4 Hz. Nonetheless, the acquisition of a single frame within fractions of a second could provide valuable information. Surgeons could check for remaining PpIX in resection cavities or look for smaller PpIX accumulations in infiltration zones or LGG.

A further speed up could be achieved by the normalized difference method, which calculates a lifetime weighted intensity image while only relying on a single sensor readout. However, as no absolute lifetime values are obtained, the task on where to set the border between non-pathological and tumorous tissue would remain challenging. One possibility to overcome this drawback would be to repetitively calibrate the normalized difference method with single FLIM acquisitions. In practice, two sensor readouts instead of one could be acquired in a specified time period. Another difficulty is that the optimal working point for maximal lifetime contrast depends on the respective lifetime delay of the sample. In this study, the lifetime delay of the HGG was about 7 to 8 ns higher than for the LGG. Thus, the optimal phase shift $\beta$ for maximum lifetime contrast was reduced from 210$^{\circ }$ to 190$^{\circ }$ for the HGG and LGG, respectively. Finally, the normalized difference approach does depend on fluorescence intensity and low fluorescence intensities slightly above noise floor are more difficult to contrast. Also, optimizing the phase shift for tissue with different lifetimes in real-time during surgery could be challenging. With respect to the sensitivity for the detection of lower concentrations of PpIX, FLIM imaging certainly should be preferred over the normalized difference. Yet, applications where only a single lifetime is measured and no absolute values are needed could profit from the normalized difference approach.

Our findings have to be seen under the limitation that imaging was performed in the laboratory under low-light conditions. In principle, the homodyne detection scheme should reject any DC part of ambient light when measuring the phase and thereby the fluorescence lifetime. Nonetheless, the relatively bright light in the laboratory would have occasionally saturated the sensor and thereby increased the noise respective to the actual measurement signal. Fluorescence guided resections, however, often are performed under similar low-light conditions, which might be a valid surrounding for phase based lifetime measurements. The effect of the ambient light present in a surgical suite on the lifetime measurements however still needs to be investigated.

Currently, the limiting factor for imaging speed is the detection sensitivity of our system. The power of the modulated excitation is limited by laser safety regulations and the working distance of 250 mm restricts the maximal object-space numerical aperture (NA). In this work, the NA was 0.07, based on the large aperture of the photo lens. However, one needs to consider that commercial surgical microscopes have an even lower NA in the range of 0.02 due to the stereoscopic optical design. Therefore, more sensitive cameras are needed. Novel and fast time-gated multi-tap sensors [36] would allow for shorter exposure times by capturing more than 2 phase frames at each sensor readout. Yet, each additional tap must share the number of photons available. More sensitive, fast gated cameras [37] could be used for better exploitation of the photon budget. Furthermore, time-of-flight imaging is a flourishing field, currently boosted by technological trends like autonomous driving [38]. Thus, technological advances from other disciplines may further benefit the development towards real-time wide-field fluorescence lifetime imaging.

6. Conclusion

We investigated the real-time capability of multi-tap CMOS camera based macroscopic FLIM imaging for future surgical guidance in glioma surgery. Real-time visualization with 12 Hz was feasible for HGG by reducing the number of sensor readouts acquired for calculating lifetime maps. Tissue exhibiting lower levels of fluorescence was limited to imaging rates of 4 Hz. Therefore, the acquisition of two sensor readouts or 4 phase frames seems to be the most promising method to achieve the best compromise between visualization speed and accuracy of fluorescence lifetime estimations. Fluorescence lifetime maps on macroscale could provide valuable information to the surgeon for the delineation of tumorous tissue in glioma surgery. Current developments in multi-tap sensing are promising and might eventually allow for real-time visualization with increased sensitivity, especially relevant for LGG.

A. Derivation of equation (8)

The normalized difference is given by

(10)$$ \eta = \frac{I_A - I_B}{I_A + I_B} $$

The integrals for $I_{A}$ and $I_{B}$ are:

I_{\textrm{A}} = \int_{0}^{T/2} I_f \cdot \left(\frac{1}{2} + \frac{1}{2}\cos{\left(\frac{2\pi}{T} \cdot t + \Phi_\tau + \beta \right)}\right) + I_s\;dt

and

I_{\textrm{B}} = \int_{T/2}^{T} I_f \cdot \left(\frac{1}{2} + \frac{1}{2}\cos{\left( \frac{2\pi}{T} \cdot t + \Phi_\tau + \beta \right)}\right) + I_s\;dt

Solving the integrals yields:

(11)$$ I_{\textrm{A}} = \frac{T}{4} \Bigl(I_f + 2I_s\Bigr) - \frac{T\cdot I_f}{2\pi} \cdot \sin (\Phi_\tau + \beta) $$

and

(12)$$ I_{\textrm{B}} = \frac{T}{4} \Bigl(I_f + 2I_s\Bigr) + \frac{T\cdot I_f}{2\pi} \cdot \sin (\Phi_\tau + \beta) \label{eq:etaC} $$

We insert $I_{A}$ (Eq. (11)) and $I_{B}$ (Eq. (12)) in Eq. (10):

\eta = -\frac{2}{\pi} \cdot \frac{I_f}{ I_f + 2I_s} \cdot \sin \left( \Phi_\tau + \beta \right) \;\sim\; \frac{I_f}{ I_f + 2I_s} \cdot \sin \left( \Phi_\tau + \beta \right)

Funding

Carl Zeiss Meditec AG; Österreichisches Bundesministerium für Digitalisierung und Wirtschaftsstandort; Österreichische Nationalstiftung für Forschung, Technologie und Entwicklung; H2020 Marie Skłodowska-Curie Actions (721766); Österreichischen Akademie der Wissenschaften (25262); European Research Council (ERC StG 640396 OPTIMALZ).

Acknowledgments

This project has received funding from the European Union Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement 721766 as well as from the innovation board of the Carl Zeiss Meditec AG. T.R. is recipient of a DOC Fellowship of the Austrian Academy of Sciences at the Institute of Neurology (25262). The financial support by the Austrian Federal Ministry for Digital and Economic Affairs and the National Foundation for Research, Technology and Development is gratefully acknowledged.

Disclosures

The authors declare no potential conflict of interests.

References

1. S. Walter, S. Susanne, W. Simon, S. Herbert, F. Clemens, G. Claudia, E. G. Alwin, K. Rainer, and J. R. Hans, “Intraoperative detection of malignant gliomas by 5-aminolevulinic acid-induced porphyrin fluorescence,” Neurosurgery 42(3), 518–526 (1998). [CrossRef]

2. W. Stummer, U. Pichlmeier, T. Meinel, O. D. Wiestler, F. Zanella, H.-J. Reulen, and A.-G. S. Group, “Fluorescence-guided surgery with 5-aminolevulinic acid for resection of malignant glioma: a randomised controlled multicentre phase iii trial,” Lancet Oncol. 7(5), 392–401 (2006). [CrossRef]

3. G. Widhalm, S. Wolfsberger, G. Minchev, A. Woehrer, M. Krssak, T. Czech, D. Prayer, S. Asenbaum, J. A. Hainfellner, and E. Knosp, “5-aminolevulinic acid is a promising marker for detection of anaplastic foci in diffusely infiltrating gliomas with nonsignificant contrast enhancement,” Cancer 116(6), 1545–1552 (2010). [CrossRef]

4. C. G. Hadjipanayis, G. Widhalm, and W. Stummer, “What is the surgical benefit of utilizing 5-aminolevulinic acid for fluorescence-guided surgery of malignant gliomas?” Neurosurgery 77(5), 663–673 (2015). [CrossRef]

5. N. Sanai, L. A. Snyder, N. J. Honea, S. W. Coons, J. M. Eschbacher, K. A. Smith, and R. F. Spetzler, “Intraoperative confocal microscopy in the visualization of 5-aminolevulinic acid fluorescence in low-grade gliomas,” J. Neurosurg. 115(4), 740–748 (2011). [CrossRef]

6. A. Johansson, G. Palte, O. Schnell, J.-C. Tonn, J. Herms, and H. Stepp, “5-aminolevulinic acid-induced protoporphyrin ix levels in tissue of human malignant brain tumors,” Photochem. Photobiol. 86(6), 1373–1378 (2010). [CrossRef]

7. P. A. Valdés, F. Leblond, V. L. Jacobs, B. C. Wilson, K. D. Paulsen, and D. W. Roberts, “Quantitative, spectrally-resolved intraoperative fluorescence imaging,” Sci. Rep. 2(1), 798 (2012). [CrossRef]

8. P. A. Valdés, V. Jacobs, B. T. Harris, B. C. Wilson, F. Leblond, K. D. Paulsen, and D. W. Roberts, “Quantitative fluorescence using 5-aminolevulinic acid-induced protoporphyrin ix biomarker as a surgical adjunct in low-grade glioma surgery,” J. Neurosurg. 123(3), 771–780 (2015). [CrossRef]

9. G. Widhalm, J. Olson, J. Weller, J. Bravo, S. J. Han, J. Phillips, S. L. Hervey-Jumper, S. M. Chang, D. W. Roberts, and M. S. Berger, “The value of visible 5-ala fluorescence and quantitative protoporphyrin ix analysis for improved surgery of suspected low-grade gliomas,” J. Neurosurg. 1, 1–10 (2019). [CrossRef]

10. J. Bravo, J. Olson, S. Davis, D. Roberts, K. Paulsen, and S. Kanick, “Hyperspectral data processing improves ppix contrast during fluorescence guided surgery of human brain tumors,” Sci. Rep. 7(1), 9455 (2017). [CrossRef]

11. P. A. Valdes, V. L. Jacobs, B. C. Wilson, F. Leblond, D. W. Roberts, and K. D. Paulsen, “System and methods for wide-field quantitative fluorescence imaging during neurosurgery,” Opt. Lett. 38(15), 2786–2788 (2013). [CrossRef]

12. B. Montcel, L. Mahieu-Williame, X. Armoiry, D. Meyronet, and J. Guyotat, “Two-peaked 5-ala-induced ppix fluorescence emission spectrum distinguishes glioblastomas from low grade gliomas and infiltrative component of glioblastomas,” Biomed. Opt. Express 4(4), 548–558 (2013). [CrossRef]

13. M. Marois, J. J. Bravo, S. C. Davis, and S. C. Kanick, “Characterization and standardization of tissue-simulating protoporphyrin ix optical phantoms,” J. Biomed. Opt. 21(3), 035003 (2016). [CrossRef]

14. L. Alston, D. Rousseau, M. Hebert, L. Mahieu-Williame, and B. Montcel, “Nonlinear relation between concentration and fluorescence emission of protoporphyrin ix in calibrated phantoms,” J. Biomed. Opt. 23(09), 1 (2018). [CrossRef]

15. N. A. Markwardt, N. Haj-Hosseini, B. Hollnburger, H. Stepp, P. Zelenkov, and A. Rühm, “405 nm versus 633 nm for protoporphyrin ix excitation in fluorescence-guided stereotactic biopsy of brain tumors,” J. Biophotonics 9(9), 901–912 (2016). [CrossRef]

16. L. Marcu and B. A. Hartl, “Fluorescence lifetime spectroscopy and imaging in neurosurgery,” IEEE J. Sel. Top. Quantum Electron. 18(4), 1465–1477 (2012). [CrossRef]

17. Y. H. Sun, N. Hatami, M. Yee, J. E. Phipps, D. S. Elson, F. Gorin, R. J. Schrot, and L. Marcu, “Fluorescence lifetime imaging microscopy for brain tumor image-guided surgery,” J. Biomed. Opt. 15(5), 056022 (2010). [CrossRef]

18. S. R. Kantelhardt, D. Kalasauskas, K. König, E. Kim, M. Weinigel, A. Uchugonova, and A. Giese, “In vivo multiphoton tomography and fluorescence lifetime imaging of human brain tumor tissue,” J. Neuro-Oncol. 127(3), 473–482 (2016). [CrossRef]

19. V. I. Shcheslavskiy, M. V. Shirmanova, V. V. Dudenkova, K. A. Lukyanov, A. I. Gavrina, A. V. Shumilova, E. Zagaynova, and W. Becker, “Fluorescence time-resolved macroimaging,” Opt. Lett. 43(13), 3152–3155 (2018). [CrossRef]

20. W. Becker, “Fluorescence lifetime imaging–techniques and applications,” J. Microsc. 247(2), 119–136 (2012). [CrossRef]

21. M. T. Erkkilä, B. Bauer, N. Hecker-Denschlag, M. J. Madera Medina, R. A. Leitgeb, A. Unterhuber, J. Gesperger, T. Roetzer, C. Hauger, W. Drexler, G. Widhalm, and M. Andreana, “Widefield fluorescence lifetime imaging of protoporphyrin ix for fluorescence-guided neurosurgery: An ex vivo feasibility study,” J. Biophotonics 12(6), e201800378 (2019). [CrossRef]

22. R. Franke and G. A. Holst, “Frequency-domain fluorescence lifetime imaging system (pco. flim) based on a in-pixel dual tap control cmos image sensor,” in Imaging, Manipulation, and Analysis of Biomolecules, Cells, and Tissues XIII, vol. 9328 (International Society for Optics and Photonics, 2015), p. 93281K.

23. H. Chen, G. Holst, and E. Gratton, “Modulated cmos camera for fluorescence lifetime microscopy,” Microsc. Res. Tech. 78(12), 1075–1081 (2015). [CrossRef]

24. J. R. Lakowicz, Principles of Fluorescence Spectroscopy (Springer Science & Business Media, 2013).

25. J. W. Goodman, Statistical Optics (John Wiley & Sons, 2015).

26. S. R. Kantelhardt, H. Diddens, J. Leppert, V. Rohde, G. Hüttmann, and A. Giese, “Multiphoton excitation fluorescence microscopy of 5-aminolevulinic acid induced fluorescence in experimental gliomas,” Lasers Surg. Med. 40(4), 273–281 (2008). [CrossRef]

27. J. A. Russell, K. R. Diamond, T. J. Collins, H. F. Tiedje, J. E. Hayward, T. J. Farrell, M. S. Patterson, and Q. Fang, “Characterization of fluorescence lifetime of photofrin and delta-aminolevulinic acid induced protoporphyrin ix in living cells using single-and two-photon excitation,” IEEE J. Sel. Top. Quantum Electron. 14(1), 158–166 (2008). [CrossRef]

28. R. M. Ballew and J. Demas, “An error analysis of the rapid lifetime determination method for the evaluation of single exponential decays,” Anal. Chem. 61(1), 30–33 (1989). [CrossRef]

29. A. Kim, M. Khurana, Y. Moriyama, and B. C. Wilson, “Quantification of in vivo fluorescence decoupled from the effects of tissue optical properties using fiber-optic spectroscopy measurements,” J. Biomed. Opt. 15(6), 067006 (2010). [CrossRef]

30. M. Jermyn, K. Mok, J. Mercier, J. Desroches, J. Pichette, K. Saint-Arnaud, L. Bernstein, M.-C. Guiot, K. Petrecca, and F. Leblond, “Intraoperative brain cancer detection with raman spectroscopy in humans,” Sci. Transl. Med. 7(274), 274ra19 (2015). [CrossRef]

31. J. McGinty, N. P. Galletly, C. Dunsby, I. Munro, D. S. Elson, J. Requejo-Isidro, P. Cohen, R. Ahmad, A. Forsyth, A. V. Thillainayagam, M. A. Neil, P. M. French, and G. W. Stamp, “Wide-field fluorescence lifetime imaging of cancer,” Biomed. Opt. Express 1(2), 627–640 (2010). [CrossRef]

32. M. V. Shirmanova, M. Lukina, E. B. Kisileva, V. V. Fedoseeva, V. V. Dudenkova, E. V. Zagaynova, W. Becker, and V. I. Shcheslavskiy, “Interrogation of glioma metabolism on macroscale by flim,” in Multiphoton Microscopy in the Biomedical Sciences XIX, vol. 10882 (International Society for Optics and Photonics, 2019), p. 1088209.

33. B. E. Sherlock, J. E. Phipps, J. Bec, and L. Marcu, “Simultaneous, label-free, multispectral fluorescence lifetime imaging and optical coherence tomography using a double-clad fiber,” Opt. Lett. 42(19), 3753–3756 (2017). [CrossRef]

34. B. K. Hendricks, N. Sanai, and W. Stummer, “Fluorescence-guided surgery with aminolevulinic acid for low-grade gliomas,” J. Neuro-Oncol. 141(1), 13–18 (2019). [CrossRef]

35. J. Y. Chen and J. E. Thropp, “Review of low frame rate effects on human performance,” IEEE Trans. Syst., Man, Cybern. A 37(6), 1063–1076 (2007). [CrossRef]

36. M.-W. Seo, Y. Shirakawa, Y. Kawata, K. Kagawa, K. Yasutomi, and S. Kawahito, “A time-resolved four-tap lock-in pixel cmos image sensor for real-time fluorescence lifetime imaging microscopy,” IEEE J. Solid-State Circuits 53(8), 2319–2330 (2018). [CrossRef]

37. H. Ingelberts, T. Lapauw, P. Debie, S. Hernot, and M. Kuijk, “A proof-of-concept fluorescence lifetime camera based on a novel gated image sensor for fluorescence-guided surgery,” in Molecular-Guided Surgery: Molecules, Devices, and Applications V, vol. 10862 (International Society for Optics and Photonics, 2019), p. 108620C.

38. F. Rosique, P. J. Navarro, C. Fernández, and A. Padilla, “A systematic review of perception system and simulators for autonomous vehicles research,” Sensors 19(3), 648 (2019). [CrossRef]

Sample	ROI	16 phase frames	8 phase frames	4 phase frames
Sample	ROI	$τ$ [ns]	$τ$ [ns]	$τ$ [ns]
Glioblastoma (Fig. 4)	A	$11.92 \pm 0.73$	$11.67 \pm 0.80$	$11.11 \pm 1.48$
Grade IV - HGG	B	$4.11 \pm 0.44$	$3.96 \pm 0.52$	$3.81 \pm 0.91$
Oligodendroglioma (Fig. 6)	A	$4.75 \pm 0.43$	$3.99 \pm 0.71$	$3.80 \pm 0.80$
Grade II - LGG	B	$2.80 \pm 0.27$	$2.27 \pm 0.33$	$1.85 \pm 0.51$
Brain parenchyma (Fig. 5)	A	$1.58 \pm 0.41$	$1.47 \pm 0.56$	$1.32 \pm 0.48$
non-pathological	B	$2.07 \pm 0.19$	$1.91 \pm 0.22$	$1.50 \pm 0.39$

Sample	ROI	Optimal WP	Least favorable WP
Sample	ROI	Norm. diff. [rel.]	Norm. diff. [rel.]
Glioblastoma (Fig. 7(a,b))	A	$0.18 \pm 0.02$	$0.64 \pm 0.02$
Grade IV - HGG	B	$- 0.20 \pm 0.04$	$0.77 \pm 0.02$
Oligodendroglioma (Fig. 7(c,d))	A	$0.04 \pm 0.05$	$0.65 \pm 0.08$
Grade II - LGG	B	$- 0.05 \pm 0.04$	$0.74 \pm 0.05$
Brain parenchyma (Fig. 7(e,f))	A	$0.00 \pm 0.05$	$0.81 \pm 0.06$
Non-pathological	B	$0.04 \pm 0.04$	$0.85 \pm 0.03$

Sample	ROI	16 phase frames	8 phase frames	4 phase frames
Sample	ROI	$τ$ [ns]	$τ$ [ns]	$τ$ [ns]
Glioblastoma (Fig. 4)	A	$11.92 \pm 0.73$	$11.67 \pm 0.80$	$11.11 \pm 1.48$
Grade IV - HGG	B	$4.11 \pm 0.44$	$3.96 \pm 0.52$	$3.81 \pm 0.91$
Oligodendroglioma (Fig. 6)	A	$4.75 \pm 0.43$	$3.99 \pm 0.71$	$3.80 \pm 0.80$
Grade II - LGG	B	$2.80 \pm 0.27$	$2.27 \pm 0.33$	$1.85 \pm 0.51$
Brain parenchyma (Fig. 5)	A	$1.58 \pm 0.41$	$1.47 \pm 0.56$	$1.32 \pm 0.48$
non-pathological	B	$2.07 \pm 0.19$	$1.91 \pm 0.22$	$1.50 \pm 0.39$

Sample	ROI	Optimal WP	Least favorable WP
Sample	ROI	Norm. diff. [rel.]	Norm. diff. [rel.]
Glioblastoma (Fig. 7(a,b))	A	$0.18 \pm 0.02$	$0.64 \pm 0.02$
Grade IV - HGG	B	$- 0.20 \pm 0.04$	$0.77 \pm 0.02$
Oligodendroglioma (Fig. 7(c,d))	A	$0.04 \pm 0.05$	$0.65 \pm 0.08$
Grade II - LGG	B	$- 0.05 \pm 0.04$	$0.74 \pm 0.05$
Brain parenchyma (Fig. 7(e,f))	A	$0.00 \pm 0.05$	$0.81 \pm 0.06$
Non-pathological	B	$0.04 \pm 0.04$	$0.85 \pm 0.03$

Towards real-time wide-field fluorescence lifetime imaging of 5-ALA labeled brain tumors with multi-tap CMOS cameras

Abstract

Corrections

1. Introduction

2. Theory

2.1 Discrete frequency domain fluorescence lifetime imaging

2.2 Multi-tap detector based fluorescence lifetime reconstruction

2.3 Normalized difference imaging

3. Material and methods

3.1 Fluorescence lifetime imaging system

3.2 Brain tissue samples

3.3 Simulated surgery

4. Results

4.1 Imaging rate limitations for PpIX fluorescence lifetime imaging

4.2 Dark noise analysis

4.3 PpIX fluorescence lifetime imaging of 5-ALA labeled high-grade glioma

4.4 PpIX fluorescence lifetime imaging of 5-ALA labeled low-grade glioma

4.5 Normalized difference imaging of 5-ALA labeled glioma samples

4.6 Illustration of lifetime visualization with 12 Hz

5. Discussion

6. Conclusion

A. Derivation of equation (8)

Funding

Acknowledgments

Disclosures

References

Supplementary Material (1)

Cited By

Figures (8)

Tables (2)

Equations (16)

Biomedical Optics Express