Open Access
Add to BibSonomy
Abstract
Histopathological image analysis performed by a trained expert is currently regarded as the gold-standard for the diagnostics of many pathologies, including cancers. However, such approaches are laborious, time consuming and contain a risk for bias or human error. There is thus a clear need for faster, less intrusive and more accurate diagnostic solutions, requiring also minimal human intervention. Multiphoton microscopy (MPM) can alleviate some of the drawbacks specific to traditional histopathology by exploiting various endogenous optical signals to provide virtual biopsies that reflect the architecture and composition of tissues, both in-vivo or ex-vivo. Here we show that MPM imaging of the dermoepidermal junction (DEJ) in unstained fixed tissues provides useful cues for a histopathologist to identify the onset of non-melanoma skin cancers. Furthermore, we show that MPM images collected on the DEJ, besides being easy to interpret by a trained specialist, can be automatically classified into healthy and dysplastic classes with high precision using a Deep Learning method and existing pre-trained convolutional neural networks. Our results suggest that deep learning enhanced MPM for in-vivo skin cancer screening could facilitate timely diagnosis and intervention, enabling thus more optimal therapeutic approaches.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
As a result of their inherent optical sectioning capabilities and intrinsic contrast mechanisms, Multiphoton Microscopies (MPM) have emerged over the past three decades as very powerful tools for the label-free characterization of tissue morphology, functionality and biochemical composition [1–3], in-vivo and ex-vivo. Among these, Two-Photon Excitation Fluorescence (TPEF) microscopy [4] and Second-harmonic Generation (SHG) microscopy [5] have demonstrated their usefulness for exploring important properties of tissues, allowing the assessment of anatomical and functional states, and the identification of valuable pathological cues [6–9].
TPEF involves the simultaneous absorption of two photons with combined energy sufficient to induce an electronic transition to an excited electronic state [4]. Interestingly, TPEF allows to image in-vivo, ex-vivo or in-vitro the emission of various endogenous fluorophores such as reduced nicotinamide adenine dinucleotide (NADH), flavin adenine dinucleotide (FAD), melanin and others. Subsequently, TPEF microscopy allows a non-invasive assessment of cell morphology, size variation of cell nuclei, blood vessel hyperplasia, or inflammatory reaction related aspects, which are important for assessing the state of a tissue [6,10–12]. In SHG, two incident photons are combined into a single emitted photon with halved energy via a nonlinear process involving virtual states [13]. One of the main applications of SHG for tissue characterization and diagnostics is imaging of collagen [14], which is the main structural protein in the extracellular matrix of mammalian tissues. Investigating collagen distribution with SHG enables a precise and non-invasive assessment of extracellular matrix modifications, which represent a hallmark of cancers [15,16], and of many other pathologies [17].
The usefulness of TPEF and SHG to characterize human skin has been demonstrated both ex-vivo [18,19] and in-vivo [8,20–24]. Subsequently, multiphoton tomographs capable of providing virtual biopsies in-vivo have become available in many medical centers across the world [2]. The utility of MPM techniques for characterizing skin, or other organs/parts of the human body, is manifold and most importantly they can (i) enhance our understanding of tissue anatomy and functionality, (ii) enable fast and accurate tissue characterization ex-vivo and in-vivo. Because of these advantages, MPM is likely to soon become one of the central elements of in-vivo skin tissue characterization frameworks, while also representing a powerful tool to complement traditional diagnostics techniques such as immunohistochemistry or brightfield microscopy of Hematoxylin and Eosin (H&E) stained tissues. Recent advances in digital staining, where images taken by other modalities (including MPM) are transformed into virtual H&E images [25–27], facilitate the interpretation of MPM images by histopathologists, which we anticipate to massively boost the penetration of these techniques into the clinical practice in the near future.
In this work we perform MPM imaging of transversal tissue sections of human origin containing the dermoepidermal junction (DEJ) [28], which separates the dermis and the epidermis, two morphologically distinct compartments that interact in several ways and at different levels to create, control, or restore tissue homeostasis. It is known that processes occurring near the DEJ coordinate the growth and differentiation of the epidermis and are essential in demonstrating the complicated pathogenesis of epidermal tumors, irrespective of their benign or malignant nature. Our interest in characterizing this region is two-fold: (a) the DEJ lies at a depth accessible with MPM systems developed for clinical in-vivo applications [24,29] (b) during epidermal carcinogenesis important changes take place in the DEJ, which can be linked to early hyperplastic and neoplastic phases. These changes can be classified by their appearance, their extent and their frequency, the most prominent ones being related to the destructive modifications occurring in the basal lamina when dealing with an invasive neoplastic proliferation (even in early stages). The first part of our experiment shows that many of these subtle changes, which are difficult to assess with conventional microscopy, are available with label-free MPM imaging of the DEJ.
The second part of our experiment finds motivation in the fact that manual evaluation approaches for histopathological image analysis and diagnostics are both time consuming and prone to errors [30–32]. To address this, we show that TPEF and SHG images collected on the DEJ can be analyzed to automatically and precisely distinguish between healthy and dysplastic skin tissues by using Deep Learning [33], a family of machine learning techniques that are currently transforming many fields of science, including dermatology [34,35], and medicine in general [36–38]. The use of Deep Learning methods that can efficiently characterize MPM data sets is still in an early stage. However, the efforts implemented recently illustrate very well the possibilities derived from using in tandem MPM imaging and Deep Learning. For example, the work presented in [39] was successful in addressing the classification of normal and cancerous murine ovarian tissues, Lin et al. [40] focused on automatically identifying different stages of hepatocellular carcinoma, achieving classification accuracies >90%, and the method of Yu et al. [41] addressing automated diagnostics of liver fibrosis staging in a Wistar Rat model by transfer learning yielded values of up to 0.85–0.95 for the area under receiver operating characteristic (AUROC). The utility of MPM oriented Deep Learning methods is not restricted to addressing fixed or freshly excised tissues samples. For example, although our method is developed and tested on MPM data sets collected on fixed transversal tissue sections, with additional fine-training it can be easily adapted to address in-vivo skin characterization assays for cancer screening/diagnostics based on clinically validated multiphoton tomographs capable to scan epithelial tissues in both horizontal and vertical directions [29].
2. Methods
2.1 Image acquisition and processing
Combined SHG and TPEF imaging (to which we refer throughout the paper as MPM imaging) was performed using an upright Leica TCS-SP confocal laser-scanning microscope modified for nonlinear optical imaging. We used for excitation a Ti:Sapphire laser (Chameleon Vision II, Coherent) tuned at 860 nm, with ∼140 fs pulses and a repetition rate of 80 MHz. A Leica HCX PL FLUOTAR 40X/0.75 objective was used for focusing the laser beam on the sample and for collecting the backward-generated MPM signals (Fig. 1, representation adapted from [42]). The average power reaching the sample plane was kept below 15 mW. Images were acquired with a linear laser beam polarization obtained by using a polarization stage generator (PSG) comprised of an achromatic quarter-wave plate (AQWP05M-980, Thorlabs) and an achromatic half-wave plate (AHWP05M-980, Thorlabs), mounted in motorized rotation stages (PRM1/MZ8, Thorlabs) and placed in the laser beam path before the microscope. Three different input polarizations at 0°, 60° and 120° were used. The spectral detection available with the Leica TCS SP microscope allowed us to collect three channels simultaneously: the SHG channel (425–435 nm) and two TPEF channels – 440–490 nm corresponding to intrinsic fluorescence of reduced pyridine nucleotides (NAD(P)H) and 510–600 nm detecting the flavin adenine dinucleotide (FAD) fluorescence. The SHG/TPEF images were the Kalman average of four consecutive frames. The final SHG image was obtained by averaging the three SHG images acquired at different laser beam polarizations, resulting in a polarization independent SHG image [43]. Image processing was performed using FIJI [44], and consisted of applying a 0.5 radius mean filter to reduce noise. In addition, contrast was automatically enhanced, and a 0.7 gamma correction was applied in order to enhance the visibility of low intensity collagen fibers in the SHG images. The TPEF images corresponding to the NAD(P)H and FAD configurations were processed similarly, and the final TPEF image was formed by averaging the two separate TPEF NAD(P)H and FAD images. A composite RGB image was obtained by inserting the processed SHG and TPEF images into the blue and red channels, respectively.
Brightfield-microscopy (BM) images of H&E stained samples were collected using a Leica DM 3000 LED brightfield microscope, equipped with an MC 190 HD camera. A Leica HC PL Fluotar 5x/0.15 objective was used to record overlapping image tiles that were stitched together to form large mosaics representing the entire sample slides. The H&E stained sample regions corresponding to those imaged with MPM were also imaged with BM at high magnification with a Leica N Plan L 50X/0.5 objective.
2.2 MPM image classification with deep learning
Typically, very large data sets are needed to train deep convolutional neural networks from scratch and to overcome problems related to overfitting [45]. To address this, we used a pre-trained network which alleviates the cumbersome problem of generating large scale MPM training data sets [39]. Thus, for classifying the MPM images collected in this experiment we employed the GoogLeNet pre-trained Convolutional Neural Network, originally trained to perform 1000-fold multi-class classification of images by using a database consisting of ~1.2 million annotated images [46,47]. This way, we were able to use a relatively small data set of MPM images of the DEJ to fine-train the network for performing binary classification of target MPM images. Given that annotated training data was available, a supervised learning scheme was employed [48]. Prior to performing fine-training, we replaced the final classification layers of the original network (layers after ‘pool5’) with new fully connected (FC), Softmax and classification output layers, so that the modified version can accurately classify the MPM images of the DEJ. The workflow of the approach is illustrated in Fig. 2.
Fig. 2. Convolutional Neural Network based workflow for binary classification of MPM images. The input images are fed to the pre-trained network (GoogLeNet) that first performs feature extraction effectively transforming the data into a more optimal representation. Subsequent image classification is performed by the FC, Softmax and classification layers that, contrary to the convolutional layers, are trained from scratch during the fine-training process.
To further address overfitting problems raised by the relatively small training data set [47], we employed data augmentation, which is known to help in this matter [49]. We augmented MPM training data in two ways: (i) by casting horizontal and vertical reflections (a commonly met strategy), and (ii) with a novel strategy that combines horizontal/vertical reflections with the image pyramid in the Gaussian-Scale Space [50]. Here, each layer of the image pyramid (5 in total: σ = [0.5, $\sqrt {2}/2$, 1, $\sqrt {2}/2$, 2]) was horizontally and vertically reflected.
2.3 Sample preparation and imaging
The skin tissue samples used in this experiment consist in (i) lesions typical to squamous cell carcinoma (SCC), which has been regarded as an ideal prototype for lesions with a malignant/invasive character, (ii) lesions that are considered premalignant/non-invasive/"in situ”, e.g. from patients diagnosed with actinic keratosis or Bowen disease. Normal skin-tissue fragments have also been included in the study and were obtained from healthy regions close to the resection margins of the considered malignant/premalignant lesions. For each of the three investigated tissue categories we have imaged 14 pairs of histological slides, each corresponding to a distinct case. To obtain a pair of histological slides, from a formaldehyde-fixed paraffin-embedded histological block, two skin tissue sections were consecutively cut; one was left unstained for MPM imaging, while the other was stained with H&E for conventional histopathology. Using BM mosaics collected from the latter one at low magnification, trained histopathologists marked the positions of the DEJ, and MPM images were collected at random sites across the marked DEJ on the unstained sample pair. The regions imaged with MPM were also imaged with BM at high magnification for ground-truth availability. Each of the MPM images collected on the unstained tissue fragments was manually registered to the BM mosaics assembled from tiles collected with the high magnification objective (50X), covering the corresponding tissue region on the pair H&E stained slide. This operation was performed by using the “Align Image by Line ROI” plugin in the Fiji Open Source Software [44]. The small differences that can be observed to occur between the morphology of the MPM and BM imaged tissue pairs (e.g. Figure 3(c)) result from structural variations across the histological tissue block. The employed imaging protocol is generically presented in Fig. 3.
Fig. 3. Schematic demonstration of the employed imaging protocol. (a) Photographs of two histology slides (75 × 25 mm) containing tissue fragments consecutively cut from a histological block, (left) stained with H&E, and (right) unstained. (b) Large mosaic depicting the entire histology slide assembled by stitching overlapping BM images collected at low magnification (5X obj.) (c) Example of MPM images (40X obj.) collected on the unstained samples at random positions across the DEJ. All acquired MPM images were registered to high magnification BM images (50X obj.) collected on the corresponding regions of the H&E stained sample, in order to re-confirm that they indeed depict the DEJ, which is captured transversally. Scale bar in SHG/TPEF images: 50 µm.
3. Results and discussion
3.1 MPM imaging of the DEJ for tissue state assessment
In Fig. 4 we present a set of MPM images (overlaid TPEF and SHG signals) of the DEJ, collected on normal and dysplastic unstained epithelial tissues (transversal sections, see Section 2.3), accompanied by BM images collected on the corresponding regions of the pair H&E stained tissues. A version of the MPM images without annotation masks is available in [51].
Fig. 4. Pairs of MPM and BM images of the DEJ collected on unstained (MPM) and H&E stained (BM) normal and dysplastic epithelial tissues. MPM Field of view: 250 × 250 µm2.
The MPM images in Fig. 4 are showed under a pseudo-coloring scheme: blue-color for collagen-rich tissues (providing contrast for SHG), red-color for autofluorescent tissue regions (probed by TPEF), and violet-color for co-localized SHG and TPEF signals. The displayed images demonstrate the utility of MPM signals collected on the DEJ in being helpful to a histopathologist in his/her task to assess the skin tissues state. In Fig. 4(a-c) we can observe a normal appearance of the DEJ, as the MPM images show a complex, yet intact, collagen framework mainly of the basement membrane of the epidermis (Fig. 4(a)) and of the (superficial) papillary dermis (Fig. 4(b),(c)). In case of the latter, the bright TPEF signals highlight the red blood cells in the capillary network. The delicate walls of the capillary are also highlighted by a continuous red line corresponding to TPEF signals. The epidermis exhibits homogeneous TPEF signals determined by cytokeratins inside the keratinocytes. We note that those signals have a monotonous cytoplasmic pattern in the squamous cells of the epidermis due to the keratin content. The keratinization is a dynamic process exhibiting a gradient of keratin content from the basal layer to the stratum corneum. Subsequently, increased TPEF signals in this layer correspond to a greater content of keratin. The corneous layer exhibits stronger TPEF emission having more mature keratin (corneocytes) – blue arrow in Fig. 4(b),(c); the epidermis has an overall honeycomb appearance. Figure 4(d)-(f) depict images of the DEJ in dysplastic tissues. In Fig. 4(d),(e) the strong TPEF signals in the papillary dermis originate from the hemoglobin in the red blood cells, which has been previously documented [52,53]. The presence of cells with abnormal individual keratinization (dyskeratotic cells) is shown by TPEF signals similar of those of the red blood cells which are marked with green arrows in Fig. 4(e). TPEF signals also indirectly outline the nuclear contour of the squamous cells, which is important for assessing their state (the irregular nuclear contour is an important feature of neoplastic lesions, in situ or malignant). The increased nuclear ratio, dyskeratotic cells and parakeratosis are common features of actinic keratosis. In the case of the basement membrane the collagen framework is still visible and suggests an in-situ lesion, but the rest of the collagen framework of the papillary dermis has a more fragmented pattern suggesting a degenerative process (prolonged solar exposure). Figure 4(f) is representative for the MPM images collected on tissues affected by actinic keratosis, showing the usefulness of this technique to highlight the degeneration and fragmentation of the collagen fibers due to solar elastosis (area marked by dotted line).
A landmark study [54] showed that the basement membrane, one of the main components allowing the identification of the DEJ in the case of healthy and dysplastic tissues, is lost in the case of invasive tumors of the skin. The MPM images collected on malignant tissues, Fig. 5, containing relevant borders of such tumors, are in accordance with this previous study, the DEJ no longer being visible. A version of these images without annotation masks is available in [51]. These images contain nonetheless features that allow a histopathologist to assess the tissue state, and tumor invasion patterns. For example, in Fig. 5(a) we can observe a region corresponding to normal epidermis (left- area marked by dotted line) demonstrated by monotonous cytoplasmic TPEF signals, and to the basement membrane nearby, easily observable based on strong SHG signals. On the right, we can observe similar cytoplasmic signals in a moderate differentiated SCC; few keratin pearls (green circles in Fig. 5(a)-(c)) or dyskeratotic cells are seen (blue arrows Fig. 5(a)-(c)). We observe here that the large irregular nuclei are indirectly highlighted by TPEF (yellow dotted line in Fig. 5(a)-(c)). This front of invasion shows a pushing borders invasion pattern on the remaining structures. The collagen in the remaining dermis is fragmented (shown by SHG dot-like blue signals and marked by orange diamonds in Fig. 5(a)-(c)). Interestingly, solar elastosis has a strong, almost homogenous red signal in TPEF suggesting that this process promotes the presence of endogenous chromophores in regions affected by this condition. Figure 5(a)-(c) are also representative for SCC. In these three images we can easily observe the large dyskeratotic cells (blue arrows) with strong TPEF signals. Large, individual or grouped cells with squamous differentiation are found in the papillary dermis, admixed with red blood cells (with very strong TPEF signals – marked by red circles) and lymphocytes (rounded small cells with a dimmer appearance – marked by blue circles); irregular, parakeratotic pearls can also be seen (green circles). The collagen framework, highlighted by blue signals from SHG, is partially destroyed, suggesting an invasive lesion.
Fig. 5. Pairs of MPM and BM images of upper tumor borders collected on unstained (MPM) and H&E stained (BM) malignant epithelial tissues. MPM Field of view: 250 × 250 µm2.
Following our analysis of the MPM images collected on epithelial tissues, it can be observed that the DEJ region can be easily identified in the case of healthy and dysplastic tissues. In the case of malignant tissues, components of the DEJ are lost, this region being compromised and no longer identifiable. MPM imaging of transversal tissue sections containing the DEJ holds thus potential for screening/diagnostics, in fixed, fresh and in-vivo samples, allowing the histopathologist to differentiate healthy from dysplastic (unstained) tissues. Such utility is of great importance especially with respect to in-vivo assays, since identifying dysplastic lesions is often difficult with non-invasive methods, and patients are reluctant to allow the physician to resort to excisional biopsy when they are not convinced of the risk. Furthermore, such non-invasive in-vivo screening/diagnostic methods based on transversal (xz scanning direction) MPM imaging of the DEJ would represent a key tool for patients who cannot be subjected to excisional biopsy without the risk for complications [55], e.g. patients suffering of hemophilia [56], or in patients where cutaneous excision may result in defects that are difficult to correct by plastic surgery [57].
3.2 Automated identification of healthy and dysplastic tissues with MPM and deep learning
The first part of our experiment showed that the DEJ can be easily identified in MPM images collected on transversal sections of healthy and dysplastic tissues and provides important cues for a histopathologist to assess the tissue state. In malignant tissues the DEJ is compromised and hence not identifiable. Considering these, we hypothesized that an important utility of investigating the DEJ with MPM systems dedicated to clinical imaging, e.g. [8,22,58,59], would refer to potential assays that aim to screen patients with dysplastic modifications of the skin, which are difficult to implement with traditional non-invasive modalities. To further explore this utility, in the second part of our experiment we implemented and evaluated a Deep Learning method that augments the potential of MPM imaging of the DEJ, aiming to achieve automated and precise classification of tissues either as healthy or dysplastic.
The employed Deep Learning image classification method (detailed in Section 2.2) was inspired from a recent work [39], and dealt with a total of 358 MPM images from healthy (n = 14) and dysplastic (n = 14) unstained tissue sections, collected to contain the DEJ (see Methods). Images were randomly divided into validation (30%) and training sets (70%), the latter being augmented with the two strategies: (i) by reflecting the original images horizontally and vertically, and (ii) by repeatably blurring these horizontal and vertical reflections by using a five-layer Gaussian image pyramid, (see Section 2.2). The first augmentation strategy resulted in 1000 training images, while application of both yielded a set consisting of 5000 images. The test set always consisted of 108 images. This Deep Learning classification experiment was repeated 25 times, each time using a random selection of the validation/training images.
Previous work showed that better classification accuracy of MPM images of (ovarian) tissues is achieved by exploiting images with merged (summed) TPEF and SHG signals, compared to addressing solely TPEF or SHG images [39]. Here, we have extended the number of MPM signals for automated tissue classification, by additionally including in our evaluation framework the metabolic redox ratio (REDOX), which is calculated based on the TPEF emission originating from FAD and NADH molecules in the tissue [60,61]. We have thus experimented Deep Learning classification of MPM derived images representing the following summed-up signals: a) SHG + REDOX, b) TPEF + REDOX, c) SHG + TPEF + REDOX, d) SHG + TPEF. To better highlight the performance of our MPM oriented Deep Learning classification approach, we compared its results with those obtained by using the exact same strategy on the registered (high-magnification) BM images collected on the H&E stained slides (in practice, we substituted the MPM images with their BM counterparts in the training/validation sets). The resulting mean sensitivities, specificities and accuracies along with their standard deviations are shown in Fig. 6.
Fig. 6. Calculated classification sensitivity (a), specificity (b) and accuracy (c) with the error bars corresponding to the respective standard deviations. (a)-(c) MPM Image classification performance is on average improved ∼4% when the data augmentation includes also a 5-level Gaussian pyramid (red markers), compared to data augmentation not utilizing the Gaussian pyramid (blue markers). (a)–(c) Highest MPM image classification sensitivity (93.5 ± 2.3%), specificity (95.0 ± 2.4%) and accuracy (94.2 ± 1.6%) are achieved by using combined TPEF + SHG images for training/validation.
The classification performance of the trained network (GoogLeNet) for all four different MPM signal permutations was in general outstanding (∼90% or better). The best classification sensitivity (93.5 ± 2.3%, see Fig. 6(a)), specificity (95.0 ± 2.4%, see Fig. 6(b)) and accuracy (94.2 ± 1.6%, see Fig. 6c) were achieved using combined TPEF + SHG images. Use of “TPEF + SHG + REDOX” images resulted in best classification specificity (95.7 ± 2.8%, see Fig. 6(b)) alongside with excellent sensitivity (92.1 ± 3.6%) and accuracy (93.7 ± 1.7%). Overall, the Deep Learning classification results obtained for the MPM images are comparable with the best results obtained (under Gaussian image pyramid augmentation) by subjecting their BM counterparts (collected on the corresponding H&E stained tissues) to the same classification framework: specificity: 95.2 ± 3.6%, sensitivity: 95.8 ± 3.6% and accuracy: 95.4 ± 2.5%. We consider this to be very interesting, given that the Signal-to-Noise Ratio (SNR) of BM images collected on H&E tissues is significantly higher compared to the SNR of MPM images collected on unstained tissues.
In the case of the MPM images, the classification performance was on average improved ∼4% when the Gaussian image pyramid scheme was employed, compared to the case when data augmentation was done only with horizontal and vertical reflections of the original images. A similar improvement could also be observed in the case of the BM images, where using the training image set augmented with the first scheme (without Gaussian image pyramid), resulted in lower classification performance: specificity 90.6 ± 5.5%, sensitivity: 93.9 ± 5.1% and 91.8 ± 4% accuracy. While the number of training images (5000 vs 1000) could also explain this improvement, we believe that the main underlying reason refers to the fact that different layers of the Gaussian image pyramid approximate images of the original objects collected at different scales [62,63] (the reason why we adopted this data augmentation strategy). Therefore, by including layers of the image pyramid in the training process approximates exposing the network to images collected at different magnifications, conferring thus scale invariance to the classification framework. The advantage of scale invariance for the performance of Deep Learning classification of MPM images collected on epithelial tissues derives from the fact that the morphology of skin varies with age [64], anatomical site [65,66] and other factors. Convolving laser-scanning microscopy images (in our case the MPM images used for training) with a Gaussian filter is also known to suppress noise [67], which might as well have contributed to the observed classification performance increase.
4. Conclusions
In this work we have focused our attention on MPM imaging of transversal fixed tissue sections containing the DEJ, a region of the skin which is known to harbor important processes and modifications that are relevant with respect to the pathogenesis of epidermal tumors. We showed that MPM images contain features that are easy to interpret, which allow assessing the integrity of the DEJ, and differentiating healthy from dysplastic tissues. We regard this as being of great medical interest because compromised DEJ structures are a major hallmark of cancer progression and invasiveness and identifying these at a very early stage of the disease in a non-invasive manner compatible with in-vivo deployment has great importance for timely implementing the appropriate therapeutic strategies. Secondly, we have shown that MPM images of the DEJ, besides being easily interpreted by a trained expert, can also be automatically classified with Deep Learning either as healthy or dysplastic. To this end, we have demonstrated a Deep Learning approach based on the GoogLeNet network, which provides real-time image classification, with sensitivity, specificity and accuracy all exceeding 90%. The results obtained for MPM classification of unstained tissues were found to be comparable to the those obtained when using the exact same Deep Learning classification framework in association with the BM images collected on H&E stained corresponding tissue regions. Considering that the employed Deep Learning method provides comparable results for two different modalities (BM and MPM) that yield images with distinct characteristics in terms of content, quality and appearance, suggests in our opinion that it might also perform very well when transferred to address skin tissue images collected with other relevant techniques (e.g. Reflectance Confocal Laser Scanning Microscopy, Optical Coherence Tomography, etc.).
The utility of the demonstrated methodology for automated classification of MPM data sets collected on the DEJ (in unstained tissues) is threefold: (i) it can be used to characterize MPM data sets collected with bench-top systems on fixed, excised, samples (acting as a complementary tool to traditional histopathology protocols); (ii) it can be easily adapted to characterize freshly excised, and unstained, tissues (imaged with bench-top MPM systems or clinical multiphoton tomographs) to provide skin tissue diagnostic very fast, next to the patient, avoiding wait times associated to tissue fixation, staining and expert interpretation, and (iii) with additional fine-training it can be easily adapted to address in-vivo characterization approaches for real-time skin diagnostics in association with clinical multiphoton tomographs, which are already available in many institutions worldwide. In such in-vivo assays, a target region of the patient’s skin should be scanned with a multiphoton tomograph in xz direction (transversally), and once the DEJ is visible, an MPM image containing it could be instantly classified as healthy or dysplastic with a Deep Learning strategy derived/inspired from our proposed method. With respect to the feasibility of such applications, it is important to mention that while the technical performance of latest-hour bench-top MPM systems exceeds the performance of available clinical multiphoton tomographs, the intensity of MPM signals in fresh/in-vivo tissues is significantly higher compared to the case of fixed tissues [68], which can compensate this aspect. Overall, our results show that MPM and Deep Learning are likely to play a huge role in the forthcoming years in terms of speeding up and improving the current methodologies used in skin cancer screening and diagnosis.
Funding
Unitatea Executiva pentru Finantarea Invatamantului Superior, a Cercetarii, Dezvoltarii si Inovarii (Romania's Executive Agency for Higher Education, Research, Development and Innovation Funding UEFISCDI) (PN-III-P1-1.1-TE-2016-2147 CORIMAG, PN-III-P2-2.1-PED-2016-1252 MICAND); EU's Horizon 2020 Framework Programme (777222 Attract [HARMOPLUS]); European Regional Development Fund (Project No. P_36_611, MySMIS code 107066); Academy of Finland (308596, 320165).
Acknowledgments
MJH acknowledges the Flagship of Photonics Research and Innovation of the Academy of Finland. The authors thank Mr. Tiberiu Totu (UPB) and Ms. Roxana Buga (UPB) for their help with acquisition of BM mosaics. SGS acknowledges the support of the Nvidia Corporation via their Academic Hardware Grant and the support of the CA15124 NEUBIAS COST Action, which facilitated interactions relevant for this work.
Disclosures
The authors declare that there are no conflicts of interest related to this article.
References
1. E. E. Hoover and J. A. Squier, “Advances in multiphoton microscopy technology,” Nat. Photonics 7(2), 93–101 (2013). [CrossRef]
2. K. König, Multiphoton Microscopy and Fluorescence Lifetime Imaging, Applications in Biology and Medicine (Degruyer, 2018).
3. R. M. Williams, W. R. Zipfel, and W. W. Webb, “Multiphoton microscopy in biological research,” Curr. Opin. Chem. Biol. 5(5), 603–608 (2001). [CrossRef]
4. P. T. So, C. Y. Dong, B. R. Masters, and K. M. Berland, “Two-photon excitation fluorescence microscopy,” Annu. Rev. Biomed. Eng. 2(1), 399–429 (2000). [CrossRef]
5. P. J. Campagnola and L. M. Loew, “Second-harmonic imaging microscopy for visualizing biomolecular arrays in cells, tissues and organisms,” Nat. Biotechnol. 21(11), 1356–1360 (2003). [CrossRef]
6. W. R. Zipfel, R. M. Williams, R. Christie, A. Y. Nikitin, B. T. Hyman, and W. W. Webb, “Live tissue intrinsic emission microscopy using multiphoton-excited native fluorescence and second harmonic generation,” Proc. Natl. Acad. Sci. 100(12), 7075–7080 (2003). [CrossRef]
7. W. R. Zipfel, R. M. Williams, and W. W. Webb, “Nonlinear magic: multiphoton microscopy in the biosciences,” Nat. Biotechnol. 21(11), 1369–1377 (2003). [CrossRef]
8. M. Balu, C. B. Zachary, R. M. Harris, T. B. Krasieva, K. König, B. J. Tromberg, and K. M. Kelly, “In vivo multiphoton microscopy of basal cell carcinoma,” JAMA Dermatol. 151(10), 1068–1074 (2015). [CrossRef]
9. O. J. Muensterer, S. Waldron, Y. J. Boo, C. Ries, L. Sehls, F. Simon, L. Seidmann, J. Birkenstock, and J. Gödeke, “Multiphoton microscopy: a novel diagnostic method for solid tumors in a prospective pediatric oncologic cohort, an experimental study,” Int. J. Surg. 48, 128–133 (2017). [CrossRef]
10. R. K. Benninger and D. W. Piston, “Two-photon excitation microscopy for the study of living cells and tissues,” Curr. Protoc. Cell Biol. 59(1), 1–24 (2013). [CrossRef]
11. M. C. Skala, J. M. Squirrell, K. M. Vrotsos, J. C. Eickhoff, A. Gendron-Fitzpatrick, K. W. Eliceiri, and N. Ramanujam, “Multiphoton microscopy of endogenous fluorescence differentiates normal, precancerous, and cancerous squamous epithelial tissues,” Cancer Res. 65(4), 1180–1186 (2005). [CrossRef]
12. S. G. Stanciu, S. Xu, Q. Peng, J. Yan, G. A. Stanciu, R. E. Welsch, P. T. So, G. Csucs, and H. Yu, “Experimenting liver fibrosis diagnostic by two photon excitation microscopy and bag-of-features image classification,” Sci. Rep. 4(1), 4636 (2015). [CrossRef]
13. P. J. Campagnola and C. Y. Dong, “Second harmonic generation microscopy: principles and applications to disease diagnosis,” Laser Photonics Rev. 5(1), 13–26 (2011). [CrossRef]
14. X. Chen, O. Nadiarynkh, S. Plotnikov, and P. J. Campagnola, “Second harmonic generation microscopy for quantitative analysis of collagen fibrillar structure,” Nat. Protoc. 7(4), 654–669 (2012). [CrossRef]
15. C. Bonnans, J. Chou, and Z. Werb, “Remodelling the extracellular matrix in development and disease,” Nat. Rev. Mol. Cell Biol. 15(12), 786–801 (2014). [CrossRef]
16. P. Lu, V. M. Weaver, and Z. Werb, “The extracellular matrix: a dynamic niche in cancer progression,” J. Cell Biol. 196(4), 395–406 (2012). [CrossRef]
17. E. W. Raines, “The extracellular matrix can regulate vascular cell migration, proliferation, and survival: relationships to vascular disease,” Int. J. Exp. Pathol. 81(3), 173–182 (2001). [CrossRef]
18. J. Paoli, M. Smedh, and M. B. Ericson, “Multiphoton laser scanning microscopy–a novel diagnostic method for superficial skin cancers,” Semin. Cutaneous Med. Surg. 28(3), 190–195 (2009). [CrossRef]
19. J. Paoli, M. Smedh, A.-M. Wennberg, and M. B. Ericson, “Multiphoton laser scanning microscopy on non-melanoma skin cancer: morphologic features for future non-invasive diagnostics,” J. Invest. Dermatol. 128(5), 1248–1255 (2008). [CrossRef]
20. T. Y. Sun, A. M. Haberman, and V. Greco, “Preclinical advances with multiphoton microscopy in live imaging of skin cancers,” J. Invest. Dermatol. 137(2), 282–287 (2017). [CrossRef]
21. R. Cicchi, D. Kapsokalyvas, and F. S. Pavone, “Clinical nonlinear laser imaging of human skin: a review,” BioMed Res. Int. 2014, 1–14 (2014). [CrossRef]
22. M. J. Koehler, M. Speicher, S. Lange-Asschenfeldt, E. Stockfleth, S. Metz, P. Elsner, M. Kaatz, and K. König, “Clinical application of multiphoton tomography in combination with confocal laser scanning microscopy for in vivo evaluation of skin diseases,” Exp. Dermatol. 20(7), 589–594 (2011). [CrossRef]
23. E. Dimitrow, M. Ziemer, M. J. Koehler, J. Norgauer, K. König, P. Elsner, and M. Kaatz, “Sensitivity and specificity of multiphoton laser tomography for in vivo and ex vivo diagnosis of malignant melanoma,” J. Invest. Dermatol. 129(7), 1752–1758 (2009). [CrossRef]
24. R. B. Saager, M. Balu, V. Crosignani, A. Sharif, A. J. Durkin, K. M. Kelly, and B. J. Tromberg, “In vivo measurements of cutaneous melanin across spatial scales: using multiphoton microscopy and spatial frequency domain spectroscopy,” J. Biomed. Opt. 20(6), 066005 (2015). [CrossRef]
25. T. W. Bocklitz, F. S. Salah, N. Vogler, S. Heuke, O. Chernavskaia, C. Schmidt, M. J. Waldner, F. R. Greten, R. Bräuer, and M. Schmitt, “Pseudo-HE images derived from CARS/TPEF/SHG multimodal imaging in combination with Raman-spectroscopy as a pathological screening tool,” BMC cancer 16(1), 534 (2016). [CrossRef]
26. N. Borhani, A. J. Bower, S. A. Boppart, and D. Psaltis, “Digital staining through the application of deep neural networks to multi-modal multi-photon microscopy,” Biomed. Opt. Express 10(3), 1339–1350 (2019). [CrossRef]
27. Y. Rivenson, H. Wang, Z. Wei, K. de Haan, Y. Zhang, Y. Wu, H. Günaydın, J. E. Zuckerman, T. Chong, and A. E. Sisk, “Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning,” Nat. Biomed. Eng. 3(6), 466–477 (2019). [CrossRef]
28. R. A. Briggaman and C. E. Wheeler Jr, “The epidermal-dermal junction,” J. Invest. Dermatol. 65(1), 71–84 (1975). [CrossRef]
29. H. G. Breunig, B. Sauer, A. Batista, and K. König, “Rapid vertical tissue imaging with clinical multiphoton tomography,” Proc. SPIE 10679, 81 (2018). [CrossRef]
30. S. Chatterjee, “Artefacts in histopathology,” Int. J. Oral Maxillofac. Pathol. 18(4), 111 (2014). [CrossRef]
31. B. J. Reid, R. C. Haggitt, C. E. Rubin, G. Roth, C. M. Surawicz, G. Vanbelle, K. Lewin, W. M. Weinstein, D. A. Antonioli, H. Goldman, W. Macdonald, and D. Owen, “Observer Variation in the Diagnosis of Dysplasia in Barretts Esophagus,” Hum. Pathol. 19(2), 166–178 (1988). [CrossRef]
32. L. Brown, “Improving histopathology turnaround time: a process management approach,” Curr. Diagn. Pathol. 10(6), 444–452 (2004). [CrossRef]
33. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature 521(7553), 436–444 (2015). [CrossRef]
34. A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, “Dermatologist-level classification of skin cancer with deep neural networks,” Nature 542(7639), 115–118 (2017). [CrossRef]
35. H. A. Haenssle, C. Fink, R. Schneiderbauer, F. Toberer, T. Buhl, A. Blum, A. Kalloo, A. B. H. Hassen, L. Thomas, and A. Enk, “Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists,” Ann. Oncol. 29(8), 1836–1842 (2018). [CrossRef]
36. A. Rajkomar, J. Dean, and I. Kohane, “Machine Learning in Medicine,” N. Engl. J. Med. 380(14), 1347–1358 (2019). [CrossRef]
37. M. Wainberg, D. Merico, A. Delong, and B. J. Frey, “Deep learning in biomedicine,” Nat. Biotechnol. 36(9), 829–838 (2018). [CrossRef]
38. G. Hinton, “Deep learning—a technology with the potential to transform health care,” JAMA 320(11), 1101–1102 (2018). [CrossRef]
39. M. J. Huttunen, A. Hassan, C. W. McCloskey, S. Fasih, J. Upham, B. C. Vanderhyden, R. W. Boyd, and S. Murugkar, “Automated classification of multiphoton microscopy images of ovarian tissue using deep learning,” J. Biomed. Opt. 23(06), 1 (2018). [CrossRef]
40. H. Lin, C. Wei, G. Wang, H. Chen, L. Lin, M. Ni, J. Chen, and S. Zhuo, “Automated classification of hepatocellular carcinoma differentiation using multiphoton microscopy and deep learning,” J. Biophotonics 12(7), e201800435 (2019). [CrossRef]
41. Y. Yu, J. Wang, C. W. Ng, Y. Ma, S. Mo, E. L. S. Fong, J. Xing, Z. Song, Y. Xie, and K. Si, “Deep learning enables automated scoring of liver fibrosis stages,” Sci. Rep. 8(1), 16016 (2018). [CrossRef]
42. S. G. Stanciu, F. J. Ávila, R. Hristu, and J. M. Bueno, “A study on image quality in polarization-resolved second harmonic generation microscopy,” Sci. Rep. 7(1), 15476 (2017). [CrossRef]
43. L. Gao, L. Jin, P. Xue, J. Xu, Y. Wang, H. Ma, and D. Chen, “Reconstruction of complementary images in second harmonic generation microscopy,” Opt. Express 14(11), 4727–4735 (2006). [CrossRef]
44. J. Schindelin, I. Arganda-Carreras, E. Frise, V. Kaynig, M. Longair, T. Pietzsch, S. Preibisch, C. Rueden, S. Saalfeld, and B. Schmid, “Fiji: an open-source platform for biological-image analysis,” Nat. Methods 9(7), 676–682 (2012). [CrossRef]
45. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Commun. ACM 60(2), 84–90 (2017). [CrossRef]
46. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015), 1–9.
47. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, (Ieee, 2009), 248–255.
48. D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio, “Why does unsupervised pre-training help deep learning?” J Mach. Learn. Res. 11, 625–660 (2010).
49. J. Wang and L. Perez, “The effectiveness of data augmentation in image classification using deep learning,” Convolutional Neural Networks Vis. Recognit (2017).
50. E. H. Adelson, C. H. Anderson, J. R. Bergen, P. J. Burt, and J. M. Ogden, “Pyramid methods in image processing,” RCA engineer 29, 33–41 (1984).
51. M. J. Huttunen, R. Hristu, A. Dumitru, M. Costache, and S. G. Stanciu, “Investigating and Assessing the Dermoepidermal Junction with Multiphoton Microscopy and Deep Learning,” bioRxiv, 743054 (2019).
52. Q. Sun, W. Zheng, J. Wang, Y. Luo, and J. Y. Qu, “Mechanism of two-photon excited hemoglobin fluorescence emission,” J. Biomed. Opt. 20(10), 105014 (2015). [CrossRef]
53. W. Zheng, D. Li, Y. Zeng, Y. Luo, and J. Y. Qu, “Two-photon excited hemoglobin fluorescence,” Biomed. Opt. Express 2(1), 71–79 (2011). [CrossRef]
54. S. Barsky, G. Siegal, F. Jannotta, and L. Liotta, “Loss of basement membrane components by invasive tumors but not by their benign counterparts,” Lab. Invest. ; a journal of technical methods and pathology 49, 140–147 (1983).
55. K. Abhishek and N. Khunger, “Complications of skin biopsy,” J Cutan Aesthet Surg 8(4), 239 (2015). [CrossRef]
56. J. Chapin, J. Bamme, F. Hsu, P. Christos, and M. DeSancho, “Outcomes in patients with hemophilia and von Willebrand disease undergoing invasive or surgical procedures,” Clin. Appl. Thromb./Hemostasis 23(2), 148–154 (2017). [CrossRef]
57. A. Bayat, D. McGrouther, and M. Ferguson, “Skin scarring,” BMJ 326(7380), 88–92 (2003). [CrossRef]
58. M. Weinigel, H. Breunig, M. Kellner-Höfer, R. Bückle, M. Darvin, M. Klemp, J. Lademann, and K. König, “In vivo histology: optical biopsies with chemical contrast using clinical multiphoton/coherent anti-Stokes Raman scattering tomography,” Laser Phys. Lett. 11(5), 055601 (2014). [CrossRef]
59. K. König, “Clinical multiphoton tomography,” J. Biophotonics 1(1), 13–23 (2008). [CrossRef]
60. I. Georgakoudi and K. P. Quinn, “Optical imaging using endogenous contrast to assess metabolic state,” Annu. Rev. Biomed. Eng. 14(1), 351–367 (2012). [CrossRef]
61. M. C. Skala, K. M. Riching, A. Gendron-Fitzpatrick, J. Eickhoff, K. W. Eliceiri, J. G. White, and N. Ramanujam, “In vivo multiphoton microscopy of NADH and FAD redox states, fluorescence lifetimes, and cellular morphology in precancerous epithelia,” Proc. Natl. Acad. Sci. 104(49), 19494–19499 (2007). [CrossRef]
62. J. Sporring, M. Nielsen, L. Florack, and P. Johansen, Gaussian Scale-space Theory (Springer Science & Business Media, 2013), Vol. 8.
63. L. M. Florack, B. M. ter Haar Romeny, J. J. Koenderink, and M. A. Viergever, “Scale and the differential structure of images,” Image Vision Comput. 10(6), 376–388 (1992). [CrossRef]
64. M. Branchet, S. Boisnic, C. Frances, C. Lesty, and L. Robert, “Morphometric analysis of dermal collagen fibers in normal human skin as a function of age,” Arch. Gerontol. Geriatr. 13(1), 1–14 (1991). [CrossRef]
65. M. Huzaira, F. Rius, M. Rajadhyaksha, R. R. Anderson, and S. González, “Topographic variations in normal skin, as viewed by in vivo reflectance confocal microscopy,” J. Invest. Dermatol. 116(6), 846–852 (2001). [CrossRef]
66. T. Gambichler, R. Matip, G. Moussa, P. Altmeyer, and K. Hoffmann, “In vivo data of epidermal thickness evaluated by optical coherence tomography: effects of age, gender, skin type, and anatomic site,” J. Dermatol. Sci. 44(3), 145–152 (2006). [CrossRef]
67. G. Van Kempen, L. Van Vliet, P. Verveer, and H. Van Der Voort, “A quantitative comparison of image restoration methods for confocal microscopy,” J. Microsc. 185(3), 354–365 (1997). [CrossRef]
68. F. Poulon, M. Zanello, A. Ibrahim, A. Zaylaa, P. Varlet, B. Devaux, and D. A. Haidar, “Comparison between fresh and fixed human biopsies using spectral and lifetime measurements: Fluorescence analysis using one and two photon excitations,” in 2015 International Conference on Advances in Biomedical Engineering (ICABME), (IEEE, 2015), 25–28.
