Open Access
Add to BibSonomy
Abstract
Fast and reliable quantification of cone photoreceptors is a bottleneck in the clinical utilization of adaptive optics scanning light ophthalmoscope (AOSLO) systems for the study, diagnosis, and prognosis of retinal diseases. To-date, manual grading has been the sole reliable source of AOSLO quantification, as no automatic method has been reliably utilized for cone detection in real-world low-quality images of diseased retina. We present a novel deep learning based approach that combines information from both the confocal and non-confocal split detector AOSLO modalities to detect cones in subjects with achromatopsia. Our dual-mode deep learning based approach outperforms the state-of-the-art automated techniques and is on a par with human grading.
© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
The ability to quantify the photoreceptor mosaic geometry is useful for the study, diagnosis, and prognosis of diseases that affect photoreceptors such as achromatopsia (ACHM), age-related macular degeneration, retinitis pigmentosa (RP)/Usher syndrome, Stargardt disease, choroideremia, and blue-cone monochromacy [1, 2], or for evaluating subclinical photoreceptor disruption from head trauma [3]. Adaptive optics (AO) ophthalmoscopes reveal the photoreceptor mosaic in the living human retina [4–12], and have been used to study its geometry in healthy [4, 13–19] and pathologic [3, 20–26] eyes.
The most common of these AO systems is the adaptive optics scanning light ophthalmoscope (AOSLO), which is often implemented with reflectance confocal [5] and, more recently, also non-confocal modalities [27–31]. Reflectance confocal AOSLO relies on intact and functional outer segment structure [32] to reveal the smallest photoreceptors in the retina - rods and foveal cones [16]. Non-confocal AOSLO modalities have recently been developed to utilize the multiple-scattered light not detected by the confocal channel to provide an alternate source of information [27–30]. Split detector [29] is one of the most widely used non-confocal AOSLO detection configurations, which reveals the cone inner segment mosaic [29].
Quantitative analysis of the photoreceptor mosaic in AO images requires the localization of each individual photoreceptor. Unfortunately, even in normal subjects with clear ocular media, there is ambiguity in the identification of individual photoreceptors in AOSLO images [29, 33], which is the all-important first step in the interpretation of these images. Manual identification of these photoreceptors is highly subjective [34] and very time consuming, both factors severely limiting the translation of photoreceptor imaging into a practical clinical tool. Several automated methods have been developed for detecting cones in ophthalmic AO images [33, 35–48]. However, most of these have not been tested on truly representative images of diseased eyes or have shown comparatively poor performance [33, 48]. Typical images from diseased eyes often exhibit substantially lower contrast and signal-to-noise ratio than those from healthy subjects, as well as disease-dependent features that might not be accounted for in algorithms designed or trained for healthy retinas. Thus, the general consensus of the AO research community can be summarized as “… today's automated approaches to identify cones should be used with caution, if at all,…” [32].
AOSLO imaging is already being used to select candidates for and predict the effectiveness of gene therapy [32, 49] for conditions such as ACHM, a retinal condition characterized by a lack of cone function resulting in color blindness, photophobia, nystagmus, and severely reduced visual acuity [50]. Unfortunately, quantification of cone photoreceptors in ACHM AOSLO images is especially challenging, even for human graders [34]. In confocal AOSLO images of healthy eyes, cones appear as bright spots in the image, whereas in ACHM they appear as dark spots [51]. As the rods appear to waveguide normally, it is sometimes possible to indirectly infer the presence of a cone when seeing a dark spot circumscribed by a ring of reflective rods, however this becomes challenging in images closer to the central fovea, where rod numerosity declines. Non-confocal split detector AOSLO imaging reveals remnant cone inner segment structures in areas that lack reflectivity in confocal AOSLO [29, 52] (Figs. 1(a) and 1(b)), showing potential for predicting therapeutic outcomes [32, 49], and thus making automated detection of these cone structures desirable. Even though visualization of cones is possible with this imaging modality, there is often uncertainty in identifying cone locations due to the relatively poor contrast seen in typical images such as that shown in Fig. 1(c). It has been recently suggested that combining multiple modalities could improve the reliability/accuracy/other for cone identification [32], and it has been shown that multiple AOSLO modalities could improve performance in other image processing tasks such as mosaicking [53]. As seen in Fig. 1(d), simultaneously captured confocal AOSLO images can help resolve some ambiguities seen in the matching split detector image, even with cones lacking intensity in ACHM subjects.
Fig. 1 Dual-mode AOSLO cone imaging in ACHM subjects. (a) Split detector AOSLO image near the fovea of an ACHM subject. (b) Simultaneously captured confocal AOSLO image from the same location as (a). (c) Split detector AOSLO image at 12° from the fovea in another subject with ACHM. (d) Simultaneously captured confocal AOSLO image from the same location as (c). Orange arrows point to ambiguous locations in the split detector image (c) that can be seen to be cones based on the dark circles in the confocal image (d). Scale bars: 20 μm.
As with other computer vision tasks, automated analyses of AOSLO images with deep learning convolutional neural networks (CNNs) that learn features directly from training data are expected to outperform classic machine learning based techniques. CNNs have been utilized in numerous ophthalmic image processing applications [46, 54–63]. In our previous work [46], we developed the first CNN based AOSLO image analysis method for detecting cones, demonstrating superiority to existing state-of-the-art techniques. Here, we expand on this work by combining the complimentary confocal and non-confocal AOSLO information to improve performance in low contrast images of diseased retinas.
The organization of the paper is as follows. We first introduce a novel dual-modality deep learning AOSLO segmentation paradigm for identification of cones. We then demonstrate that our method that incorporates dual-mode information from confocal and split detector AOSLO images outperforms a comparable deep learning method that only uses a single AOSLO imaging modality. Finally, we show that the dual-mode deep learning based method outperforms the state-of-the-art automated techniques and is on a par with human grading.
2. Methods
Our proposed algorithm for identification of cones, shown in Fig. 2, is comprised of a training and a testing phase. In the training phase, a set of reflectance confocal and split detector AOSLO image pairs was broken into small patches. A subset of all patches was classified (labeled) as cone and non-cone, based on manual markings. These labeled patches were used to train a CNN classifier, which was then utilized to generate probability maps from all overlapping patches in the images, which in turn allowed optimization of the parameters used for detecting cones. The trained CNN was then used to detect cones in previously unseen image pairs without known labels.
2.1 Data set
The images for our data set were captured using a previously described AOSLO system [10, 29], which acquires both confocal and split detector modalities simultaneously and with perfect spatial registration. For each subject, a series of image sequences, using a 1.0° and/or 1.75° field of view, were captured over a range of eccentricities (from the fovea to 12° along the temporal and superior meridian). The images in this data set were strip-registered using between 5 and 69 frames, as previously described [29, 64], although the number of frames averaged within a given strip of the processed image could be anywhere between 1 and 69. Within each sequence pair, transformation for both split detector and confocal modalities was identical. Lateral scale/sampling for each subject was determined using axial length measurements from an IOL Master (Carl Zeiss Meditec Inc., Dublin, California, USA). Regions of interest (ROIs) were extracted from these images and used for analysis.
The final data set contained 200 split detector and confocal AOSLO image pairs (size 100 × 100 μm2 each) from 16 subjects with ACHM. There were an uneven number of image pairs per subject, with a minimum number of 6 and maximum of 29. For three subjects, the split detector images had the opposite orientation as the rest of the set. Thus, the split detector and confocal images from these subjects were horizontally flipped so that all images would have the same orientation. All image pairs had their cones marked by two masked manual graders, DC and SB. Manual grading was done primarily on the split detector images, with the corresponding confocal images used to resolve any ambiguous cases. Over 9,200 cones were marked by the first manual grader (DC) over the entire data set. We resized all images (confocal and split detector) using cubic interpolation so they have a pixel size of 0.5 × 0.5 μm (200 × 200 pixels per image) and adjusted the manual markings to match this change. This resizing was performed due to the large differences in pixel size between images from 1.0° and 1.75° field of view scans.
2.2 Image preprocessing and patch extraction
We first normalized all split detector and confocal images so that their intensity values stretched between 0 and 255. We then extracted cone and non-cone patch pairs from the training images to be used to train the CNN in a similar fashion as presented in Cunefare et al. [46]. In brief, for each training image pair, we used the first set of expert manual markings to define the cone locations, and extracted patches of 33 × 33 pixels centered around each marking in both the confocal and split detector images. The patch size was chosen so that individual patches would be large enough to encompass any cone. The non-cone locations are naturally more challenging to define as there were no manual annotations of non-cone areas, so we used Voronoi diagrams [35], in which the center of each cell is defined by a cone location, in order to find non-cone locations. As the Voronoi edges are equidistant to the two nearest cone markings, they are generally located in the space between cones. Therefore, we created the non-cone patches by randomly selecting a single point from each Voronoi edge, rounding to the nearest pixel value, and extracting patches of 33 × 33 pixels around this position from both the split detector and corresponding confocal images. Patches that would extend outside the bounds of the image were not used. For each training image pair, the first set of manual markings was used to generate the Voronoi diagram, as shown in Figs. 3(a) and 3(b). Note that all manually marked cones were used to generate the Voronoi diagram, which differs from [46] where marked cones too close to the edges were not included when generating the Voronoi diagram. Example paired patches are shown in Figs. 3(c) and 3(d).
Fig. 3 Extraction of labeled patches from AOSLO image pairs. (a) Cropped split detector AOSLO image. (b) Simultaneously captured cropped confocal AOSLO image from the same location. Voronoi diagram overlain in cyan, manually marked cones are shown in green, and randomly generated locations along Voronoi edges are shown in yellow. (c) Example cone patch pair from position shown in purple in (a) and (b). (d) Example non-cone patch pair from position shown in red in (a) and (b).
2.3 Convolutional neural network
We built upon the single-mode Cifar [65, 66] based network used in Cunefare et al. [46] to incorporate dual-mode data. The network architecture, shown in Fig. 4, is similar to late fusion architectures that have been used in other classification problems with multiple input images [67, 68]. As such, we named this network the late fusion dual-mode CNN (LF-DM-CNN). The late fusion network was chosen empirically over early fusion architectures based on results across our data set. The network incorporates convolutional, batch normalization, pooling, rectified linear units (ReLU), fully connected, concatenation (i.e. fusion), and soft-max layers. The convolutional layers convolve an input of size W × H × D (before padding) with N kernels of size F × F × D with a stride of 1 to get an output of size W × H × N, where the output can be considered a stack of N feature maps. For each of these N feature maps, the CNN adds a potentially different bias value. We set the kernel size, F, to be 5 throughout the network. Figure 5 displays the trained filters from the first convolutional layer for both the split detector and confocal paths. Batch normalization layers [69] normalize their inputs based on mean and variance statistics, and are used to reduce internal covariate shift, which can decrease overfitting during training. The pooling layers apply either a max or average operation over the first two dimensions of the input in a 3 × 3 window while down-sampling by a factor of 2 in the first two dimensions. ReLU layers [70] non-linearly transform their inputs by setting all negative values to 0, which speeds up the training process and improves the performance of the network [71]. Fully connected layers output multiple nodes that each result from the weighted sum of all the values from the previous layer, with an additional bias term added for each node. The concatenation layer combines the two 64 × 1 vectors output from the confocal and split detector paths into a single 128 × 1 vector. Finally, the soft-max [72] layer takes a 2 × 1 vector from the last fully connected layer and applies the soft-max function which normalizes each value to be between 0 and 1, and makes the sum of all values 1. These values can be thought of as the probability of the input patch pair belonging to a cone or non-cone location.
Fig. 4 Proposed late fusion dual-mode CNN (LF-DM-CNN) architecture, which consists of the following layers: convolutional (Conv(N,F) where N is the number of kernels, and F is the kernel size in the first two dimensions), fully connected (FC(X) where X is the number of output nodes) batch normalization (BatchNorm), max pooling (MaxPool), average pooling (AvePool), ReLu, concatenation, and soft-max.
Fig. 5 Filter weights from the first convolutional layer in the LF-DM-CNN for the (a) split detector and (b) confocal paths.
Before the network could be used to detect cones, the weight and biases needed to be learned using the labeled patch pairs. The initial weights for the network were randomly initialized, and the bias terms were set to zero similarly to [65]. The weights and biases were then learned using stochastic gradient descent to minimize cross-entropy loss [73]. All of the training data was split into mini-batches with 100 patch pairs per mini-batch, and each iteration of the gradient descent occurred over a single mini-batch. This was repeated for all mini-batches (known as an epoch), and we trained over 45 epochs. Data augmentation was applied by randomly vertically flipping both patches in a pair 50% of the time the patch pair is seen, in order to effectively increase our training data amount. The weight learning rates were set initially to 0.001 for all convolutional and fully connected layers except the last fully connected layer, where it was set to 0.0001. Bias learning rates were similarly set to 0.1 for all layers except the last fully connected layer where it was set to 0.01. All learning rates were reduced by a factor of 10 twice over training, at the beginning of epochs 31 and 41. Weight decay was set to 0.0001. All noted training hyper-parameters were set empirically based on a different online data set introduced in [46].
2.4 Cone localization
In the final step of our method, for each split detector and confocal AOSLO image pair, the trained CNN generated a probability map from which the cone locations were inferred as follows. We first normalized the intensity values to be between 0 and 255 for both images. For each pixel position in the corresponding split detector and confocal images, we extracted a pair of 33 × 33 pixel patches. We applied 16 pixel symmetric padding to the images that mirrored the intensity values near the borders of the images to account for patches that would have extended past the image boundaries. The paired patches were then input into the trained LF-DM-CNN to determine the probability of that location being positioned on a cone. We then used these probabilities to generate a single probability map the same size as one of the images as shown in Fig. 6(c). We smoothed the map by convolving with a Gaussian filter with standard deviation σ to remove spurious maxima. Next, we applied the extended-maxima transform using MATLAB’s imextendedmax function [74], which finds maximal regions where the probability difference in the region is less than or equal to H and outputs these regions in a binary map (Fig. 6(d)). We found all connected clusters in the binary map to use as potential candidates for cone positions, and eliminated weak candidates by removing any cluster whose maximum value in the filtered probability map was less than a threshold T. Finally, we found the center of mass of all remaining clusters which are considered to be the cone positions. The values of σ, H, and T were all automatically set by maximizing the average Dice’s coefficient (explained in 2.5) across the training images over a set of potential parameter combinations.
Fig. 6 Detection of cones in split detector and confocal AOSLO image pairs. (a) Split detector AOSLO image. (b) Simultaneously captured confocal AOSLO image from the same location. (c) Probability maps generated from (a) and (b) using the trained LF-DM-CNN. (d) Extended maxima of (c). (e-f) Detected cones marked in green on the split detector image shown in (a) and on the confocal image shown in (b).
2.5 Validation and comparison to the state-of-the-art
We validated our method against the current gold-standard of manual grading and alternative automatic cone detection methods. We used leave-one-subject-out cross validation to evaluate our method, which means that for each subject, all images from the other subjects were used for training the network and cone localization parameters, and all images from that subject were used as the validation data set. Thus, there was no overlap between subjects or images used for training and testing of the algorithm. The first set of manual markings by the more experienced grader (DC) was used for training.
For comparison to the state-of-the-art cone detection methods, we first evaluated the performance of Bergeles et al. [48], which was designed for detecting cones in split detector images and tested on subjects with Stargardt disease. We validated this algorithm across the entire split detector data set. We horizontally flipped the split detector images to match the orientation used in [48], and flipped the detected cone coordinates back to the original orientation of the images. The parameters for diseased images in their software were used. We also evaluated the software developed in Cunefare et al. [46] using the trained networks and optimization parameters learned from healthy split detector (SD-CNN) and confocal (C-CNN) AOSLO images exactly as reported in [46] across our split detector and confocal data sets, respectively. Additionally, we evaluated the performance of Cunefare et al. [46] after training new networks and parameters on the current ACHM split detector and confocal images (SD-CNN-ACHM and C-CNN-ACHM) using leave-one-subject-out cross validation.
To quantify the performance of the different methods, we first matched the automatically detected cones to the cones marked by the first grader one-to-one for each image pair in a similar fashion to Cunefare et al. [46]. To summarize, an automatic cone was considered a true positive if it was located within some distance d of a manually marked cone. The value d was set to the smaller between 0.75 of the median spacing between manually marked cones in the image and 8 μm. The upper limit was used to account for images with sparse cone mosaics due to disease, and was chosen to be smaller than the maximum value of d found in healthy eyes in [46]. Automatically detected cones that were not matched to a manually marked cone were considered false positives, and manually marked cones that did not have a matching automatically detected cone were considered false negatives. In the case that a manually marked cone matched to more than one automatically detected cone, only the automatically marked cone with the smallest distance to the manually marked cone was considered a true positive, and the remaining were considered false positives. To remove border artifacts, we did not analyze marked cones within 7 pixels (3.5 μm) of the edges of the images. After matching, for each image pair the number of automatically marked cones (NAutomatic) and manually marked cones (NManual) can then be expressed as:
where NTP is the number of true positives, NFP is the number of false positives, and NFN is the number of false negatives. For each image pair, we then calculated the true positive rate, false discovery rate, and Dice’s coefficient [75, 76] as: The second set of manual markings (SB) was compared to the first set of manual markings in the same way to assess inter-observer variability.3. Results
Figure 7 shows a representative example of each automated method tested as well as the second set of manual markings in comparison to the first set of manual markings. In the marked images, automatically detected cones that were matched to a manually marked cone (true positives) are shown in green, cones missed by the automatic algorithm (false negatives) are shown in cyan, and automatically detected cones with no corresponding manually marked cone (false positive) are shown in red. Figure 8 displays examples of the performance of the single modality Cunefare et al. [46] method with the SD-CNN-ACHM and our proposed method using the dual–mode LF-DM-CNN architecture. Instances where the LF-DM-CNN, which uses multimodal information, correctly marks ambiguous locations in the split detector image where the single mode SD-CNN-ACHM method does not are indicated by orange arrows.
Fig. 7 Performance of the automated cone detection algorithms on an ACHM image pair. (a) Split detector AOSLO image. (b) Simultaneously captured confocal AOSLO image from the same location. (c-i) Comparison to the first manual markings (with Dice’s coefficients) for (c) the second manual markings (0.915), (d) Bergeles et al. [48] (0.667), (e) C-CNN [46] (0.178), (f) SD-CNN [46] (0.800), (g) C-CNN-ACHM (0.835), (h) SD-CNN-ACHM (0.907), and (i) our proposed method using the LF-DM-CNN network (0.932). Green points denote true positives, cyan denotes false negatives, and red denotes false positives.
Fig. 8 Comparison of our dual-mode method to the single-mode Cunefare et al. [46] method with the SD-CNN-ACHM. Split detector AOSLO images from different subjects with ACHM are shown in the top row, and the corresponding simultaneously captured confocal AOSLO images are shown in the row second from the top. Comparisons to the first manual markings for the single-mode SD-CNN-ACHM are shown in the second row from the bottom, and our method using the dual-mode LF-DM-CNN are shown in the bottom row. Green points denote true positives, cyan denotes false negatives, and red denotes false positives. Orange arrows point to ambiguous locations in the split detector images. Dice’s coefficients for the SD-CNN-ACHM are 0.914 in (a), 0.867 in (b), and 0.815 in (c). Dice’s coefficients for the LF-DM-CNN are 0.986 in (a), 0.929 in (b), and 0.897 in (c).
Table 1 summarizes the performance of the automated methods in comparison to the first (more experienced) manual grader, as well as the variability between the two graders over the 200 ACHM image pairs in our data set. A large increase in performance can be seen by training Cunefare et al. [46] on ACHM images before testing. Our proposed method using the LF-DM-CNN architecture had the best performance in terms of Dice’s coefficient. C-CNN and SD-CNN had higher true positive rates at the cost of substantially worse false discovery rates.
Table 1. Average performance of automatic methods and second manual marking with respect to the first manual marking across the data set (standard deviations shown in parenthesis).
We implemented and ran all methods in MATLAB 2016b (The MathWorks, Natick, MA) with MatConvNet [66] 1.0-beta23 for training and evaluating the CNN’s. We ran all experiments on a desktop PC with an i7-5930K CPU at 3.5 GHz, 64 GB of RAM, and a GeForce GTX TITAN X GPU. The average run time of our method for cone detection after training was 21.0 seconds using the LF-DM-CNN on 200 × 200 pixel image pairs. The average run times for Cunefare et al. [46] with the C-CNN, SD-CNN, C-CNN-ACHM, and SD-CNN-ACHM, and the method of Bergeles et al. [48] were 10.2, 10.4, 10.2, 10.0, and 10.2 seconds, respectively. The total training time (including training the network and learning optimization parameters) for the LF-DM-CNN was under 3 hours. The mean values and standard deviations of the automatically chosen cone detection parameters σ, H, and T across the validation groups for our method were 1.69 ± 0.25, 0.07 ± 0.03, and 0.77 ± 0.07.
4. Discussion
We developed an automatic CNN based method that incorporates both confocal and non-confocal split detector AOSLO images to detect cone photoreceptors in challenging but representative ACHM images. We showed that performance of this method that incorporates dual-mode information in its CNN architecture outperformed a similar method using only a single-mode CNN architecture, and showed that our method had good agreement with the current gold standard of manual grading. The method was tested on a set of images taken from 16 different subjects and at a range of retinal eccentricities.
The proposed LF-DM-CNN architecture outperformed the other methods tested, including the state-of-the-art Bergeles et al. [48] method. It should be noted that the parameters of [48] were set experimentally based on Stargardt images. The performance might be improved by optimizing the parameters for ACHM images. This highlights the utility of a method that automatically learns features and parameters from training data, such as our CNN method, instead of requiring manual adjustment. Our method also had comparable performance to a second set of manual markings when the first set of manual markings was considered to be the gold-standard.
Additionally, Table 1 shows that our method using a dual-mode architecture, LF-DM-CNN, outperformed the other methods which only take advantage of a single imaging modality. The C-CNN and SD-CNN [46] had higher true positive rates but much worse false discovery rates. This is likely due to these networks being trained on healthy eyes with more regular and higher density cone mosaics, and substantially different features in the case of the confocal images. The SD-CNN-ACHM greatly outperformed the C-CNN-ACHM, which might suggest that split detector images provide more information than the confocal images in determining cone locations. Additionally, for the final fully connected layer of the LF-DM-CNN which combines the information from the two modalities, 56.4% of the weight magnitude was associated with the split detector information and 43.6% was associated with the confocal information (averaged across the validation groups). This is not surprising considering that the manual markings were done primarily using the split detector images with the confocal images used to resolve ambiguities, and that split detector AOSLO is able to visualize residual cone structures in ACHM, whereas confocal does not. Even so, from Fig. 1 and Fig. 8 it can be seen that the confocal AOSLO images can provide complimentary information which can be used to resolve ambiguities in the simultaneously captured split detector AOSLO images.
Figures 8(b) and 8(c) show examples of cones correctly detected by the single mode SD-CNN network but missed by the dual-mode LF-DM-CNN network. In these cases, the cone-like structures are more prominent in the split detector images than in the confocal images. Thus, the single mode SD-CNN is more sensitive to these features than the LF-DM-CNN method. Indeed, as shown in Table 1, the LF-DM-CNN is overall more reliable for detecting cones.
A limitation of our study is that our method was only trained and tested on images of ACHM. However, our CNN based deep learning method learns features directly from training data, which should allow our method to accurately detect cones in other disease cases by simply changing the training data. This is supported by the fact that our similar previous single-mode method ([46]), which was originally tested on healthy images, performed well on ACHM split detector images by only changing the training data set. Further, we expect that this algorithm can be easily modified for detection of photoreceptors on alternative high-resolution imaging techniques such as computational AO [77], and en face AO optical coherence tomography images of photoreceptors, which are similar in structure to AOSLO images [78], as well as for other retinal neurons such as ganglion cells which have recently been visualized with ophthalmic AO technology [30, 79].
Funding
Foundation Fighting Blindness (BR-CL-0616-0703-Duke); unrestricted grant from Research to Prevent Blindness to Duke University and Stanford University; National Institutes of Health (NIH) (R21EY027086, P30EY005722, R01EY025231, U01EY025477, R01EY017607, P30EY001931, P30EY026877 and T32EB001040).
Acknowledgment
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Disclosures
The authors declare that there are no conflicts of interest related to this article.
References and links
1. J. Carroll, M. Neitz, H. Hofer, J. Neitz, and D. R. Williams, “Functional photoreceptor loss revealed with adaptive optics: An alternate cause of color blindness,” Proc. Natl. Acad. Sci. U.S.A. 101(22), 8461–8466 (2004). [CrossRef] [PubMed]
2. K. M. Litts, R. F. Cooper, J. L. Duncan, and J. Carroll, “Photoreceptor-based biomarkers in AOSLO retinal imaging,” Invest. Ophthalmol. Vis. Sci. 58(6), BIO255 (2017). [CrossRef] [PubMed]
3. K. E. Stepien, W. M. Martinez, A. M. Dubis, R. F. Cooper, A. Dubra, and J. Carroll, “Subclinical photoreceptor disruption in response to severe head trauma,” Arch. Ophthalmol. 130(3), 400–402 (2012). [CrossRef] [PubMed]
4. A. Roorda and D. R. Williams, “The arrangement of the three cone classes in the living human eye,” Nature 397(6719), 520–522 (1999). [CrossRef] [PubMed]
5. A. Roorda, F. Romero-Borja, W. Donnelly Iii, H. Queener, T. Hebert, and M. Campbell, “Adaptive optics scanning laser ophthalmoscopy,” Opt. Express 10(9), 405–412 (2002). [CrossRef] [PubMed]
6. R. J. Zawadzki, S. M. Jones, S. S. Olivier, M. Zhao, B. A. Bower, J. A. Izatt, S. Choi, S. Laut, and J. S. Werner, “Adaptive-optics optical coherence tomography for high-resolution and high-speed 3D retinal in vivo imaging,” Opt. Express 13(21), 8532–8546 (2005). [CrossRef] [PubMed]
7. D. Merino, C. Dainty, A. Bradu, and A. G. Podoleanu, “Adaptive optics enhanced simultaneous en-face optical coherence tomography and scanning laser ophthalmoscopy,” Opt. Express 14(8), 3345–3353 (2006). [CrossRef] [PubMed]
8. C. Torti, B. Považay, B. Hofer, A. Unterhuber, J. Carroll, P. K. Ahnelt, and W. Drexler, “Adaptive optics optical coherence tomography at 120,000 depth scans/s for non-invasive cellular phenotyping of the living human retina,” Opt. Express 17(22), 19382–19400 (2009). [CrossRef] [PubMed]
9. R. D. Ferguson, Z. Zhong, D. X. Hammer, M. Mujat, A. H. Patel, C. Deng, W. Zou, and S. A. Burns, “Adaptive optics scanning laser ophthalmoscope with integrated wide-field retinal imaging and tracking,” J. Opt. Soc. Am. A 27(11), A265–A277 (2010). [CrossRef] [PubMed]
10. A. Dubra and Y. Sulai, “Reflective afocal broadband adaptive optics scanning ophthalmoscope,” Biomed. Opt. Express 2(6), 1757–1768 (2011). [CrossRef] [PubMed]
11. R. S. Jonnal, O. P. Kocaoglu, Q. Wang, S. Lee, and D. T. Miller, “Phase-sensitive imaging of the outer retina using optical coherence tomography and adaptive optics,” Biomed. Opt. Express 3(1), 104–124 (2012). [CrossRef] [PubMed]
12. M. Laslandes, M. Salas, C. K. Hitzenberger, and M. Pircher, “Increasing the field of view of adaptive optics scanning laser ophthalmoscopy,” Biomed. Opt. Express 8(11), 4811–4826 (2017). [CrossRef] [PubMed]
13. A. Roorda and D. R. Williams, “Optical fiber properties of individual human cones,” J. Vis. 2(5), 404–412 (2002). [CrossRef] [PubMed]
14. Y. Kitaguchi, K. Bessho, T. Yamaguchi, N. Nakazawa, T. Mihashi, and T. Fujikado, “In vivo measurements of cone photoreceptor spacing in myopic eyes from images obtained by an adaptive optics fundus camera,” Jpn. J. Ophthalmol. 51(6), 456–461 (2007). [CrossRef] [PubMed]
15. T. Y. Chui, H. Song, and S. A. Burns, “Adaptive-optics imaging of human cone photoreceptor distribution,” J. Opt. Soc. Am. A 25(12), 3021–3029 (2008). [CrossRef] [PubMed]
16. A. Dubra, Y. Sulai, J. L. Norris, R. F. Cooper, A. M. Dubis, D. R. Williams, and J. Carroll, “Noninvasive imaging of the human rod photoreceptor mosaic using a confocal adaptive optics scanning ophthalmoscope,” Biomed. Opt. Express 2(7), 1864–1876 (2011). [CrossRef] [PubMed]
17. M. Pircher, J. S. Kroisamer, F. Felberer, H. Sattmann, E. Götzinger, and C. K. Hitzenberger, “Temporal changes of human cone photoreceptors observed in vivo with SLO/OCT,” Biomed. Opt. Express 2(1), 100–112 (2010). [CrossRef] [PubMed]
18. O. P. Kocaoglu, S. Lee, R. S. Jonnal, Q. Wang, A. E. Herde, J. C. Derby, W. Gao, and D. T. Miller, “Imaging cone photoreceptors in three dimensions and in time using ultrahigh resolution optical coherence tomography with adaptive optics,” Biomed. Opt. Express 2(4), 748–763 (2011). [CrossRef] [PubMed]
19. M. Lombardo, S. Serrao, and G. Lombardo, “Technical factors influencing cone packing density estimates in adaptive optics flood illuminated retinal images,” PLoS One 9(9), e107402 (2014). [CrossRef] [PubMed]
20. S. S. Choi, N. Doble, J. L. Hardy, S. M. Jones, J. L. Keltner, S. S. Olivier, and J. S. Werner, “In vivo imaging of the photoreceptor mosaic in retinal dystrophies and correlations with visual function,” Invest. Ophthalmol. Vis. Sci. 47(5), 2080–2092 (2006). [CrossRef] [PubMed]
21. J. L. Duncan, Y. Zhang, J. Gandhi, C. Nakanishi, M. Othman, K. E. H. Branham, A. Swaroop, and A. Roorda, “High-resolution imaging with adaptive optics in patients with inherited retinal degeneration,” Invest. Ophthalmol. Vis. Sci. 48(7), 3283–3291 (2007). [CrossRef] [PubMed]
22. S. S. Choi, R. J. Zawadzki, M. A. Greiner, J. S. Werner, and J. L. Keltner, “Fourier-domain optical coherence tomography and adaptive optics reveal nerve fiber layer loss and photoreceptor changes in a patient with optic nerve drusen,” J. Neuroophthalmol. 28(2), 120–125 (2008). [CrossRef] [PubMed]
23. S. Ooto, M. Hangai, A. Sakamoto, A. Tsujikawa, K. Yamashiro, Y. Ojima, Y. Yamada, H. Mukai, S. Oshima, T. Inoue, and N. Yoshimura, “High-resolution imaging of resolved central serous chorioretinopathy using adaptive optics scanning laser ophthalmoscopy,” Ophthalmology 117(9), 1800–1809 (2010). [CrossRef] [PubMed]
24. D. Merino, J. L. Duncan, P. Tiruveedhula, and A. Roorda, “Observation of cone and rod photoreceptors in normal subjects and patients using a new generation adaptive optics scanning laser ophthalmoscope,” Biomed. Opt. Express 2(8), 2189–2201 (2011). [CrossRef] [PubMed]
25. Y. Kitaguchi, S. Kusaka, T. Yamaguchi, T. Mihashi, and T. Fujikado, “Detection of photoreceptor disruption by adaptive optics fundus imaging and fourier-domain optical coherence tomography in eyes with occult macular dystrophy,” Clin. Ophthalmol. 5, 345–351 (2011). [CrossRef] [PubMed]
26. J. Lammer, S. G. Prager, M. C. Cheney, A. Ahmed, S. H. Radwan, S. A. Burns, P. S. Silva, and J. K. Sun, “Cone photoreceptor irregularity on adaptive optics scanning laser ophthalmoscopy correlates with severity of diabetic retinopathy and macular edemacone mosaic irregularity in diabetic eyes on AOSLO,” Invest. Ophthalmol. Vis. Sci. 57(15), 6624–6632 (2016). [CrossRef] [PubMed]
27. T. Y. P. Chui, D. A. Vannasdale, and S. A. Burns, “The use of forward scatter to improve retinal vascular imaging with an adaptive optics scanning laser ophthalmoscope,” Biomed. Opt. Express 3(10), 2537–2549 (2012). [CrossRef] [PubMed]
28. D. Scoles, Y. N. Sulai, and A. Dubra, “In vivo dark-field imaging of the retinal pigment epithelium cell mosaic,” Biomed. Opt. Express 4(9), 1710–1723 (2013). [CrossRef] [PubMed]
29. D. Scoles, Y. N. Sulai, C. S. Langlo, G. A. Fishman, C. A. Curcio, J. Carroll, and A. Dubra, “In vivo imaging of human cone photoreceptor inner segments,” Invest. Ophthalmol. Vis. Sci. 55(7), 4244–4251 (2014). [CrossRef] [PubMed]
30. E. A. Rossi, C. E. Granger, R. Sharma, Q. Yang, K. Saito, C. Schwarz, S. Walters, K. Nozato, J. Zhang, T. Kawakami, W. Fischer, L. R. Latchney, J. J. Hunter, M. M. Chung, and D. R. Williams, “Imaging individual neurons in the retinal ganglion cell layer of the living eye,” Proc. Natl. Acad. Sci. U.S.A. 114(3), 586–591 (2017). [CrossRef] [PubMed]
31. K. A. Sapoznik, T. Luo, A. de Castro, L. Sawides, R. L. Warner, and S. A. Burns, “Enhanced retinal vasculature imaging with a rapidly configurable aperture,” Biomed. Opt. Express 9(3), 1323–1333 (2018). [CrossRef] [PubMed]
32. A. Roorda and J. L. Duncan, “Adaptive optics ophthalmoscopy,” Annu Rev Vis Sci 1(1), 19–50 (2015). [CrossRef] [PubMed]
33. D. Cunefare, R. F. Cooper, B. Higgins, D. F. Katz, A. Dubra, J. Carroll, and S. Farsiu, “Automatic detection of cone photoreceptors in split detector adaptive optics scanning light ophthalmoscope images,” Biomed. Opt. Express 7(5), 2036–2050 (2016). [CrossRef] [PubMed]
34. M. A. Abozaid, C. S. Langlo, A. M. Dubis, M. Michaelides, S. Tarima, and J. Carroll, “Reliability and repeatability of cone density measurements in patients with congenital achromatopsia,” in Advances in Experimental Medicine and Biology, C. Bowes Rickman, M. M. LaVail, R. E. Anderson, C. Grimm, J. Hollyfield, and J. Ash, eds. (Springer International Publishing, 2016), pp. 277–283.
35. K. Y. Li and A. Roorda, “Automated identification of cone photoreceptors in adaptive optics retinal images,” J. Opt. Soc. Am. A 24(5), 1358–1363 (2007). [CrossRef] [PubMed]
36. B. Xue, S. S. Choi, N. Doble, and J. S. Werner, “Photoreceptor counting and montaging of en-face retinal images from an adaptive optics fundus camera,” J. Opt. Soc. Am. A 24(5), 1364–1372 (2007). [CrossRef] [PubMed]
37. D. H. Wojtas, B. Wu, P. K. Ahnelt, P. J. Bones, and R. P. Millane, “Automated analysis of differential interference contrast microscopy images of the foveal cone mosaic,” J. Opt. Soc. Am. A 25(5), 1181–1189 (2008). [CrossRef] [PubMed]
38. A. Turpin, P. Morrow, B. Scotney, R. Anderson, and C. Wolsley, “Automated identification of photoreceptor cones using multi-scale modelling and normalized cross-correlation,” in Image analysis and processing – ICIAP 2011, G. Maino and G. Foresti, eds. (Springer Berlin Heidelberg, 2011), pp. 494–503.
39. R. Garrioch, C. Langlo, A. M. Dubis, R. F. Cooper, A. Dubra, and J. Carroll, “Repeatability of in vivo parafoveal cone density and spacing measurements,” Optom. Vis. Sci. 89(5), 632–643 (2012). [CrossRef] [PubMed]
40. F. Mohammad, R. Ansari, J. Wanek, and M. Shahidi, “Frequency-based local content adaptive filtering algorithm for automated photoreceptor cell density quantification,” in Proceedings of IEEE International Conference on Image Processing, (IEEE, 2012), 2325–2328. [CrossRef]
41. S. J. Chiu, C. A. Toth, C. Bowes Rickman, J. A. Izatt, and S. Farsiu, “Automatic segmentation of closed-contour features in ophthalmic images using graph theory and dynamic programming,” Biomed. Opt. Express 3(5), 1127–1140 (2012). [CrossRef] [PubMed]
42. S. J. Chiu, Y. Lokhnygina, A. M. Dubis, A. Dubra, J. Carroll, J. A. Izatt, and S. Farsiu, “Automatic cone photoreceptor segmentation using graph theory and dynamic programming,” Biomed. Opt. Express 4(6), 924–937 (2013). [CrossRef] [PubMed]
43. R. F. Cooper, C. S. Langlo, A. Dubra, and J. Carroll, “Automatic detection of modal spacing (Yellott’s ring) in adaptive optics scanning light ophthalmoscope images,” Ophthalmic Physiol. Opt. 33(4), 540–549 (2013). [CrossRef] [PubMed]
44. L. Mariotti and N. Devaney, “Performance analysis of cone detection algorithms,” J. Opt. Soc. Am. A 32(4), 497–506 (2015). [CrossRef] [PubMed]
45. D. M. Bukowska, A. L. Chew, E. Huynh, I. Kashani, S. L. Wan, P. M. Wan, and F. K. Chen, “Semi-automated identification of cones in the human retina using circle Hough transform,” Biomed. Opt. Express 6(12), 4676–4693 (2015). [CrossRef] [PubMed]
46. D. Cunefare, L. Fang, R. F. Cooper, A. Dubra, J. Carroll, and S. Farsiu, “Open source software for automatic detection of cone photoreceptors in adaptive optics ophthalmoscopy using convolutional neural networks,” Sci. Rep. 7(1), 6620 (2017). [CrossRef] [PubMed]
47. J. Liu, H. Jung, A. Dubra, and J. Tam, “Automated photoreceptor cell identification on nonconfocal adaptive optics images using multiscale circular voting,” Invest. Ophthalmol. Vis. Sci. 58(11), 4477–4489 (2017). [CrossRef] [PubMed]
48. C. Bergeles, A. M. Dubis, B. Davidson, M. Kasilian, A. Kalitzeos, J. Carroll, A. Dubra, M. Michaelides, and S. Ourselin, “Unsupervised identification of cone photoreceptors in non-confocal adaptive optics scanning light ophthalmoscope images,” Biomed. Opt. Express 8(6), 3081–3094 (2017). [CrossRef] [PubMed]
49. A. M. Dubis, R. F. Cooper, J. Aboshiha, C. S. Langlo, V. Sundaram, B. Liu, F. Collison, G. A. Fishman, A. T. Moore, A. R. Webster, A. Dubra, J. Carroll, and M. Michaelides, “Genotype-dependent variability in residual cone structure in achromatopsia: Toward developing metrics for assessing cone health,” Invest. Ophthalmol. Vis. Sci. 55(11), 7303–7311 (2014). [CrossRef] [PubMed]
50. S. Kohl, D. Zobor, W.-C. Chiang, N. Weisschuh, J. Staller, I. Gonzalez Menendez, S. Chang, S. C. Beck, M. Garcia Garrido, V. Sothilingam, M. W. Seeliger, F. Stanzial, F. Benedicenti, F. Inzana, E. Héon, A. Vincent, J. Beis, T. M. Strom, G. Rudolph, S. Roosing, A. I. Hollander, F. P. M. Cremers, I. Lopez, H. Ren, A. T. Moore, A. R. Webster, M. Michaelides, R. K. Koenekoop, E. Zrenner, R. J. Kaufman, S. H. Tsang, B. Wissinger, and J. H. Lin, “Mutations in the unfolded protein response regulator ATF6 cause the cone dysfunction disorder achromatopsia,” Nat. Genet. 47(7), 757–765 (2015). [CrossRef] [PubMed]
51. M. A. Genead, G. A. Fishman, J. Rha, A. M. Dubis, D. M. O. Bonci, A. Dubra, E. M. Stone, M. Neitz, and J. Carroll, “Photoreceptor structure and function in patients with congenital achromatopsia,” Invest. Ophthalmol. Vis. Sci. 52(10), 7298–7308 (2011). [CrossRef] [PubMed]
52. C. S. Langlo, E. J. Patterson, B. P. Higgins, P. Summerfelt, M. M. Razeen, L. R. Erker, M. Parker, F. T. Collison, G. A. Fishman, C. N. Kay, J. Zhang, R. G. Weleber, P. Yang, D. J. Wilson, M. E. Pennesi, B. L. Lam, J. Chiang, J. D. Chulay, A. Dubra, W. W. Hauswirth, and J. Carroll, “Residual foveal cone structure in cngb3-associated achromatopsia,” Invest. Ophthalmol. Vis. Sci. 57(10), 3984–3995 (2016). [CrossRef] [PubMed]
53. M. Chen, R. F. Cooper, G. K. Han, J. Gee, D. H. Brainard, and J. I. W. Morgan, “Multi-modal automatic montaging of adaptive optics retinal images,” Biomed. Opt. Express 7(12), 4899–4918 (2016). [CrossRef] [PubMed]
54. P. Liskowski and K. Krawiec, “Segmenting retinal blood vessels with deep neural networks,” IEEE Trans. Med. Imaging 35(11), 2369–2380 (2016). [CrossRef] [PubMed]
55. Q. Li, B. Feng, L. Xie, P. Liang, H. Zhang, and T. Wang, “A cross-modality learning approach for vessel segmentation in retinal images,” IEEE Trans. Med. Imaging 35(1), 109–118 (2016). [CrossRef] [PubMed]
56. H. Fu, Y. Xu, S. Lin, D. W. Kee Wong, and J. Liu, “Deepvessel: Retinal vessel segmentation via deep learning and conditional random field,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer International Publishing, 2016), 132–139. [CrossRef]
57. V. Gulshan, L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, S. Venugopalan, K. Widner, T. Madams, J. Cuadros, R. Kim, R. Raman, P. C. Nelson, J. L. Mega, and D. R. Webster, “Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs,” JAMA 316(22), 2402–2410 (2016). [CrossRef] [PubMed]
58. M. J. van Grinsven, B. van Ginneken, C. B. Hoyng, T. Theelen, and C. I. Sánchez, “Fast convolutional neural network training using selective data sampling: Application to hemorrhage detection in color fundus images,” IEEE Trans. Med. Imaging 35(5), 1273–1284 (2016). [CrossRef] [PubMed]
59. M. D. Abràmoff, Y. Lou, A. Erginay, W. Clarida, R. Amelon, J. C. Folk, and M. Niemeijer, “Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning,” Invest. Ophthalmol. Vis. Sci. 57(13), 5200–5206 (2016). [CrossRef] [PubMed]
60. S. P. K. Karri, D. Chakraborty, and J. Chatterjee, “Transfer learning based classification of optical coherence tomography images with diabetic macular edema and dry age-related macular degeneration,” Biomed. Opt. Express 8(2), 579–592 (2017). [CrossRef] [PubMed]
61. L. Fang, D. Cunefare, C. Wang, R. H. Guymer, S. Li, and S. Farsiu, “Automatic segmentation of nine retinal layer boundaries in OCT images of non-exudative AMD patients using deep learning and graph search,” Biomed. Opt. Express 8(5), 2732–2744 (2017). [CrossRef] [PubMed]
62. X. Fei, J. Zhao, H. Zhao, D. Yun, and Y. Zhang, “Deblurring adaptive optics retinal images using deep convolutional neural networks,” Biomed. Opt. Express 8(12), 5675–5687 (2017). [CrossRef] [PubMed]
63. A. G. Roy, S. Conjeti, S. P. K. Karri, D. Sheet, A. Katouzian, C. Wachinger, and N. Navab, “ReLayNet: retinal layer and fluid segmentation of macular optical coherence tomography using fully convolutional networks,” Biomed. Opt. Express 8(8), 3627–3642 (2017). [CrossRef] [PubMed]
64. A. Dubra and Z. Harvey, “Registration of 2D images from fast scanning ophthalmic instruments,” in Biomedical Image Registration (Springer Berlin Heidelberg, 2010), 60–71.
65. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, (Springer, 2012), 1097–1105.
66. A. Vedaldi and K. Lenc, “Matconvnet: Convolutional neural networks for matlab,” in Proceedings of the 23rd ACM international conference on Multimedia, (ACM, Brisbane, Australia, 2015), pp. 689–692. [CrossRef]
67. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, (IEEE, 2014), 1725–1732. [CrossRef]
68. A. Eitel, J. T. Springenberg, L. Spinello, M. Riedmiller, and W. Burgard, “Multimodal deep learning for robust RGB-D object recognition,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2015), 681–687. [CrossRef]
69. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proceedings of the International Conference on Machine Learning, (2015), 448–456.
70. V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), (IEEE, 2010), 807–814.
71. K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun, “What is the best multi-stage architecture for object recognition?” in IEEE 12th International Conference on Computer Vision, (IEEE, 2009), 2146–2153. [CrossRef]
72. C. M. Bishop, Pattern Recognition and Machine Learning (Springer, 2006).
73. J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Netw. 61(10), 85–117 (2015). [CrossRef] [PubMed]
74. P. Soille, Morphological Image Analysis: Principles and Applications (Springer Science & Business Media, 2013).
75. L. R. Dice, “Measures of the amount of ecologic association between species,” Ecology 26(3), 297–302 (1945). [CrossRef]
76. T. Sørensen, “A method of establishing groups of equal amplitude in plant sociology based on similarity of species and its application to analyses of the vegetation on danish commons,” Biol. Skr. 5(1), 1–34 (1948).
77. N. D. Shemonski, F. A. South, Y.-Z. Liu, S. G. Adie, P. S. Carney, and S. A. Boppart, “Computational high-resolution optical imaging of the living human retina,” Nat. Photonics 9(7), 440–443 (2015). [CrossRef] [PubMed]
78. M. Pircher and R. J. Zawadzki, “Review of adaptive optics OCT (AO-OCT): principles and applications for retinal imaging [Invited],” Biomed. Opt. Express 8(5), 2536–2562 (2017). [CrossRef] [PubMed]
79. Z. Liu, K. Kurokawa, F. Zhang, J. J. Lee, and D. T. Miller, “Imaging and quantifying ganglion cells and other transparent neurons in the living human retina,” Proc. Natl. Acad. Sci. U.S.A. 114(48), 12803–12808 (2017). [CrossRef] [PubMed]
