Open Access
Add to BibSonomy
Abstract
Hyperspectral imaging is an emerging technique that allows measurement of spectral absorption at each point of a scene, thus offering the capability to identify and characterize important biomarkers for clinical practice and therapeutic research, as well as enhancing image identification of important structures. So far, few hyperspectral cameras have been used for retinal scanning because of the need to acquire the image in a fraction of a second. Here, and to the best of our knowledge, we present a novel concept of a snapshot hyperspectral camera suited for retinal imaging. We demonstrate the technique by presenting the optical density spectrum of a healthy patient’s retina in the 450–700 nm range, together with the spectral response of several retinal features.
© 2024 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
The retina is a central part of the vision. As an extension of the brain and part of the of the central nervous system (CNS) [1], the retina provides an excellent opportunity for multi and hyperspectral imaging to gather information on diseases that affect the CNS as well as providing a direct access to the retinal blood vessels [2]. A multispectral imaging system displays a discrete set of wavebands while hyperspectral displays an effectively continuous set of wavebands. Such systems have been widely used in dermatology where they have improved the ability to diagnose skin cancer [3].
However feasible in theory, a hyperspectral camera for ophthalmology usage needs to be quick in order to be practically useful [4–6]. Acquiring the image needs to be fast, ideally in less than 50 ms to avoid eye motion. It cannot be more than 200 ms to avoid the effect of the pupillary reflex and allow imaging without mydriatics. Furthermore, the amount of light entering the eye needs to be limited not to cause damage to the retina.
Several snapshot hyperspectral imaging systems have been developed over the year such as computed tomographic imaging spectrometer [7] or coded aperture snapshot spectral imagers (CASSI) [8]. These methods requires to solve an inverse problem which can introduce artefact in the images. Other emerging methods such as dual comb imaging could provide hyperspectral images with an acquisition time compatible with ophthalmology [9].
The organization IMEC (Kapeldreef 7, B-3001 Leuven, Belgium) has developed a snapshot hyperspectral camera that provides another approach for snapshot hyperspectral imaging; it uses mosaic filter to generate a hyperspectral image with 16 wavebands [4,10].
The company Cubert (Science Park II Lise-Meitner Strasse 8/1 D-89081, Ulm) has developed a snapshot hyperspectral camera but to the best of our knowledge it has only been used for dermatology applications [11].
It is worth noting that the company Optina diagnostic (Montreal, H4T 1Z2 QC, Canada) uses a scanning technique where different wavelengths are successively used to illuminate the eye [12]. This technology takes such a long time that it requires to either limit the number of bands scanned or to use of mydriatic drops, which limits its potential use outside of ophthalmology clinics and hospitals.
Here we present a novel concept of snapshot hyperspectral imaging based on diffraction and light field cameras [13], that requires a simple calibration to retrieve the hyperspectrum instead of solving an inverse problem. It allows to image a retina in the visible range with 150x150 pixels over a 30 degree field of view, and 36 bands with 12 nm spectral resolution and 7 nm sampling in the 450-700 nm spectral region.
2. Principle
2.1 Hyperspectral imaging system
Figures 1 and 2 show the perspective and side view of the hyperspectral camera concept. The scene (here the retina) is illuminated and then, the scattered reflected rays are collected by a first lens assembly. This assembly is composed of two lenses with matching focal points where an iris is located. This assembly ensures that the light emerging from it is collimated before it enters the next part of the instrument.
Then a stack of a transmission diffraction grating and a microlens array diffracts and focuses the light onto a CMOS sensor which collects the light. Each of the microlens constitute together with the grating a miniature spectrometer that analyses a specific region of the image [14,15]. In order to avoid interferences between the different orders of adjacent lenslets, the grating is rotated by 8 degree with respect to the microlens array. What is more, the diffraction grating and microlens array must be placed as close as possible from one another to so that the lenslets do not perform spectral imaging of the entire field of view but only of a singular point, in this work this distance is of 1 mm.
2.2 Theory
To determine the spectral resolution we consider the case where the light incoming onto the grating and lenses is perfectly collimated. In this case, each wavelength is focused onto the CMOS sensor with a focal spot size of $1.22\times \frac {\lambda *f}{D}$ with $\lambda$ the wavelength, f the lens focal length and D the (micro)lens diameter. After the grating, each wavelength is diffracted according to law $\sin \theta = \frac {\lambda }{a}$ where a is the grating period. Two different wavelengths can be resolved if they are separated on the sensor by more than the size of their focal spot, thus:
Spatial resolution. The spatial resolution is determined by the microlenses’ diameter which determines the pixel size of the hypercube. Therefore one must use large camera sensors to get sufficiently high resolution. For this camera we use the XIMEA CB200 sensor.
Light filtering and scaling law. The divergence of the light beam after the second lens is determined by $\frac {d_{iris}}{f_{lens}}$, which is the ratio between the iris diameter and the second lens focal length. This must be smaller than the resolving power of the microlenses in order not to affect the resolution.
2.3 Resolution measurement
The spectral resolution is determined by the spot size generated by a monochromatic wave on the CMOS sensor. It is measured with 2 methods:
- 1. Measuring the focal spot size of the zeroth order. A Gaussian fit is then used to calculate the full width half maximum.
- 2. By illuminating the camera with spectrally filtered light. This was done at 3 wavelengths (450, 532 and 635 nm, all 3 with 10 nm width FWHM). The results are shown in Fig. 3.
Both methods lead to a spectral resolution of $12 \pm 1$ nm.
2.4 Results
For this camera we used a grating of periodicity $1.666\,\mu$m and microlenses of $148\,\mu$m diameter and the MLA had a size of 25x25mm. This should result in principle in hypercubes with the following specifications: 168x168 pixels and 7.5 nm spectral resolution at 550 nm. However, in order to get sufficient amount of light into the camera the iris diameter was increased such that the divergence of the light was above the resolving power of the microlenses. This resulted in a measured spectral resolution of 12 nm (FWHM) over the full 450-700 nm range. What is more, because of limitations in the mount holder for the gratings, the images have 150x150 pixels. The extraction of the hypercube for visualization by the medical professional takes less than 1s.
3. Method
3.1 Participants
The camera was stationed at Sundets Ögonläkare, a privately owned clinic in Helsingborg, Sweden. Healthy subjects without any known retinal disease were recruited either by personal information or by advertisement. After written informed consent, images with optical coherence tomography (OCT) (Topcon Maestro, Japan) were taken of the retina where after images with the hyperspectral camera (Mantis Photonics AB, Sweden) were taken. Only the fundus images provided by the topcon Maestro were used for this work. Tropicamide and phenylephrine 10% was used as mydriatics and given in the beginning of the appointment. This study was approved by the Swedish Ethical Review Authority (DN:2022-04391-01) and performed in accordance with the tenets of the Declaration of Helsinki.
3.2 Hyperspectral camera
The snapshot hyperspectral camera is connected with a C-mount to a TL-230T relay lens mounted on top of a Topcon TRC50-IX fundus camera as shown in Fig. 4. The lens in the camera where chosen such that the hyperspectral camera provides images of a 30x30 degree field-of-view of the retina. The hyperspectral camera spectral response and the transmission through the Topcon fundus camera are calibrated using a spatially and spectrally uniform white target. The spatial resolution was done by measuring the optic disc with the corneal microscope. Comparison with the hyperspectral images of the same eyes showed that the camera provides images with $46 \pm 5\,\mu$m per pixel.
Fig. 4. Scheme of the setup used for the clinical trial. In dark blue in the outline of the Topcon TRC50 camera and in light blue of mantis hyperspectral camera. On top is a photograph of the hyperspectral camera attached to the relay lens adapter.
3.3 Spectral analysis
The optical density spectrum is defined as
with $I_{ret}$ and $I_{ref}$ the measured intensity of the retina and of a white spectral reference.3.4 Macular pigment optical density
The macular pigment is a yellowish deposition in the macula whose lower levels have been associated with risk of developing Age-related Macular Degeneration (AMD). We calculate the Macular Pigment Optical Density (MPOD) using the light absorption at two different wavelengths [16–19]):
3.5 Reconstructed color image
It is possible to reconstruct a color image by combining 3 slices at different wavelengths [20]. Here we combine with equal weights the slices at 485 (blue channel), 527 (green) and 646 nm (red). An outlier removal filter was applied to improve the image quality.
4. Result
4.1 Hyperspectral images
Figure 5 shows a healthy patient’s retina at eight different wavelengths between 450 and 700 nm. The contrast of features like blood vessels changes for different wavelengths.
Fig. 5. Hypercube. Image of the same retina taken in the same acquisition at 8 different wavelengths in the visible range.
4.2 Spectral response
Figure 6 shows the optical density spectrum at 4 different locations in the retina: the optic disk, the macula a vein and an artery. As expected the optic disk (white) has a much lower and flat ODS than the macula (dark). While performing an oxygenation map is out of the scope of this article one clearly notice a bump in the ODS around 5-600 nm, such feature is associated with the spectrum of blood. The difference between the vein and the arteries is caused by the different oxygenation level.
4.3 Macular pigment optical density
In Fig. 7 is shown the result of the MPOD calculation in a young individual (left) and in an old individual (right). As expected the young individual presents a strong peak of macular pigment at the center of the retina while the older individual has a more spread and less dense (about three times less) pigment distribution.
Fig. 7. Image of the Macular Pigment Optical Density overlayed with the corresponding retinal image at 548 nm. a) Young healthy eye. b) Old eye. c) and d) MPOD only.
4.4 Reconstructed color image
In Fig. 8 is shown an image of a retina with dry AMD. The reconstructed image using narrow spectral slices substantially enhances the retinal abnormalities thus easing the diagnostic. This is expected since the spectral slices captured by the hyperspectral camera are spectrally much narrower than the Bayer filters of a color camera. Thus by carefully selecting the wavelengths combined, one can choose the wavelengths where specific features will exhibit the highest contrast. Note that the color image was taken with a Topcon maestro system and thus the optical system is not optically equivalent for both images, this could also affect the image quality and thus the comparison.
Fig. 8. Image of a retina with dry AMD. a) Standard color fundus image and b) reconstructed color image from the 485, 527 and 646 nm slices.
5. Discussion
The hyperspectral camera developed in this work can measure and display hypercubes. This already presents an interest in clinical settings as it can help the practitioner to identify features (vessels, drusen, naevi etc) by selecting the suited wavelength where these features have the highest contrast. Another useful information given by hyperspectral camera is the spectrum at a given location or within a given area. From this spectrum one can determine the chemical composition as in the case of MPOD [16], blood oxygenation [21,22] or protein deposition [12]. Chemical changes can precede and induce structural changes. Therefore, measuring the chemical composition could help detecting diseases before irreversible damages appears thus being important information for preventive medicine. This is particularly true for blood oxygenation which has been shown to be a biomarker in several diseases [2]. Several techniques have been developed to extract this information such as spectral unmixing [23] or spectral angle mapper [24]. The advantage of these techniques is that they use the full spectrum and thus exploit the full potential of hyperspectral imaging. What is more, in the case of blood oxygenation they could overcome the limitation of 2 or 3 wavelengths algorithms [21]. Finally, retinal features can be classified based on their spectrum. Such work has been submitted for naevi [25] and is in preparation for retinal drusen.
Future improvement of the camera will include:
- 1. The use of larger camera sensors to improve the spatial resolution.
- 2. Improved optical components such as aspheric doublet lenses to reduce optical aberrations.
- 3. The use of holographic gratings to reduce stray light.
The presented camera concept is suitable for ophthalmology because it can produce a high spectral range and spectral resolution image in a snapshot. This reduces movement artefacts while producing a rich dataset for both human expert and machine analysis.
In conclusion we have developed a new design very well suited for ophthalmology that will allow measuring the chemical composition of different eye features in the eye. Future improvements are planned that will improve both the spectral and spatial resolution.
Disclosures
JA and DG are co-founders of Mantis Photonics AB and are employed by the company together with MAK. DG is a board member of the company. MS is a board member and a shareholder of Mantis Photonics AB.
Data availability
Data underlying the results presented in this paper are not publicly available because the participants of this study did not give written consent for their data to be shared publicly.
References
1. A. London, I. Benhar, and M. Schwartz, “The retina as a window to the brain—from eye research to CNS disorders,” Nat. Rev. Neurol. 9(1), 44–53 (2013). [CrossRef]
2. E. Stefansson, O. B. Olafsdottir, T. S. Eliasdottir, et al., “Retinal oximetry: metabolic imaging for diseases of the retina and brain,” Prog. Retin Eye Res. 70, 1–22 (2019). [CrossRef]
3. F. Vasefi, N. MacKinnon, and D. Farkas, “Chapter 16. Hyperspectral and multispectral imaging in dermatology,” in Imaging in Dermatology, M. R. Hamblin, P. Avci, and G. K. Gupta, eds. (Academic, 2016), pp. 187–201.
4. S. Lemmens, J. V. Eijgen, K. V. Keer, et al., “Hyperspectral imaging and the retina: worth the wave?” Trans. Vis. Sci. Tech. 9(9), 9 (2020). [CrossRef]
5. G. Lua and B. Fei, “Medical hyperspectral imaging: a review,” J. Biomed. Opt. 19(1), 010901 (2014). [CrossRef]
6. E. R. Reshef, J. B. Miller, and D. G. Vavvas, “Hyperspectral imaging of the retina: a review,” Int. Ophthalmol. Clin. 60(1), 85–96 (2020). [CrossRef]
7. W. R. Johnson, D. W. Wilson, W. Fink, et al., “Snapshot hyperspectral imaging in ophthalmology,” J. Biomed. Opt. 12(1), 014036 (2007). [CrossRef]
8. D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006). [CrossRef]
9. K. Shibuya, T. Minamikawa, Y. Mizutani, et al., “Scan-less hyperspectral dual-comb single-pixel-imaging in both amplitude and phase,” Opt. Express 25(18), 21947–21957 (2017). [CrossRef]
10. “Ximea hyperspectral camera,” https://www.ximea.com/en/usb3-vision-camera/hyperspectral-usb3-cameras-mini.
11. R. Leon, B. Martinez-Vega, H. Fabelo, et al., “Non-invasive skin cancer diagnosis using hyperspectral imaging for in-situ clinical support,” J. Clin. Med. 9(6), 1662 (2020). [CrossRef]
12. X. Hadoux, F. Hui, J. Lim, et al., “Non-invasive in vivo hyperspectral imaging of the retina for potential biomarker use in Alzheimer’s disease,” Nat. Commun. 10(1), 4227 (2019). [CrossRef]
13. N. Hagen, R. T. Kester, L. Gao, et al., “Snapshot advantage: a review of the light collection improvement for parallel high-dimensional measurement systems,” Opt Eng. pp. 1–8 (2012).
14. D. Guenot and O. Lundh, “Optical spectrometer and method for spectrally resolved two-dimensional imaging of an object,” (2022). US Patent 17/623,291.
15. D. Guenot, “An optical spectrometer and a method for spectrally resolved two-dimensional imaging of an object,” (2023). WO2023140771A1.
16. J. Kaluzny, H. Li, W. Liu, et al., “Bayer filter snapshot hyperspectral fundus camera for human retinal imaging,” Curr. Eye Res. 42(4), 629–635 (2017). [CrossRef]
17. C. Creuzot-Garcher, P. Koehrer, C. Picot, et al., “Comparison of two methods to measure macular pigment optical density in healthy subjects,” Invest. Ophthalmol. Vis. Sci. 55(5), 2941–2946 (2014). [CrossRef]
18. A. A. Fawzi, N. Lee, J. H. Acton, et al., “Recovery of macular pigment spectrum in vivo using hyperspectral image analysis,” J. Biomed. Opt. 16(10), 106008 (2011). [CrossRef]
19. D. Christaras, H. Ginis, A. Pennos, et al., “Objective method for measuring the macular pigment optical density in the eye,” Biomed. Opt. Express 10(7), 3572–3583 (2019). [CrossRef]
20. A. C. S. Tan, M. Fleckenstein, S. Schmitz-Valckenberg, et al., “Clinical application of multicolor imaging technology,” Ophthalmologica. 236(1), 8–18 (2016). [CrossRef]
21. L. E. MacKenzie and A. R. Harvey, “Oximetry using multispectral imaging: theory and application,” J. Opt. 20(6), 063501 (2018). [CrossRef]
22. S. Parhl, “Optical absorption of hemoglobin,” Https://omlc.org/spectra/hemoglobin/index.html.
23. A. Merdasa, J. Berggren, K. Tenland, et al., “Oxygen saturation mapping during reconstructive surgery of human forehead flaps with hyperspectral imaging and spectral unmixing,” Microvasc. Res. 150, 104573 (2023). [CrossRef]
24. R. Yuhas, A. Goetz, and J. Boardman, “Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm,” JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop (1992).
25. M. Selvander, J. Alexander, and D. Guenot, “Neavi characterization using hyperspectral imaging, a pilot study,” Current Eye Research (2024).
