Open Access
Add to BibSonomy
Abstract
Computational lensless imaging technology implements the thinning of the optical system by replacing optical imaging with image reconstruction processing. The conventional optical design uses a single coded mask and an image sensor; however, researchers have recently proposed optical designs incorporating multiple stacked coded apertures for multidimensional and wide-field imaging. Here, we investigate the effects of multilayering the coded aperture on the performance of two-dimensional spatial imaging. Through simulations and optical experiments, we demonstrate that multilayering the coded aperture enhances the condition number of the optical system’s transmission matrix and consequently improves the accuracy of image reconstruction in lensless imaging.
© 2024 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. INTRODUCTION
Computational lensless imaging is a type of computational imaging that realizes imaging by combining a coded optical system and image reconstruction processing without a lens [1–4]. The concept is illustrated in Fig. 1. In conventional optical imaging, the object is directly imaged onto the image sensor by the lens, whereas in computational lensless imaging, the object is imaged with optical encoding onto the image sensor by the coded aperture, and it is computationally decoded to reconstruct the image. By eliminating the lens, it is possible to realize a significant thinning of the camera’s optical system. Additionally, by actively using the degrees of freedom of a system that includes optical encoding, computational lensless imaging is also employed to compactly realize three-dimensional (3D) imaging [5,6], multispectral imaging [7,8], polarization imaging [9,10], diffractive neural network [11,12], and privacy-protected imaging [13–15].
Fig. 1. Concept of optical imaging, single-layer lensless imaging, and multilayer lensless imaging. In optical imaging, the object is directly imaged onto the image sensor by the lens. In lensless imaging, the object is imaged with optical encoding onto the image sensor by the coded aperture, and it is computationally decoded to reconstruct the image by solving the inverse problem. Multilayer lensless imaging involves stacked coded apertures in the optical system.
To record the spatial information of an image in an invertible manner, a coded aperture or phase mask is placed in front of the image sensor. In this method, the spatial information incident on the camera is characterized by spatial intensity/phase modulation. Such a coded optical system stabilizes the inverse problem for image reconstruction processing, and it is possible to obtain reconstructed images that are visually recognizable in real environments. Representative examples of coded apertures include the MURA aperture [16], random aperture [17], separable doubly Toeplitz aperture [18], M-sequence aperture [19], Fresnel zone aperture (FZA) [6,20,21], and radial aperture [22], and representative examples of phase masks include multiple scattering medium [23,24], random mirror [25], spiral grating [26], diffuser [5,27,28], multilevel diffractive optical element (DOE) [29], random lens array [30], and polymer fabricated by grayscale lithography [31]. The combination of a single coded element and a single image sensor is the general design of a lensless imaging optical system. In addition, multidimensional information can also be acquired by combining additional physical encoders, such as a rolling-shutter-based image sensor [32], a linear valuable filter [33], and so on.
Recently, research on the multilayering of coded optical elements has been developed to improve the performance of coded imaging systems. As an early work in this technology, Wetzstein et al. proposed a glasses-free 3D display using a refreshable multilayer lensless coded-aperture system and image optimization processing [34]. Recently, not only displays, but also applications to sensing systems have been studied. Zhang et al. proposed a compressive temporal imaging method by multilayering the coded aperture, which is a combination of a dynamic low-resolution device and a static high-resolution element [35]. Arguello et al. proposed an efficient compressive spectral imaging method by combining a DOE and a color-coded aperture in a lensless optical system [36]. Igarashi et al. proposed an imaging method that expands the field of view by multilayering the DOE in a lensless optical system and its optimal design [37]. By multilayering the DOE, the degrees of freedom of the optical design are expanded, and consequently, a field of view impossible with a single layer was implemented in the imaging system.
As mentioned above, the multilayering of coded masks is effective for expanding the degrees of freedom of the optical system and improving the functionalities of coded imaging. However, to the best of our knowledge, it has not yet been clarified whether the multilayering of coded masks itself is effective in improving the performance of two-dimensional (2D) spatial imaging. This study investigates how the imaging performance of a lensless camera changes when the coded aperture is multilayered. Through simulations and optical experiments, we demonstrate that the multilayering of the coded aperture improves the condition number of the matrix that represents the forward model of imaging, and we demonstrate the improvement in imaging performance.
The contributions of our study are as follows:
- • We analyze how the number of layers of coded apertures affects imaging performance, specifically the condition number of the transmission matrix of the optical system and the quality of image reconstruction, in the optical design of lensless cameras under fixed conditions of light-use efficiency and optical system thickness.
- • Through simulations and optical experiments, we quantitatively and experimentally demonstrate that, under conditions where the point spread function (PSF) includes a significant amount of diffraction blur, increasing the number of layers of coded apertures contributes to the improvement of the condition number and the quality of image reconstruction.
2. SYSTEM DESIGN
Image transformation through the propagation of incoherent light through an arbitrary optical system is known to be linear with respect to intensity [38]. Therefore, the optical systems’ measurement process by the optical system can be mathematically represented by a matrix. This matrix is called the transmission matrix (TM) [39,40]. The concepts of an imaging system, a single-layer lensless camera, and a multilayer lensless camera are shown in Fig. 1. The multilayer lensless camera is a new concept proposed to improve the performance of lensless cameras by multilayering the coded aperture. In this section, we describe the optical systems of the lensless camera and the multilayer lensless camera, and the image reconstruction model. The optical design and parameter definitions used in this section for the conventional single-layer lensless camera and multilayer lensless camera are shown on the left and right of Fig. 2, respectively.
A. Lens Camera
In the lens camera, the optical system is composed of a lens and an image sensor, as shown on the left of Fig. 1. The image obtained by the image sensor is ideally the spatial information of the object itself. The imaging process of the imaging system can be modeled as follows when ignoring aberrations:
In the imaging system, the identity matrix is used as the TM ideally; however, optical imaging is necessary. Therefore, at least the thickness of the focal length is required in the optical system. The focal length scales as follows with the effective aperture diameter when the F-number is fixed:
where ${F_{\rm n}}$ is the F-number, $f$ is the focal length, and $\Phi$ is the effective aperture diameter. Therefore, it is theoretically impossible to thin the optical system, while maintaining the aperture diameter in the imaging optical system.B. Lensless Camera
To overcome the thinning limit of the optical system based on the scaling law of the lens camera, a lensless camera was proposed to replace optical imaging with optical encoding and image reconstruction processing [2]. The optical system of the current lensless camera is generally composed of a single coded mask and a single image sensor, as shown at the center of Fig. 1. The coded aperture, which is a type of coded mask, is an optical element that spatially modulates the intensity of light incident in the measurement and functions as an optical encoder of spatial information. The measured coded image is reconstructed into an object image by solving the inverse problem of the measurement using a computer. To solve the inverse problem, it is common to measure the TM using calibration and estimate the unknown object image using the known TM and the measured image. The imaging process of the lensless camera can be modeled as follows:
This model assumes that the PSF is shift invariant and that diffraction blur can be approximated by a 2D Gaussian function. This approximation model is valid and reasonable in that it can simply describe the forward process of measurement, assuming natural light incidence within an ordinary field of view. However, it is important to note that when this assumption does not hold, for instance, when the aim is to accurately model the effects of wide-angle light incidence and/or when it is necessary to model the PSF using the accuracy of wave optics, it is necessary to consider a more accurate forward model that takes into account the angular characteristics of the measurement system and the propagation of light waves.
In lensless cameras, the TM is not required to be the identity matrix; the only requirement is that it is reversible. Therefore, optical imaging is not necessary, and the thickness of the optical system is not limited by the focal length. Consequently, the lensless camera can realize a significant thinning of the optical system.
In lensless cameras, the optical resolution limit (${\delta _{{\rm optical}}}$) is defined by the blur amount of the PSF, which is represented by $\sigma (L)$:
By contrast, the sampling resolution limit (${\delta _{{\rm sampling}}}$) is defined by the sampling interval of sensor $d$ and the magnification factor of the system as follows:
where $L = {z_{{\rm img}}}$ is the thickness of the optical system. Therefore, the two resolutions are in a tradeoff relationship with $L$ [17]. To improve the effective resolution of the measurement, it is desirable to resolve this tradeoff.C. Multilayer Lensless Camera
The multilayer lensless camera discussed in this study is an optical system composed of multiple stacked coded apertures and a single image sensor, as shown on the right of Fig. 1. We compare the single-layer case and multilayer case by fixing the thickness of the optical system as a matter of fairness. The observation process using this optical system can be modeled as follows:
where ${\boldsymbol H} \in {\mathbb{R}^{{N_{\rm g}} \times {N_{\rm f}}}}$ is the TM. In the multilayer lensless optical system, the impulse response is affected by various coded apertures; hence, the submatrix of the TM is generally not a Toeplitz matrix. In the components of the impulse response, the component that shifts most sensitively to the shift of the point light source is the shadow of the most object-side coded aperture. The ratio of these shift speeds corresponds to the magnification of the multilayer lensless camera.To discuss the spatial resolution of the multilayer system, the case is considered in which $K - 1$ coded apertures are inserted at equal intervals inside the single-layer lensless camera with thickness $L$, as shown on the right of Fig. 2. If each coded aperture and image sensor is called a layer, the distance between layers in the camera is divided by $K$; hence, the variance of the Gaussian distribution that represents the diffraction blur function of the most sensor-side coded aperture is $\sigma ({L/K})$. Thus, the upper limit of the sampling resolution is still defined by the shadow of the most object-side aperture; however, the upper limit of the optical resolution is improved by the influence of the shadow of the aperture placed closer to the sensor. Therefore, the multilayer lensless camera can resolve the resolution tradeoff in the single-layer lensless camera, and as a result, it is expected to improve the properties of the matrix.
D. Inverse Problem for Image Reconstruction
To reconstruct the object image from the measured coded image obtained by the lensless optical system, it is necessary to solve the inverse problem of the measurement process formulated by Eq. (9). The most straightforward approach is to measure the matrix ${\boldsymbol H}$ using calibration and solve the inverse problem using its pseudo-inverse matrix. However, this method is unstable against noise; hence, the inverse problem is actually replaced by an error minimization problem as follows:
3. SIMULATION
To investigate the imaging performance of the multilayer lensless camera, we assumed the optical system of the lensless camera shown in Figs. 1 and 2 obtained the TM using ray-tracing simulation and then simulated imaging using the TM. We quantitatively evaluated the performance difference between the single-layer lensless camera and the multilayer lensless camera from the viewpoint of the condition number of the TM and image reconstruction accuracy.
A. Setup
In the simulation, binary random apertures were used as the coded aperture. The pattern of the random aperture was selected as the worst value of the column–vector correlation of the TM of the single-layer lensless optical system after 100 random generations, that is, the minimum incoherence [42]. To evaluate the imaging performance of the single- and multilayer cases, the thickness of the entire optical system $L$ was set to the same value in both cases. In the multilayer case, $K - 1$ random apertures were added to the optical system at equal intervals. For simplicity, the thickness of the aperture was ignored. Additionally, to evaluate the imaging performance, the same light-use efficiency was used, in which the ratio of the total area of the entire coded optical system to the area of the sum of all the small apertures, that is, the aperture ratio, was set to the same value for both optics. In this simulation, the aperture ratio was set to 25%, that is, in the single-layer case, a coded aperture with an aperture ratio of 25% was used, and in the two-layer case, two coded apertures with an aperture ratio of 50% were used. The resolution of the aperture, observed data, and object image was set to $32 \times 32$. The object was simplified to be located on a plane at a distance ${z_{{\rm obj}}}$ from the aperture, and ${z_{{\rm obj}}} = L$ was set. This corresponds to imaging at unity magnification.
B. Ray Tracing for Simulation
First, the TM was obtained using ray tracing simulation for the designed optical system. Ray tracing was performed as shown in Algorithm 1. In the algorithm, ${\rm PS}(i)$ is the $i$-th point light source in the object space, $\textit{proj}(\cdot)$ represents the geometric projection of the light intensity distribution onto the next plane, ${{\boldsymbol S}_k}$ represents the light intensity distribution on the $k$-th plane, ${\boldsymbol S}_k^\prime$ represents the modulated ${{\boldsymbol S}_k}$ by a coded aperture, $\odot$ represents the Hadamard product, and ${\boldsymbol H}[;,i]$ represents the $i$-th column of TM. The input pixels in the object space were scanned in a raster, and the result of the light originally emitted from each pixel propagating to the next physical element (a coded aperture or sensor) was calculated sequentially, taking into account modulation by the coded aperture, to obtain the impulse response for the number of pixels in the object. The impulse response was arranged as a column vector and inserted into the TM.
Algorithm 1. Ray tracing for the simulated acquisition of TM. CA represents the coded aperture, and PS represents the point light source
C. Result: Matrix and Condition Number
Figure 3 shows the TM with various numbers of coded apertures. $\sigma$ was defined in Section 2.B as the variance of the Gaussian blur kernel, which physically corresponds to the magnitude of diffraction blur that arises from the propagation of incoherent light. In a lensless optical system, $\sigma$ increases linearly with the increase in the propagation distance of light. In this experiment, $\sigma$ is considered as a quantity relative to the pixel size; hence, it is defined in units of pixels. Because the kernel is defined in continuous space and then discretized by sampling, $\sigma$ can assume decimal values when it is set. The submatrix is enlarged to visualize the local structure of the matrix. In the single-layer lensless optical system, the submatrix is a Toeplitz matrix because of the shift invariance of the system. By contrast, in the multilayer lensless optical system, the submatrix is not a Toeplitz matrix because of the more complex propagation and modulation process. As the number of coded apertures increased, the observation became more affected by the shadow of the aperture closer to the sensor, and the results confirmed that the observation contained less-blurry codes. This effect was more pronounced as $\sigma$ increased.
Fig. 3. TM with various numbers of coded apertures (CAs). The submatrix is enlarged to visualize the local structure of the matrix (yellow).
Figure 4 shows the plot of singular values of the TM of Fig. 3. This figure is a plot with the singular values arranged in ascending order along the horizontal axis. The index on the horizontal axis in the figure represents the ordinal position of each singular value, with 1 being the smallest. Note that the vertical axis of Fig. 4 is displayed on a logarithmic scale. In lensless imaging, the captured data are affected by noise, and the smaller the singular value, the larger the error in reconstructing the object image from the captured data. From Fig. 4, the change in the number of coded apertures had little effect on the singular values when $\sigma \lt 1.0$. By contrast, when $\sigma \ge 1.0$, the singular values significantly improved as the number of coded apertures increased. The reason for this is that in optical systems with a large $\sigma$, the effect of being able to measure higher frequencies due to multilayering becomes more pronounced. The scenario in which $\sigma$ is small corresponds to the scenario in which blur caused by light propagation between the coded apertures is slight, such as imaging with very short wavelengths (e.g., X-rays or $\gamma$-rays), or when the distance between the coded apertures is short. In the case of imaging using natural light, a certain amount of diffraction blur is inevitable; hence, $\sigma$ becomes large. Therefore, these results mean that multilayering is effective for improving singular values, particularly in the case of imaging under natural light.
Fig. 4. Singular values of TM. The vertical axis represents the normalized magnitude of singular values, and the horizontal axis represents the index of singular values. The vertical axis is displayed on a logarithmic scale.
The condition number is a useful index for numerically evaluating the quality of singular values. The condition number is defined as the ratio of the maximum singular value to the minimum singular value (${\lambda _{{\max}}}/{\lambda _{{\min}}}$), where ${\lambda _{{\max}}}$ and ${\lambda _{{\min}}}$ are the maximum and minimum singular values of the TM, respectively. The condition number represents how much noise is amplified in the transformed space when solving the inverse problem. Therefore, the smaller the condition number, the more stable the system. Table 1 shows the condition number corresponding to the TM of Fig. 3. From the table, the condition number decreased as the number of coded apertures increased when $\sigma \ge 1.0$. This result also suggests that the multilayer lensless optical system improved the stability of the system against noise.
Table 1. Condition Number of TM
D. Result: Imaging
To investigate how the improvement of the condition number of the TM affects the image quality of actual lensless imaging, an imaging simulation experiment was conducted, and the reconstruction results were quantitatively evaluated. The measurement process was based on Eq. (1), and the captured data were generated by applying the TM to the object ground truth (GT: Ground Truth) (on the left of Fig. 5). The captured data were added with 40 dB additive white Gaussian noise (AWGN). The object image was reconstructed by solving the regularized error minimization problem of Eq. (10) using the noisy captured data and TM as input. The regularization term was 2D TV [41]. The minimization problem was solved using the TwIST algorithm [43]. In the simulation, the regularization parameter of TwIST $\tau$ was set to ${10^{- 6}}$. The total iteration count for the algorithm was set to ${10^4}$. The reconstruction process was performed using a computer equipped with an Intel Core i9-12900 K CPU, NVIDIA GeForce RTX 3090 GPU, 128GB RAM, and MATLAB, which took 31.0 s of computation time per reconstruction process.
Fig. 5. Simulation results of imaging. GT and captured data generated by the forward model with GT and TM, and reconstructed data with various numbers of coded apertures and $\sigma$. Before reconstruction, 40 dB AWGN was added to the captured data. In each reconstructed image, the PSNR is also displayed.
The reconstructed images and their evaluation results are shown on the right of Fig. 5. The peak signal-to-noise ratio (PSNR) was used as the evaluation scale of the results. From the results, the PSNR, which represents the reconstruction accuracy, improved as the condition number of the TM improved. Additionally, the results visually confirmed that the greater the number of coded apertures, the more frequency image information was reconstructed, suggesting that the multilayering of the coded aperture in a lensless camera improved the imaging performance.
In the case of the two-layer model of Fig. 5, contrary to theoretical expectations, the larger the value of $\sigma$, the better the PSNR. This is because a large $\sigma$ value led to a failure in reconstructing high-frequency components and because of the object’s uniform background, this failure unintentionally contributed to the improvement of the PSNR.
E. Discussion
In the simulation, comparing one, two, and four-layer coded apertures, increasing the number of layers improved imaging performance. However, in practice, increasing layers indefinitely is not possible. For simplicity, aperture thickness and resulting vignetting were ignored in the simulation; however, these are significant in real devices with many layers. The effects are minor with few layers but cannot be ignored with more layers because they potentially worsen light-use efficiency in actual devices beyond ideal assumptions. The optimal layer count in real camera devices must consider efficiency, particularly for oblique light.
4. OPTICAL EXPERIMENT
To experimentally verify the effect of the multilayering of the coded aperture, a prototype that could be configured for a single- or two-layer lensless camera was constructed, and optical experiments for TM acquisition and imaging were conducted.
A. Prototype and Experimental Setup
Figure 6 shows the prototype that could be configured for a single- or two-layer lensless camera, and the experimental setup. The optical system of the camera was constructed by directly attaching an aluminum aperture holder to a board-level camera (BFS-U3-50S50C-BD2 by FLIR) with an exposed light-receiving surface. One or two coded aperture could be inserted into this aperture holder, and a single- or two-layer lensless camera was constructed by inserting implemented coded apertures into the holder. Each coded aperture was created using patterning after chromium was deposited on the glass plate to implement the spatial distribution of the intensity transmittance. The measured image was acquired as 12-bit data and resized to $32 \times 32$ pixels. In the optical experiment, random apertures were used as in the simulation. The aperture ratio of the entire optical system of the single- and two-layer lensless camera was set to 50%. The aperture area was designed to be $20 \times 20\;{\rm mm} $, and the size of each small aperture of the mask was set to 50 µm. The thickness of the optical system was set to $L = 6\;{\rm mm} $, and the distance from the object to the aperture was set to ${z_{{\rm obj}}} = 150\;{\rm mm} $. A $32 \times 32$ image displayed on a liquid crystal display (LCD) was used as the subject. The point display position of the monitor was scanned, and the observed data at each position were acquired to experimentally obtain the TM. Note that background-subtraction processing was applied to each measured impulse-response image to mitigate the background light of the LCD.
Fig. 6. (Top) Prototype that could be configured for a single or two-layer lensless camera and example of a coded aperture. (Bottom) Experimental setup.
B. Result: Condition Number
Figure 7 shows the singular values of the experimental TM. Although the improvement was not as large as in the simulation, the singular values quantitatively improved as the number of coded apertures increased, as in the simulation. The condition number was $43.6 \times {10^5}$ in the single-layer case and $8.88 \times {10^5}$ in the two-layer case. This result suggests that the multilayer lensless optical system experimentally improved the singular values and condition number of the TM. Note that the improvement effect on the condition number in the optical experiment was smaller than that obtained in the simulation. This is because of the thickness of the cover glass of the image sensor device, which resulted in an unavoidable gap between the coded aperture and the actual light-receiving surface, thereby losing high-frequency information significantly even with multilayering of the aperture.
Fig. 8. Experimental results of imaging. Calibrated TM, close-up of submatrix of the TM, GT of imaged subject, captured data, reconstructed data, and their binarized data.
C. Result: Imaging
The coded image of the object was captured using the single- and two-layer lensless camera prototypes, and the object image was reconstructed using the measured TM. The object was a $32 \times 32$ binary image displayed on a liquid crystal monitor. The reconstruction algorithm was the TwIST algorithm, as in the simulation. In the optical experiment, unlike the simulation, the GT of the object on the image plane was not available. Therefore, the object was set to a simple binary shape (+), and the reconstruction accuracy was quantitatively evaluated using the matching rate of the binarized reconstructed image and the object.
Figure 8 shows the experimental imaging results. On the left are the experimentally calibrated TMs and close-ups of their submatrices. For the two-layer optical system, we confirmed that the TM deviates from being a Toeplitz matrix, as demonstrated in the simulation results. From the reconstructed images, it can be visually confirmed that the reconstruction accuracy improved, particularly in the background region, as a result of the increase in the number of coded apertures. The binarized reconstructed images in the right column also indicate an improvement in reconstruction accuracy caused by the multilayering of coded apertures. Note that the threshold value for binarization was determined so that the ratio of the number of white pixels to black pixels in the output binary image would be the same as the ratio in the input binary image generated for the target. The matching rate improved from 0.77 in the single-layer case to 0.81 in the two-layer case. These results experimentally confirm that the condition number of the TM and the reconstruction accuracy of the coded image improved as a result of the increase in the number of coded apertures under the conditions of fixed light-use efficiency and camera thickness.
5. LIMITATIONS
A. Computational Cost of Matrix-based Imaging
The main limitation of the multilayer lensless camera is that matrix operations are required for image reconstruction because of the lack of shift invariance. In the single-layer lensless camera, the forward model of the measurement process can be modeled as a 2D convolution using a single PSF, because shift invariance can be assumed. In this case, the PSF information is less than the data volume of a single object image, and the spatial and temporal computational cost is low because the convolution theorem can be used to perform fast calculations in the frequency space. By contrast, the multilayer lensless camera requires matrix operations. This is one of the image reconstruction methods used in the computational imaging field, for example, imaging through scattering media or waveguides [23,25,39,40,44], where the number of elements in the TM is restricted to a small value. This means that high-resolution imaging is difficult to achieve. To address this issue, further research is required on matrix factorization [19,45], computational methods using the sparsity of the matrix [46], and distributed computing [47].
B. Implementation of the Coded Mask
In this study, the effect of multilayering was verified using a coded aperture with a random pattern. This implementation method is simple to implement in the optical system; however, it sacrifices light-use efficiency and PSF design freedom [29]. This is a factor that limited the performance of the multilayer lensless camera in this study. To address this issue, additional research is required on the design and implementation of multilayer coded optical systems using phase masks (phase-modulation optical elements), such as multilevel DOEs.
C. Alignment of Coded Apertures
Stacking multiple coded apertures requires meticulous attention to alignment. To address this, in this study, holders were prefabricated to fix the multiple coded apertures, removing horizontal placement variability. However, when an optical system is constructed in a simpler and adjustable manner without using specially fabricated holders, maintaining alignment after TM calibration becomes crucial. Future research may focus on alignment-correcting image processing and/or designing shift-resistant coded masks.
6. CONCLUSION
In this study, through simulations and optical experiments, we evaluated the imaging performance of a multilayer lensless camera with an optical design that has multiple stacked coded apertures. Under the condition of fixed light-use efficiency and camera thickness, we compared the imaging performance of single- and multilayer lensless cameras and showed that the multilayering of the coded aperture contributed to the improvement of the singular values and condition number of the TM. We also showed that the accuracy of image reconstruction in lensless imaging was improved by multilayering.
In this study, because of the limitation of computational resources, we verified the effect of multilayering on relatively low-resolution imaging of $32 \times 32$ pixels. In future work, we will verify the effect of multilayering on high-resolution imaging by considering computational methods using the sparsity of the matrix and distributed computing. Additionally, we verified the effect of multilayering using a coded aperture with a random pattern. In future work, we will develop a multilayer lensless camera with high light-use efficiency that exploits phase masks. Additionally, we will investigate the design of specialized coding patterns for multilayer systems achieving performance that surpasses random patterns for specific imaging parameters, similar to MURA and FZA in single-layer systems.
Lensless cameras enable spatial information measurement through thin optical systems; however, they often encounter challenges, such as the degradation of image quality and the subsequent reduction in object recognition accuracy caused by the requisite use of coded imaging and image reconstruction processes. The findings of this study will assist in improving the quality of imaging in lensless cameras without sacrificing the thinness of the optical system or light utilization efficiency. More specifically, they can serve as a new optical design method for constructing imaging devices suitable for scenarios in which accurate spatial information must be measured in confined spaces, such as in pipe inspections or medical cameras.
Funding
Japan Society for the Promotion of Science (JPMJFR206K); Fusion Oriented Research for Disruptive Science and Technology (22H01992).
Disclosures
The authors declare no conflicts of interest.
Data availability
The data generated and analyzed in the presented research may be obtained from the authors upon reasonable request.
REFERENCES
1. V. Boominathan, J. K. Adams, M. S. Asif, et al., “Lensless Imaging: A computational renaissance,” IEEE Signal Process. Mag. 33,23–35 (2016). [CrossRef]
2. V. Boominathan, J. T. Robinson, L. Waller, et al., “Recent advances in lensless imaging,” Optica 9, 1–16 (2022). [CrossRef]
3. N. Antipa, “Losing the lens,” Opt. Photon. News 32, 26–33 (2021). [CrossRef]
4. Y. Wang and Z. Duan, “Advances in mask-modulated lensless imaging,” Electronics 13, 617 (2024). [CrossRef]
5. N. Antipa, G. Kuo, H. Reinhard, et al., “DiffuserCam: lensless single-exposure 3D imaging,” Optica 5, 1–9 (2018). [CrossRef]
6. T. Shimano, Y. Nakamura, K. Tajima, et al., “Lensless light-field imaging with Fresnel zone aperture: quasi-coherent coding,” Appl. Opt. 57, 2841–2850 (2018). [CrossRef]
7. S. K. Sahoo, D. Tang, and C. Dang, “Single-shot multispectral imaging with a monochromatic camera,” Optica 4, 1209–1213 (2017). [CrossRef]
8. K. Monakhova, K. Yanny, N. Aggarwal, et al., “Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array,” Optica 7, 1298–1307 (2020). [CrossRef]
9. S. Elmalem and R. Giryes, “A Lensless Polarization Camera,” in OSA Imaging and Applied Optics Congress 2021 (3D, COSI, DH, ISA, pcAOP) (Optica Publishing Group, 2021), vol. 2021, paper CTh7A.1.
10. N. Baek, Y. Lee, T. Kim, et al., “Lensless polarization camera for single-shot full-Stokes imaging,” APL Photon. 7, 116107 (2022). [CrossRef]
11. X. Lin, Y. Rivenson, N. T. Yardimci, et al., “All-optical machine learning using diffractive deep neural networks,” Science 361, 1004–1008 (2018). [CrossRef]
12. S. Chen, P. Xiang, H. Wang, et al., “Optical-electronic neural network for multi-modality and high-accurate lensless optics design and image reconstruction,” Opt. Eng. 63, 013102 (2024). [CrossRef]
13. Z. W. Wang, V. Vineet, F. Pittaluga, et al., “Privacy-preserving action recognition using coded aperture videos,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (IEEE, 2019), pp. 1–10.
14. M.-H. Wu, Y.-T. C. Lee, and C.-H. Tien, “Lensless facial recognition with encrypted optics and a neural network computation,” Appl. Opt. 61, 7595–7601 (2022). [CrossRef]
15. C. Henry, M. S. Asif, and Z. Li, “Privacy Preserving Face Recognition with Lensless Camera,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2023), pp. 1–5.
16. S. R. Gottesman and E. E. Fenimore, “New family of binary arrays for coded aperture imaging,” Appl. Opt. 28, 4344–4352 (1989). [CrossRef]
17. T. Nakamura, K. Kagawa, S. Torashima, et al., “Super field-of-view lensless camera by coded image sensors,” Sensors 19, 1329 (2019). [CrossRef]
18. M. J. DeWeert and B. P. Farm, “Lensless coded aperture imaging with separable doubly Toeplitz masks,” Opt. Eng. 54, 023102 (2014). [CrossRef]
19. M. S. Asif, A. Ayremlou, A. Sankaranarayanan, et al., “FlatCam: thin, lensless cameras using coded aperture and computation,” IEEE Trans. Comput. Imaging 3, 384–397 (2017). [CrossRef]
20. T. Nakamura, T. Watanabe, S. Igarashi, et al., “Superresolved image reconstruction in FZA lensless camera by color-channel synthesis,” Opt. Express 28, 39137–39155 (2020). [CrossRef]
21. J. Wu, H. Zhang, W. Zhang, et al., “Single-shot lensless imaging with fresnel zone aperture and incoherent illumination,” Light Sci. Appl. 9, 53 (2020). [CrossRef]
22. J. R. C. S. A. V. Silva Neto, T. Nakamura, Y. Makihara, et al., “Extended depth-of-field lensless imaging using an optimized radial mask,” IEEE Trans. Comput. Imaging 9, 857–868 (2023). [CrossRef]
23. P. Potuluri, M. Xu, and D. J. Brady, “Imaging with random 3D reference structures,” Opt. Express 11, 2134–2141 (2003). [CrossRef]
24. T. Nakamura, R. Horisaki, and J. Tanida, “Compact wide-field-of-view imager with a designed disordered medium,” Opt. Rev. 22, 19–24 (2015). [CrossRef]
25. R. Fergus, A. Torralba, and W. T. Freeman, “Random Lens Imaging,” MIT computer science and artificial intelligence laboratory technical report (2006).
26. P. R. Gill and D. G. Stork, “Lensless ultra-miniature imagers using odd-symmetry spiral phase gratings,” in Imaging and Applied Optics (2013), paper CW4C.3.
27. Z. Cai, J. Chen, G. Pedrini, et al., “Lensless light-field imaging through diffuser encoding,” Light Sci. Appl. 9, 143 (2020). [CrossRef]
28. G. Pedrini, R. Li, L. Cao, et al., “Lensless imaging in one shot using the complex degree of coherence obtained by multiaperture interferences,” Opt. Lett. 49, 718–721 (2024). [CrossRef]
29. V. Boominathan, J. K. Adams, J. T. Robinson, et al., “PhlatCam: designed phase-mask based thin lensless camera,” IEEE Trans. Pattern Anal. Mach. Intell. 42, 1618–1629 (2020). [CrossRef]
30. F. Tian and W. Yang, “Learned lensless 3D camera,” Opt. Express 30, 34479–34496 (2022). [CrossRef]
31. K. C. Lee, J. Bae, N. Baek, et al., “Design and single-shot fabrication of lensless cameras with arbitrary point spread functions,” Optica 10, 72–80 (2023). [CrossRef]
32. N. Antipa, P. Oare, E. Bostan, et al., “Video from stills: lensless imaging with rolling shutter,” in IEEE International Conference on Computational Photography (ICCP) (IEEE, 2019), pp. 1–8.
33. T. Kim, K. C. Lee, N. Baek, et al., “Aperture-encoded snapshot hyperspectral imaging with a lensless camera,” APL Photon. 8, 066109 (2023). [CrossRef]
34. G. Wetzstein, D. Lanman, M. Hirsch, et al., “Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting,” ACM Trans. Graph. 31, 80 (2012). [CrossRef]
35. Z. Zhang, C. Deng, Y. Liu, et al., “Ten-mega-pixel snapshot compressive imaging with a hybrid coded aperture,” Photon. Res. 9, 2277–2287 (2021). [CrossRef]
36. H. Arguello, S. Pinilla, Y. Peng, et al., “Shift-variant color-coded diffractive spectral imaging system,” Optica 8, 1424–1434 (2021). [CrossRef]
37. T. Igarashi, M. Naruse, and R. Horisaki, “Incoherent diffractive optical elements for extendable field-of-view imaging,” Opt. Express 31, 31369–31382 (2023). [CrossRef]
38. J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill, 1996).
39. S. Popoff, G. Lerosey, M. Fink, et al., “Image transmission through an opaque material,” Nat. Commun. 1, 81 (2010). [CrossRef]
40. Y. Choi, C. Yoon, M. Kim, et al., “Optical imaging with the use of a scattering lens,” IEEE J. Sel. Top. Quantum Electron. 20, 6800213 (2014). [CrossRef]
41. L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D 60, 259–268 (1992). [CrossRef]
42. E. Candès and J. Romberg, “Sparsity and incoherence in compressive sampling,” Inverse Probl. 23, 969–985 (2007). [CrossRef]
43. J. M. Bioucas-Dias and M. A. T. Figueiredo, “A new twist: two-step iterative shrinkage/thresholding algorithms for image restoration,” IEEE Trans. Image Process. 16, 2992–3004 (2007). [CrossRef]
44. X. Chen, N. Tagami, H. Konno, et al., “Computational see-through screen camera based on a holographic waveguide device,” Opt. Express 30, 25006–25019 (2022). [CrossRef]
45. T. Monno, X. Quan, K. Ohta, et al., “Two-photon speckle excitation fluorescence imaging using non-negative matrix factorization,” Proc. SPIE 12608, 126081B (2023). [CrossRef]
46. S. Li, C. Saunders, D. J. Lum, et al., “Compressively sampling the optical transmission matrix of a multimode fibre,” Light Sci. Appl. 10, 88 (2021). [CrossRef]
47. D. J. Brady, W. Pang, H. Li, et al., “Parallel cameras,” Optica 5,127–137 (2018). [CrossRef]
