Open Access
Add to BibSonomy
Abstract
A reciprocal 360-deg three-dimensional light-field image acquisition and display system was designed using a common catadioptric optical configuration and a lens array. Proof-of-concept experimental setups were constructed with a full capturing part and a truncated display section to demonstrate that the proposed design works without loss of generality. Unlike conventional setups, which record and display rectangular volumes, the proposed configuration records 3D images from its surrounding spherical volume in the capture mode and projects 3D images to the same spherical volume in the display mode. This is particularly advantageous in comparison to other 360-deg multi-camera and multiple projector display systems that require extensive image and physical calibration. We analyzed the system and showed quality measures such as angular resolution and space bandwidth product based on design parameters. The issue due to pixel size difference between the available imaging sensor and the display was also addressed. A diffractive microlens array matching the sensor size was used in the acquisition part, whereas a vacuum cast lens array matching the display size was used in the display part with scaled optics. The experimental results demonstrate the proposed system design works well and is in good agreement with the simulation results.
© 2019 Optical Society of America
1. INTRODUCTION
Holography is a very well-known technique for true 3D image acquisition and display [1]. However, it has fundamental drawbacks preventing it being widely available for out-of-the-lab applications [2]. Integral imaging is an auto-stereoscopic and multi-view method used for 3D imaging and display under incoherent light illumination [3]. It can overcome conventional stereoscopic and auto-stereoscopic display limitations, and many advances in this technology have been demonstrated [4]. Moreover, it has been shown that this method is indeed related to holography [5]. Light-field imaging [6–8] is rapidly developing as an alternative to conventional methods [9]. A light field is mathematically defined as a 4D function, which gives the positional and directional intensity distribution [10]. A 3D light-field system can potentially record and display high-resolution 3D images compared to integral imaging. It has also been shown that the light field is equivalent to integral imaging when certain restrictions are imposed on the light-field data [11]. In this paper, we will base our design on the assumption that these two methods are similar. One important similarity between integral imaging and light-field imaging is the capturing and displaying methods used in these systems. Figure 1 shows a generic approach for recording and displaying rays using a microlens array. The directions of the arrows are shown for both capture and display parts. The plane of the microlens array and the plane of recording are assumed to be equivalent to the planes describing a 3D light field. However, when a microlens (or alternatively a pinhole) array is used, only those rays passing through the center of each lens array are recorded. Therefore, we have a subset of the whole light-field data.
Fig. 1. Generic light-field data recording and display method using microlens arrays. Arrows toward (s,t) plane represent recording, and away from (s,t) plane represent display parts.
It is possible to capture the light-field data using multi-camera array systems or a single sensor camera with micro-lens arrays, depending on the application needs [9]. A plenoptic camera can be used to image the 3D light-field data. A focused plenoptic camera has a main lens, which images the scene/object onto an intermediate imaging volume. This focused image is then imaged a second time by the microlens array onto the camera sensor [12]. The acquisition part is relatively straightforward compared to the display part. The captured data can be used for numerical reconstructions and computational refocusing [13,14].
Unlike generic integral imaging displays, the 3D light-field displays are demonstrated using a directional diffuser screen [15] or complex parallax barrier methods [16]. In these approaches, the common goal is to recreate directional light rays. Scanning type displays can demonstrate 3D images in a cylindrical 360-deg volume, whereas integral type displays generate 3D images in a limited angular section [15]. The scanning type light-field displays are fundamentally limited, as they require mechanical movement. Integral type displays require multiple projectors to display in a rectangular volume. Although they can be configured to display 3D images in a full 360-deg or truncated cylindrical volume as in scanning type, they are quite bulky.
Generic 3D light-field acquisition, based on a focused plenoptic camera [17], and reciprocal display systems are planar configurations that generate 3D images in a rectangular volume as depicted in Figs. 2(a) and 2(b). The reciprocity in integral-imaging-based capture and display systems was studied earlier [18–20] in a different context. The common aim was to record and display a live stream of 3D data. However, these 3D display systems provide only a limited field of view within a limited viewing angle.
Fig. 2. Generic light-field system for 3D image (a) acquisition and (b) display based on focused plenoptic camera configuration.
It is desirable to be able to record and display 3D images in a 360-deg volume. There is a trending demand and work on 360-deg camera systems [21] that offer a new way of image acquisition, especially for virtual reality content creation. However, most of these designs require bulky and expensive multi-camera rigs with sophisticated calibration procedures. In addition, they require extensive image and physical calibration with planar approximation of hypothetical spherical sensors [22].In addition, several recent optical designs with 360-deg displays have been demonstrated. In a design, given in [23], multiple imaging sources are configured around a small volume to reconstruct a 3D image, whereas in another design, [24], the display does not reconstruct the 3D image in the free space at all.
We have recently reported a novel design for a 360-deg 3D light-field system with a single plenoptic camera and a display [25] that uses a common optics to record and display 3D images. In this paper, we expand the design with more detailed optical analysis, simulations, physical experiments, and discussion of challenges arising during system demonstration. Our initial design aimed to have a reciprocal optical acquisition and display structure, as shown in Fig. 3. A catadioptric optical relay system, i.e., a parabolic mirror and a field lens combination, first images a surrounding 3D volume to an intermediate imaging volume. Then, a beam splitter allows a sensor to capture the 3D information present, as depicted in Fig. 3(a). Our design assumes exact matching of physical sensor and display parameters. Hence, the captured information is processed and then displayed through the same beam splitter. Therefore, 3D images can be relayed back to the exact same locations as their 3D object counterparts. Due to the practical limitations, such as the difference between the pixel size and count between the camera sensor and the display, we separated sub-systems and demonstrated the results independently. The working principle of our “360-deg” design is that the system captures the rays coming from points on the surface of the objects, which are placed around the periphery of the system in the acquisition mode, and the image points of the 3D images are reconstructed at their respective 3D positions again at the periphery of the system in the display mode. This is different from other systems, such as the one demonstrated in [26], which captures the 360-deg information around the 3D object with a light-field camera and two planar mirrors and reconstructs the 3D information using a holographic display. The use of the curved mirror is also studied in different cases such as the one in [27]. A hyperbolic mirror was placed in front of a conventional camera to capture 360-deg circumference in a single shot, and a numerical 3D reconstruction of a recorded scene was demonstrated using light-field calculation techniques. In our case, our system captures the light-field data intrinsically using the lens array. Another work, [28], was published just after our reported study, where a parabolic mirror was used to create a 360-deg image by making use of holographic reconstruction. Although this work has a similar aim to our method, it does not include the capturing stage, which would be extremely difficult if holographic methods would be used for recording.
Fig. 3. Reciprocal 3D light-field acquisition and display system design with common optics. (a) Capturing. (b) Displaying. Arrows indicate the direction where the light rays travel. Left side of each sub-figure is assumed to have the catadioptric optics.
We structured the paper as follows. In Section 2, we demonstrate design principles and constructed optical setups. In Section 3, we model and simulate the design with certain parameters. In Section 4, experimental results are presented based on these physical parameters. Finally, we discuss the limitations and potential improvements of the system and draw our conclusions.
2. 3D LIGHT-FIELD ACQUISITION AND DISPLAY SYSTEM DESIGN
Our proposed system can record a spherical volume around itself and display the true 3D images of the objects in their respective physical places, as shown in the conceptual sketch in Fig. 4(a). Acquisition and display parts of the proposed system are shown in Figs. 4(b) and 4(c), respectively. In both setups, a single conventional 3D light-field acquisition/display device is used to record/display all 360-deg information. The display can project multiple 3D images to multiple observers. Alternatively, a single 3D image can also be displayed, and an observer can view different parts of the same 3D image. Our system design exploits catadioptric optical systems [29] together with a diffractive optical element (DOE). An array of DOE microlenses, which have smaller diameter and shorter focal length compared to conventional ones, offers a compact low-cost solution for capturing high-resolution light-field images even under incoherent light illumination. The common reciprocal optical configuration allows to record from its surrounding volume in the capture mode and project 3D images to the same volume in the display mode when the sensor at the imaging plane is replaced by a display.
Fig. 4. (a) Proposed design for a 360-deg 3D light-field system in display mode. The 3D images are displayed in a spherical volume rather than a rectangular volume. (b) Light-field camera and (c) display of the proposed design.
A. 360-deg 3D Light-Field Camera Design
The microlens array of the capturing part is the crucial optical component, which is normally manufactured by conventional methods. However, the design and manufacturing of these components are significantly costly. Using off-the-shelf microlens arrays is an option, although it is not cost effective and, more importantly, their fixed physical catalog parameters are a limiting factor in the overall system design. Therefore, alternative approaches, such as diffractive optical components, would better fit in the proposed design case. It has been shown that diffractive lenses, realized as binary masks or by reconfigurable spatial light modulators, can be used to replace conventional microlens arrays [30,31]. Although binary masks might give low diffraction efficiency, phase-only liquid-crystal-on-silicon devices give significantly better-quality results [5].
In [14], a focused plenoptic camera design to record high-quality light-field images is demonstrated. An off-the-shelf high-resolution camera and a manual lens pair are converted to a plenoptic camera by inserting a micro photon sieve array (mPSA) between the lens and the camera sensor. Based on this camera design, our proposed 360-deg light-field camera design comprises this focused plenoptic camera and a parabolic mirror with a geometry that can collect light from the volume surrounding it. The field lens and mirror combination can also be regarded as a catadioptric imaging system that images a large surrounding 3D volume in front of a microlens array, which captures sub-images of the scene/objects in the imaged 3D volume, as in a conventional integral imaging setup [3].
B. 360-deg 3D Light-Field Display Design
In principal, the display part traces all the rays in the reverse optical path of the acquisition part because of the common optical configuration. Therefore, the light rays are as if they are coming from the original object. The proposed system is similar to an integral imaging display. However, the additional mirror helps to cover the surrounding volume. The rays from different objects are recorded from different angles. The replayed rays reconstruct the objects in their original physical locations as real images. Hence, the observers are able to visualize the objects from their respective positions, as in Fig. 4(c). One example geometry for the mirror is a convex shape, as shown in Fig. 4, although it may not be the optimal geometry.
The reader should note that the optimization of the mirror geometry and the optical setup is out of the scope of this paper and left as a future study. The mirror may introduce certain distortions to the relayed images from the intermediate imaging volume. It should be noted that distortions incurred on the images this way can be corrected using additional optics or by image processing. A simple convex mirror creates divergent rays. However, with additional optics, it is possible to control the rays to converge. In the display part, all the rays are assumed to travel back the same optical path, although we do not have all the rays recorded with their exact directional information. The conventional 2D displays scatter the light with a certain angle. We assume that at least two rays from the source trace back to the desired direction as convergent rays to create a point, while others can be considered as a background noise on the reconstructed image.
It is also possible to computationally generate the light-field images and use the display part only. Ideally, a priori information on the optical distortion of the system needs to be known, such that a pre-distortion can be applied on the generated light-field data. If this pre-distortion is not applied, image reconstruction is still possible with some distortion.
One major practical issue we had in the display part is the pixel size mismatch between the camera sensor and the display. Due to this difference, we had to construct a scaled setup for the display part. A 4K mobile phone (Sony Xperia XZ Premium) screen used to display approximately a quarter of the entire image captured by the full frame camera sensor. A matching lens array was designed in house based on the design rules given by [32] and manufactured by Materialise UK Ltd, using the vacuum casting method. Each element in the array had a size with a focal length of approximately 32 mm. The manual camera lens was replaced by a stack of Fresnel lenses. A single Fresnel lens, with a 300 mm focal length and 300 mm diameter, did not give the f-number required to bend the rays towards the edges of the parabolic mirror. Therefore, with a stack of three Fresnel lenses, we achieved one third of the focal length of a single lens while maintaining the same aperture size. With the large lenslet size and long focal length, the reconstructed images created a coarse 3D reconstruction when it passed through the Fresnel lens stack.
C. Analysis of the Designs
In this sub-section, we give a basic analysis of this complex system. An earlier review study [33] discusses various physical parameters such as resolution, viewing angle, and depth range, from a display-only perspective. A detailed analysis can further be carried out together with the optimization of the overall system.
In Fig. 5(a), we start with a conventional light-field capturing setup. At this stage, we assume circular symmetry around the optical axis. A point source at the object reference plane is imaged by a field lens to the intermediate imaging plane. Then, the image at the intermediate imaging plane is captured by individual microlenses and mapped to the pixels of the sensor. Assuming that the point source at the object reference plane has Lambertian scattering, the field lens is able to collect only a portion of the uniformly distributed light rays it receives. The total angle, , is determined roughly by the numerical aperture, , of the lens. However, we use the imaging distance parameter, , instead. The rays then travel with this limited angle and reach the microlens array, where the rays are further limited, , and then imaged on the sensor. If the number of microlenses receiving the rays from the field lens in the array is , then
gives the angular resolution of the capturing system. To find the spatial resolution, we need to determine the point spread function of the microlenses, which is given by for a diffraction limited lens, where is the focal length, is the wavelength, and is the diameter. If the pixel size of the sensor is smaller than this value, there is no one-to-one mapping of the points, which reduces the captured image resolution, and it is the case for these systems. However, for a mathematical point of view, we assume we can record the rays individually by a single pixel.Fig. 5. Schematics of a light-field capture system are shown (a) without a mirror, (b) with a (half) conical mirror, and (c) with a (half) parabolic mirror.
In the display stage, we replace the sensor with the display, and we trace the rays back to the object plane from the display pixels. The pixel period of the display normally determines spatial bandwidth, and the number of pixels determines the spatial extent. These two parameters give the space-bandwidth product (SBP) of the display system, which is a measure used to quantify the performance of holographic displays [34–36]. This can be extended in our case. However, we cannot apply the same definition on the display as holographic displays, because the pixels on the display are imaged through the microlens array, which decreases the spatial extent and limits spatial bandwidth. In this case, we can define the SBP on the intermediate imaging plane. The angular resolution gives the spatial bandwidth, and the number of points imaged by a single lens gives the spatial extent, , where is the number of pixels that a single lens images. Therefore, we can approximate
If we place a conical mirror between the field lens and the object reference plane, it changes the direction of the imaging, as shown in Fig. 5(b). The rays from a point source on the object reference plane expand in a conical volume and are reflected from the mirror surface. However, we still have a similar imaging performance if we assume that the solid angle of the conical volume is small enough to be regarded as a ray. (On a certain plane cut, the angle would be ). Moreover, we are able to capture the 360-deg periphery of the surrounding volume. The rectangular field of view of the system is translated to a spherical one. However, this does not increase the SBP of the display system, simply because the parameters of the system do not change.The more complicated situation is when we place a parabolic mirror as depicted in Fig. 5(c). In this case, the ray bundle coming from the object point assumes a more divergent path. The convex mirror creates a virtual point at a closer distance than the origin of the point. Moreover, this creates a distortion on the virtual image. In this case, a narrower ray bundle reaches the field lens. Therefore, the information density from the object point is decreased.
Although in the display stage it is possible to reconstruct the 3D image by reversing the ray paths, the quality of the image is decreased in the case where a mirror is used.
3. DESIGN VERIFICATION AND SIMULATION RESULTS
Optical simulations are carried out using ZEMAX to explore and verify the optical design. The light source used in the simulations is a 2D rectangular LED, in which the emitting angle and light distribution can be controlled by varying the model parameters. Light is uniformly emitted from the source surface at the designed angle and intensity distribution. The optical components are simulated with paraxial lenses, so the wavelength of light rays within the visible range does not have an effect on the ray-tracing results. The default design wavelength of 550 nm is used for all ray-tracing simulations.
A. Simulation of the Acquisition Part
The schematics of acquisition setup is illustrated in Fig. 6(a). The LED light source with a dimension of and a diverging angle of 30° (FWHM) is used as the object. Three black stripes, each of , are placed right in front of the LED source and they block light rays from the covered sections. A ZEMAX “standard surface” is used as the parabolic mirror. The radius of curvature is set to 338.4 mm and the conic constant set to (); these values match the physical dimensions of the real component. The camera/field lens has a maximum effective aperture of 11.1 mm in diameter, and it is simulated as a paraxial lens. A paraxial lens array is used to simulate the mPSA. The camera sensor (Sony A7R II, 42 MPx) has a dimension of and resolution of with a pixel size of 4.52 μm. In ZEMAX, it is simulated as a “detector,” and the smallest pixel size with the given dimension is 6 μm.
Fig. 6. (a) Schematics of the capture system with a parabolic mirror, a field lens, and a lens array. (b) Simulated ray-tracing results on the detector. (c) Chart with the minimum resolvable pixel size against the object distance.
The distance between the lens array and the camera sensor is fixed to the imaging distance of 12.5 mm. The LED light source (object) is about 270 mm away from the parabolic mirror surface, which is 150 mm away from the field lens. The camera lens aperture is set to f/1.8 to allow more light rays collected by the sensor. The simulated image on the detector is shown in Fig. 6(b) with an inset showing a single enlarged sub-image; all sub-images are clearly seen. Because the object is at only one side of the parabolic mirror, the corresponding sub-images are congregated towards one side of the sensor. The object that is imaged can be as close as 40 mm away from the field lens, with its image distance 40 mm from the field lens. However, it is observed that an object distance of greater than 100 mm gives sharper images on the sensor. An object at infinity would be imaged as a point on the sensor, and it would not be possible to resolve its details. With the field lens and lens array in use, the smallest pixel size at the intermediate plane is calculated as 18.08 μm. An object at 2 m away would have a resolvable pixel size of 1.8 mm, as shown in Fig. 6(c); the resolvable pixel size increases linearly with the object distance.
Two different letter-shaped light sources (“F” and “S”) are used as objects in ZEMAX. In simulations, the detector, lens array, camera/field lens, and parabolic mirror are placed at the locations shown in Fig. 6(a). A lens entrance aperture with a diameter of 70 mm is inserted to mimic the real camera lens, and the two objects are symmetrically placed on either side of it with a distance of 70 mm from the parabolic mirror. The simulated (captured) sub-images of two letters are clearly seen in Fig. 7. Since both letters are symmetrically placed around the camera lens optical axis, their sub-images are imaged symmetrically on the camera sensor.
B. Simulation of the Display Part
As the test image, the sub-image set shown in Fig. 8 is generated. Two differently colored letters, “A” and “B,” at two different depths are generated using wave propagation. In [37], a method to convert pseudoscopic images to orthoscopic images was given. A 3D data set was input to a two-step wave propagation algorithm. In that, it is shown that the points on the reference plane should be imaged closer to each other compared to the points at the far away plane. For example, a 2D letter on the closer plane should be imaged as smaller sub-images, whereas a 2D letter at the far away plane should have larger images. Therefore, when the 3D reconstruction is done, the corresponding images appear in the right order. In this paper, in the generation of the sub-images, we obtained the images at the same distance both for “A” and “B” in order to generate an all in-focus image set. Obviously, the physical reconstructions of the objects would be at the same distance if we used this sub-image set directly. Therefore, the sub-images of the letter “A” slightly () enlarged in size. This brought the physical reconstruction distance closer to the lens array while keeping the resolution of the images high. Therefore, we obtained orthoscopic reconstruction of the sub-image set. For the ZEMAX simulations, the sub-image set for each letter was separated from the combined images. Each sub-image set for each letter was input to the simulations separately.
The schematics of the display setup is illustrated in Fig. 9(a). The same rectangular LED source used in imaging simulation is used here to simulate a 4K mobile display (Sony Xperia XZ Premium, density). The LED source has a 5° FWHM diverging angle so that most of light rays go through the corresponding lens, although in practice the mobile display has a much larger scattering value.
Fig. 8. Sets of sub-images with different colors and distances are generated using wave propagation.
Fig. 9. (a) Schematics of the displaying part with the sub-images and lens array, with simulation results (insets) of the integrated image at the intermediate plane and the viewer’s plane. (b) Display setup. A 4K mobile phone display (Sony Experia XZ) is used to display 2D sub-images. A lens array, fabricated by vacuum casting with elements and 32 mm focal length each, matching the displayed sub-images, is used to reconstruct the 3D images. A stack of Fresnel lenses with a focal length and diameter of 300 mm is used to obtain a low f-number equivalent lens. A parabolic mirror, used to mitigate the blind spots in traffic, with 300 mm diameter is placed after the Fresnel lens stack.
An effective section ( elements) of the larger lens array ( elements) is simulated as an array of paraxial lenses, each with a square period of and the effective focal length (EFL) of 29.7 mm. The EFL is different from the designed value of 32 mm, in that the physical lens array was fabricated by vacuum casting and has a thickness of 5.5 mm. The simulated paraxial lens on the other hand does not have a thickness; in order to address this difference, a different EFL value is chosen in the simulation to produce the same imaging distance as the experimentally obtained one. As mentioned in Section 2, three identical Fresnel lenses are placed close to one another and used as the field lens, with each Fresnel lens having an EFL of 300 mm. The resultant EFL is 102.6 mm. The field lens is simulated as a paraxial lens. Such an arrangement provides a small f-number of and can reduce the volume of the display system.
The same parabolic mirror from the imaging simulation is used in the display simulation. The distance between each part of the display system is annotated in Fig. 9(a), and the simulated images are included as insets to the location where detectors are placed.
With the setup of 42 mm between the sub-images and lens array, the integrated images of letters “A” and “B” are formed at 93 mm and 108.5 mm away from the lens array, respectively. The reconstructed images are then relayed by the field lens and the parabolic mirror to the viewer’s direction. The letter “A” seen from the viewer’s point of view is about 103 mm from the intersection of light rays and parabolic mirror, while the letter “B” is about 208 mm away. The final simulated images are shown as “simulated images” in Fig. 9(a), they are about 3.9 times the displayed sub-images for “A” and 7.8 times for “B,” respectively.
4. EXPERIMENTAL RESULTS
After design verification by simulations, we conducted physical experiments to confirm and qualify the actual setups. Note that in the demonstration of the display part, the setup was limited by the available components at the time of the experiments. It should be obvious to the reader that the display and the lens pair can be rotated around the optical axis to display from other angles to demonstrate another 3D image from a different viewing point, since the mirror is circularly symmetrical around the optical axis. In our case, in order to match the captured image size, we would need at least four 4K mobile screens and matching lens arrays to be able to cover most of the mirror surface from all angles in a 360-deg circumference.
A. Acquisition Part Experiments
Figure 10(a) shows the schematic of the 360-deg camera setup, and Fig. 10(b) shows the physical camera setup. Objects are placed on either side of the parabolic mirror. The reflected images are imaged using the manual lens (Samyang 20 mm F1.8 ED AS UMC) to an intermediate imaging volume. This volume is imaged onto the (42 Mpx, ) camera sensor using the ( element) mPSA with each element having a focal length of 10 mm and a diameter of . The mPSA is designed for a single wavelength, .
Using the setup in Fig. 10(b), we captured the sub-image set shown in Fig. 11. An external flash light source synchronizes with the camera and is used to illuminate while capturing the images. A green filter is used to partially filter the spatial bandwidth of the light source to bring closer to the mPSA design wavelength. The raw images suffer from some undiffracted light. Due to this, similar image processing steps in [14] are applied on the raw images in order to improve the visibility of the sub-images. We add one extra step to remove the red points appearing due to the flash light reflected from the mirror surface. Since the red dots resemble “red-eye” effect when the subject is a person, we use a manual red-eye removal tool to get rid of this degradation. The processing steps are as follows: (1) separate each color channel of the image; (2) apply a local histogram equalization over the sub-images; (3) apply Wiener filtering over the sub-images; (4) apply red-eye removal at the bright red spots on the image where necessary.
Fig. 11. Captured (left) and processed (right) sub-image sets. Insets on the left and in the middle show the portions from corresponding images. The one on the right shows after red-eye removal applied. Although we have recorded the images using green color filter, we obtained full color images after the processing steps. This is due to the poor spatial filtering of the filter used over the flash light source.
In Section 3, all the simulations are done with a large camera aperture of f/1.8 in order to have more rays hitting the detector. A wide aperture field lens allows more light to enter to the optical system while forming sharper images in the intermediate imaging volume. However, in the physical experiments, due to the low diffraction efficiency of the mPSA, the captured sub-images suffer from the background noise because of the high-intensity light used for illumination even with low ISO settings. Therefore, the size of the aperture should be chosen such that the undiffracted light is reduced to allow the sub-images to be seen clearly. The effect of the aperture size on the captured images is shown in Fig. 12. With f/4, the letter “F” has a sharper edge than that of f/1.8, but with f/11, half of the letter is lost due to the limited number of light rays entering the camera.
Fig. 12. Single sub-images cropped from the same location of the recorded images with lens aperture sizes of (a) f/1.8, (b) f/4, and (c) f/11.
B. Display Part Experiments
The test image in Fig. 8 is displayed on the mobile phone display. In order to demonstrate the 3D reconstruction with the data, we placed a diffuser at two different imaging distances after the lens array and recorded the reconstructed images shown in Figs. 13(a) and 13(b). We later demonstrated the real image reconstruction after the rays were reflected from the parabolic mirror surface. However, we faced one of the limitations of the available optical parts for the demonstration. Figure 13(c) shows the reflected images from the parabolic mirror surface. The coarse lens array did not provide enough continuous views to reconstruct the entire image or enough rays to make the entire image visible from a single viewing point. Moreover, the parabolic mirror further spread out these limited numbers of rays, which in turn decreased the number of rays entering the camera or eye pupil. Therefore, it required to move the camera or the eye pupil in order to observe the entire reconstructions.
Fig. 13. Reconstructed real images after light rays are reflected from the mirror surface. (a) Focus on the letter “B” and (b) focus on letter “A” when a narrow angle (15° Luminit) holographic diffuser is placed at the intermediate imaging planes. (c) Observed reconstructed images reflected from the parabolic mirror surface without any diffuser in the intermediate imaging plane. (d) Observed reconstructions realized on the diffuser, which is placed at the reconstruction distance after the rays are reflected from the mirror surface.
C. Displayed Image Improvement for Continuous Viewing
Due to the limitations mentioned in the previous sub-section, we placed a simple narrow angle (15°) holographic diffuser at the reconstruction volume after the parabolic mirror to improve the visibility of the reconstructions. This is an approach similar to the one demonstrated in [38]. Each sub-image and corresponding lens in the array can be regarded as an image projector. The light rays are scattered from the diffuser surface making the reconstructed images visible from wider angles, as shown in Fig. 13(d).
Finally, in order to confirm the reciprocity of the entire system, we used the processed image in Fig. 11, captured by the acquisition part given in Fig. 10. We successfully observed a reconstructed real image, Fig. 14, using the display part shown in Fig. 9(b). We noticed that the use of a largely white background introduces significant color dispersion, which happens as a combined result of the use of a mPSA, poor color filtering of the display, and the parabolic mirror, which spreads the rays further. Although, there were many imperfections and difficulty to match the acquisition and display part, we demonstrated that the physical system design works as the numerical verifications. The system can further be improved to mitigate these imperfections with optimized parameters and components.
Fig. 14. Reconstruction of the captured image “F” realized on a diffuser after it is reflected from the parabolic mirror surface. (left) Full color. (right) Green color only.
5. DISCUSSION
As demonstrated by the experiments and supporting simulations, the designed system provides a 360-deg acquisition and display. However, it needs further optimization of the optics and improvement of the image processing steps in order to fully achieve high-quality 3D image reconstruction. One of the drawbacks is the image sensor and 2D display pixel size mismatch. This is rather a matter of improvements in the available 2D display technologies, which is a common problem in any 3D display application regardless of the used approach, i.e., holographic or light field. The recent developments in flat panel display technologies, such as LCD and OLED, enable higher resolution and larger size displays.
A detailed analysis, which takes the capture and display mismatches into account, of the light-field displays is given in [39]. Display pixel configuration plays an important role for accurate reconstruction of the 3D images. Moreover, the display configuration has to match the capturing sensor pixel configuration. As a common industry standard, a Bayer pattern is used in most of the camera sensors. This is also true for the camera sensor we used according to the basic specification analysis given in [40]. We assume in our work that the stored data are compatible in terms of the image being captured and displayed. However, the display part is the physical optical interface effecting the results. Figure 15 shows the display pixel configuration of the display we used in the experiments. There are two important corollaries using this display. Our assumption was that the pixels are arranged in a regular rectangular array. In the actual display, they are arranged in a hexagonal array. Although as a standard display, this reduces artifacts such as Moire effects due to small pixel periods and improves the spatial frequency utilization, we did not take this into account in our optical configuration. The lens array geometry should be configured according to the display pixel configuration for better pixel–lens utilization. Second, the blue colored pixels have the worst performance, as this color filter allow partial leakage from the green channel, as it can be seen in the right-most image in Fig. 15. In fact, there were slight leakages in other channels as well, although they were not this severe. This was one of the main problems we had in the reconstruction of the color images and the main reason in the displayed image quality when Figs. 13 and 14 are compared. Since we had more image color control, the reconstructed images of computer-generated data resulted in better visibility.
Fig. 15. Display pixel configuration. From left to right, full color and respective single channel colors isolated from the full color image are shown. The full color image is shot under a 60× microscope objective. The dashed boxes represent a single pixel composed of sub-pixels of each individual color channel. The blue channel leaks through the green pixels. This is visible when the picture is zoomed in.
As is obvious from the above corollaries, the off-the-shelf devices do not provide exactly what is mathematically required, although they provide good enough results to prove the concepts. Therefore, these should be considered when optimizing this system. We will also investigate the other possible pixel configurations, such as circular ones used in the circular smart watches in our future study.
Another drawback of the current system is two different types of lens arrays used in data capturing and reconstructions. Microlens arrays are crucial components in light-field imaging of the 3D data. Hence, they should be tailored according to the needs of the system. With the new diffractive optical techniques applied to the lens array designs [14], it is possible to achieve better performance in such systems. It will improve the image acquisition and display quality while reducing the size of the systems.
One last drawback in the current system is the parabolic mirror. The main source of geometric distortions on images in the system is caused by this component. There are several ways to mitigate the effect of it by optical optimization, digital image processing, or both. The pure optical optimization of the components is much more complicated in many cases, as the system uses unconventional optics. In this case, camera calibration methods in image acquisition and pre-distortion using a known distortion function in image display are practical.
6. CONCLUSION
A 360-deg 3D light-field acquisition and display system was designed and demonstrated using a common optical setup comprising a focused plenoptic configuration and a catadioptric aperture. The use of a paraboloidal mirror is essential for the acquisition and display of a spherical volume surrounding the system. A diffractive optical component, mPSA, is used as a microlens array in the acquisition part of the proposed design. Due to the difference between the pixel sizes of the acquisition sensor and the display, a scaled version of the acquisition optics is used in the display part. A vacuum cast lens array and a high-resolution mobile phone display are used for reconstructing 3D images. Without loss of generality, a section of the entire image is displayed to demonstrate that the proposed design works. Both the images captured using the proposed design and generated numerically are used in the display setup. The display successfully reconstructed 3D images in their relevant locations as expected. The experimental results were verified by simulations of the system. Such a design is envisaged to be used in video conferencing and immersive gaming.
Funding
University of Cambridge, Centre for Advanced Photonics and Electronics (Acorn Blue Sky Research Award 2016).
Acknowledgment
Portions of this work were presented at the OSA Imaging and Applied Optics Congress in 2018 [25]. The authors declare that there are no conflicts of interest related to this paper.
REFERENCES
1. D. Gabor, “A new microscopic principle,” Nature 161, 777–778 (1948). [CrossRef]
2. P. Benzie, J. Watson, P. Surman, I. Rakkolainen, K. Hopf, H. Urey, V. Sainov, and C. von Kopylow, “A survey of 3DTV displays: techniques and technologies,” IEEE Trans. Circuits Syst. Video Technol. 17, 1647–1658 (2007). [CrossRef]
3. G. Lippmann, “La photographie integrale,” C.R. Hebd. Seances Acad. Sci. 146, 446–451 (1908).
4. X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, “Advances in three-dimensional integral imaging: sensing, display, and applications [Invited],” Appl. Opt. 52, 546–560 (2013). [CrossRef]
5. A. O. Yontem and L. Onural, “Integral imaging based 3D display of holographic data,” Opt. Express 20, 24175–24195 (2012). [CrossRef]
6. M. Levoy and P. Hanrahan, “Light field rendering,” in 23rd Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1996), pp. 31–42.
7. E. Y. Lam, “Computational photography with plenoptic camera and light field capture: tutorial,” J. Opt. Soc. Am. A 32, 2021–2032 (2015). [CrossRef]
8. M. Martínez-Corral and B. Javidi, “Fundamentals of 3D imaging and displays: a tutorial on integral imaging, light-field, and plenoptic systems,” Adv. Opt. Photon. 10, 512–566 (2018). [CrossRef]
9. J. Y. Son, H. Lee, B. R. Lee, and K. H. Lee, “Holographic and light-field imaging as future 3-D displays,” Proc. IEEE 105, 789–804 (2017). [CrossRef]
10. N. Balram, “Fundamentals of light field imaging and display systems,” in Display Week (SID, 2016).
11. E. Sahin and L. Onural, “A comparative study of light field representation and integral imaging,” Imaging Sci. J. 58, 28–31 (2010). [CrossRef]
12. A. Lumsdaine and T. Georgiev, “The focused plenoptic camera,” in IEEE International Conference on Computational Photography (ICCP)(2009).
13. J. Tumblin, “Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing,” ACM Trans. Graph. 26, 69 (2007). [CrossRef]
14. A. O. Yontem and D. Chu, “Design of micro photon sieve arrays for high resolution light-field capture in plenoptic cameras,” in 3DTV-Conference: The True Vision—Capture, Transmission and Display of 3D Video (3DTV-CON) (2018).
15. X. Liu and H. Li, “The progress of light field 3-D displays,” Inf. Disp. 30, 6–14 (2014).
16. G. Wetzstein, D. Lanman, M. Hirsch, and R. Raskar, “Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting,” ACM Trans. Graph. 31, 80 (2012). [CrossRef]
17. R. Ng, M. Levoy, M. Br’edif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a handheld plenoptic camera,” Stanford Technical Report CTSR 2005-02 (2005).
18. F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Real-time pickup method for a three-dimensional image based on integral photography,” Appl. Opt. 36, 1598–1603 (1997). [CrossRef]
19. Y. Jeong, J. Kim, J. Yeom, C.-K. Lee, and B. Lee, “Real-time depth controllable integral imaging pickup and reconstruction method with a light field camera,” Appl. Opt. 54, 10333–10341 (2015). [CrossRef]
20. J. Kim, S. Moon, Y. Jeong, C. Jang, Y. Kim, and B. Lee, “Dual-dimensional microscopy: real-time in vivo three-dimensional observation method using high-resolution light-field microscopy and light-field display,” J. Biomed. Opt. 23, 066502 (2018). [CrossRef]
21. Filmora, 2018, https://filmora.wondershare.com/virtual-reality/top-10-professional-360-degree-cameras.html.
22. G. Krishnan and S. K. Nayar, “Towards a true spherical camera,” Proc. SPIE 7240, 724002 (2009). [CrossRef]
23. L. Onural, “Design of a 360-degree holographic 3D video display using commonly available display panels and a paraboloid mirror,” Proc. SPIE 10126, 101260I (2017). [CrossRef]
24. T. Kawakami, M. Date, M. Sasai, and H. Takada, “360-degree screen-free floating 3D image in a crystal ball using a spatially imaged iris and rotational multiview DFD technologies,” Appl. Opt. 56, 6156–6167 (2017). [CrossRef]
25. A. O. Yöntem and D. Chu, “Design for 360-degree 3D light-field camera and display,” in Imaging and Applied Optics (3D, AO, AIO, COSI, DH, IS, LACSEA, LS&C, MATH, pcAOP), OSA Technical Digest (Optical Society of America, 2018), paper 3Tu5G.6.
26. Y. Jeong, S. Moon, J. Jeong, G. Li, J. Cho, and B. Lee, “One shot 360-degree light field capture and reconstruction with depth extraction based on optical flow for light field camera,” Appl. Sci. 8, 890 (2018). [CrossRef]
27. H. Todoroki and H. Saito, “Light field rendering with omni-directional camera,” Proc. SPIE 5150, 1159–1168 (2003). [CrossRef]
28. Y. Sando, K. Satoh, T. Kitagawa, M. Kawamura, D. Barada, and T. Yatagai, “Super-wide viewing-zone holographic 3D display using a convex parabolic mirror,” Sci. Rep. 8, 11333 (2018). [CrossRef]
29. B. Braker and E. D. Moore, “Catadioptric projector systems, devices, and methods,” U.S. patent application 20170023780A1 (January 1, 2017).
30. J.-S. Jang and B. Javidi, “Three-dimensional integral imaging with electronically synthesized lenslet arrays,” Opt. Lett. 27, 1767–1769 (2002). [CrossRef]
31. M. Hain, W. Spiegel, M. Schmiedchen, T. Tschudi, and B. Javidi, “3D integral imaging using diffractive Fresnel lens arrays,” Opt. Express 13, 315–326 (2005). [CrossRef]
32. P. Nussbaum, R. Volkel, H. P. Herzig, M. Eisner, and S. Haselbeck, “Design, fabrication and testing of microlens arrays for sensors and microsystems,” Pure Appl. Opt. 6, 617–636 (1997). [CrossRef]
33. M. Yamaguchi, “Light-field and holographic three-dimensional displays [Invited],” J. Opt. Soc. Am. A 33, 2348–2364 (2016). [CrossRef]
34. D. Mendlovic and A. W. Lohmann, “Space-bandwidth product adaptation and its application to superresolution: fundamentals,” J. Opt. Soc. Am. A 14, 558–562 (1997). [CrossRef]
35. B. L. Reddy, M. P. Phinehas, and A. Nelleri, “Space bandwidth product analysis of digital holography and image reconstruction process,” in International Conference on Nextgen Electronic Technologies: Silicon to Software (ICNETS2), Chennai, India, March 2017, pp. 194–197.
36. F. Yaras, H. Kang, and L. Onural, “Multi-SLM holographic display system with planar configuration,” in 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video, Tampere Finland, June 2010, pp. 1–4.
37. A. O. Yontem, “Three-dimensional integral imaging based capture and display system using digital programmable Fresnel lenslet arrays,” Ph.D. thesis (Bilkent University, 2012).
38. J.-H. Lee, J. Park, D. Nam, S. Young Choi, D.-S. Park, and C. Y. Kim, “Optimal projector configuration design for 300-Mpixel multi-projection 3D display,” Opt. Express 21, 26820–26835 (2013). [CrossRef]
39. R. Bregovic, P. T. Kovács, T. Balogh, and A. Gotchev, “Display-specific light-field analysis,” Proc. SPIE 9117, 911710 (2009). [CrossRef]
40. Apotelyth, 2018, https://www.apotelyt.com/compare-camera/sony-a7r-ii-vs-sony-a850.
