Open Access
Add to BibSonomy
Abstract
We propose an integral 3D display system that enhances the viewing zone and depth range of reconstructed 3D images using eye-tracking technology and a lens array with a long focal length. To reduce crosstalk due to flipped images caused by the movement of the viewer, a lens arrangement that forms a horizontally widened optical viewing zone by rotating the lens array is designed. Unlike the conventional design, the viewing angle expanded 2.9 times to 81.4° horizontally and 1.7 times to 47.6° vertically, and the depth range doubled. Results of the evaluation experiment confirmed that crosstalk was sufficiently suppressed.
© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Integral imaging is a three-dimensional (3D) imaging and display technology that is based on the principle of integral photography [1]. It has full parallax horizontally and vertically, and it can reproduce continuous and natural 3D images in the depth direction. Integral imaging is a type of light field display. The light rays equivalent to the light emitted from the real object are reproduced in space by the corresponding combination of the elemental image of an elemental image array (EIA) and the elemental lens of a lens array. In principle, it is suggested that the “vergence-accommodation conflict” that causes visual fatigue and discomfort in the stereoscopic method can be reduced in the integral method [2,3]. Therefore, as a future 3D imaging system, integral imaging is expected to be applied in industries such as broadcasting, medical care, entertainment, and advertising, and various studies have been conducted in this regard [4–12].
The integral method reproduces numerous parallaxes, and it is necessary to use a display device with a large number of pixels and a narrow pixel pitch for high-performance display [13]. Therefore, it is currently difficult to improve the display performance by only combining the display and the lens array as in the conventional method. Several studies have been conducted to expand the viewing angle and improve the quality of integral 3D images by combining multiple devices systematically. For example, to improve display performance, studies have been conducted to increase the number of effective pixels by using multiple display panels [14–17] or multiple projectors [18–23]. As it is possible to handle an extremely large number of pixels using multiple display devices, it is possible to reliably improve display performance. However, as the system becomes more complicated, the cost increases, and the depth of the whole device becomes larger. These are major challenges that hinder its practical applicability.
In a 3D display system for personal use, applying eye-tracking technology is suitable for achieving high performance. This is because the 3D image can be presented according to the viewer's eye position, and pixels can be effectively used. Further, as the eye-tracking technology uses a camera, such as a webcam or depth camera, the system as a whole can be configured compactly and at a low cost. Hence, the technology has high practical prospects.
A major drawback of integral imaging is the flipping effect [24,25]. This is a phenomenon by which when the viewer deviates from the primary viewing zone, they enter the secondary viewing zone formed by the adjacent elemental image and the elemental lens, causing a flipped image to be perceived. This flipping effect significantly reduces the visibility of 3D images. In integral 3D systems with eye-tracking, the viewing zone can be dynamically controlled according to the eye position. However, when the viewer watches 3D images while moving, the viewer's eye often deviates from the primary viewing zone due to errors in eye detection and a delay in EIA rendering. Consequently, in the area between the primary and secondary viewing zones, mixed images are perceived as crosstalk.
Therefore, we propose a system that uses eye-tracking technology to increase the viewing angle and quality of integral 3D image efficiently. Furthermore, to suppress the image flipping caused by the movement of the viewer, we propose a viewing zone formation and lens array arrangement suitable for the eye-tracking display.
2. Related work
Various eye-tracking studies have been performed in 3D display systems of a stereoscopic method [26–28]. In the display system, as only a small number of viewpoint images are formed, a high-resolution image can be displayed. In the eye-tracking system, the calculation cost on the system is required to detect the eye position and generate an image accordingly. Therefore, in a stereoscopic method, crosstalk occurs in the viewed image due to system latency when the viewer moves quickly. Further, in recent years, in other 3D display types, such as multi-view and super multi-view, a method of improving display performance through eye-tracking has been proposed [29–32]. However, owing to the method that uses a lenticular lens or parallax barrier, the parallax is optically limited to the horizontal direction only.
In the integral 3D display using eye-tracking technology, as the area having large amounts of viewpoint information is formed as the viewing zone, the crosstalk caused by the movement of the viewer can be suppressed in both horizontal and vertical directions. Several studies that combined an integral 3D display and eye-tracking technology have been reported [33–35]. These studies succeeded in expanding the viewing angle by using eye-tracking. However, because of the lens array design of a wide optical viewing zone (OVZ), the light ray density and display performance are low. In addition, the formation of an OVZ that is robust to the movement of the viewer has not been examined.
Therefore, our proposed method aims to enhance both the viewing zone and image quality using the lens array of a long focal length, increasing the light ray density, and using eye-tracking technology. Then, using the method of changing the aspect ratio of the OVZ with a honeycomb-structure or square-structure lens array, OVZ formations suitable for the eye-tracking system are discussed. As there is a trade-off relationship between increasing the light ray density and the crosstalk caused by the viewer’s movement, an optical design guideline for efficiently forming the OVZ is important.
3. Proposed system
3.1 Enhancement of viewing zone and image quality using eye-tracking technology and the lens array of a long focal length
As shown in Fig. 1(a), in the conventional integral 3D display, a viewing zone is formed by the corresponding combination of the elemental image of an EIA and elemental lens of the lens array. This viewing zone is called an OVZ, and its angle θova is called an optical viewing angle (OVA). An OVA θova is expressed by the following equation using elemental image size e and elemental lens focal length f.
If the position of the elemental image is static, the OVZ is also static. An OVA of approximately 28° or more is required for simultaneous viewing by multiple viewers at a viewing distance of approximately 700 mm; thus, it is necessary to use a lens array with a short focal length. Consequently, a lower light ray density reduces the depth range so that the images become blurrier the further they are from the lens array plane.
Fig. 1. Integral 3D display system (a) without eye-tracking (conventional design) and (b) with eye-tracking (proposed design).
Our proposed system improves both the viewing angle and the image quality by limiting the display to a single user using eye-tracking technology (Fig. 1(b)). The position of the viewer's eyes is estimated by analyzing the images captured by a camera. Then, the elemental image is changed according to the viewer’s movement to dynamically control the OVZ, thus increasing the overall viewing zone. The overall viewing zone formed by the eye-tracking system is called the system viewing zone (SVZ), and its angle θsva is called the system viewing angle (SVA).
Furthermore, we propose a method to increase the light ray density and improve the quality of the viewed image using a lens array with a long focal length and narrowing the OVA. This method can improve both the overall viewing angle and quality of 3D images. The differences between the conventional and proposed designs are presented in Table 1.
Table 1. Differences between the conventional and proposed designs.
The system configuration is shown in Fig. 2. The system consists of a camera for eye-tracking, a high-speed EIA generator, a high pixel density display panel, and a lens array. A webcam or depth camera can be used for eye-tracking. First, the image of the viewer is acquired using the camera for eye-tracking. The image is analyzed to estimate the 3D positions of the viewer’s eyes. An EIA corresponding to the eye position is then generated. Finally, the generated EIA is displayed on the display panel equipped with the lens array. By performing this processing for each frame, it is possible to form an OVZ according to the viewer's eye position and present a wide SVZ and a high-quality integral 3D image to the viewer.
3.2 Formation of an OVZ that suppresses the occurrence of crosstalk due to flipped image
In integral imaging using a flat-panel display and a lens array, a flipping effect occurs when the viewer deviates from the primary viewing zone formed by the corresponding elemental image and elemental lens [24,25]. Figure 3 shows the configuration of the viewing zones formed by the elemental image and the lens array, as well as examples of 3D images when viewed from each viewpoint. A secondary viewing zone is formed by the adjacent elemental image and the elemental lens, and a flipped image is seen in the secondary viewing zone. Therefore, crosstalk due to flipped images occurs between the primary viewing zone and the secondary viewing zone. The crosstalk zone is formed because of lens aberration, an error in placement of the lens array, and because the number of pixels of the elemental image corresponding to the elemental lens is not an integral multiple. A 3D image in the crosstalk zone is mixed with the flipped image in the secondary viewing zone, yielding multiple images and poor visibility.
In our proposed system, the light ray density can be increased using a lens array with a long focal length, but the OVZ is reduced. As shown in Fig. 4(a), when the OVZ is too narrow and the viewer moves at a high speed, crosstalk due to the flipped image is likely to occur because of the eye position estimation error and EIA rendering delay. By giving the OVZ a sufficient margin as shown in Fig. 4(b), the effects of the eye position estimation error and EIA rendering delay can be reduced, and a robust system with less crosstalk can be constructed. That is, there is a trade-off relationship between the improvement of the light ray density by narrowing the OVZ and the perceived amount of crosstalk.
Fig. 4. Occurrence of crosstalk due to flipped image when (a) OVZ is too small and (b) OVZ has a sufficient margin.
We therefore propose a novel lens arrangement of a lens array for forming an OVZ suitable for the eye-tracking integral 3D display system with high image quality. The lens array is often a honeycomb or square structure. In the integral 3D display, the aspect ratio of the elemental image and OVZ become the same. Thus, by changing the lens arrangement of the lens array and the shape of the corresponding elemental image, the horizontal and vertical ratio of the OVZ can be changed flexibly [36]. This principle enables the formation of a low-crosstalk OVZ suitable for eye-tracking.
Considering that human eyes are positioned horizontally, it is desirable that the shape of the OVZ is horizontally oriented. Further, when a stationary display is viewed, the position of the viewer's eyes is considered to fluctuate more horizontally than vertically [37]. We consider the case of using a lens array having a honeycomb-structure, as shown in Figs. 5(a) and 5(b). The relationship between the elemental lens and elemental image (red rectangle) when the rotation of the lens array is 0° is shown in Fig. 5(a); the aspect ratio of the OVZ is $2/\sqrt 3 :1$. By rotating the lens array by 30°, the EIA can be arranged as shown in Fig. 5(b), and the aspect ratio of the OVZ is $2\sqrt 3 :1$. Hence, the OVZ can be expanded in the horizontal direction, and an OVZ that is robust against horizontal movement can be formed. Similarly, we consider the case of using a square-structure lens array as shown in Figs. 5(c) and 5(d). In the case of the 0° rotation, the arrangement of the elemental lens and elemental image match as shown in Fig. 5(c), and the aspect ratio of OVZ is 1:1. When rotated by 45° as shown in Fig. 5(d), the aspect ratio of the OVZ becomes 2:1, and the OVZ in the horizontal direction can be widened.
Fig. 5. Method for controlling the aspect ratio of OVZ by rotating the lens array. (a) 0° rotation and (b) 30° rotation of a honeycomb-structure lens array and (c) 0° rotation and (d) 45° rotation of a square-structure lens array.
Table 2 summarizes the elemental image sizes and OVZ aspect ratios for each lens array arrangement. The elemental lens pitch is p, and the horizontal and vertical sizes of the elemental image are eh and ev, respectively. The size of the elemental image is slightly enlarged with respect to the lens pitch, making the OVZ most enlarged when viewed at a certain viewing distance. The magnification ratio m is expressed by the following equation using the viewing distance L from the lens array plane and focal length f of the elemental lens.
In this manner, even with the same lens structure, the aspect ratio of the OVZ can be easily changed by rotating the lens and changing the arrangement. An appropriate aspect ratio of the OVZ can be selected according to the viewing mode. For example, when a square-structure lens array is used at a 45° rotation, the aspect ratio of the OVZ can be easily changed to 2:1 or 1:2. Therefore, the arrangement is suitable for viewing on a portable terminal. In an experiment described in Section 5.3 that investigated the relationship between the viewer’s movement and the occurrence of crosstalk due to the flipped image, a 30° rotation lens array with a honeycomb structure and the largest horizontal ratio was used to change the margin of the OVZ to various values.
Table 2. Elemental iImage sizes and OVZ aspect ratios for each lens array arrangement
4. Implementation
To implement our proposed system, we constructed a system using a camera for the eye position detection, workstation for the eye position analysis and an EIA generation, high pixel density liquid crystal display (LCD), and a lens array. To perform comparative experiments with the device of the proposed design, a device of conventional design with a static OVZ was also constructed. Table 3 details the specifications of the prototype devices. For the high pixel density display panel, a 4K resolution 9.6-inch LCD with a pixel density of 457.7 ppi was used. A square-structure lens array with a lens pitch of 0.5 mm and focal length of 1.0 mm was used in the conventional design, whereas a honeycomb-structure lens array with a lens pitch of 0.5 mm and focal length of 2.0 mm was used in the proposed design. The lens array arrangement of the conventional design was 0° rotation. The lens array arrangement of the proposed design was 30° rotation to enhance the OVZ horizontally.
Table 3. Specifications of prototypes of conventional and proposed design.
For eye-tracking, a camera was attached to the top of the 3D display, the viewer’s image was captured, and the eye position was estimated using a face detection program. The camera used for eye-tracking had 60 frames per second (fps), 640 × 360 resolution, and 78° lens angle of view. We used Dlib [38], an open-source machine learning library, to detect face landmarks. The 3D position of both eyes was estimated by solving the perspective-n-point (PnP) problem based on the coordinates of the acquired face landmarks. When solving the PnP problem, the interpupillary distance was set to 64 mm. By comparison with the actual measurement, the estimation error was suppressed below 10 mm in all XYZ directions. Further, by acquiring the 3D position of the eyes, the virtual camera array could move not only in the XY-plane but also in the Z direction (depth direction). Hence, the OVZ could be optimized even when the viewer moved back and forth.
To control the OVZ according to the eye position of the viewer, an EIA is generated and updated for each frame. As a method for rendering an EIA at high speeds using computer graphics technology, a method for rearranging pixels of multi-viewpoint images at high speeds by graphics processing unit (GPU) parallel processing and generation of an EIA has been proposed [39–41]. As shown in Fig. 6, a 3D model is placed in a virtual space, and multi-viewpoint images from various directions are captured by a virtual camera array. The projection direction of each lens-shifted camera is set toward the center of the lens array. An EIA is generated by mapping the pixels of the acquired multi-viewpoint images according to the arrangement of the lens array. To change the EIA in accordance with the eye position, the virtual camera array is dynamically moved in 3D so that the center of the virtual camera array coincides with the center of the OVZ. Thus, the system is controlled in a way that the center of the OVZ is always located at the center of both eyes of the viewer.
Real-time EIA generation was performed using a workstation with Intel Core i7-7800X 3.50 GHz CPU, 64 GB memory, and NVIDIA Quadro P6000 GPU. In a scene with 2,602,342 polygons, when the number of virtual cameras for EIA generation was 16 horizontally and 4 vertically, the rendering speed of the EIA was approximately 62.6 fps, including the calculation of the eye position estimation. It was possible to generate the EIA according to the eye position in real time. Unity, which is a cross-platform game engine, was used to implement the EIA generation program. As the EIA is dynamically changed for each frame, the eye-tracking display can be naturally performed not only in the still image scene, but also in the moving image scene.
5. System evaluation
5.1 Viewing zone enhancement
Figure 7 illustrates an integral 3D image when viewed from the upper, lower, left, and right ends in the case of the conventional and proposed designs, with and without eye-tracking. In the conventional design, a square-structure lens array (0° rotation) with a lens pitch of 0.5 mm and focal length of 1.0 mm was used. The viewing angle (OVA) was 28.1°, both horizontally and vertically. As the conventional design had no eye-tracking function, the viewing zone (OVZ) was static. On the other hand, in the proposed design, a honeycomb-structure lens array (30° rotation) with a lens pitch of 0.5 mm and focal length of 2.0 mm was used. The OVA (without tracking) was 24.4° horizontally and 7.2° vertically, and the viewing angle (SVA) was expanded to 81.4° horizontally and 47.6° vertically using eye-tracking. This is horizontally 2.9 times and vertically 1.7 times the conventional design and horizontally 3.3 times and vertically 6.6 times the proposed design without eye-tracking. The SVA is determined by the lens angle of view of the eye-tracking camera used. The lens angle of view of the eye-tracking camera used in the prototype was 78°. In the proposed method, the SVA can be further expanded using a camera with a wider-angle lens.
Fig. 7. Integral 3D image when viewed from the upper, lower, left, and right ends in the case of (a) conventional design, (b) proposed design (without eye-tracking), and (c) proposed design (with eye-tracking).
5.2 Depth range enhancement of 3D images
The depth range of reconstructed 3D images was evaluated in the prototype devices of the conventional and proposed designs. Although the lens pitch of the lens array used for each prototype device was the same (0.5 mm), the focal length differences doubled (1 mm and 2 mm, respectively). Therefore, in the proposed design the light ray density doubled both horizontally and vertically.
In integral 3D displays, the viewing spatial frequency β refers to the spatial frequency when the projected image is viewed from the viewpoint. It is expressed by the following equation using the spatial frequency αp determined by the pixel pitch.
where L is the viewing distance, z is the distance from the lens array to the display position of the 3D image (positive is in front), f is the focal length of the lens array, and p is the pixel pitch. The maximum spatial frequency is restricted by the Nyquist frequency βn at which the 3D image is sampled by the elemental lens, and it is represented by the following equation. where Pl indicates the lens pitch. The upper-limit spatial frequency γ is expressed by the following equation by using the viewing spatial frequency β and Nyquist frequency βn [13]. Figure 8 is a graph showing the relationship between the displayed position of the integral 3D image and upper-limit spatial frequency in the prototype devices of the conventional and proposed designs. These are shown in the horizontal and vertical directions. The viewing distance was assumed to be 700 mm. As a result of the simulation, the depth range of reconstructed 3D images improved both horizontally and vertically at all spatial frequencies. For example, at the spatial frequency of 12.2 cycles/deg, the depth range of the reconstructed 3D images improved to approximately twice, from 18 mm to 36 mm, in both horizontal and vertical directions. The spatial frequency given in cycles/deg denotes the perceived spatial resolution at the viewing distance of 700 mm.Fig. 8. Upper-limit spatial frequency characteristics in (a) the horizontal and (b) vertical directions.
In order to investigate how much spatial frequency can actually be resolved according to the reconstructed image distance, the EIA of the wedge chart was displayed on the display of the conventional and proposed designs, and the shooting experiment was performed while changing the position of the reconstructed image through the lens array. Figures 9(a) and 9(b) illustrate the reconstructed images of a wedge chart displayed −40, −20, 0, 20, and 40 mm away from the position of the lens array in the horizontal and vertical directions. Figures 9(c) and 9(d) are the enlarged images at 40 mm. In the display of the conventional design, as shown in Figs. 9(c) and 9(d), the aliasing occurs at the spatial frequency of 2.6 cycles/deg both horizontally and vertically. This aliasing indicates that at the spatial frequency of more than 2.6 cycles/deg, the image could not be properly resolved at a position of 40 mm. On the other hand, in the proposed design, the aliasing does not occur up to a spatial frequency of 5.2 cycles/deg at the position of 40 mm. The actual measurement and the simulation of the upper-limit spatial frequency shown in Fig. 8 yielded the same results with respect to the spatial frequency at which aliasing occurs. As shown in Fig. 8, at the spatial frequency of 5.2 cycles/deg, the depth range of the reconstructed 3D images improves to approximately twice, from 42 mm to 85 mm, in both horizontal and vertical directions. Thus, it was confirmed that the lens array with a long focal length improved the depth range of the reconstructed 3D images.
Fig. 9. (a) Horizontal and (b) vertical resolution characteristics when a reconstructed image of a wedge chart is displayed at various depth positions using the display of the conventional and proposed designs. (c) and (d) are the enlarged images of (a) and (b) in the blue and red rectangles.
5.3 Evaluation of OVZ margin
Regarding the margin of the OVZ, we conducted an experiment to evaluate the required margin in the proposed system for stable viewing. As shown in Fig. 10(a), the apparatus was configured using an eye-tracking integral 3D display, a mask mounted with a camera, and a slider capable of tracking shooting. A camera was attached to an eye of the mask to treat it as a virtual human eye. The mask was attached to the slider stage for tracking shooting that can fix the gazing point, and the video was shot while moving the mask horizontally. As shown in Fig. 10(b), in the integral 3D display, a white image was displayed in the peripheral part of the OVZ, and a black image was displayed in the central part of the OVZ. By changing the ratio of the black and white images, the size of the margin of the OVZ could be changed virtually. The shooting was performed at a position 700 mm away from the 3D display.
Fig. 10. (a) Experimental setup to evaluate OVZ margin. (b) Method of changing the margin of OVZ. The margin was virtually changed by changing the ratio of black and white images in the OVZ. (c) Calculation of crosstalk index value according to eye movement speed. Display example when eye movement is slow (left) and fast (right).
As shown in Fig. 10(c), when the mask moved slowly, the screen was stably displayed in black. On the other hand, when the mask moved quickly, part of the screen was displayed as a white image because of the effect by eye detection error and EIA rendering delay. The ratio of white display on the screen corresponds to the degree of occurrence of crosstalk due to the flipped image. Here, the average luminance value of the display screen in the captured image is considered as a crosstalk index value Iv. The eye movement speed v was derived by estimating the camera position (i.e., eye position) for every frame using a chess board placed under the display and then calculating based on differences between each frame.
Based on the eye position coordinates and image on the screen for each frame, the average luminance value, Lv, of the display screen according to the eye speed was derived. Figure 11 illustrates the relationship between eye movement speed v and crosstalk index value Iv when the margin of the OVZ was changed. The margin of the OVZ was calculated with an interpupillary distance of 64 mm. In each margin, crosstalk index value Iv was normalized using average luminance value Lmin when the movement speed is 0 and average luminance value Lmax when a white image is displayed. Crosstalk index value Iv is represented by the following equation.
From the graph, it can be seen that when the margin of the OVZ is 101 mm or more, the crosstalk index value is 0.1 or less, and almost no crosstalk due to the flipped image occurs even when watching while moving at 340 mm/s. When using the 30° rotation lens array of the proposed design, the OVZ margin was 120 mm, making stable viewing possible with almost no crosstalk due to the flipped image. On the other hand, when the margin of the OVZ was 63 mm or less, the crosstalk index value was 0.7 or more at 340 mm/s, resulting in poor visibility. As the margin of the OVZ is 55.8 mm for 0° rotation and 120 mm for 30° rotation, the proposed lens array arrangement can significantly reduce the occurrence of crosstalk caused by the viewer’s movement. Such relationship between the margin and crosstalk will be useful as a design guideline when designing an eye-tracking integral 3D display in the future.Fig. 11. Relationship between eye movement speed v and crosstalk index value Iv when the OVZ margin is changed.
The degree of occurrence of crosstalk due to the flipped image depends on the latency of the system. The total latency of the system is caused by the sensing time of the camera; the computer processing time for detecting the viewer’s eyes and rendering the EIA according to the eye position; the data communication time between the camera, the workstation, and the display; and the time for refreshing the LCD. The average computer processing time for detecting the viewer's eyes and rendering the EIA was 16.0 ms when the number of virtual cameras was 16 horizontally and 4 vertically. Therefore, by adding the time consumption for sensing the camera with a frame rate of 60 fps, refreshing the LCD with a refresh rate of 60 Hz, and communicating the data, a total latency of approximately 60 ms occurs [42,43]. When a viewer moves at a high speed, motion blur causes an increase in the number of frames in which the eyes cannot be detected, resulting in further delay. By shortening the exposure time to reduce eye detection errors due to motion blur, or by analyzing more images using a camera with a high frame rate, the required OVZ margin can be suppressed.
6. Conclusion
In this study, we proposed a method that efficiently suppresses crosstalk due to the flipped image in eye-tracking integral 3D display systems by rotating the lens array and changing lens arrangement to widen the OVZ in the horizontal direction. We investigated the relationship between the movement speed of the viewer’s eye and the amount of crosstalk when the margin of the OVZ was changed, and we found that the proposed lens arrangement is effective in reducing crosstalk caused by the viewer’s movements. In the comparative experiments using prototype devices of the conventional and proposed designs, it was confirmed that the SVZ expanded by controlling the OVZ according to eye position, and the depth range of reconstructed 3D images improved by increasing the light ray density using a lens array with a long focal length. The proposed system can enhance the viewing zone and depth range of 3D images, as well as reduce crosstalk due to the flipped image in a well-balanced manner. As the device configuration is simple, it can be applied to future practical 3D display systems.
Disclosures
The authors declare no conflicts of interest.
References
1. M. G. Lippmann, “Épreuves, réversibles donnant la sensation du relief,” J. Phys. 7(1), 821–825 (1908). [CrossRef]
2. S. Yano, M. Emoto, and T. Mitsuhashi, “Two factors in visual fatigue caused by stereoscopic HDTV images,” Displays 25(4), 141–150 (2004). [CrossRef]
3. Y. Kim, J. Kim, K. Hong, H. K. Yang, J. H. Jung, H. Choi, S. W. Min, J. M. Seo, J. M. Hwang, and B. Lee, “Accommodative response of integral imaging in near distance,” J. Display Technol. 8(2), 70–78 (2012). [CrossRef]
4. Y. Igarashi, H. Murata, and M. Ueda, “3-D display system using a computer generated integral photograph,” Jpn. J. Appl. Phys. 17(9), 1683–1684 (1978). [CrossRef]
5. N. Davies, M. McCormick, and L. Yang, “Three-dimensional imaging systems: a new development,” Appl. Opt. 27(21), 4520–4528 (1988). [CrossRef]
6. F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Real-time pickup method for a three-dimensional image based on integral photography,” Appl. Opt. 36(7), 1598–1603 (1997). [CrossRef]
7. H. Arimoto and B. Javidi, “Integral three-dimensional imaging with digital reconstruction,” Opt. Lett. 26(3), 157–159 (2001). [CrossRef]
8. B. Javidi, I. Moon, and S. Yeom, “Three-dimensional identification of biological microorganism using integral imaging,” Opt. Express 14(25), 12096–12108 (2006). [CrossRef]
9. J. Arai, M. Okui, T. Yamashita, and F. Okano, “Integral three-dimensional television using a 2000-scanning-line video system,” Appl. Opt. 45(8), 1704–1712 (2006). [CrossRef]
10. K. Suehiro, M. Yoshimura, Y. Haino, M. Sato, J. Arai, M. Kawakita, and F. Okano, “Integral 3D TV using ultrahigh-definition D-ILA device,” Proc. SPIE 6803, 680318 (2008). [CrossRef]
11. M. Kawakita, H. Sasaki, J. Arai, M. Okui, F. Okano, Y. Haino, M. Yoshimura, and M. Sato, “Projection-type integral 3-D display with distortion compensation,” J. Soc. Inf. Disp. 18(9), 668–677 (2010). [CrossRef]
12. J. Arai, M. Kawakita, T. Yamashita, H. Sasaki, M. Miura, H. Hiura, M. Okui, and F. Okano, “Integral three-dimensional television with video system using pixel-offset method,” Opt. Express 21(3), 3474–3485 (2013). [CrossRef]
13. H. Hoshino, F. Okano, and I. Yuyama, “Analysis of resolution limitation of integral photography,” J. Opt. Soc. Am. A 15(8), 2059–2065 (1998). [CrossRef]
14. B. Lee, S. W. Min, and B. Javidi, “Theoretical analysis for three-dimensional integral imaging systems with double devices,” Appl. Opt. 41(23), 4856–4865 (2002). [CrossRef]
15. N. Okaichi, M. Miura, J. Arai, M. Kawakita, and T. Mishina, “Integral 3D display using multiple LCD panels and multi-image combining optical system,” Opt. Express 25(3), 2805–2817 (2017). [CrossRef]
16. N. Okaichi, M. Kawakita, H. Sasaki, H. Watanabe, and T. Mishina, “High-quality direct-view display combining multiple integral 3D images,” J. Soc. Inf. Disp. 27(1), 41–52 (2019). [CrossRef]
17. H. Sasaki, N. Okaichi, H. Watanabe, M. Kano, M. Miura, M. Kawakita, and T. Mishina, “Color moiré reduction and resolution enhancement of flat-panel integral three-dimensional display,” Opt. Express 27(6), 8488–8503 (2019). [CrossRef]
18. H. Liao, M. Iwahara, N. Hata, and T. Dohi, “High-quality integral videography using a multiprojector,” Opt. Express 12(6), 1067–1076 (2004). [CrossRef]
19. Y. Takaki and N. Nago, “Multi-projection of lenticular displays to construct a 256-view super multi-view display,” Opt. Express 18(9), 8824–8835 (2010). [CrossRef]
20. J. Y. Jang, D. Shin, B. G. Lee, and E. S. Kim, “Multi-projection integral imaging by use of a convex mirror array,” Opt. Lett. 39(10), 2853–2856 (2014). [CrossRef]
21. M. Kawakita, S. Iwasawa, R. Lopez-Gulliver, and N. Inoue, “Glasses-free large-screen three-dimensional display and super multiview camera for highly realistic communication,” Opt. Eng. 57(06), 1 (2018). [CrossRef]
22. N. Okaichi, M. Miura, H. Sasaki, H. Watanabe, J. Arai, M. Kawakita, and T. Mishina, “Continuous combination of viewing zones in integral three-dimensional display using multiple projectors,” Opt. Eng. 57(6), 061611 (2018). [CrossRef]
23. H. Watanabe, N. Okaichi, T. Omura, M. Kano, H. Sasaki, and M. Kawakita, “Aktina Vision: Full-parallax three-dimensional display with 100 million light rays,” Sci. Rep. 9(1), 17688 (2019). [CrossRef]
24. C. G. Luo, C. C. Ji, F. N. Wang, Y. Z. Wang, and Q. H. Wang, “Crosstalk-free integral imaging display with wide viewing angle using periodic black mask,” J. Display Technol. 8(11), 634–638 (2012). [CrossRef]
25. W. Zhang, X. Sang, X. Gao, X. Yu, C. Gao, B. Yan, and C. Yu, “A flipping-free 3D integral imaging display using a twice-imaging lens array,” Opt. Express 27(22), 32810–32822 (2019). [CrossRef]
26. K. Perlin, S. Paxia, and J. Kollin, “An autostereoscopic display,” Proc. SIGGRAPH 2000, 319–326 (2000). [CrossRef]
27. N. A. Dodgson, “On the number of viewing zones required for head-tracked autostereoscopic display,” Proc. SPIE 6055, 60550Q (2006). [CrossRef]
28. S. Ishizuka, T. Mukai, and H. Kakeya, “Multi-phase convex lens array for directional backlights to improve luminance distribution of autostereoscopic display,” IEICE Trans. Electron. E98.C(11), 1023–1027 (2015). [CrossRef]
29. K. H. Yoon, M. K. Kang, H. Lee, and S. K. Kim, “Autostereoscopic 3D display system with dynamic fusion of the viewing zone under eye tracking: principles, setup, and evaluation [Invited],” Appl. Opt. 57(1), A101–A117 (2018). [CrossRef]
30. S. Lee, J. Park, J. Heo, B. Kang, D. Kang, H. Hwang, J. Lee, Y. Choi, K. Choi, and D. Nam, “Autostereoscopic 3D display using directional subpixel rendering,” Opt. Express 26(16), 20233–20247 (2018). [CrossRef]
31. Y. Watanabe and H. Kakeya, “A full-HD super-multiview display based on adaptive time-division multiplexing parallax barrier,” ITE Trans. on MTA 8(4), 230–237 (2020). [CrossRef]
32. G. J. Lv, B. C. Zhao, and F. Wu, “Real-time viewpoint generation-based 3D display with high-resolution, low-crosstalk, and wide view range,” Opt. Eng. 59(10), 103104 (2020). [CrossRef]
33. G. Park, J. Hong, Y. Kim, and B. Lee, “Enhancement of viewing angle and viewing distance in integral imaging by head tracking,” Proc. Digital Holography and Three-Dimensional Imaging, DWB27 (2009).
34. G. Park, J. H. Jung, K. Hong, Y. Kim, Y. H. Kim, S. W. Min, and B. Lee, “Multi-viewer tracking integral imaging system and its viewing zone analysis,” Opt. Express 17(20), 17895–17908 (2009). [CrossRef]
35. Z. L. Xiong, Q. H. Wang, S. L. Li, H. D. Deng, and C. C. Ji, “Partially-overlapped viewing zone based integral imaging system with super wide viewing angle,” Opt. Express 22(19), 22268–22277 (2014). [CrossRef]
36. M. Miura, J. Arai, T. Mishina, M. Okui, and F. Okano, “Integral imaging system with enlarged horizontal viewing angle,” Proc. SPIE 8384, 83840O (2012). [CrossRef]
37. Y. Sawahata and T. Morita, “Estimating depth range required for 3-D displays to show depth-compressed scenes without inducing sense of unnaturalness,” IEEE Trans. Broadcast. 64(2), 488–497 (2018). [CrossRef]
38. D. E. King, “Dlib-ml: A machine learning toolkit,” J. Mach. Learn. Res. 10, 1755–1758 (2009).
39. S. W. Min, K. S. Park, B. Lee, Y. Cho, and M. Hahn, “Enhanced image mapping algorithm for computer-generated integral imaging system,” Jpn. J. Appl. Phys. 45(No. 28), L744–L747 (2006). [CrossRef]
40. B. N. R. Lee, Y. Cho, K. S. Park, S. W. Min, J. S. Lim, M. C. Whang, and K. R. Park, “Design and implementation of a fast integral image rendering method,” ICEC, 135–140 (2006).
41. K. S. Park, S. W. Min, and Y. Cho, “Viewpoint vector rendering for efficient elemental image generation,” IEICE Trans. Inf. Syst. E90-D(1), 233–241 (2007). [CrossRef]
42. H. Imai, M. Imai, Y. Ogura, and K. Kubota, “Eye-position tracking stereoscopic display using image-shifting optics,” Proc. SPIE 2653, 49–55 (1996). [CrossRef]
43. H. Zhang, M. Chen, X. Li, K. Li, X. Chen, J. Wang, H. Liang, J. Zhou, W. Liang, H. Fan, R. Ding, S. Wang, and D. Deng, “Overcoming latency with motion prediction in directional autostereoscopic displays,” J. Soc. Inf. Disp. 28(3), 252–261 (2020). [CrossRef]
