Open Access
Add to BibSonomy
Abstract
Electroholography can produce natural 3D scenes and has gained recognition as an ideal 3D technology. However, insufficient computational power has made it difficult to achieve real-time electroholography. In this paper, we developed a compact special-purpose system for calculating phase-only holograms. We implemented the developed system using a system on a chip embedded with a processor and logic circuit blocks. Our system successfully computed holograms of 1,920×1,080 pixels from a point-cloud with 32,500 points at 10 frames per second. The system is 147 times faster than a personal computer (with 6 CPU cores).
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Holography [1] is a technique that uses the interference and diffraction of light. Electronically realized holography is called electroholography [2], which was presented in the 1990s. Electroholography can produce natural three-dimensional (3D) scenes and has gained recognition as an ideal 3D technology [3,4]. In electroholography, computer-generated holograms (CGHs) are displayed on a spatial light modulator (SLM), and it is possible to create a movie by switching CGHs at high speed. However, the lack of the resolution and size of the SLMs and the insufficient computational power have made it difficult to achieve real-time electroholography [5]. In recent years, the pixel range of SLMs has approached approximately 1 μm and is approaching practical use [6,7]; larger screens have been developed, and 8K4K SLMs are now available [8]. Also, as the number of screen pixels increases, the computational load of CGHs increases. Speeding CGH computation is one of the most critical issues that should be solved.
Many works on improving the computation of CGHs using a look-up table [9,10], recurrence relation [11,12], a wavefront-recording plane [13,14], a sparsity-based model [15,16], a patch model [17], a polygon model [18,19], ray-wavefront conversion [20,21], and layer-based [22,23] methods have been reported. Moreover, there are approaches using graphics processing units (GPUs) [24–26] and, application specific integrated circuits (ASICs) [27] and field-programmable gate arrays (FPGAs) [28–31] to accelerate CGH computation. The acceleration of software-based methods were limited to to a few 10 to 100 times, but hardware-based methods [32] are very effective because they have achieved CGH computation speed that is 1-100 times faster than commercial computers.
In this study, we developed a compact special-purpose system using a system on a chip (SoC) design to accelerate hologram computation. Recently, holographic head-mounted displays (HMDs) have attracted much attention because they can sufficiently satisfy human depth cues [33–35]. In our previous works [36–38], we have implemented small special-purpose computers for HMD applications. In [38], we have presented a special-purpose computer for amplitude holograms using the Xilinx ZYNQ MPSoC (ZYNQMP). ZYNQMP is an SoC that incorporates an embedded CPU (ARM CPU) and an FPGA. In our previous work [38], an FPGA-based CGH circuit was used as the accelerator for hologram computation, and an embedded CPU performed the tasks of an operating system, auxiliary processing (e.q. the rotation of point-cloud), and controlled the FPGA-based CGH circuit, making it possible to realize high-speed CGH computation on a single chip. Computer systems that can accelerate hologram computation are a large computer cluster that uses GPUs or FPGAs [26,29,39]. Hence, they are not suitable for HMD applications. On the other hand, our system is a stand-alone system that is implemented on a single chip, and it is expected to be applied to HMDs.
The difference between this study and our previous study [38] is that we used a recurrence relation algorithm [40] to develop circuits for phase-only CGHs. The proposed special-purpose computer system is 147 times faster than the Intel Core i5-8400 with six CPU cores and 66 times faster than the NVIDIA Jetson TX1 with 256 GPU cores. We have successfully developed a system to compute and display 1,920$\times$1,080-pixel phase-only CGHs at 10 frames per second (fps) from a point-cloud with 32,500 points.
2. Principle of computer-generated hologram
2.1 Computer-generated hologram
A point-cloud-based complex hologram can be expressed under the condition of $z_j \gg x_j, y_j$ as follows:
where $x_a$ and $y_a$ are coordinates on a CGH, $M$ is the number of point-clouds, $A_j$ is the amplitude of a point light source, $\rho _j$ is $1/\lambda |z_j|$, $\lambda$ is the wavelength, and $x_j, y_j,$ and $z_j$ are the coordinates of the point-clouds, respectively. Equation 1 comprises the amplitude and phase and produces an ideal image reconstruction. However, commercial SLMs can only display either the amplitude or phase of light. Phase-only holograms are generated by extracting the phase part from complex holograms by: where ${\textrm{Re}}\{u_c\}$ and $\textrm {Im}\{u_c\}$ denote the operators adopting real and imaginary parts of a complex amplitude. Phase-only holograms have an advantage over amplitude-only holograms because of their efficient light usage and the absence of conjugate light [41].2.2 Recurrence relation algorithm
A hardware-effective computation called recurrence relation algorithm [40] is described. In this study, we defined recurrence relation algorithm:
where $x_0$ denote most left side coordinates in CGH and $\Delta _{0j}$ denotes an initial value. In the recurrence relation algorithm, the phase $\theta _{0j}$ is initially calculated using Eq. (2). The $n$-th phase $\theta _{nj}$ is calculated by the following recurrence relation: $\Delta _{nj}$ is updated byBy recalculating Eq. (6) and Eq. (7) along the $x$-axis, we can obtain phases-only holograms by addition. When this is implemented in hardware, we can build efficient circuits that use fewer resources. Also, the recurrence relation algorithm can be performed using pipeline operations.
3. Implementation of special-purpose computer
3.1 Hardware
We used Zynq UltraScale+ MPSoC ZCU102 Evaluation Kit (ZCU102), released by Xilinx, for implementing our proposed system. ZCU102 is an SoC with an embedded CPU and FPGA (XCZU9EG-2FFVB1156). Table 1 shows the specification of the embedded CPU. Also, the specification of the FPGA will be shown later (Table 2). The clock frequencies of the embedded CPU and FPGA are 1,200 MHz and 375 MHz, respectively.
Table 1. Embedded CPU of ZCU102.
Table 2. Resource usage.
3.2 Implementation of system on chip
Figure 1 shows the block diagram of the CGH computation system in ZYNQMP. The embedded CPU executed an operating system (i.e., an Ubuntu 18.04.3 with Linux Kernel 4.19.0), performed auxiliary processing, and controlled the FPGA. The auxiliary processing is the calculation of $\rho _j=1/2\lambda |z_j|$, which includes the division. Implementing the division in FPGAs led to an increase in resource usage and a decrease in operating frequency. So instead of using the hardware divider, we used the embedded CPU to calculate $\rho _j$. Then, the calculated $\rho _j$ is sent to the FPGA, and the FPGA computes CGHs. Another auxiliary processing is the rotation of point-cloud. The proposed system implemented on a single chip can be executed in a stand-alone system, which is more compact. However, most of the CGH computing systems using FPGAs and GPUs tend to be large in size, since they use a cluster system with FPGAs or GPUs [24–26,29,30].
3.3 Implementation of field-programmable gate array
We implemented a CGH computational circuit in an FPGA based on the recurrence relation algorithm. The block diagram of the computational circuit is shown in Fig. 2. The circuit receives the total number of point-clouds, the CGH size, and the point-cloud coordinates that contain the calculated $\rho _j$ from the embedded CPU, respectively. The point-cloud is stored in Block RAMs, which are FPGA memory blocks. The CGH computation units are executed in parallel along the $x$-axis of the CGH. In Fig. 2, the basic phase unit (BPU) computes Eq. (2), Eq. (4), and Eq. (5). The additional phase units (APUs) perform Eq. (1), Eq. (6), and Eq. (7), which are the additive phase in the recurrence relation algorithm. The system hopes to achieve a CGH of 1,920$\times$1,080 pixels, so we implemented one BPU and 1,919 APUs as a single-line computing unit. To achieve high-speed calculations, 1,920 computational units were used to perform parallel computations efficiently.
Computed complex holograms are sent to a normalization unit (NU) through a multiplexer (mux). In the NU, the real and imaginary parts are respectively normalized to 5 bits to address the arctan look-up table, which is used to calculate arctan operations. Computed phase holograms are stacked in a first-in-first-out manner and connected to a frame buffer for displaying outputs, subsequently sending outputs to an SLM.
For the calculation of cosine, sine, and arctan, we used a look-up table method to store calculated results in memory in advance and quickly compute them by address-referencing. The required bit accuracy for the CGH computation in our circuit was based on a previous study [30]: a 6-bit accuracy for cosine and sine, and a 10-bit input for arctan–5 bits for real and imaginary parts, respectively.
4. Results
Table 2 shows the hardware resource usage of our compact special-purpose system implemented on a ZCU102. "Flip Flop" is fast one-bit memory, but the capacity of memory is small. "Look-up Table" provides logic functions. The circuit consists of Flip-Flops and Look-up tables. "Block RAM" is large memory on the FPGA. Block RAMs are mainly used for maintaining point cloud data.
The operating frequency of the FPGA decreases as the number of units increases because of the difficulty of the placement and routing of FPGA compilation. Note that the FPGA we used could only operate at the frequency of 375 MHz, 250 MHz, and 100 MHz. Figure 3 shows the operating frequency as the function of the number of units. The best computational performance can be evaluated by the operating frequency $\times$ the number of units. Thus, we chose 375 MHz and 1,920 units to obtain the best computational performance.
4.1 Calculation time
The theoretical computation time of the calculation circuit (Fig. 2) while neglecting pipeline delay, is given by
where $M$ is the number of point-clouds, and $K$ is the number of pixels in the hologram. When the number of point clouds is 32,500, and the hologram has 2,073,600 pixels (1,920$\times$1,080-pixel), the computation time is approximately 0.1 seconds.Table 3 shows the actual computation time for FPGA–ZCU102, GPU–NVIDIA Jetson TX1, and CPU–Intel Core i5-8400. The proposed system’s computing speed was almost as fast as the theory. We measured the computation time for the CPU (a desktop PC) and GPU under the same conditions. For the desktop PC, we used Core i5-8400 CPU 2.80 GHz, Ubuntu 18.04.4 (Linux Kernel 4.15.0-96), and the Intel C compiler 19.0.5.281. In the environment, we used the recurrence relation algorithm, N-LUT [42] and Frenel approximation of Eqs. (1)–(3) as the comparison. All the CPU cores were utilized and executed in parallel; for the GPU, we used the NVIDIA Jetson TX1 (256 CUDA Cores) on the NVIDIA CUDA compiler 8.0. In the environment, we compared the recurrence relation algorithm and Frenel approximation as the comparison. All the GPU cores were used and executed in parallel. As shown in Table 3, the proposed special-purpose system is 147 times faster than the CPU and 61 times faster than the GPU.
Table 3. Calculation time: fps denotes frame per second.
4.2 Image quality
We used the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) for image quality validation. Figure 4(a) shows an original 3D data and Figs. 4(b) and 4(c) show numerically reconstructed images from a phase-only hologram obtained using the CPU and the proposed system. The computation conditions used for the evaluation are as follows: the wavelength was 633 nm, the pixel pitch of the hologram was 8 µm, the distance between the holograms and point cloud was 0.5 m, and the resolution of the hologram was 1,920$\times$1,080 pixels. Comparing Figs. 4(b) and 4(c), the PSNR and SSIM are 29.4 dB and 0.96, respectively.
The proposed system was connected to an optical system, as shown in Fig. 5. The point-clouds are stored in a secure digital memory card on the ZCU102 and read from the embedded CPU. The rotated point-cloud data is transferred to a special-purpose circuit in the FPGA. The FPGA performs hologram computation in real-time and displays the holograms on an SLM. Figures 6(a) and 6(b) show the optical reconstructions of a phase-only hologram obtained using the CPU and the proposed system. For this, we used a phase-modulated based SLM (HoloEye PLUTO-2) with a pixel pitch of 8 µm and a resolution of 1,920$\times$1,080 pixels. A 532 nm laser was used as the light source. The output lens in Fig. 5 was used to capture reconstructed images with a camera. We have successfully developed a system that can compute 1,920$\times$1,080 pixels of CGHs at 10 fps from 32,500 points.
Fig. 6. Comparison of 3D data and optical reconstructions obtained using the CPU and proposed system (Visualization 1).
5. Conclusion and future works
In this study, we developed a compact special-purpose system for phase-only holograms. The system used an SoC embedded with a CPU and an FPGA. We implemented a dedicated computational circuit for the system, using the recurrence relation algorithm [40]. Consequently, the proposed system can generate CGHs of 1,920$\times$1,080 pixels from a point-cloud with 32,500 points at 10 fps. The proposed special-purpose system was 147 times faster than the Intel Core i5-8400 and 66 times faster than the NVIDIA Jetson TX1 GPU, designed to be used in embedded devices.
The proposed system implemented an embedded CPU and a special-purpose computer on a single chip. The entire system can be easily integrated into an optical system. Although NVIDIA Jetson TX1 is a dedicated compact computer for embedded devices, the proposed system is 66 times faster than the NVIDIA Jetson TX1. The NVIDIA Jetson is a low power design, with catalog specs ranging from 10 to 30 W. In the proposed system, the power consumption is 13.9 W. They have roughly equivalent power consumption. Evaluation results show the usefulness of the proposed system in holographic HMD applications.
In this study, we used an evaluation board as a prototype for implementing the proposed system. The size of the evaluation board is 25 cm $\times$ 25 cm, while the size of the SoC in the evaluation board is only 3.5 cm $\times$ 3.5 cm. Therefore, it is possible to embed the proposed system into HMDs. For future works, we plan to develop a dedicated board with LCDs and FPGAs, as done in [36,37], and evaluate the board for HMD applications.
Funding
Yazaki Memorial Foundation for Science and Technology; Japan Society for the Promotion of Science (19H01097, 19H04132, 20J10202).
Disclosures
The authors declare no conflicts of interest.
References
1. D. Gabor, “A new microscopic principle,” Nature 161(4098), 777–778 (1948). [CrossRef]
2. P. St-Hilaire, S. A. Benton, M. E. Lucente, M. L. Jepsen, J. Kollin, H. Yoshikawa, and J. S. Underkoffler, “Electronic display system for computational holography, in Practical Holography IV, vol. 1212S. A. Benton, ed., International Society for Optics and Photonics (SPIE, 1990), pp. 174–182.
3. S. Reichelt, R. Häussler, G. Fütterer, and N. Leister, “Depth cues in human visual perception and their realization in 3D displays,” in Three-Dimensional Imaging, Visualization, and Display 2010 and Display Technologies and Applications for Defense, Security, and Avionics IV, vol. 7690B. Javidi, J.-Y. Son, J. T. Thomas, and D. D. Desjardins, eds., International Society for Optics and Photonics (SPIE, 2010), pp. 92–103.
4. P. W. M. Tsang and T.-C. Poon, “Review on the state-of-the-art technologies for acquisition and display of digital holograms,” IEEE Transactions on Ind. Informatics 12(3), 886–901 (2016). [CrossRef]
5. M. Lucente, “Interactive three-dimensional holographic displays: seeing the future in depth,” SIGGRAPH Comput. Graph. 31(2), 63–67 (1997). [CrossRef]
6. K.-i. Aoshima, N. Funabashi, K. Machida, Y. Miyamoto, K. Kuga, T. Ishibashi, N. Shimidzu, and F. Sato, “Submicron magneto-optical spatial light modulation device for holographic displays driven by spin-polarized electrons,” J. Disp. Technol. 6(9), 374–380 (2010). [CrossRef]
7. Y. Isomae, Y. Shibata, T. Ishinabe, and H. Fujikake, “Design of 1-μm-pitch liquid crystal spatial light modulators having dielectric shield wall structure for holographic display with wide field of view,” Opt. Rev. 24(2), 165–176 (2017). [CrossRef]
8. K. Wakunami, P.-Y. Hsieh, R. Oi, T. Senoh, H. Sasaki, Y. Ichihashi, M. Okui, Y.-P. Huang, and K. Yamamoto, “Projection-type see-through holographic three-dimensional display,” Nat. Commun. 7(1), 12954–7 (2016). [CrossRef]
9. M. E. Lucente, “Interactive computation of holograms using a look-up table,” J. Electron. Imaging 2(1), 28–35 (1993). [CrossRef]
10. S. Jiao, Z. Zhuang, and W. Zou, “Fast computer generated hologram calculation with a mini look-up table incorporated with radial symmetric interpolation,” Opt. Express 25(1), 112–123 (2017). [CrossRef]
11. K. Matsushima and M. Takai, “Recurrence formulas for fast creation of synthetic three-dimensional holograms,” Appl. Opt. 39(35), 6587–6594 (2000). [CrossRef]
12. H. Yoshikawa, “Fast computation of fresnel holograms employing difference,” Opt. Rev. 8(5), 331–335 (2001). [CrossRef]
13. T. Shimobaba, N. Masuda, and T. Ito, “Simple and fast calculation algorithm for computer-generated hologram with wavefront recording plane,” Opt. Lett. 34(20), 3133–3135 (2009). [CrossRef]
14. A. Symeonidou, D. Blinder, A. Munteanu, and P. Schelkens, “Computer-generated holograms by multiple wavefront recording plane method with occlusion culling,” Opt. Express 23(17), 22149–22161 (2015). [CrossRef]
15. T. Shimobaba and T. Ito, “Fast generation of computer-generated holograms using wavelet shrinkage,” Opt. Express 25(1), 77–87 (2017). [CrossRef]
16. D. Blinder and P. Schelkens, “Accelerated computer generated holography using sparse bases in the stft domain,” Opt. Express 26(2), 1461–1473 (2018). [CrossRef]
17. Y. Ogihara and Y. Sakamoto, “Fast calculation method of a cgh for a patch model using a point-based method,” Appl. Opt. 54(1), A76–A83 (2015). [CrossRef]
18. K. Matsushima, M. Nakamura, and S. Nakahara, “Silhouette method for hidden surface removal in computer holography and its acceleration using the switch-back technique,” Opt. Express 22(20), 24450–24465 (2014). [CrossRef]
19. J.-P. Liu and H.-K. Liao, “Fast occlusion processing for a polygon-based computer-generated hologram using the slice-by-slice silhouette method,” Appl. Opt. 57(1), A215–A221 (2018). [CrossRef]
20. S. Igarashi, T. Nakamura, and M. Yamaguchi, “Fast method of calculating a photorealistic hologram based on orthographic ray–wavefront conversion,” Opt. Lett. 41(7), 1396–1399 (2016). [CrossRef]
21. H. Sato, T. Kakue, Y. Ichihashi, Y. Endo, K. Wakunami, R. Oi, K. Yamamoto, H. Nakayama, T. Shimobaba, and T. Ito, “Real-time colour hologram generation based on ray-sampling plane with multi-gpu acceleration,” Sci. Rep. 8(1), 1500 (2018). [CrossRef]
22. J.-S. Chen and D. Chu, “Improved layer-based method for rapid hologram generation and real-time interactive holographic display applications,” Opt. Express 23(14), 18143–18155 (2015). [CrossRef]
23. H. Zhang, L. Cao, and G. Jin, “Computer-generated hologram with occlusion effect using layer-based processing,” Appl. Opt. 56(13), F138–F143 (2017). [CrossRef]
24. N. Masuda, T. Ito, T. Tanaka, A. Shiraki, and T. Sugie, “Computer generated holography using a graphics processing unit,” Opt. Express 14(2), 603–608 (2006). [CrossRef]
25. B. J. Jackin, S. Watanabe, K. Ootsu, T. Ohkawa, T. Yokota, Y. Hayasaki, T. Yatagai, and T. Baba, “Decomposition method for fast computation of gigapixel-sized fresnel holograms on a graphics processing unit cluster,” Appl. Opt. 57(12), 3134–3145 (2018). [CrossRef]
26. H. Sannomiya, N. Takada, K. Suzuki, T. Sakaguchi, H. Nakayama, M. Oikawa, Y. Mori, T. Kakue, T. Shimobaba, and T. Ito, “Real-time spatiotemporal division multiplexing electroholography for 1,200,000 object points using multiple-graphics processing unit cluster,” Chin. Opt. Lett. 18(7), 070901 (2020). [CrossRef]
27. Y.-H. Seo, Y.-H. Lee, and D.-W. Kim, “Asic chipset design to generate block-based complex holographic video,” Appl. Opt. 56(9), D52–D59 (2017). [CrossRef]
28. P. Tsang, J. Liu, T. Poon, and K. Cheung, “Fast generation of hologram sub-lines based on field programmable gate array,” in Digital Holography and Three-Dimensional Imaging (Optical Society of America, 2009), p. DWC2.
29. T. Sugie, T. Akamatsu, T. Nishitsuji, R. Hirayama, N. Masuda, H. Nakayama, Y. Ichihashi, A. Shiraki, M. Oikawa, and N. Takada, “High-performance parallel computing for next-generation holographic imaging,” Nat. Electron. 1(4), 254–259 (2018). [CrossRef]
30. T. Nishitsuji, Y. Yamamoto, T. Sugie, T. Akamatsu, R. Hirayama, H. Nakayama, T. Kakue, T. Shimobaba, and T. Ito, “Special-purpose computer horn-8 for phase-type electro-holography,” Opt. Express 26(20), 26722–26733 (2018). [CrossRef]
31. Y. Yamamoto, S. Namba, T. Kakue, T. Shimobaba, T. Ito, and N. Masuda, “Special-purpose computer for digital holographic high-speed three-dimensional imaging,” Opt. Eng. 59(05), 1 (2020). [CrossRef]
32. Y. Wang, D. Dong, P. J. Christopher, A. Kadis, R. Mouthaan, F. Yang, and T. D. Wilkinson, “Hardware implementations of computer-generated holography: a review,” Opt. Eng. 59(10), 1 (2020). [CrossRef]
33. A. Maimone, A. Georgiou, and J. S. Kollin, “Holographic near-eye displays for virtual and augmented reality,” ACM Trans. Graph. 36(4), 1–16 (2017). [CrossRef]
34. E. Murakami, Y. Oguro, and Y. Sakamoto, “Study on compact head-mounted display system using electro-holography for augmented reality,” Trans. Inst. Electron., Inf. Commun. Eng., Sect. E E100.C(11), 965–971 (2017). [CrossRef]
35. M. Makowski, R. Trybus, S. Fiderkiewicz, and J. Starobrat, “Overcoming the limitations of spatial light modulators for holographic near-eye displays,” in The 9th Laser Display and Lighting Conference, vol. 11520 (2020), p. 1152001.
36. T. Ito and T. Shimobaba, “One-unit system for electroholography by use of a special-purpose computational chip with a high-resolution liquid-crystal display toward a three-dimensional television,” Opt. Express 12(9), 1788–1793 (2004). [CrossRef]
37. T. Shimobaba, A. Shiraki, N. Masuda, and T. Ito, “Electroholographic display unit for three-dimensional display by use of special-purpose computational chip for holography and reflective lcd panel,” Opt. Express 13(11), 4196–4201 (2005). [CrossRef]
38. Y. Yamamoto, N. Masuda, R. Hirayama, H. Nakayama, T. Kakue, T. Shimobaba, and T. Ito, “Special-purpose computer for electroholography in embedded systems,” OSA Continuum 2(4), 1166–1173 (2019). [CrossRef]
39. Y. Yamamoto, H. Nakayama, N. Takada, T. Nishitsuji, T. Sugie, T. Kakue, T. Shimobaba, and T. Ito, “Large-scale electroholography by horn-8 from a point-cloud model with 400,000 points,” Opt. Express 26(26), 34259–34265 (2018). [CrossRef]
40. T. Shimobaba and T. Ito, “An efficient computational method suitable for hardware of computer-generated hologram with phase computation by addition,” Comput. Phys. Commun. 138(1), 44–52 (2001). [CrossRef]
41. L. Lesem, P. Hirsch, and J. Jordan, “The kinoform: a new wavefront reconstruction device,” IBM J. Res. Dev. 13(2), 150–155 (1969). [CrossRef]
42. S.-C. Kim and E.-S. Kim, “Effective generation of digital holograms of three-dimensional objects using a novel look-up table method,” Appl. Opt. 47(19), D55–D62 (2008). [CrossRef]
