1. Introduction

Compressive light field display technique has attracted considerable attention due to its advantages of high spatial bandwidth utilization, simple structure, and strong compatibility with current 2D display technique. It compresses and decomposes high-dimensional light field information into multiple-layer two-dimensional (2D) images, and utilizes multi-layer 2D displays to output these images to reconstruct the light field of three-dimensional (3D) images [1–6]. Different from holographic display techniques, compressive light field display technique does not require holographic encoding and coherent illumination [7,8]. It overcomes the trade-off problems of spatial-angular resolution which is commonly found in traditional light field display systems [9–11]. This technique enhances the spatial bandwidth utilization of the display system through pixel multiplex. It can effectively address the vergence accommodation conflicts issue [12,13], and present fascinating 3D effects to the audiences.

Lots of improvements have been made in light field decomposition algorithm since the introduction of compressive light field technology by A. Loukianitsa et al. in 2002 [14]. For instance, D. Lanman optimized the light field decomposition technique using non-negative matrix factorization (NMF) and joint algebraic reconstruction (SART) algorithms, which improve the image quality and decomposition efficiency. But the algorithm is only applicable to static scenes [15,16]. E. Vargas et al. improved the imaging performance by using end-to-end optimization and the time-multiplexed coded aperture technique [17]. J. Zhang et al. developed a light field decomposition algorithm using limited-memory broyden-fletcher-goldfarb-shanno technique, but the algorithm requires more than 50 iterations to converge [18]. F. Heide proposed a relatively efficient adaptive algorithm which reduces rendering time and bandwidth requirements, but it still requires more than 20 iterations to converge [19]. Using tomographic algorithms and high order non-negative tensor factorization, G. Wetzstein’s team achieved the time-multiplexed decomposition of multiple-frame light field images, resulting in the display of dynamic compressive light fields. Although a frame rate of 35fps is achieved, it has low resolution and few viewpoints, whose light field only consists of 3 × 3 × 320 × 240 pixels [20]. Using the internal and external correlation of light field frames, X. Cao's team succeeded in reducing the resolution of unknown information and they improved the rendering accuracy and reduced the number of iterations. The light field video decomposition was further accelerated using CUDA-based parallel algorithms, but it still fails to achieve real-time decomposition [21]. The L.-Q. Weng’s team proposed a light field decomposition processor that combines the half-block-based factorization process with sparse-ray sampling and INT-hybrid methods to address the challenges of insufficient memory bandwidth and complex computing. This algorithm can decompose the light field with data volume of 7 × 7 × 1280 × 720 pixels at a frame rate of 30fps [22]. But the implementation of the algorithm is complex. Although significant progress have been made in image decomposition algorithms for compressive light field display, these algorithms still have limitations in processing real-time rendering tasks with high-resolution light field images.

To address the above issues, a real-time decomposition technique for compressive light field display based on multiplex correlations is proposed. By optimizing the initial values of iterations using spatial correlations of pixel multiplex light fields, the iteration times are significantly reduced, thereby accelerating convergence. Using stochastic gradient descent (SGD) algorithm, a method for solving the NMF problem is developed. Parallel compression and decomposition of light fields is achieved by combining the correlation of rays and the parallel characteristic of GPUs. By establishing a corresponding Hashtable using the sign distance field (SDF) transformation in sheared camera frustum space, the memory addresses of light field data are reordered, thereby improving the addressing efficiency of the compression and decomposition process. Using the proposed techniques, a rendering pipeline was made, whose decomposition rate exceeded 30 fps. A compressive light field display system was built, vivid 3D images were reconstructed by using the decomposed images and display system.

2. Principles

2.1 Light field recording and display principles of compressive light fields

Figure 1 shows the principles of the light field recording and display processes for compressed light fields. A sheared camera array is used to record light field information. As shown in Fig. 1(a), the cameras are located on z = l plane. The camera's focal plane is located on z = 0 plane which is coplanar with the zero parallax plane. The parameters n and f represent the distances from the cameras’ near and far clipping planes to the viewpoint, respectively. The parameters w and h are the height and width of the near clipping plane, Δx_view and Δy_view represent the absolute offset of the viewpoint of sheared camera relative to the Z-axis. The optical axes of all cameras in the array converge to the point O. The convergence optical axis configuration of sheared camera array can record more effective images and reduce invalid pixels in the focal plane of each camera than camera array with parallel optical axes. It can improve the bandwidth utilization of light field recording system and obtain more parallax information. The perspective projection matrix of a sheared camera is given by:

Figure 1(b) shows the display principle of the compressive light field display system, whose structure consists of two stacked display layers, Near and Far. The observers are located on the z = l plane. The Near display layer is located on the z = z_near plane and coordinates on this plane are represented by (s, t), while the Far display layer is located on the z = z_far plane, and coordinates on this plane are represented by (u, v). The light field information of the 3D model is decomposed into two image layers through a compression and decomposition algorithm, and the two images are loaded on the Near and Far display layers respectively. Observers at different locations of the viewing zone near the z = l plane will receive different light rays, corresponding to different parallax information. Then, 3D effects can be perceived by receiving these parallax cues.

 

Fig. 1. Principles of compressive light fields. (a) Light field recording. (b)Light field display.

Download Full Size | PDF

Each ray in the light field illustrated in Fig. 1 corresponds to a pixel on the light field images recorded by the cameras. For instance, the recorded brightness of red light ray corresponding to a pixel of 5th view is I₀(x, y). It intersects with the Near display layer, the camera's focal plane, and the Far display layer at points (u, v), (x, y), and (s, t), respectively. When the light field is reconstructed, a uniform backlight is usually used to illuminate the Near and Far image layer. The brightness of the reconstructed light ray, I_mul(x, y), after it passes through the two display layers can be described as following:

2.2 Initial value optimization using multiplex correlation of light rays

The most challenging problem of compressive light filed display is to find the solution of Eq. (2). Equation (2) is an over-determined equation. There is not a unique nor exact analytical solution for this over-determined equation. Iterative algorithms such as NMF provide an effective way to solve this problem. These algorithms start iteration for optimization with initial estimates and converge to an optimal solution. Traditional algorithms use random initial values to iteration for the various and uncertain target values in the beginning. The traditional methods need a lot of iterations and significant computation time to converge due to the large deviation between random initial estimates and target values. In the proposed technique multiplex correlation has been exploited to optimize the initial value of the NMF algorithm. It significantly reduces the iterations and improves the computational efficiency.

For compressive light field display, the number of light rays recorded in the light field is much greater than the total number of pixels on each display layer. Therefore, pixel multiplex is required to achieve light field display. As illustrated in Fig. 2, whether in the Near display layer or the Far display layer, multiple rays from different cameras will pass through the same pixel, so these rays have strong spatial correlation. The transmittance of a pixel is jointly determined by the light rays passing through that pixel. This phenomenon is known as pixel multiplex correlation. According to the multiplex correlation, the initial values of pixels in the Near and Far display layers are set to the geometric mean of the brightness of each ray with multiplex correlation. Their initial values are respectively represented as follows:

 

Fig. 2. Multiplex correlation of light rays.

Download Full Size | PDF

The initial values described in Eq. (4) have more correlations with the target light field and are closer to the optimal values. Therefore, fewer number of iteration is needed in theory. Combining the optimized initial values with the parallel NMF algorithm based on stochastic gradient descent (SGD) optimizer, the compression and decomposition speed of compressive light field can be significantly improved.

2.3 Parallel NMF algorithm using SGD optimizer and optimized initial value

The NMF algorithm is an effective and meaningful solver that can decompose high-dimensional light field onto multi-layer images. Taking the two-layer display planes shown in Fig. 1(b) as example, the iterative formula of the NMF algorithm is given as follows [15]:

 

Fig. 3. Parallel NMF algorithm flowchart using SGD optimizer and optimized initial values.

Download Full Size | PDF

Firstly, the target light field captured by the camera array and the optimized initial values calculated using the multiplex correlation are given to the iterative loop. Then the optimization objective is iterated utilizing the SGD optimizer. Within the iterative loop, the gradient value of the optimization objective needs to be first calculated, and iteration is performed. The hyper-parameters α control the iteration step size, and hyper-parameters λ is used to control the model complexity and prevent over-fitting. In multiplicative light field, the gradient value of the optimization objective contains two independent variables. Two-step iterations are required to update the values of Near and Far layers respectively using the control variable method. During the iteration of I_near, I_far is considered as constant, and vice versa. Iteration is stopped and the optimized transmittance is outputted when the difference between adjacent iteration results is less than a preset value or the number of iterations exceeds the preset value. In practice, we use compute shader to execute one-dimensional linear iterative tasks in parallel.

The advantages of this algorithm are as follows: (a) The number of iterations to achieve convergence is significantly reduced. By using the optimized initial values which are closer to the optimal values than random initial values. (b) The computation load of iterations is further reduced using SGD optimizer to solve the NMF process. (c) The iterative calculation process of the algorithm is accelerated by using the parallel nature of GPUs.

2.4 Addressing optimization in compression and decomposition process

In the decomposition process, we need to index the light field information by tracking the positions of each ray passing through the multi-layer images to be decomposed. Huge light rays require massive addressing calculations. Finding an efficient addressing algorithm is very important to improve the speed of compression decomposition algorithm. However, the density of the rays hitting on Near layer is higher than that on Far layer, due to the perspective effect of the sheared camera. The offset of the intersection of the same light ray on different layers is different, as shown in the left diagram of Fig. 4. When reading and indexing the different layer images, the memory pointers need to jump back and forth constantly. The column-major storage method of layer images can lead to more jumping and discontinuous memory addresses. It is an important reason for the slow decomposition computation speed.

 

Fig. 4. The sign distance field based on the frustum space of the sheared camera and its reordering and mapping relation.

Download Full Size | PDF

To address this issue, we define the frustum space of the sheared camera as the SDF, as the space surrounded by the blue dashed lines shown in the left diagram of Fig. 4. The signed distance between points in the frustum space and its nearest layer image to be decomposed is used to index the information of the light field. A corresponding Hashtable is created to reorder the memory addresses of pixels on the layer images, so as to ensure that the pointer can be quickly addressed in the memory with the minimum fixed offset. The actual memory reordering operation is based on the Hashtable implementation, which enables search, insertion, and deletion operations with an average time complexity of O(1). This data structure is ideal for indexing light field information. Using the SDF function of the frustum of the sheared camera as the Hash function, we remap the memory addresses of information in the light field. Specifically, the mapping relationship between pixel coordinates (s_i, t_i) in the left diagram of Fig. 4 and their positions (s₀, t₀) in the Hashtable is as follows:

When a Hash collision occurs, it indicates that rays from different sheared cameras hit the same pixel. Based on the design principle of the Hash function, the number of Hash collisions for the same pixel is no more than the number of viewpoints. In this way, the address offset of the intersections between one light ray and the multi-layer images are fixed, ensuring continuous and efficient memory addressing during the iteration stage. This improves the speed and efficiency of the entire computational process.

2.5 Real-time decomposition and rendering pipeline

Figure 5 is the real-time decomposition and rendering pipeline designed according to the principles described above. It can render the compressive light fields using 3D model data directly. The pipeline consists of three key stages: instantiation preprocessing, compression and decomposition, and screen rendering.

 

Fig. 5. Real-time decomposition and rendering pipeline.

Download Full Size | PDF

During the instantiation preprocessing stage, 3D model data is read and the camera and display layer parameters are set at first. Then, instantiation parallel rendering techniques are used to record the light field information of the 3D model. The address of the light field data is reordered by implementing the SDF transformation according to the camera and display layer parameters. The information is stored in an efficient way for addressing. Next, based on multiplex correlations of light rays, a set of optimized initial values is pre-computed for the subsequent compression and decomposition stage. The core stage of the entire pipeline employs a parallel NMF algorithm using the SGD optimizer and optimized initial value to perform the compression and decomposition of the light field. The core logic is implemented in compute shader, which handles highly parallelized iterations. During the screen rendering stage, the optimized multi-layer images obtained from the compression and decomposition stage are displayed on the Far and Near display layers.

The whole pipeline converts the 3D model data into four-dimensional light field information, and then compresses and decomposes it into multi-layer transmittance images, so that users can visually observe the 3D scene using multi-layer displays directly. When the 3D model or camera array changes, the rendering pipeline is activated to refresh the multi-layer displays with the latest optimized multi-layer images.

3. Experiments

3.1 Rendering effect tests

To demonstrate the effectiveness of the proposed technique, firstly, the advantages of the optimized initial values based on multiplex correlation on the algorithm efficiency were analyzed. Secondly, the computation efficiency of the traditional serial NMF algorithm and the parallel NMF algorithm using the SGD optimizer was compared. Thirdly, compression and decomposition efficiency of the algorithm in handling with different models and resolution images was tested. The parameters of the compressive light field and computational hardware are shown in Table 1.

Table 1. Parameters of compressive light field and computation hardware

View Table

To analyze the impact of optimized initial values on the efficiency of decomposition, the properties of the optimal initial value are firstly analyzed. As shown in Fig. 6, a Mask model and Cat model were used in the tests. The first column of Fig. 6 shows the light field images sampled by a 5 × 5 camera array at equal intervals within a 10° FOV range. The second and third columns in Fig. 6 show a set of optimized initial values calculated using the multiplex correlation of the light rays according to Eq. (4). The fourth and fifth columns in Fig. 6 are the convergence values of the decomposition algorithm. Compared to random initial values, optimized initial values are closer to the convergence values. In addition, the optimized initial values for the Near and Far layers implied the depth position information of the models relative to the display layers. For example, in the second column of Fig. 6, the head of the cat model and the teeth and nose of the mask model near the Near display layer are clear, while the tail of the cat model and the horns of the mask model away from the Near display layer are blurred. A similar pattern can be observed in the third column, the horns of the mask model and the tail of the cat model near the Far display layer is clear, while the head of the cat model and the teeth and nose of the mask model away from the Far display layer are blurred. These patterns are consistent with the mapping rules of the compressive light field.

 

Fig. 6. The optimized initial values and iterative results using multiplex correlation.

Download Full Size | PDF

The number of iterations using optimized initial values and random numbers as initial values was compared using the same iterative algorithm. To get the fastest convergence speed and ensure the efficiency and accuracy of the model during training, the hyper-parameters are set to be α=0.02 and λ=0.00001 respectively. Figure 7 is the trends of average peak signal-to-noise ratio (PSNR) of the compressive light field with respect to the number of iteration when the optimized initial values and random initial values are respectively used in the NMF algorithm for iteration. Red and blue lines represent the Mask and Cat models, respectively. The $\blacktriangle$ symbols denote the results using optimized initial values, while the ○ symbols denote the results using random initial values. The $\blacksquare$ symbols denote the results of traditional NMF algorithm using random initial values. It can be found that whether mask model or cat model, the algorithms using random initial values requires 6-7 iterations or more (traditional NMF algorithm) to approach the optimal value, while the algorithm using optimized initial values only need 2-3 iterations to approach the optimal value. Therefore, the proposed algorithm using multiplex correlation can significantly reduce the number of iterations and improve the convergence speed. After optimization using the proposed method, the two models both reached a PSNR of over 30 dB. For the same model, whether using optimized initial values or random initial values, the PSNR of the optimization results is very close. This demonstrates the correctness of proposed method. The PSNR of convergence value of the Mask model is 30.446 dB, while that of the Cat model is slightly higher at 31.680 dB due to its fewer texture details. Although there is a slight difference in the convergence values of the two models, their trends are same, which demonstrates the robustness of the algorithm.

 

Fig. 7. The trends of PSNR with respect to the number of iteration.

Download Full Size | PDF

To meet the requirements of real-time display, the computational efficiency of the entire pipeline was tested using different resolutions images. At the same time, the computation efficiency of pipeline using the traditional serial NMF algorithm and the parallel NMF algorithm using the SGD optimizer was compared. As shown in Fig. 8, the horizontal axis represents different resolutions. Images with three different resolutions, including 256 × 256, 512 × 512, and 1024 × 1024, were tested respectively. The vertical axis represents the time required for the pipeline to run one rendering cycle. The green bars represent the time consumption of the parallel NMF algorithm using optimized initial values and SGD optimizer. The blue bars represent the time consumption of the traditional serial NMF algorithm using random initial values. The time consumption shown in bar of Fig. 8 includes the time from reading the light field images to outputting the optimized values. It can be found that the proposed algorithm is faster than the traditional NMF algorithm in various resolution conditions. Moreover, as the resolution increases, the increase in time consumption of SGD-NMF algorithm is much less than that of traditional NMF algorithm. In other words, for high-resolution light field decomposition problems, the advantage of the proposed SGD-NMF algorithm is more pronounced. At a resolution of 1024 × 1024, the computational time for rendering each frame of the compressive light field image using the SGD-NMF algorithm is 28.91 milliseconds, achieving a speed of 34 fps. Compared to the traditional NMF algorithm, there is a 15.24 times speed improvement, indicating that the proposed algorithm and pipeline can achieve real-time compression and decomposition of light fields.

 

Fig. 8. The computation time for rendering one frame of compressive light field image with different resolutions.

Download Full Size | PDF

3.2 Compressive light field display

To demonstrate the correctness of the decomposition results, digital reconstruction experiments and optical 3D reconstruction experiments using the compressive light field display system were conducted respectively. Figure 9 shows the rendering and digital reconstruction images of the Mask and Cat models within a 10° field of view (FOV). The “Top View” is a bird's-eye view of the display system and the 3D model. The two red lines indicate the positions of the Near and Far display layers. The Near and Far display layers show the transmittance images generated by the decomposition pipeline. The middle three columns in Fig. 9 show the comparison between the rendering images and reconstruction images of the 1st, 13th, and 25th (top left, center, bottom right) viewpoints respectively. The rightmost column in Fig. 9 shows the enlarged views of the red-boxed of the reconstruction results from different viewpoints. The position changes of both the eyes of the cat model and the teeth of the mask model show noticeable parallax effects. That’s means the compressive light field system can display 3D images using the decomposition results successfully.

 

Fig. 9. The compressive light field decomposition and digital reconstruction results of the Cat and Mask models.

Download Full Size | PDF

Optical 3D reconstruction experimental system is shown in Fig. 10(a), the display system consists of a backlight and two LCD. A lens array is used to limit the divergence angle of backlight to within 15°, it can make the reconstructed 3D image brighter. The two LCD display Near and Far transmittance images respectively. The transmittance images have a resolution of 1024 × 1024 pixels. Figure 10(b) contains two frames taken from Visualization 1 and two frames taken from Visualization 2, respectively. The cat images were taken from the viewpoint 11th and 15th, and the mask images were taken from viewpoint 3th and 23th. The results show that vivid 3D images can be optically reconstructed using the images rendered by the proposed pipeline and the display system.

 

Fig. 10. Experiment system and imaging results: (a) Optical 3D reconstruction experiment system. (b) Optical reconstruction 3D images: Four frames taken from Visualization 1 and 2 with different viewpoints.

Download Full Size | PDF

4. Discussion

There is grid noise in the experimental results. It is moiré patterns generated by the periodic pixel structure between the two screens. When the two screens are completely aligned, the minimum amount of moiré patterns can be observed. The use of a black background will also reduce the visible moiré patterns as shown in Fig. 11.

 

Fig. 11. The comparative experiment of the visible moiré patterns using white and black background. (a) 3D model, (b) Optical 3D display results of the model.

Download Full Size | PDF

The 3D images appear blurry in the edge viewing angles due to an increase in the misalignment error of the two LCD. Asymmetric light rays of edge viewing angles and diffraction of the LCD can also reduce the reconstruction image quality. Additive light field display technique using holographic projection screens can be used to improve these issues in future work.

5. Conclusions

A real-time compression and decomposition technique for compressive light fields based on multiplex correlation was proposed and demonstrated to address the challenge of compression and decomposition of high-dimensional light field information in compressive light field display technology. Firstly, the iteration initial values were optimized using the spatial correlation of pixel multiplex light fields, which significantly reduced the number of iterations. Then, the addressing efficiency of the compression and decomposition process was improved by reordering the memory addresses of the light field data using the SDF transformation. And then, parallel compression and decomposition of the light field was achieved by combining the SGD optimizers and GPUs. Furthermore, a rendering pipeline was constructed, and experiments were conducted to validate the proposed pipeline. The experimental results show that for a two-layer compressive light field with 5 × 5 viewpoints and a resolution of 1024 × 1024, the proposed algorithm only requires 2-3 iterations to approach the optimal solution, and its decomposition frame rate exceeds 30 fps. Compared to traditional NMF algorithms, the speed is improved by 15.24 times. Finally, a compressive light field display system was made. Vivid 3D display was successfully realized, which validates the feasibility of the technique.