Optical see-through head-mounted displays (OST-HMDs) display digital content for users while allowing the real world to be observed without latency. Nowadays, OST-HMDs have been commonly used for augmented reality (AR), while the absence of mutual occlusion and the narrow field of view (FOV) significantly restrict their application scenarios. Without the mutual occlusion capability, OST-HMDs display highly transparent content under excessive illumination. A narrow FOV prevents the large-scale scene from being perceived entirely by users. AR searching tasks also become more difficult when OST-HMDs fail to support a wide FOV [1]. To date, OST-HMDs that provide mutual occlusion have been actively developed [2,3], while it is still a challenge for researchers to expand the occlusion-capable FOV into the range of common wide-view OST-HMDs [4–6].

Among a few common methods for conducting mutual occlusion, hard-edge occlusion, also known as per-pixel occlusion, fixes a spatial light modulator (SLM) at an inner focal plane of an optical see-through architecture, so that it allows virtual content to be displayed with a sharp occlusion pattern [7–9]. However, the implementation of lenses in those hard-edge occlusion systems brings inevitable drawbacks when the FOV expands (e.g., severe distortion and blurred occlusion). A well-known approach to avoid disadvantages caused by the nature of lenses is called soft-edge occlusion. Maintaining a raw see-through image, mutual occlusion is conducted by SLMs directly positioned before users’ eyes. This approach does expand the occlusion-capable FOV by ignoring the restriction from the numerical aperture (NA) of lenses, but the out-of-focus occlusion pattern becomes another severe issue [10,11].

Recently, we proposed an enhanced soft-edge occlusion method [12]. In addition to a transmissive liquid crystal display (LCD) working as an SLM, a double-ellipsoidal-mirror system is installed before the user’s eyes that transmits a wide see-through scene. Incident beams are shrunk by a pinhole mask positioned at the shared focus of the paired ellipsoidal mirrors. Thus, mutual occlusion can be rendered in a large FOV and with a precision close to the pixel pitch of the transmissive LCD. However, the pixelated occlusion here needs a small pinhole aperture to reduce the beam diameter, which also reduces system luminance simultaneously. Besides, the transmissive LCD deteriorates the image quality because of severe diffraction and low transmittance. What is more, there is a minimum vertical parallax of 203.2 mm caused by the system layout.

In this Letter, we introduce a double-parabolic-mirror (DPM) structure that supports wide-view hard-edge occlusion. Different from the enhanced soft-edge occlusion approach, mutual occlusion in this design is conducted by a liquid crystal on silicon (LCoS) device positioned at a focal plane of a relay lens group, thereby addressing the conflict between occlusion precision and system luminance. Meanwhile, the implementation of the LCoS benefits the image quality from minor diffraction and high reflectance. Furthermore, a loop structure is designed that makes the system vertically parallax-free.

The schematic diagram of the proposed system is shown in Fig. 1. The entire imaging system bounded with the orange rectangle is drawn above. Red chief rays from various directions enter the system through the focus of the first parabolic mirror. In terms of the well-known property of the parabolic mirror, these chief rays at large incident angles are transformed into parallel rays before reaching the LCoS. Off-axis aberrations, such as coma, are thereby significantly reduced even with a large FOV. Although the usage of the parabolic mirror introduces an extra aberration, it can be compensated by adding the paired parabolic mirror in the imaging system [13]. A four-lens structure that consists of two paired convex lenses is designed to adjust the optical path of the system for eliminating the vertical parallax. Furthermore, the magnification of the input image is controlled by setting the focal lengths of these lenses, since existing LCoSs are usually dimension limited. A roof prism on the top-left corner flips the propagation image left-and-right that ensures the eye, which is located at the focus of the second parabolic mirror, observes the input image correctly. Moreover, an adjustable pinhole mask is used as an aperture stop to balance the brightness and resolution of the display image. According to our experiment, a high pinhole aperture results in better image brightness but lower resolution (see Visualization 1) with worse occlusion (see Visualization 2).

 

Fig. 1. Schematic diagram of the proposed system. Rays from the real scene and the LCD are marked as red and blue lines.

Download Full Size | PDF

The projector for projecting virtual images is separately drawn in the blue rectangle below. Virtual images are merged with the real scene by a polarizing beam splitter upon the LCoS. Similar to the imaging process of the real scene, a lens pair is implemented to shrink the original digital image given by the LCD in the projector, so that the requirement of the pixel density of the LCD for displaying high-resolution virtual images is attenuated. And the tradeoff between the brightness and resolution of the virtual image is also controlled by a pinhole between the lens pair (see Visualization 3). Overall, the proposed system enables that sharp virtual images and hard-edge occlusion are rendered simultaneously with a wide see-through view.

 

Fig. 2. Basic optical layout of the proposed system.

Download Full Size | PDF

A basic structure of our system is shown in Fig. 2. Besides the two paired parabolic mirrors, optical layouts for the real scene and virtual display are both built with four convex lenses from ${L_1}$ to ${L_4}$. In order to make a sharp occlusion mask for the see-through view while rendering virtual content with sufficient resolution, ${L_1}$ and ${L_2}$ are chosen with different parameters, where ${L_{1r}}$, ${L_{2r}}$ are used for the see-through view and ${L_{1v}}$, ${L_{2v}}$ are used for the virtual display. The beam from an arbitrary direction of the real scene enters the system through the focus of the first parabolic mirror ${f_{\text{in}}}$ with the angle $\theta$, then exits to the user pupil through the focus of the second parabolic mirror ${f_{\text{out}}}$, and it is focused at points ${i_1}$, ${i_2}$, and ${i_3}$ through the optical path. ${P_v}$ and ${P_o}$ are the LCD for virtual display and the LCoS for occlusion rendering, respectively. ${d_1}$ and ${d_5}$ are distances from ${f_{\text{in}}}$ and ${f_{\text{out}}}$ to ${L_1}$ and ${L_2}$, respectively. And ${d_2}$, ${d_3}$, and ${d_4}$ are the spacings between ${L_1}$, ${L_2}$, ${L_3}$, and ${L_4}$, respectively. Resulting from the fact that chief rays through ${f_{\text{in}}}$ are transformed into parallel rays, and the mechanism of the LCoS, the first lens pair of ${L_1}$ and ${L_2}$ is designed as a double telecentric lens by placing the pinhole plane ${P_m}$ at the shared focal plane. This gives the lens spacing

We simplify the equation by substituting Eq. (3) into Eq. (6):

The system expects to let the exit beam at ${f_{\text{out}}}$ be recovered into parallel, which guarantees the image to be well-focused by the user’s eye. Therefore, we have

As a result, the optical layout should follow the constraint that is given as

Equations (1–4) and (9) determine the configuration of the proposed system altogether. Since ${s_1}$ varies with different incident angles, it is obvious that the system performance also differs from each FOV.

The theoretical FOV of the proposed system is calculated as Fig. 3. The parabolic mirror used for computation has a focal length of 6 mm and a height of 25 mm. The grey area shows the ideal FOV that is achieved when all the light through ${f_{\text{in}}}$ is transmitted by the system. Owing to the insufficient clear aperture of lenses (23 mm), the FOV is shrunk into a light-blue area around ${\rm H}{114^ \circ} \times {\rm V} 120^\circ$. With further considering the dimension of the LCoS, the FOV shrinks to the dark-blue area with an average width of ${114^ \circ}$ and a nonsymmetrical height from ${32^ \circ}$ to ${-}{63^ \circ}$. Contours with the beside legend indicate the defocus of real scene pixels on the LCoS plane regarding each FOV, which leads to the FOV-dependent occlusion performance and image quality.

 

Fig. 3. Theoretical FOV with defocus contours of the proposed system. The grey, light-blue, and dark-blue areas indicate FOVs calculated from dimensions of the parabolic mirror, lens aperture, and the LCoS, respectively. Contours with the legend beside indicate the defocus of real scene pixels of each FOV at the plane ${P_o}$.

Download Full Size | PDF

Figure 4 shows polychromatic modulation transfer function (MTF) curves given by ZEMAX in both tangential (T) and sagittal (S) planes. The simulated system is set to focus at the lower central view and to work with the pinhole aperture of 1 mm. In general, the MTF curves show the best performance around the central FOV and become worse with the FOV expanding. MTF30 (where the average modulation of T plane and S plane drops to 30%) for the FOV of ${\rm H}{0^ \circ} \times {\rm V} 0^\circ$ occurs at the spatial frequency of 30.4 cpd (cycles per degree). Along with the vertical direction, MTF30 drops rapidly in the upper vision because of the severe defocus, as shown in Fig. 3. It varies from 9.8 cpd at ${\rm H}{0^ \circ} \times {\rm V} 5^\circ$ to 1.6 cpd at ${\rm H}{0^ \circ} \times {\rm V} 15^\circ$. Conversely, the image quality tends to be much more stable in the lower vision. The MTF curve always keeps above 30% until 35 cpd. It even has MTF30 at 6.1 cpd for the FOV of ${\rm H}{0^ \circ} \times {\rm V} - 30^\circ$. Regarding horizontal direction, MTF30 at ${\rm H} {15^ \circ} \times {\rm V}{0^ \circ}$ and ${\rm H}{30^ \circ} \times {\rm V}{0^ \circ}$ are given with the spatial frequency of 23.9 cpd and 18.3 cpd, respectively. When the vision further expands to the theoretical maximum of our prototype, MTF30 is found at 9.8 cpd for the FOV of ${\rm H}{55^ \circ} \times {\rm V}{0^ \circ}$. In general, the proposed system can provide decent image quality for the AR experience with most of the lower vision, which is also indicated by the smoother variation of defocus on the ${P_o}$ plane in Fig. 3. A source image given with a FOV of ${\rm H}{85.3^ \circ} \times {\rm V}{64.0^ \circ}$ is shown at the bottom left of Fig. 4, and the simulated image and the grid distortion by ZEMAX are adjacently placed. A maximum distortion of 2.3% occurs at the top-left and top-right corners of the FOV.

 

Fig. 4. (top) MTF curves for both tangential and sagittal planes. (bottom left) Source image is given with an FOV of ${\rm H}{85.3^ \circ} \times {\rm V}{64.0^ \circ}$. (bottom middle) Simulated image and (bottom right) the grid distortion calculated by ZEMAX.

Download Full Size | PDF

The side view (left) and the front view (right) of a monocular benchtop prototype are shown in Fig. 5. The rays from the real scene and the LCD are labeled as red arrows and blue arrows, respectively. The paired parabolic mirrors have the same focal length of 6 mm and height of 25 mm. All lenses used in the prototype have a diameter of 25 mm and a clear aperture of 23 mm. The LCoS working as the SLM has the dimension of $ 15.36\; {\rm mm} \times 7.64 \; {\rm mm}$, the resolution of $1920 \times 1080$, and the reflectance of 73% (at 633 nm). It is noticeable that though the form-factor is increased by implementing the two parabolic mirrors, around 3/4 of them are redundant due to the limited dimension of other components. With the polarizing beam splitter used as the optical combiner, we calculate the see-through efficiency based on the ZEMAX simulation, which is 0.6%, 5.1%, and 14.2% at different pinhole apertures of 1 mm, 3 mm, and 5 mm (compared to naked eyes with a pupil of 2 mm). Similarly, the eye-box also changes with the pinhole aperture variation since it is the projection of the pinhole by the following optical components. With the above three settings of the pinhole aperture, the eye-box is given with the average diameters of 0.4 mm, 1.2 mm, and 1.8 mm, respectively. The paired lenses ${L_{1r}}$ and ${L_{2r}}$ have the focal length of 50 and 25 mm, which gives the magnification ${M_{12}}$ as 1/2, thus allowing a wider FOV with the dimension-limited LCoS. Accordingly, the focal length of the following paired lenses ${L_3}$ and ${L_4}$ is set as 50 and 100 mm to keep a unit magnification for the see-through view. Owing to the attributes of the LCD (the active area is $51.84\; {\rm mm} \times 51.84\; {\rm mm}$, and the resolution is $1440 \times 1440$) used for projecting virtual images, the lens pair in the projector has the focal length 75 mm for ${L_{1v}}$ and 25 mm for ${L_{2v}}$ that gives the magnification of 1/3. The virtual image is rendered based on OpenGL with the algorithm for compensating the optical aberration by single parabolic mirror imaging and the global magnification of 2/3 for virtual display. Vision areas of the real scene are selectively focused by shifting the LCoS plane up-and-down (see Visualization 4), and the virtual image is focused by moving the convex lens ${L_{2v}}$ horizontally.

 

Fig. 5. Side view (left) and front view (right) of the prototype.

Download Full Size | PDF

Figure 6 shows hard-edge occlusion conducted by our prototype. A FOV of ${\rm H}{83.5^ \circ} \times {\rm V}{53.1^ \circ}$ is recorded by a webcam with a minimum illumination of 0.01 lux. The redundant parts of parabolic mirrors and the additional mounts prevent a sufficient calibration, which leads to extra distortion in recorded images. The pinhole aperture for the real scene and the virtual display are set as 1 mm and 3 mm, respectively. The real scene directly recorded by the camera at the entrance pupil is shown in Fig. 6A, and the see-through view recorded through the prototype is shown in Fig. 6B. The vertical parallax has been mostly eliminated. A red teapot with lighting effects is used as the target image as shown in Fig. 6C. The bright spot by specular lighting and the dark surface by diffuse lighting is bounded with blue- and yellow-dotted lines, respectively. When the virtual content is displayed without occlusion, which is shown in Fig. 6D, the image becomes highly transparent, and the teapot also looks less realistic because effects from both diffuse lighting and specular lighting are missed (see Visualization 5). To solve the problem, a sharp occlusion pattern is given by the LCoS as shown in Fig. 6E. Again, the virtual image is projected with the same illumination by the LCD while keeping the background blocked by the occlusion pattern, and the recorded scene is shown in Fig. 6F. The teapot becomes not only more visible with the opaque surface but also more realistic with the bright spot and the dark bottom being perceived (see Visualization 6).

 

Fig. 6. Hard-edge occlusion is demonstrated by the prototype with the recorded FOV of ${\rm H}{83.5^ \circ} \times {\rm V}{53.1^ \circ}$.

Download Full Size | PDF

In conclusion, we introduce a wide-view AR display with hard-edge occlusion based on the DPM structure. A benchtop prototype is built, and an occlusion-capable FOV of ${\rm H}{83.5^ \circ} \times {\rm V}{53.1^ \circ}$ is demonstrated. As OST-HMDs may work under excessive ambient illumination, hard-edge occlusion is necessary for keeping the visibility and enhancing the reality of the AR display.