1. Introduction

Freeform optical surfaces offer the potential to establish compact optical systems of exceptional performance and image quality. They find extensive applications across various domains such as optical and laser processing, semiconductor manufacturing, aerospace, and automotive production [1,2]. Currently, primary methodologies for the measurement of freeform optical surfaces encompass coordinate measuring and interferometry. However, the point-by-point scanning principle of coordinate measuring limits its measurement speed, while the null detection principle of the interferometry limits its applicability [3–6]. In contrast, phase measuring deflectometry (PMD) is a technique that relies on inverse ray tracing to ascertain the surface gradient distribution of the surface under test (SUT), thereby enabling the precise reconstruction of its three-dimensional (3D) morphology [3]. PMD has advantages including non-null detection, system simplicity and adaptability, in-situ measurement, and robust stability [7–10]. While the measurement accuracy of PMD for flat specular elements approaches the precision of interferometry at the submicron scale, maintaining the same level of accuracy for surfaces characterized by high local slopes still poses a challenge. In these scenarios, the rays reflected from the SUT can easily surpass the physical boundaries of the display screen, leading to measurement failures [11]. Hence, the measurement of freeform surfaces exhibiting significant local slopes emerges as a pivotal challenge for the future advancement of PMD technology.

To address this challenge, researchers have recently investigated various approaches to enhance the measurement capacity of PMD for specular surfaces with large local slopes [12–18]. For instance, Liu et al. introduced a curved screen to replace the conventional planar screen, constructing a curved screen model via stereo vision system monitoring, and subsequently virtually transforming the curved screen into an extensive planar surface to broaden the range of gradient measurements [12]. Carvalho et al. employed a 45-degree cone mirror to transform the reflected wavefront from cylindrical surfaces into planar wavefronts, facilitating the measurement of cylindrical roundness [15]. Blazer et al. introduced Cavlectometry, which involved utilizing multiple screens or arrays of projectors to create a surrounding source closure around the SUT, enabling the application of inverse ray tracing for measuring surfaces with high slopes [18]. Moreover, Graves et al. proposed an infinite deflectometry, wherein the SUT was securely positioned on a precision rotating platform and subjected to scanning through a series of sub-apertures while undergoing continuous rotation, thereby enabling the measurement of high-slope surfaces [17]. Despite the advancements in enhancing the measurement capability for freeform specular surfaces, these PMD adaptions still suffer from issues such as limited detectable surface types, decreased measurement accuracy, escalated system complexity and prolonged measurement durations. Furthermore, the above methods usually rely on monocular viewpoint, which is susceptible to the challenge of height-gradient ambiguity [7]. To resolve this issue, supplementary techniques e.g., laser trackers are necessitated to obtain precise 3D surface topography, introducing operational complexity. Implementing binocular approaches (e.g., stereo deflectometry) to perform a thorough depth search has high potential of determining the absolute depth of the SUT, but this demands significant computational resources [19].

Here, we propose a multi-view stitching PMD (MVS-PMD) for the measurement of freeform specular surfaces. It involves employing multiple cameras positioned at distinct locations to capture the deformed fringes resulting from reflections across different regions of the SUT. Each camera and screen form an independent PMD system, facilitating the reconstruction of the 3D morphology within the captured view. The 3D morphologies are then precisely stitched together, addressing the excessive local slope challenge encountered in conventional PMD setups. Throw a multistage calibration process, high-precision system parameters are obtained. The proposed MVS-PMD framework retains the simple inverse ray tracing measurement model, and expands the slope range measurable with minor escalated system complexity and prolonged measurement durations. In this work, we established an MVS-PMD simulation model, validating its efficacy in measuring freeform 3D morphologies. Furthermore, we constructed an MVS-PMD prototype that successfully measured a high-curvature spherical lens, curved mobile phone screen, and freeform acrylic sheet.

2. Principle

Figure 1(a) illustrates the schematic of the proposed MVS-PMD. Initially, the patterns deflected by the SUT are captured by $M$ cameras concurrently. Each camera records the $m$-th ($1 \leq m \leq M$) region, which intersects with one of the other patterns. Subsequently, since the measurement accuracy depends on precise calibration of system parameters, thus the optical parameters and physical positions of components in MVS-PMD are precisely calibrated through a multistage calibration process. Following this, the screen projects two distinct sets of sinusoidal fringe patterns onto the SUT, the $M$ cameras capture the distorted patterns reflected by the SUT, to establish the correspondence between pixels in each cameras and pixels on the screen. The MVS-PMD strategically harness the advantage of both monocular and binocular approaches. Within the intersecting region, multiple sets of reference points are selected through feature matching algorithms. For each set, the stereo deflectometry technique is employed to ascertain their absolute depths. These absolute positions are then incorporated as prior knowledge into the monocular iterative reconstruction model. This enables us to rapidly reconstruct the absolute 3D surface topography for each field of view. By employing an iterative gradient-based approach, the surface gradient of the SUT within region $m$ can be efficiently determined, and subsequent integration yields metrological information for $M$ regions. Ultimately, the 3D morphologies from all $M$ regions are stitched together, obtaining the comprehensive appearance of the SUT.

 

Fig. 1. Principle and schematic of MVS-PMD. a. Schematic of MVS-PMD. $\left \{x_m, y_m, z_m\right \}$ represents the coordinate of the $m$-th camera. Cam: Camera. b. Flowchart for inspecting a high-curvature SUT via MVS-PMD. c. Obtaining the absolute depth of point $P$ in the intersection region through stereo deflectometry. d. Iteratively solving for the metrology in region $m$ by SCOTS. ADP: Absolute depth plane.

Download Full Size | PDF

As depicted in Fig. 1(b), the measurement procedure involves five steps: reference point selection within the intersection regions, computation of absolute depths for these reference points, solution of the gradient distribution across the $M$ regions, integration to reconstruct the morphology of these regions, and stitch of all morphologies. To facilitate automated reference point selection, the SIFT feature matching algorithm is employed to identify distinctive features within the reflection patterns of the $m$-th region [20]. These features exhibit invariance to scale, rotation and illumination changes, thus serving as reference points for the intersection region among the remaining $M$-1 regions. After that, multiple reference points within the overlapped section are chosen and their absolute depth are computed using the stereo deflectometry [19]. Specifically, for a reference point $P$ in Fig. 1(c), suppose it appears in the views of $K$ cameras, then the ray inversely emitted from point $C_k$ ($1 \leq k \leq K$)of $\text {Cam}_k$ is reflected by point $P$ and arrives at $S_k$ of the screen. According to the reflection law, the normal vector $\boldsymbol {n}_k$ of the SUT at point $P$ can be solved by the incident ray $C_kP$ and the reflected ray $PS_k$

Subsequently, the position of reference point $P$ is searched along the reflected ray of $\text {Cam}_1$ to satisfy

Here, $(x_S, y_S)$ and $(x_C, y_C)$ represent for coordinate of the screen and camera, respectively; $\Delta z_S$ and $\Delta z_C$ represent for the displacement of a certain point in the screen and camera; $d_S$ and $d_C$ are distances that the camera locates from the point $(x, y)$. The appearance $w(x, y)$ of SUT is reconstructed and updated based on zonal integration method [22]. After several iterations, $w(x,y)$ will converge to reconstruct the appearance of SUT of a certain region. After adjusted by the calculated absolute depths of reference point, the morphologies of $M$ regions are stitched together to reconstruct the complete appearance of the SUT with high-accuracy.

3. Results

3.1 Numerical simulation

To validate the efficacy of the proposed method, we first conducted a numerical simulation based on ray tracing principle. This model replicates the interactions among a planar screen, a test object (which may possess arbitrary geometry), and multiple cameras within the world coordinate system. The established numerical simulation model forms the foundation for validating the accuracy of the MVS-PMD method and fine-tuning critical system parameters. It effectively addresses issues stemming from experimental disturbances that can hinder the precise identification of the sources of methodological and algorithmic issues.

We introduced the 3D point clouds of a planar screen, a test surface of arbitrary shape and two cameras within the world coordinate system to establish the MVS-PMD model (Fig. 2(a)). Subsequently, we configured the intrinsic and extrinsic parameters for these two cameras ($\text {Cam}_1$ and $\text {Cam}_2$). The focal lengths of the two cameras were 11.3 mm and 11.2 mm, with the camera centers situated at coordinates (94.8 mm, −291.0 mm, 427.8 mm) and (94.8 mm, 291.0 mm, 427.8 mm), respectively. The screen was centered at (−64.0 mm, 0 mm, 288.9 mm) and possessed dimensions of 392.5 mm in width and 698.4 mm in length. We simulated the real calibration process by adding a checkerboard calibration board to simply calibrate the intrinsic and extrinsic parameters of the cameras (Fig. 2(a)). Sinusoidal fringe patterns with a period of 50 pixels in two orthogonal directions were sequentially projected by the screen. To emulate the image acquisition process, we established a one-to-one correspondence between camera pixels and screen pixels through ray tracing principles. Given the predefined camera intrinsic and extrinsic parameters, the direction of each camera pixel’s outgoing rays was uniquely determined, with every ray propagating in a straight line and intersecting with the test surface. This model did not incorporate variations in light intensity, assuming uniform intensity for each ray. Once the rays reached the test surface, they continued to propagate in the reflection direction until they ultimately arrived at the screen, thereby yielding pixel coordinates and stripe grayscale values. Through this model, we can accurately emulate the distorted fringe images (Fig. 2(c)). In this numerical simulation, we generated a freeform surface as the SUT using a 21 terms Zernike polynomial fitting (Fig. 2(b)) [23]. The simulated freeform surface had a peak-to-valley (PV) value of 8.772 mm, and its slope distribution is shown in Fig. 2(d).

 

Fig. 2. Numerical simulation of MVS-PMD. a. The simulated MVS-PMD system including screen, cameras and SUT. The SUT can be calibration plates or spherical sample for calibration or measurement, respectively. b. The freeform SUT. c. The distorted fringes captured by $\text {Cam}_1$ and $\text {Cam}_2$. d. The $x$-slope and $y$-slope distribution of the simulated freeform surface. e. Reconstructed 3D surface in the field of $\text {Cam}_1$ (upper) and $\text {Cam}_2$ (bottom). f. The stitched 3D surface based on MVS-PMD. g. The reconstruction residual.

Download Full Size | PDF

The 3D morphology of the SUT was extracted using the established calibrated parameters [24,25]. The reconstructed 3D morphologies from the perspectives of $\text {Cam}_1$ and $\text {Cam}_2$ are shown in Fig. 2(e). However, a single camera’s capacity is confined to measuring a certain proportion of the SUT, making it incapable of capturing the complete morphology. By stitching the individual 3D morphologies together, we successfully reconstructed the complete 3D morphology of the freeform surface (Fig. 2(f)). The residual distribution between the measurement result and the ground truth was computed (Fig. 2(g)). The root mean squared error (RMSE) was found to be 2.1 µm, with a relative error of 0.03 %.

3.2 Experimental setup and characterization

Further, we built an MVS-PMD prototype to evaluate its measurement capacity for high-curvature specular elements (Fig. 3(a)). The sinusoidal fringe patterns projected from the screen (resolution: 1920 $\times$ 1080 pixels, pixel size: 0.36 mm, S32E360F, Samsung Electronics, Korea) illuminated the SUT placed on an SUT supporter. The distorted fringes reflected from four regions of the SUT were independently captured by four cameras (1920 $\times$ 1200, pixel size 4.8 µm, acA1920-150um, Basler AG, Germany), respectively, each paired with a lens (MVL-MF2528M-8MP, Hikvision, China). The positioning and orientations of these cameras were adjustable via camera supports, facilitating complete coverage of the SUT through their collective perspectives.

 

Fig. 3. Setup and characterization of the prototype MVS-PMD. a. The experimental setup of MVS-PMD including four cameras. b. The ceramic checkerboard (upper) and circle grid calibration board (bottom). c. The reconstructed spherical surface within four sub-apertures. The insets are the distorted fringes reflected by the spherical lens in the sub-apertures of four cameras. d. Reconstructed spherical surface based on MVS-PMD. e. The $x$-slope and $y$-slope distribution of the spherical lens. f. The reconstruction residual of spherical surface.

Download Full Size | PDF

Before measuring the optical element, we calibrated the intrinsic parameters, including the focal length of camera lenses and distortion coefficients, the extrinsic parameters of four cameras and the screen, including position and orientation parameters. As shown in Fig. 3(b), we replaced the SUT by a ceramic checkerboard calibration board (grid size: 10 mm $\times$ 10 mm, accuracy: 1 µm), and captured the checkerboard from different angles by different cameras simultaneously. We extracted the intrinsic parameter matrix $\boldsymbol {K}_m$ of $m$-th camera through Zhang’s calibration method [26]. We customized a circle grid calibration plate with the upper surface of the plate was engraved with a 4 $\times$ 4 array of circular dots (pitch: 30 mm $\times$ 30 mm, accuracy: 1 µm), as shown in Fig. 3(b), and took the coordinate system of the circular dots on the plate as the world coordinate system. By the intrinsic parameters and the Perspective-n-Point algorithm, we extracted the extrinsic parameter matrix $\left [\boldsymbol {R}_m\ \boldsymbol {t}_m\right ]$ of the $m$-th camera [27]. Then, we displayed an 11 $\times$ 11 array of circles with a diameter of 10 pixels on the screen, and obtained the pixel coordinates on the calibration plate corresponding to each circle. Combining the parameters, including intrinsic parameter matrix $\boldsymbol {K}_m$, extrinsic parameter matrix $\left [\boldsymbol {R}_m\ \boldsymbol {t}_m\right ]$, pixel size of the $m$-th camera, and screen pixel size, we extracted the extrinsic parameters $\left [\boldsymbol {R}_S\ \boldsymbol {t}_S\right ]$ of the screen.

Following the system setup and calibration, we evaluated the MVS-PMD prototype by measuring a spherical lens (GCL-010123N, Daheng Optics, China) characterized by a nominal curvature radius of 155.04 mm with a tolerance of 0.00/−0.20 mm and an aperture of 76.2 mm. A series of 9 patterns was successively projected by the screen, encompassing four-step phase-shifting fringe patterns with a period of 80 pixels in two distinct orientations and a dot array image used as a reference for phase unwrapping. The resultant reflected rays from four perspectives were captured by distinct cameras (Fig. 3(c)). All acquired data is efficiently conveyed to the computer through the GigE interface to undergo the 3D surface reconstruction. The entire data acquisition process was completed within a duration of ∼350 ms. By reconstructing the 3D morphologies of each region and stitching them together, we obtained the appearance of the spherical lens with a PV value of 2.9 mm (Fig. 3(d)). The local slope distribution of the spherical lens is represented in Fig. 3(e), which conforms to the distribution of the first-order derivatives of a quadratic surface. This reconstructed morphology was subsequently fitted to a standard sphere with a radius of 154.92 mm identical with the lens’s nominal curvature radius. The error distribution is shown in Fig. 3(f). The RMSE was computed between the measurement result and the standard sphere, which was 5.3 µm (relative error 0.18 %). The RMSEs and relative errors for various sub-apertures are listed in Table 1. The differences in reconstruction accuracy across distinct fields of view can be attributed to factors such as the effective measurement range (i.e., aperture size) and the degree of noise present in the reconstruction results. Implementing an appropriate filter can alleviate RMSE variability across different sub-apertures.

Table 1. The comparison of RMSEs and relative errors of the reconstructed spherical surface within four sub-apertures.

View Table

3.3 Measurement of a curved mobile phone screen by MVS-PMD

Furthermore, we inspected a curved screen of a mobile phone (X80, VIVO, China) using MVS-PMD. This mobile phone screen is composed of a glass, featuring a near-flat central area and smoothly transitioning into arc-shaped contours along its lateral edges. (Fig. 4(a)). We used two cameras to acquire the distorted fringe patterns reflected by the curved phone screen. Due to the high local slope of the side edge, each camera could only capture the morphology on one side of the screen. After stitching the reconstruction results from the two views, we obtained the complete 3D morphology of the curved screen (Fig. 4(b)). The 3D morphology of the curved screen is shown in Fig. 4(c). The distribution of local slopes within the effective measurement range of the curved screen by MVS-PMD was −46.1° to 51.3°, with a PV value of 2.2 mm.

 

Fig. 4. Demonstration of the curved screen measurement based on MVS-PMD. a. A photo of curved-screen mobile phone. The upper part was taken as the SUT, approximately 75.2 mm $\times$ 63.0 mm. b. Reconstructed curved screen surface in the field of $\text {Cam}_1$ (left) and $\text {Cam}_2$ (right), and the stitched surface. c. The cross-section of the curved screen at $x$ = 30 mm.

Download Full Size | PDF

3.4 Measurement of a freeform surface by MVS-PMD

Finally, we explored the capability of MVS-PMD to measure freeform surfaces. We made a freeform specular surface by heating and softening an acrylic plate. After adjusting the system layout and recalibrating the system parameters, we measured the freeform surface of the acrylic plate. We computed the slope distribution of the freeform surface in both $x$ and $y$ directions (Fig. 5(a)), with a local slope range of −6.7° to 7.7° and a PV value of 5.3 mm. The complete stitched freeform surface is presented in Fig. 5(b). We conducted Zernike polynomial fitting on this freeform surface, and the distribution of the first 30 Zernike coefficients is depicted in Fig. 5(c), excluding piston and tilt components. Furthermore, we utilized Zernike coefficients to fit the freeform surface profile, and the residual between the fitting and measurement results is demonstrated in Fig. 5(d).

 

Fig. 5. Demonstration of the freeform surface measurement based on MVS-PMD. a. The $x$-slope and $y$-slope of the freeform surface. b. Reconstructed freeform surface. c. The first 30 Zernike coefficients of the freeform surface removing the piston and tilt terms. d. The reconstruction residual of freeform surface.

Download Full Size | PDF

4. Discussion and conclusion

In this work, an MVS-PMD is provided for freeform specular surface metrology. By capturing distorted fringe patterns reflected from distinct regions of the specular SUT through multiple cameras, and stitching the reconstructed 3D morphologies of each region with high precision, accurate characterization of surfaces with substantial local slopes becomes achievable. Our method requires only a few absolute height points within the overlapping region as prior indicators, making it robust even with low field of view overlap. It significantly broadens the camera’s field of view, enabling the collection of additional information from surfaces with pronounced local curvature and facilitating the detection of freeform surfaces. The MVS-PMD is employed under the following conditions: (1) the tested surface must exhibit high reflectivity, (2) the amalgamation of camera fields of view must cover the entire SUT, with each field of view exhibiting overlapping regions with at least one other, and (3) the cameras must effectively capture clear fringe patterns.

Through numerical simulation, we developed an MVS-PMD model based on optical ray tracing, confirming its effectiveness in inspecting freeform surface, with an accuracy of up to 2.1 µm (relative error 0.03 %). Furthermore, we built a physical MVS-PMD prototype with multiple cameras, and assessed its performance on a standard spherical lens with a curvature radius of 155.04 mm, achieving a measurement precision of 5.3 µm RMSE. Additionally, we successfully measured a curved mobile phone screen, characterized by local slopes ranging from −46.1° to 51.3°, and a freeform surface featuring local slopes between −6.7° and 7.7° with a PV value of 5.3 mm. Our MVS-PMD demonstrates a certain degree of flexibility when measuring SUTs with similar local slope distributions. For surfaces characterized by markedly distinct curvature distributions, it is necessary to modify both the number and positions of the cameras to guarantee comprehensive field coverage. Following each camera adjustment, recalibration of the camera’s intrinsic and extrinsic parameters is imperative to maintain measurement precision at the micron level.

The measurement capabilities of MVS-PMD, including local slope measurement capability, accuracy and speed, can be further optimized. First, augmenting the number of cameras to collect reflected light from additional views, especially those concealed by surfaces with pronounced local slopes, would expand the attainable range of local slope measurable. For intricate SUTs, e.g., gratings, special attentions, including precise adjustment of camera parameters, utilization of high-resolution cameras and imaging lenses, implementation of advanced image processing algorithms and phase unwrapping algorithms, should be devoted to ensuring measurement accuracy and feasibility. In cases involving surfaces with exceptionally steep local slopes and small fringe periods, imaged fringes in protruding surface regions tend to be densely packed. To resolve this, multiple sets of fringe-patterned structured lights with varying periods are necessary. Additionally, for discontinuous test surfaces, the MVS-PMD requires the adoption of more sophisticated phase unwrapping algorithms, such as those based on global optimization techniques, to address phase discontinuities. Secondly, the measurement precision of MVS-PMD can be elevated by suppressing various error sources. These includes system model errors such as screen curvature and non-ideal camera imaging model, and algorithm errors such as reconstruction algorithm errors and stitching coordinate deviations. By suppressing the influence of the above error sources, it is expected to refine the measurement precision to the submicron level. Moreover, advances can be pursued in measurement speed through employing advanced structured light projection techniques such as binary coding and grayscale coding to further reduce the number of projections required for each detection [28–30], thus providing a feasible way for dynamic detection of optical elements.

Overall, the proposed MVS-PMD provides a new perspective for freeform specular surface metrology. By facilitating high-precision assessment of specular elements, it holds the potential to enhance the imaging quality, energy utilization efficiency and other attributes of optical systems. It is further expected to promote the development in fields such as biomedical imaging, semiconductor device fabrication, and optical detection.