1. INTRODUCTION

Structured illumination is a well-known and widely implemented technique for resolution enhancement in optical microscopes [1], providing images with a resolution beyond that of the imaging optics alone [2]. It is also possible to apply this technique in a telescope configuration for imaging objects at range, where structured light patterns can provide an added resolution advantage in settings where active illumination is already used, such as in active night vision systems.

In the seminal structured-illumination telescope system described by Dong et al. [3], an ensemble of images of a physically translated rough surface (e.g., a ground glass diffuser) is projected onto the object, and prior knowledge of the translations is used to recover enhanced-resolution images of the object. This method is advantageous in that extensive knowledge of the surface’s topography is unnecessary for image recovery. In fact, the topography can be recovered in addition to the enhanced object image. However, physical translation is slow, and the spatial frequency spectrum inherent to a rough surface is not easily controlled. Dong et al. applied the technique to a 2D object where the projection and detection systems were situated along different optical axes. The 2D nature of the object was important because object curvature can cause significant distortion to the projected pattern from the point of view of the camera, which breaks the assumption that the projection is a simple translation.

Another significant problem in imaging distant objects with structured illumination is object movement, since knowing the relative alignment of the illumination pattern to the object is important for image recovery [3]. One solution to this problem is image registration using the techniques of cross-correlation or template-matching. Robust template-matching requires high frame rates under nonrigid object movement, where the object can be approximated as stationary over a set of recorded images. Additionally, 3D objects require careful handling to avoid distortion of the projected patterns.

Here, we report a method that accommodates for distant, continuously moving, 3D objects by applying synchronized, high-speed projection and recording. Specifically, we apply a modified version of the incoherent Fourier ptychography algorithm described by Dong et al. [3]. An ensemble of patterns is projected onto an object, and an image is recorded of the patterned object for each pattern in the ensemble. Given knowledge of the optical transfer function (OTF) of the imaging system, and some prior knowledge about the projected patterns, the algorithm allows a user to recover an image of the object with higher resolution than is provided by the imaging optics alone. The resolution of the recovered image is also typically better than that recovered through knowledge of the OTF alone, such as through deconvolution. Our method overcomes some of the limitations of earlier systems by applying high-speed projection and camera systems with precomputed projected patterns to perform the incoherent Fourier ptychography.

2. METHODS

In this section, we describe in mathematical terms the incoherent Fourier ptychography algorithm and how it is applied for image enhancement. The algorithm takes an ensemble of low-resolution images, where the target has been illuminated by known patterns, and combines it with knowledge of the projection and camera systems to create a single, enhanced-resolution image. Specifically, we apply a modified version of the algorithm described by Dong et al. [3,4]. Let ${P_n}$ be the $n$th projected pattern mapped onto the camera detector plane, ${I_n}$ the corresponding image recorded by the camera, $T$ the OTF of the camera, and ${I_g}$ the current best guess for the true image of the target. Each ${I_n}$ is registered to account for a moving target by selecting a region of interest in ${I_1}$, performing normalized cross-correlation or template-matching with all other $n\! -\! 1$ images, and applying a translation corresponding to the cross-correlation peaks. We assume that $T$ is approximately spatially invariant, although a spatially variant ensemble could be accommodated by employing a set of $n$ OTFs, corresponding to each object location, assuming the object is always contained within a single isoplanatic patch. The guess, ${I_g}$, is iteratively updated ($n = 1,2,3,\ldots,N$) by the following series of steps:

A possible difficulty in applying the above equations is the requirement of knowing the camera OTF, $T$. In general, $T$ is complex-valued and depends on a variety of parameters of the imaging system, including aberrations and diffraction. Here, $T$ is measured by applying the same projection and detection system as used in capturing the images for incoherent Fourier ptychography. An ensemble of random patterns, where each pixel in the pattern has an intensity chosen independently from a uniform 0–255 distribution, are projected onto a blank white screen, which is placed at the object distance of interest and recorded by the camera. Given the pixel mapping from projector to camera, and the projected patterns themselves, the complex-valued camera OTF is measured by dividing each camera image Fourier spectrum by the corresponding projected image spectrum and applying an ensemble average. It is beneficial to use a projector with higher resolution than the camera so that the camera OTF can be measured at high spatial frequencies. A television screen could also be used, placed in the plane of interest, to display the patterns. Again, let ${P_n}$ be the $n$th projected pattern mapped onto the camera detector plane and let ${I_n}$ be the corresponding image recorded by the camera of the pattern on the blank screen. Then

We speculate that the ultimate resolution enhancement factor $F$ that can be achieved by the algorithm of Eqs. (1)–(3) beyond that of the camera itself is given by

3. EXPERIMENTAL DETAILS

Here, we describe two distinctly different proof-of-principle optical systems. The first system, shown in Fig. 1, uses a beam splitter to exactly align a camera and projector for imaging of a stationary 3D object at a distance of 2 m. The second system, shown in Fig. 2, uses a closely spaced camera and detector for imaging of a continuously moving, 2D object at a distance of 23 m (75 ft).

 

Fig. 1. Schematic diagram (not to scale) of the experimental system for performing high-speed structured illumination of nearby 3D objects. The distance between the beam splitter and 3D object is 2 m.

Download Full Size | PDF

 

Fig. 2. Schematic diagram (not to scale) of the experimental system for performing high-speed structured illumination of distant moving objects.

Download Full Size | PDF

In both systems, the camera and projector are registered using the following calibration procedure. Single bright pixels are sequentially projected onto a blank white target and imaged by the camera. This creates a unique spot in the image plane of the camera for each projector pixel. An ensemble of centroids is found, one for each spot. A piecewise affine transform with linear interpolation is applied to this ensemble to generate a map between projector and camera pixels. Following the calibration procedure, an ensemble of 500 pseudorandom, binary patterns is generated, which contains the known patterns $P_n^0$ (the patterns generated by the projector) that are to be projected onto the target. The patterns are completely random and uncorrelated. Each pattern is transformed using the map from the calibration step to create the $n$th projected pattern mapped onto the camera detector plane, ${P_n}$ (the patterns that would be recorded by the camera in the absence of diffraction or aberration). The patterns $P_n^0$ are also projected onto a blank screen and recorded by the imaging system, giving ${I_n}$ (the patterns actually recorded by the camera, including diffraction and aberration). The camera OTF is found by applying Eq. (4) using the transformed and recorded patterns; this can also be used in subdivided regions to account for a spatially varying intensity impulse response function. Before application of Eq. (4), the recorded images undergo frame flattening, background subtraction, and Tukey (cosine-tapered) windowing (taper parameter 0.4) about the mean such that $I_n^\prime = {\rm tukey}[{{I_n} - {\rm mean}({I_n}),0.4}] + {\rm mean}({I_n})$.

Next, after calibration in both systems, another ensemble of 500 random patterns $P_n^0$ is generated, transformed into ${P_n}$, and projected and recorded to give ${I_n}$. Here, instead of a blank screen, an object to be imaged is placed in the plane of interest. The ensemble is processed in combination with the measured OTF by applying the algorithm given by Eqs. (1)–(3) [3,5–7], looping five times over the entire 500 pattern ensemble. This step combines the set of low-resolution images of the object with the prior knowledge of the OTF and projected patterns to recover a high-resolution image of the object. In every case, the initial guess of the object is given by the ensemble average of the registered, raw images; essentially, we use the image that the camera would see under uniform illumination of the object as the initial guess.

A. 3D, Stationary Object at 2 m

The system illustrated in Fig. 1 uses a consumer electronic overhead projector system, coaligned with a scientific camera through a beam splitter. The beam splitter allows the projector and camera to have exactly the same view of the object, thereby leaving the projected patterns undistorted by the 3D object from the point of view of the camera. The projector uses an incandescent, broadband bulb source, giving white light illumination and is calibrated to provide a linear intensity response to input images. The pattern projection is performed at a speed of 1 Hz to enable rudimentary synchronization between projector and camera as well as sufficiently low shot noise in the recorded images. The object is a 3D, stationary figurine of a dinosaur, placed 2 m from the camera. The object was purposefully defocused by a small amount to demonstrate the power of the incoherent Fourier ptychography method.

Here, we recover images using two separate OTFs, one approximated by an analytic function and another recovered by Eq. (4). In the analytic case, we approximate the OTF as a sum of jinc functions, where ${\rm jinc}(x): = {J_1}(x)/x$ and ${J_1}(x)$ is the Bessel ${ J}$ function, to account for the broadband light and for the small defocus. It has been shown that the OTF of a circularly symmetric system with small defocus is approximately a jinc function [8]. Here, we use the following empirical function, which closely fits the nulls in the spatial frequency spectra of the images recorded by our particular camera system:

B. 2D, Moving Object at 23 m

Figure 2 gives a schematic of the second experimental setup. A digital micromirror device (DMD, DLi4130 1080p, ${1920} \times {1080}\;{\rm pixels}$, 10.6 µm pixel pitch) is illuminated by a laser, operating at 532 nm wavelength, and projected by an optical system (focal length $f = {300}\;{\rm mm}$, ${5}\;{\rm cm} = {2}\;{\rm in}.$ exit aperture) onto the object. A camera system ($f = {250}\;{\rm mm}$, ${3.8}\;{\rm cm} = {1.5}\;{\rm in}.$ entrance aperture) is synchronized to the DMD by a digital pulse/delay generator, enabling high-speed recording of the projected patterns combined with the object. The camera and projection optics are placed side-by-side to minimize parallax, leaving the projected patterns nearly undistorted by distant objects from the camera’s point-of-view. This setup eliminates the beam splitter used in Fig. 1, thereby eliminating the main source of spurious background light, but requires that the object be distant to keep the parallax low such that 3D objects do not significantly distort the pattern from the point of view of the camera.

Here, the random pattern projection and image capture were performed at 100 Hz over a distance of 75 ft (23 m). For imaging the object, 500 random, binary patterns were interleaved with 500 uniformly bright projections. The uniform projections were used to perform template-matching for image registration of the moving object, while the random patterns were used for ptychographic image enhancement. While the random patterns themselves could have been used for template-matching, we found the reproducibility and positioning resolution was better when using uniformly bright patterns. The object itself was a portion of a USAF 1951 resolution test pattern, which was translated in the transverse plane of the camera at a speed of 3 cm/s.

4. RESULTS AND DISCUSSION

A. 3D, Stationary Object at 2 m

The results of a representative experiment using the setup shown in Fig. 1 are given in Figs. 3 and 4. Figure 3(a) shows the average of the 500 frame image ensemble of the object, recorded under illumination by random patterns from the projection system and with a small amount of defocus on the camera; Figure 4 shows its associated Fourier spatial frequency spectrum. Figure 3(b) shows a high-resolution image of the object, taken at close range with a separate digital camera, for reference.

 

Fig. 3. Images of a stationary, 3D dinosaur figurine at different stages of processing, comparing an estimated analytical OTF with an experimentally measured OTF. (a) The 500 frame ensemble average image of the object. (b) High-resolution image of the object, taken from the same point of view as in (a), for comparison. (c) Image resulting from deconvolution of the analytical OTF from the image shown in (a). (d) Image after deconvolution of the measured OTF from (a). (e) Image recovered from the ensemble using the Fourier ptychography algorithm and the analytical OTF. (f) Image recovered from the ensemble using the algorithm and the measured OTF. (g) A $5 \times 5$ median filtered version of (e). (h) A $5 \times 5$ median filtered version of (f). The axes on each image represent the real space in the plane of the object.

Download Full Size | PDF

The remaining parts of Fig. 3 are organized by row and column. The left column given by Figs. 3(c), 3(e), and 3(g) represent images recovered using the analytic OTF, the amplitude and phase of which are given in Figs. 4(f) and 4(h). Similarly, the right column given by Figs. 3(d), 3(f), and 3(h) represent images found using the measured OTF, the amplitude and phase of which are given in Figs. 4(g) and 4(i). We note that the measured OTF includes a circular low-pass frequency filter, which is not present in the analytic OTF, with a cutoff at around ${12}\;{{\rm cm}^{- 1}}$, beyond which the measurement was overwhelmed by noise. The images in the “deconvolved” row, Figs. 3(c) and 3(d), were recovered using 10 iterations of Richardson–Lucy deconvolution of the associated OTF from the “initial” image shown in Fig. 3(a); the spatial frequency spectra associated with these images are given in Figs. 4(b) and 4(c). The images in the “recovered” row, Figs. 3(e) and 3(f), were recovered by applying Eqs. (1)–(3) to the OTF and the projected ensemble of 500 random images; the associated spatial frequency spectra are given in Figs. 4(d) and 4(e). Figures 3(e) and 3(f) were $5 \times 5$ median filtered to generate Figs. 3(g) and 3(h), respectively.

The recovered and $5 \times 5$ median filtered images contain finer details than the “deconvolved” images. The skin wrinkles and eye spot, which are visible in the reference image of Fig. 3(b), are also visible in the bottom two rows of Fig. 3, while they are absent in the “deconvolved” row. The presence of finer details in the bottom two rows indicates that the algorithm of Eqs. (1)–(3) enables the recovery of spatial frequencies beyond those that are recoverable by deconvolution alone. In this case, the gain in resolution is greater than a factor of 2, as shown by the presence of high-frequency components in Figs. 4(d) and 4(e), which are absent in Figs. 4(b) and 4(c). However, it is also clear that the algorithm can boost spurious spatial frequencies, such that the exact gain factor in resolution is not easily extracted by inspection of the frequency domain. In the image domain, these spurious spatial frequencies create a finely speckled appearance in the “recovered” images of Figs. 3(e) and 3(f), although this effect can be partially mitigated by median filtering, as shown in Figs. 3(g) and 3(h).

In this case, the analytical and numerical OTFs produce similar image results. The left and right columns of Fig. 3 are indistinguishable by eye, although the OTFs themselves are clearly somewhat different in amplitude and phase. The similarity indicates that the recovered images are not particularly sensitive to fine details of the OTFs, and that approximately correct OTFs can be used to enhance image resolution.

B. 2D, Moving Object at 23 m

The results of representative experiments using the setup shown in Fig. 2 are given in Figs. 5 and 6. Both figures represent images of the same object, as described in Section 3.B. The object is defocused in Fig. 5 and is at best focus in Fig. 6.

 

Fig. 4. Fourier spatial frequency spectra of images from Fig. 3 and associated OTFs. (a) Fourier spatial frequency spectrum of Fig. 3(a). (b) Spectrum of Fig. 3(c). (c) Spectrum of Fig. 3(d). (d) Spectrum of Fig. 3(e). (e) Spectrum of Fig. 3(f). (f) and (h) Amplitude and phase, respectively, of the analytical OTF used in recovering Figs. 3(c), 3(e), and 3(g). (g) and (i) Amplitude and phase, respectively, of the measured OTF used in recovering Figs. 3(d), 3(f), and 3(h). The axes on each image represent the frequency space in the plane of the object.

Download Full Size | PDF

Figure 5 provides images of the defocused moving object at various stages of image processing. A single frame of the 500 frame ensemble is given in Fig. 5(a), where the target is barely visible to the eye. The single frames were registered using template-matching and summed to produce the registered image given in Fig. 5(b), which represents an image of the object that would be recorded under uniform illumination, keeping the object stationary, with an integration time sufficient to minimize shot noise. Figure 5(c) gives the ideal image of the target, found by appropriately magnifying and cropping a digital representation of the bar target. Figures 5(f) and 5(g) give the measured amplitude and phase of the camera OTF in the plane of the object. This OTF was used with Richardson–Lucy deconvolution with 10 iterations to produce the image given in Fig. 5(d). The same OTF was used in the algorithm of Eqs. (1)–(3) to produce the recovered image of Fig. 5(e). The algorithm took 15 s to process 1500 frames (three loops of 500) using the Python programming language on a modern 20-core CPU to produce Fig. 5(e). Here, both the “deconvolved” and “recovered” images provide a resolution enhancement greater than 2 with respect to the “registered” image. This fact is shown by comparing Figs. 5(h)–5(k).

The resolution achieved by the images given in Figs. 5(b), 5(d), and 5(e) can be measured from the blur kernels given in Figs. 5(j), 5(h), and 5(i), respectively. Each kernel was found by applying the convolution theorem; the Fourier transform of each experimental image was divided by the Fourier transform of the ideal image, and the result was inverse Fourier transformed and is shown in amplitude. In other words, Fig. 5(b) can be found by convolving the ideal image [Fig. 5(c)] with the kernel [Fig. 5(j)] (along with the appropriate phasor, not shown) and similarly for Fig. 5(d) using Fig. 5(h) and Fig. 5(e) using Fig. 5(i). The associated kernel is shown as an inset in each part of Fig. 5 for comparison of the image and blur kernel on the same scales. Figure 5(b) has two insets, corresponding to Figs. 5(j) and 5(k). We note that Figs. 5(j) and 5(k) represent the same kernel as measured in two different ways. Figure 5(k) is the amplitude of the inverse Fourier transform of the OTF, which we expect to be a more precise measurement, which agrees in size and shape with Fig. 5(j). We determine the relative resolutions of the experimental images by comparing the characteristic length of the respective kernels. Let ${A_n}$ be the area over which the $n$th kernel has an amplitude that is greater than half of its maximum amplitude. We define the characteristic length

We find ${L_h} = 0.237\;{\rm mm} $, ${L_i} = 0.124\;{\rm mm} $, ${L_j} = 0.674\;{\rm mm} $, ${L_h}/{L_i} = 1.9$, and ${L_j}/{L_i} = 5.4$, where each subscript refers to the lettered subfigure from Fig. 5. The ratios of the characteristic lengths indicate that the “recovered” image has a resolution that is a factor 1.9 better than the “deconvolved” image and a factor 5.4 better than the “registered” image. This confirms that the Fourier ptychography method has recovered spatial frequencies outside of what is possible by deconvolution alone under these experimental conditions and has achieved a resolution enhancement greater than a factor of 2 (5.4 in this case) compared with the raw, registered image.

 

Fig. 5. Experimental images of a moving, defocused USAF 1951 bar pattern object at a distance of 23 m, under various stages of processing. (a) Single frame from the 500 frame ensemble of images recorded of the object. (b) Registered ensemble average of the 500 image frames. (c) Ideal image expected in the absence of diffraction. (d) Resulting image after deconvolving (b) with the measured OTF. (e) Resulting image of the object after processing the recorded image ensemble through the Fourier ptychography algorithm. (f) and (g) Amplitude and phase, respectively, of the measured OTF used in generating (d) and (e). (h) Amplitude of blur kernel that generates (d) from (c) via convolution [inset in (d)]. (i) Amplitude of the kernel that generates (e) from (c) [inset in (e)]. (j) Amplitude of the kernel that generates (b) from (c) [inset in (b)]. (k) Amplitude of the kernel generated by Fourier transform of the OTF given in (f) and (g) [inset in (b)]. Axes on each image represent either the real or frequency space in the plane of the object.

Download Full Size | PDF

 

Fig. 6. Experimental images of a moving USAF 1951 bar pattern object at best focus and at a distance of 23 m, under various stages of processing. (a) Registered ensemble average of the 500 image frames. (b) Ideal image expected in the absence of diffraction. (c) Registered ensemble average after deconvolution with the measured OTF. (d) Resulting image of the object after processing the recorded image ensemble through the Fourier ptychography algorithm. (e) Amplitude of blur kernel that generates (a) from (b) via convolution [inset in (a)]. (f) Amplitude of the kernel that generates (c) from (b) [inset in (c)]. (g) Amplitude of the kernel that generates (d) from (b) [inset in (d)]. The axes on each image represent the real space in the plane of the object.

Download Full Size | PDF

Images with the camera adjusted to place the moving object in best focus, as judged by eye to produce the sharpest edges, are given in Fig. 6. Figure 6(a) shows an image of the moving target that was created by template-matching registration of every frame in the 500 frame ensemble. Figure 6(b) gives an ideal image after magnifying and cropping a digital representation of the bar target. Figure 6(c) gives the result of deconvolving the measured OTF from Fig. 6(a), again using 10 iterations of Richardson–Lucy. Figure 6(d) gives the result of applying the algorithm of Eqs. (1)–(3) to the recorded ensemble and measured OTF. While Figs. 6(a) and 6(c) are almost indistinguishable by eye, Fig. 6(d) has noticeably finer details, especially in the numbers shown in the image. We found the residual blur kernels inherent to Figs. 6(a), 6(c), and 6(d), shown respectively in Figs. 6(e), 6(f), and 6(g), by applying the convolution theorem as in Fig. 5. Each kernel is shown as an inset in the corresponding image for comparison on the same scales. Applying Eq. (8), we find ${L_e} = 0.23\;{\rm mm} $, ${L_f} = 0.21\;{\rm mm} $, ${L_g} = 0.11\;{\rm mm} $, ${L_e}/{L_g} = 2.1$, and ${L_f}/{L_g} = 1.9$, where each subscript refers to the lettered subfigure from Fig. 6. The ratios of the characteristic lengths indicate again that the “recovered” image has a resolution that is a factor 1.9 better than the “deconvolved” image. While this enhancement factor is less than the theoretical maximum of 2.33, found by applying Eq. (5), the image has visibly better resolution than that of the “deconvolved” image of Fig. 6(c). One reason for the discrepancy between the theoretical and experimental resolution enhancement factors is the finite spatial resolution of the array detector used in the experiment. Interpolation of the projected image onto the camera detector plane to create ${P_n}$ resulted in some loss of information that could possibly be recovered by using a camera with higher pixel density or by using optics with greater magnification.

5. CONCLUSION

We have implemented a structured illumination system using an incoherent Fourier ptychography method to enhance images of objects. We experimentally showed that the method is applicable to 3D, distant, and moving objects and can generate resolution-enhancement factors greater than 2 in these cases. We have replaced the translating scatterer [3], stationary object [5], and low-speed, low-power [6] systems of previous work with a coherently illuminated, 100 Hz system capable of operating at greater than 20 m [9]. This work represents the first time that the technique has been used on 3D objects with large curvature, and the first time it has been applied to moving, distant targets at high projection and recording speed. We found that 500 projected patterns were sufficient to provide good results in recovered images as a proof-of-principle. A greater number of random patterns can be used to enhance image fidelity at the cost of time, and fewer patterns can be used to speed up the processing. Further work is warranted to quantitatively describe the trade-offs in number and types of patterns. We used random patterns in this case because they contain all spatial frequencies and are simple to work with. It may be that using different patterns, such as Hadamard, for example, gives a benefit. Additionally, while our moving object was entirely rigid, we expect the method to work for nonrigid objects given a fast enough projection such that the object appears rigid during one projection sequence. We expect that this work can be expanded for use in realistic field scenarios to enhance image resolution and could be used to enhance the images recorded by multiple cameras simultaneously with a single projector.