Open Access
Add to BibSonomy
Abstract
We describe a panoramic camera using one monocentric lens and an array of light field (LF) sensors to capture overlapping contiguous regions of the spherical image surface. Refractive sub-field consolidators divide the light before the image surface and concentrate the sub-images onto the optically active areas of adjacent CMOS sensors. We show the design of a 160° × 24° field-of-view (FOV) LF camera, and experimental test of a three sensor F/2.5 96° × 24° and five sensor (25 MPixel) F/4 140° × 24° camera. We demonstrate computational field curvature correction, refocusing, resolution enhancement, and depth mapping of a laboratory scene. We also present a 155° full circular field camera design compatible with LF or direct 164 MPixel sensing of 13 spherical sub-images, fitting within a one inch diameter sphere.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Established and emerging fields of virtual and augmented reality, environmental mapping of complex urban environments, and visual odometry for robotic navigation, demand wide field-of-view (FOV) depth sensing capabilities within a compact form factor. A robust system capable of functioning in diverse and dynamic conditions can be designed by using optical depth sensing combined with technologies such as light detection and ranging (LIDAR), time-of-flight, structured light, and stereo triangulation. Our goal is to combine the high numerical aperture wide-field imaging capabilities of monocentric lenses with the depth sensing and refocusing capacity of computational light field (LF) imaging [1–9] to create a compact panoramic depth sensing camera. ‘Monocentric’ lenses, made from concentric spheres of glass, enable high FOV and low F/# imaging with diffraction limited resolution [10,11]. These lenses are significantly more compact and less complex than comparable fish-eye lenses [10,12], but form their image on a spherical surface.
There has been significant effort dedicated to capturing such spherical images with conventional flat CMOS image sensors. Previous work has explored using an array of secondary imagers to relay overlapping regions of the image onto conventional image sensors [13] or coupling the spherical image to one or more CMOS sensors via straight or curved imaging fiber bundles [12,14]. Direct capture of the spherical image surface may become practical as the development of spherically curved CMOS sensors continues [15–17]. Here we explore a different approach. First, we move the challenge of focusing onto a curved image surface from the physical to the computational domain using LF imaging and space-variant refocusing to correct image curvature of a local region [18,19]. Second, we use a faceted refractive ‘field consolidator’ lens to divide a continuous wide FOV curved image onto an array of conventionally packaged CMOS sensors.
This paper is organized as follows. We describe the numerous design considerations for this panoramic LF camera in Section 2. In Section 3 we report our optimization process, the optical design, and simulated performance for an experimental proof-of-principle camera. We present our fabrication and assembly process in Section 4 before describing the LF processing method and characterizing a three sensor F/2.5 96° × 24° and five sensor F/4 140° × 24° camera in Section 5. Finally, we present a full circular 155° field lens design compatible with LF sensing, short imaging fiber bundle coupled image sensors, and spherically curved image sensors in Section 6.
2. Experimental prototype design considerations
The design of this optical system combines physical optics of image formation with LF data sensing and processing. The first design consideration is the consequence of tiling the spherical image with flat image sensors. Even sensors with no optically inactive sensing area would still leave approximately triangular gaps in the sensing of a spherical image, and the optically inactive packaging area needed for electrical connection or heat dissipation only compounds this problem. We solve this issue by breaking monocentricity and adding faceted ‘field consolidators,’ an array of positive sub-lenses located over each sensor. Most of the optical power is provided by the monocentric primary lens, but the consolidator lenses direct all of the light onto the active areas of the image sensors. The optical power of the consolidation introduces geometric aberrations and degrades the resolution of the image relative to the fully-monocentric lens, so we minimize the required power using efficient sphere-packing methods and select the sensor to maximize the ratio of optically active to inactive area.
By allowing the resulting image to curve, this consolidation can be performed by a simple aspheric singlet or doublet lens (as opposed to a full relay imaging system, as in [13]). Our initial design is based on a refractive single lens, for simplicity. A doublet can correct for chromatic (transverse) aberration at the far edges of each consolidator’s FOV. This can also be implemented with a diffractive correction on the first surface of a single element. The two-piece molded achromatic doublet, which will be discussed in Section 6, corrects chromatic aberration at a relatively low increase in complexity, but was not practical for this first proof-of-principle prototype.
Our prototype system started with an existing 12 mm focal length monocentric lens substantially similar to one used for curved fiber bundle coupling to a single full field image sensor [20]. The basic lens structure is a 3.42 mm radius spherical core of S-LAH58 (flint) glass surrounded by a 6.72 mm radius shell of S-LAH79 (crown) glass. The lens is constructed from four cemented hemispherical elements, with a physical aperture stop at the center. The lens also includes a hemispherical shell of C500 color glass infrared filter that also serves as a fixed protective dome, and a BSL7Y glass meniscus for fiber bundle mounting. The lens is diffraction limited over the visible spectrum for an aperture stop of up to F/1.35, and was mounted to allow a 120° FOV. The two specific lenses used in this project were fabricated with a F/4 and F/2.5 aperture. These objective lenses form a continuous spherical image surface with a 12.1 mm radius of curvature, but the optical system is modified for LF capture.
We decided to capture a wide aspect ratio (letterbox) panoramic FOV using a single row of image sensors. This arrangement requires sensors with efficient packing on at least one axis. We chose the 5 MPixel OmniVision OV5653, which we had used for the row of sensors in a monocentric fiber-coupled camera [12]. The original custom flex circuit achieved a horizontal fill factor (linear ratio of optically active sensing width to full packaged sensor width) of 62%. With a redesign of the flex to move all of the PCB electronics away from the sides of the chip, and using computer-controlled dicing of the flex to remove excess material to within 50 µm of the chip itself, we increased the fill factor to 70%, compared to the 71.1% limit imposed by the OV5653’s 6.505 mm wide chip itself.
The next design consideration is the position of the LF microlens array within the system. There are two well-known plenoptic camera configurations for planar image sensing. In the conventional afocal (plenoptic 1.0) LF camera, the main lens is focused at the microlens plane, and the microlenses are one focal length from the sensor, thereby focused to infinity or equivalently on the main lens. In the focused (plenoptic 2.0) LF camera, the microlens array is focused to relay the image plane of the main lens onto the sensor pixels. This opens an additional design degree of freedom, the ratio of the distance between the main lens’ image and the microlens to the distance between the microlens and image sensor. These distances, defined as $a$ and $b$ respectively in Fig. 1, should obey the basic image equation $1/a + 1/b = 1/f$ (for microlenses of focal length $f$).
We chose to use a focused (plenoptic 2.0) layout to allow us to vary $a/b$, the plenoptic ratio, to change the spatio-angular tradeoff [21,22] while further reducing the required amount of consolidation, as it produces the largest space between the main lens and the image sensor. The curved image surface from the monocentric lens produces a radially increasing plenoptic ratio, $a'/b'\;>\;a/b$, and corresponding field dependent spatial and angular frequency.
In this configuration, the microlenses direct light from sub-regions within the main lens aperture onto the focal plane. The FOV of these lenses must not exceed the numerical aperture (NA) of the main lens; otherwise light would spill over to the pixels behind neighboring microlenses. This would result in over-filling of the adjacent lenslet sub-images, creating ambiguity in the LF where perspectives become overlapped. At the same time, the FOV of the microlenses must not be too small to excessively limit the NA of the main lens, or sensor area would be dark. Such under-filling, limits the information content of the sensed image. Therefore the lenslet aperture, focal length, and position in the camera must result in an FOV that approximately matches the NA of the main lens. Further, adding a non-monocentric consolidating optic increases the NA incident on the microlens array. We must take this into consideration when choosing the lenslets.
Commodity microlens arrays consist of a thin embossed adhesive layer on a glass substrate. This poses a challenge for the system assembly, as the lenslets must be placed just over a focal length away from the sensor as shown in Fig. 1. This would not be an issue if the microlens array were to be chip-scale fabricated on the sensor, using the same wafer-scale manufacturing techniques as the single-pixel microlens on commercial CMOS image sensors. However, for system assembly using standard CMOS image sensors, our design and assembly process must take both the substrate thickness and precision spacing into consideration.
In addition, we must jointly choose the remaining parameters of the microlens array’s focal configuration (such as the array pitch, F/#, and number of pixels behind each lenslet) together with the plenoptic ratio to balance the spatial resolution with the depth resolution. These parameters can be varied so the camera can better sample the plenoptic slopes (depth), but this depth resolution comes at a cost in spatial resolution. As a reference point, the Lytro Illum LF camera used F/1.8 lenslets with 20 µm pitch and a 41 MPixel 1.4 µm pitch sensor, and approximately 14 × 14 pixels behind each lenslet. Our final design used F/1.3 hex packed embossed microlens arrays with 30 µm pitch and 40 µm focal length, with an approximately 17 × 17 array of 1.75 µm pitch pixels behind each lenslet.
3. Optical design process
We split the design process into three steps of increasing complexity and accuracy. First, we developed a MatLab toolbox LFCamExplore, publicly available at the GitHub link in [23], to explore the paraxial plenoptic design space and determine the desired range of plenoptic ratios to optimize spatial and angular sampling in the epipolar plane. We determined that a plenoptic ratio of 5 to 7 produces a suitable tradeoff between spatial and angular resolution across the FOV of a single consolidated image sensor. This acceptable range was enough to accommodate the radially increasing plenoptic ratio caused by the curved image surface and flat microlens array, even though curvature is not modeled by LFCamExplore.
We divided the remaining design process into sequential and non-sequential modeling steps using the commercial lens design program Zemax. We used sequential Zemax to optimize a single consolidator to produce the best intermediate image with sufficient field consolidation to avoid illuminating the optically inactive sensor areas. This intermediate image, analogous to the main lens image surface in Fig. 1, is formed between the monocentric lens and microlens array in the focused or plenoptic 2.0 configuration. In practice, it is formed within the microlens array glass substrate. We did not model the microlens array, as it is an inherently non-sequential object and sequential Zemax has trouble with aiming the rays through such small lenslets. However, we needed to include the effect of its aperture division, so we split the main lens’ aperture stop into sub regions to model the light captured by a single microlens. In effect, this makes the lenslets in the microlens array the system’s limiting aperture stop. We included the position and spacing of the microlens array, substrate, and sensor in this model. The sequential modeling was monochromatic at 587 nm, as we chose to use a singlet consolidator and correct the resulting transverse chromatic aberration in LF processing.
Finally, we modeled the full faceted system including microlenses in non-sequential Zemax using the visible waveband preset (486, 588, and 656 nm) under two illumination conditions. To verify pixel limited performance and ensure the capture of an adequate number of perspectives, we used collimated ray bundles at the various field angles to generate impulse responses. We also used a Lambertian source to model a flat field image to make sure we were neither over-filling or under-filling the microlens array. We repeated the sequential and non-sequential modeling and optimization until both yielded adequate results while ensuring the uncorrected chromatic aberration was acceptable.
A ray trace diagram of the resulting sequential design is seen in Fig. 2. The glass monocentric lens consists of a protective dome and a symmetric achromatic core as described in Section 2, but with the mounting meniscus omitted. The consolidator is optical grade (transparent) polystyrene, chosen for its high index of refraction of 1.59, with aspheric front and rear surfaces. The microlens array substrate, lenslets, and sensor are also shown in Fig. 2, although the lenslets are not modeled in this sequential design. The consolidator was optimized to bring the largest field angles onto the edge of the optically active area of the sensor, while minimizing spot size at the intermediate image. The lens was designed to provide a 100 µm gap between the monocentric core and consolidator when focused to infinity. This gap prevents mechanical interference and can be increased by moving the S-LAH58/79 lens core forward to optionally compensate for assembly and manufacturing tolerances. The monochromatic modulation transfer function (MTF) and geometrical spot size at the intermediate image surface are shown in Figs. 3 and 4.
Fig. 2. Sequential ray trace diagram of the monocentric lens and single consolidator. The largest (gold) and second largest (red) fields correspond to the diagonal and horizontal extremes of the sensor respectively. The lenslets are not modeled in the sequential design, but their position is shown in the inset.
Fig. 3. Monochromatic MTF plots of the sequential design at the spherical intermediate sub-image of light passing through the full (a), top (b), center (c), and bottom (d) regions of the aperture stop. The poor performance seen in the full aperture (a), which is equivalent to direct image capture, is improved by aperture division by the LF relay (b through d).
Fig. 4. Monochromatic geometrical spot diagrams for the sequential design at the intermediate image of light passing through the full, top, center, and bottom regions of the aperture stop.
A top down view of this same design modeled in non-sequential Zemax is shown in Fig. 5. Here, the illumination sources are collimated fields, equivalent to objects at an infinite conjugate distance. One interesting aspect of this design, which is apparent in Fig. 5, is the splitting of the field angle illuminating the seam of the faceted consolidators. The light is split between the adjacent consolidators and intercepts the adjoining image sensors. Any imperfection in this seam results in a slight loss in intensity and does not correspond to gaps in the image which would occur if the fields were segmented at the image surface.
The simulated intensity incident on the image sensors for both polychromatic collimated and Lambertian sources are shown in Fig. 6. The 16° horizontal field is directly illuminating the seam between two adjacent consolidators and is captured by the image sensors. When illuminating with the Lambertian source, the detected single lenslet sub-image of the monocentric pupil transitions from a full circle at the center into half-moons near the edge. Even though a significant region of the aperture stop is obscured (and directed to the adjacent image sensor) the intensity of this region is not reduced as would occur with traditional vignetting. Instead, since the microlenses are imaging the monocentric aperture stop itself, the optical energy and scene information is divided between the two sensors in full-brightness regions.
Fig. 6. Non-sequential detectors capturing light from polychromatic collimated (top) and Lambertian (bottom) sources for the center (left), horizontal extreme (middle), and diagonal extreme (right) fields. Each detector is 264.25 µm (151 pixels) square and the zoomed insets are 26.25 µm (15 pixels) square.
4. System assembly
While we chose polystyrene to be compatible with moldable plastic fabrication and diamond turning, compatibility with an anti-reflection coating (which was not necessary for our proof-of-principle prototype) may affect the choice of optical polymer for future designs. Molding would allow fabrication of a single faceted element, and is inexpensive for volume production. However, diamond turning of rotationally symmetric parts is more practical for small volume prototyping, so we integrated the linear array of field consolidators by assembly of discrete diamond-turned elements. ISP Optics made individual polystyrene consolidators to within 25 µm in diameter, center thickness, radius, and sag, and optical surfaces with less than 10 nm Ra. The fabricated optic was designed with two flat registration surfaces outside the optically active region; this allowed us to control the position of the microlens array relative to the consolidator, and the consolidator relative to the image sensor. This technique allows for the axial position of the components, including the critical 52.5 µm distance between the lenslets and sensor, to be governed by the tolerance of the diamond turning, and not be reliant on manual positioning.
The consolidator/sensor integration process required several precise steps. First, we needed to cut each of the circularly symmetric consolidators from ISP Optics into tileable wedges. We temporarily protected the optical surfaces with ‘First Contact’, a polymer normally used to clean optics, mounted the elements in a fixture at a 16° off vertical angle, cut the circular edges off with a computer controlled diamond sectioning saw, then removed the protective polymer to reveal the clean optical surfaces between the planar wedge sides.
Next we needed to expose the surface of the OV5653 image sensors. Like most commercially available small form factor CMOS image sensors, these have a protective cover-glass over the silicon to prevent contamination of the surface by dust and debris. This 400 µm thick glass substrate is glued and diced during the CMOS wafer fab and extends over the entire sensor. This cover interferes with the placement of the microlens array and therefore must be removed. We used a precision wafer dicing saw to remove the glass over the optically active region of sensors. Mounting the image sensors on custom tip and tilt stages and aligned them flat using a measuring microscope with a high magnification (low depth of field) microscope objective allowed us to cut all the way through the cover-glass and into the adhesive layer without touching the silicon. An external support microscope slide was glued to the cover-glass to secure it before the final forth cut, preventing the glass from falling into and damaging the silicon.
This dicing method left a border of cover-glass around the optically active region of the sensor, which we used as an alignment surface to correctly position the sensor with respect to the consolidator and microlens array. We also diced the left and right edges of the PCB attached behind the sensor, removing as much material as possible for tighter packing, without compromising the sensor’s functionality.
The microlens arrays fit closely into the rectangular recess in the cover glass. We purchased 30 µm pitch, 40 µm focal length, F/1.3 hex packed embossed microlens arrays, which matched the numerical aperture of the F/2.5 monocentric lens after passing through the consolidator, and diced them to size. Using an alignment microscope we centered the microlens array on the registration surface built into the consolidator during diamond turning before attaching them with UV curing adhesive. We lowered the consolidator/microlens array into the recess of the image sensor surrounded by the cover-glass border, using the glass ridge as a flat alignment surface to position the consolidator/microlens array depth relative to the silicon. While looking through a measuring microscope, we used probes attached to linear stages to create a device resembling a micromanipulator to position the consolidator/microlens array in the center of the image sensor and to rotationally align with the pixel grid. We then injected high viscosity UV cure adhesive around the optics to prevent both contamination onto the image sensor and any excess flow between the consolidator and sensor. The adhesive was cured while the consolidator was weighted down by a custom machined brass fixture to ensure the registration surfaces were in good contact and not shifted by the high viscosity adhesive.
We repeated this process with each consolidator, microlens array, and image sensor to form the components of the full panoramic consolidated LF image sensor array. Figure 7 shows CAD models of the consolidating LF image sensors, photos of the image sensor and consolidator during system assembly, and the final five sensor array, ready for integration with the monocentric lens.
Fig. 7. (a) CAD assembly of the five sensor array behind a monocentric lens with an exploded sensor stack to the lower left. (b) CAD and fabricated consolidator after wedge cut and with microlens array attached (c). Final assembled consolidating LF image sensor on flex (g) and after assembly into the full panoramic array (e).
5. System processing and characterization
The LF imager optics were designed as a five sensor F/2.5 160° × 24° camera. However, the optomechanical mount for our F/2.5 monocentric lenses, which included a servo-motor for electrically-controlled focusing, could not accommodate a sensor array containing more than three image sensors without mechanical interference. To capture photos using the full five sensor array we used an set of unmounted F/4 monocentric lens elements, which we mounted in a simple aluminum fixture that provided unobstructed access to the focal surface. The F/4 aperture, however, produces under-filling and limits the sensed angular resolution and depth information. Close-up photos of both sensor configurations are shown in Fig. 8, which also shows the scene we arranged to test both configurations. Various objects are located at distances between 0.25 m to 2.5 m across the full FOV of the panoramic camera, including objects crossing the seams between adjacent image sensors. To capture flat field frames, which are needed for the LF processing, we illuminated the aperture of the camera with a LED integrating sphere that provided Lambertian diffuse light.
Fig. 8. Photo of the lab scene taken with a fish-eye lens (left), F/2.5 three sensor camera (top right), and F/4 five sensor camera (bottom right).
Although the F/4 lens could accommodate the full five sensor array, the optical elements of the monocentric lens itself were only designed for 120° horizontal FOV. Scene information was visible out to a full 140°, but objects beyond this FOV were blocked by the lens’ edge structure. The RAW captured data (after demosaicing) for both the F/2.5 and F/4 systems are shown in Fig. 9. The lenslet sub-images for the F/2.5 camera are barely touching, indicating a good match between the numerical aperture after the consolidator to that of the microlenses. As expected, the F/4 camera had noticeable under-filling, leaving a substantial fraction of the sensor area dark, and also limiting the captured parallax information. However, the slower lens reduces the geometric aberrations caused by the consolidation, increasing the spatial resolution.
Fig. 9. RAW LF data (after demosaicing) of the lab scene for the F/2.5 three sensor camera (top) and F/4 five sensor camera (bottom) with zoomed insets showing the difference between balanced under and over-filling and under-filling for the F/2.5 and F/4 lenses respectively.
Processing the captured data into useful LFs posed some unique challenges. Ideally any LF capture can be resampled into a standard form, but it is important to select a parameterization that suits the captured data. Resampling into a two-plane parameterized LF, for example, would do a poor job of representing the wide-FOV LFs captured by our camera. We thus reparameterized our captured imagery into the recently proposed camera-centric relative spherical representation. This parameterization closely matches the geometry of the captured LF, while locally behaving similarly to a planar LF. This has the advantage that many existing tools for LF processing, including synthetic refocus and depth estimation algorithms, can be applied directly despite the wide field of view of the capture [18,19].
The combination of planar sensors and field consolidators warped the captured imagery away from the ideal spherical parameterization. This warping impacts not only depth of field, but also the LF sampling geometry [18]. We compensated for this warping in two stages. First, we found the best-fit of a grid to the flat-field image to locate the lenslet image centers, but only on the central portion of the image to avoid the unequal spacing on the outer portions of the image.
Once sliced into a 4D structure following [24], we corrected for warping in the sampling geometry following the computationally efficient resampling method proposed in [18]. The single parameter to that method, the distance between lenslets and main lens, had to be adjusted to compensate for the impact of the consolidators. We found the method to be insensitive to error in this parameter, and applied an empirical adjustment minimizing the appearance of warping in the captured scenes when viewed through a simple depth estimator [25].
With the LF decoded into a 4D structure, another challenge arose in the form of truncation of subaperture images near the field boundaries. As depicted in Fig. 10, at the seams between fields approximately half of each lenslet image appears to be missing, with the corresponding image appearing in the adjacent field. To deal with this we propose a two-step calibration process based on flat-field images. In the first step we determine which pixels are valid on a per-lenslet basis by finding a connected region of pixels with brightness similar to each central view. For this process we found a brightness threshold of 0.5 to be effective. Note that simple thresholding does not suffice and a connected region analysis is necessary, as pixels from adjacent lenslets need to be ignored.
Fig. 10. At the seams between fields, lenslet images appear truncated, because the sub-aperture views are being split between fields. In this work we deal with fields individually, compensating for this effect with weight-aware refocus, super-resolution, and depth estimation algorithms. A complete solution calibrates the capture and stitches together fields to yield a seamless LF.
In a second stage, we propose to capture checkerboard images to map out the transformation that brings adjacent fields into alignment. To better display the characteristics of our prototypes, we did not apply the corresponding transformations to the results shown here, but rather display each field separately. We leave as future work the resampling process required to obtain truly seamless results. This involves a one-time calibration where we either adapt the spherical parameterization introduced in [18] into a model-based light field camera calibration pipeline, or utilize a dense non-parametric approach, e.g. employing fixed Bernoulli noise patterns. After calibration, per-image rectification and stitching are carried out using a pre-computed 4D warp.
Dealing with fields individually raised a challenge in applying algorithms that operate over all subaperture images. Truncated lenslets yield less horizontal parallax information, impacting depth estimation, refocus, and super-resolution. To deal with this we employed the vignetting from the flat-field image as a weight channel, as well as a binary mask arising from the connected region analysis. This yields correct use of the full vertical parallax information available at the inter-field seams, as well as the increasing amount of horizontal parallax available for pixels farther from the seams.
After LF processing, the resulting panoramic images from the F/2.5 three sensor camera are shown in Fig. 11. The full LF processed image shows a single perspective generated by displaying a specific pixel from each microlens across all of the image sensors. A zoomed inset using data from different pixel locations under the microlenses correspond to varying vertically shifted perspectives. Parallax information is apparent in the occlusion and disocclusion of different background information when the perspective is shifted. The 3× digitally super-resolved panoramic image is focused to a middle depth, and the zoomed insets show refocus to near and far objects. The corresponding relative depth map for this scene is shown in the bottom of Fig. 11. A similar full LF panorama, super resolved panorama, and relative depth map for the F/4 five sensor camera is shown in Fig. 12. The amount of captured parallax information is limited due to increased vignetting caused by the reduced F/4 aperture stop.
Fig. 11. Processed LF data of the lab scene taken with the F/2.5 three sensor camera. The red region of interest from the full LF (top) shows parallax between two different vertically shifted perspectives contained within a single exposure. The green region of interest from the 3× super resolved image (middle) shows refocusing between near and far objects. The corresponding relative depth map for this image is shown at the bottom.
Fig. 12. Processed LF data of the lab scene taken with the F/4 five sensor camera. The red region of interest from the full LF (top) shows parallax between two different vertically shifted perspectives contained within a single exposure. The green region of interest from the 3× super resolved image (middle) shows refocusing between near and far objects. The corresponding relative depth map for this image is shown at the bottom.
After fabrication and test of the consolidator, we discovered that a transcription error during the design process had resulted in an inaccurate representation of the optically active area of the sensor. This caused a small gap in the seam between two adjacent image sensors, with the largest loss of information occurring near equator of the image. This is an easily fixable issue for future designs, especially if some additional room for assembly tolerance is included, without a noticeable effect on image quality. Even with this error, we achieved image continuity near the diagonal extremes of the sub-fields, where the radial distance from the center of the image sensor to the edge is larger.
Our experimental LF camera utilized previously-fabricated 12 mm focal length F/2.5 and F/4 monocentric lenses. These lenses were designed to provide a relatively small back focal length to the optical fiber bundles. However, this requires consolidator optics with large optical power to bend the sub-images towards the optically active regions of the image sensor, resulting in relatively large aberrations over the pupil of the lens. The intermediate image was highly aberrated, as shown in Fig. 3(a). Direct image capture after consolidation (e.g. by spherically-curved image sensors) would not be practical with this prototype lens system. The LF microlenses reduced the effect of these aberrations by sampling a sub-region of the pupil.
This experimental demonstration of overlapping LF sub-images, and the amended optical design, established that contiguous LF imaging is practical, but its letterbox image format is limiting in utility. Ideally, we capture a full circular FOV with a two-dimensional array of sensors tiling over the hemispherical image surface. Tiling a spherical curvature with a one-dimensional letterbox array is trivial, but two-dimensional tiling with flat focal planes is not. However, the optical complexity of the lens system is unchanged. The next step is the optical design of a full circular field imager using co-optimization of the monocentric lens and consolidator.
6. Full field design
Co-optimization of the monocentric lens and consolidator reduces the optical power needed for consolidation, and dramatically increases the potential resolution of the directly-sensed curved sub-image. The consolidated curved image formed by the fully-optimized lens can be captured by LF sensors, which allows computational refocus and provides depth information, but also incurs the loss in spatial resolution inherent in LF imaging. In addition, the fully-optimized sub-images can also be directly sensed by spherically curved sensors or short imaging fiber bundle coupled sensors, potentially retaining the full diffraction-limited resolution of a conventional image-plane sensor, but extending over the spherical FOV. Both options are of interest for an exploratory design study, intended to demonstrate the capability of the consolidated monocentric lens structure for a two-dimensional array of small format image sensors.
Such a design would be compatible with LF image capture, using the same microlens and data processing format demonstrated above, and also for direct sensing of the spherical sub-images, whether implemented by high spatial resolution imaging fiber bundle coupling to planar sensors or, ideally, with emerging spherically-curved image sensors [15–17]. Unfortunately, small format spherically curved image sensors are not yet commercially available. To allow initial exploration of this interesting design space, we proceeded under the assumption that one could fabricate a curved sensor with the same layout parameters as existing small-format sensors used in smartphone imagers.
For this design we considered image sensors with <2.0 µm pixel pitch, which deliver reasonable resolution (5 to 15 MPixel, for 1/2 to 1/4 inch format), and which provided minimal optically inactive packaging area. A square aspect ratio sensor would be preferable for two-dimensional tiling, but square images are uncommon for sensors of these sizes. We settled on OmniVision’s OV12890 1.55µm 12 MPixel 1/2.3 inch image sensor. This sensor fit well with our targeted 7-8 mm focal length system size and had high chip-on-board fill factor of 88.9% horizontal, 83.7% vertical, and 52.2% between a circle circumscribed with the sensor package to a circle inscribed within the optically active area. This is much higher than the OV5653 sensor used in the prototype, which had 70.6% horizontal, 57.0% vertical, and 38.7% circumscribed to inscribed fill factors. We assumed that custom flex circuitry could be contained entirely behind the sensor’s package and therefore not impact the fill factor, since we were able to achieve this with the horizontal fill factor of our previously integrated OV5653 image sensor.
Another design consideration for wide angle optics is vignetting of large angle fields, which can be caused by mechanical clipping of rays by lens edges and occluders near them. However, the primary source of vignetting in our monocentric system is due to the cosine theta falloff of a planar circular aperture stop. This reduces the optical signal power, and therefore the signal-to-noise ratio, and reduces the resolution due to aperture diffraction. For LF imaging, this results in loss of depth information and resolution as the angle away from the central axis increases. However, monocentric lenses can utilize a total internal reflection (TIR) virtual aperture stop, in the form of a low index or air gap between concentric shells. This can greatly reduce vignetting at large field angles, and can provide ideal angle-invariant light collection and resolution [26]. We incorporated this technique into the design.
We chose a 10 mm radius of curvature for the image sensor array based on the desired focal length and resolution (100 to 200 MPixel). We used a manual tiling scheme that reduced gaps between sensors while favoring the horizontal band of image sensors for optimal performance over the (typically) most critical areas of the scene. More optimized tiling methods may be appropriate for larger array sizes [27], but this method was sufficient for a 13 sensor tiling, shown in Fig. 13. This arrangement produces the largest gap between sensors at 30° azimuth and 26° elevation, where the largest amount of consolidation is required and the lowest performance is seen. It is this field angle which drives the optimization of the lens.
Fig. 13. CAD renderings of the 155° full field lens design with short fiber bundle coupled image sensors.
Figure 13 shows a F/2.5, 150° full circular FOV, 7.8 mm focal length, 13× OV12980 sensor, 164 MPixel, TIR aperture stop camera. These renderings show short imaging fiber bundle coupled sensors as the sensing method. The optical elements of this lens fit within a one inch diameter sphere and can be focused by axially shifting the monocentric core with respect to the faceted consolidator array. The optical ray trace, modeled geometric spot size, and MTF resolution of this design are shown in Fig. 14. The angular FOV subtended by a single 1.55 µm pitch image sensor pixel, implemented in a curved image sensor, would be 40.9 arc seconds. Note that this resolution compares favorably to the 60 arc second foveal resolution of a human eye [28], and the overall 155° circular FOV of this imager is comparable to the at most 120° full-resolution FOV accessible through eye rotation [29,30].
This design is intended to be a cost-effective and manufacturable imager, resulting in several design choices. The glass front meniscus lens is a shape that can be made with conventional glass polishing (as in the monocentric prototype objective lens). The materials chosen for the asymmetric core and both elements of the faceted consolidator are moldable plastics, compatible with high-volume injection molding. Molding of these faceted elements is required. However, the interface between the achromatic elements of the doublet consolidator was constrained to be a concentric sphere. This means that only a single side of each consolidator element is faceted, reducing alignment and fabrication complexity. Furthermore, registration surfaces within these molded elements (indicated in the CAD rendering of Fig. 13) can provide self-alignment and simplify the assembly process.
This basic structure can be adapted to the specific geometry of future commercially-curved image sensors. If needed, we can reduce the curvature of the image surfaces by reducing the angle subtended by a single image sensor. This can be accomplished by increasing the number of image sensors by reducing their size or by scaling up the lens. Both of these methods would also require increasing the number of facets on the consolidator elements.
7. Conclusion
In summary, this paper presented the concept of a panoramic LF camera with a monocentric objective lens and an array of LF sensors, where image consolidating optics direct the light away from inactive regions required for the image sensor packaging. We designed an F/2.5 160° × 24° full FOV LF camera, then assembled and characterized both a three sensor F/2.5 96° × 24° and a five sensor F/4 140° × 24° camera. We demonstrated flattening of the spherical image surface after passing through the consolidators and acquired depth information used to demonstrate refocusing, resolution enhancement, and depth mapping, and attained continuity at the diagonal extremes of the sensors in the array.
Furthermore, we reported the design of a full circular field consolidating lens design with 155° full FOV. This design demonstrates that gaps between sensors tiled on the full hemisphere can be compensated for with simple optical elements. Moreover, this design shows that when co-optimization of the monocentric lens core with achromatic doublet field consolidators result in sufficient image quality for direct or LF image capture. Cameras based on such lenses may be able to offer human-quality perception in the volume of the human eye.
Funding
Intel Corporation (1539120); National Science Foundation (IIS-1539120).
Acknowledgments
Portions of this work were published in the doctoral dissertation "Integrated Micro-Optic Imaging Systems" by Glenn M. Schuster in 2019.
Disclosures
JEF: Gojoya (I,P)
References
1. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Comput. Sci. Tech. Rep. 2, 1–11 (2005).
2. I. Ihrke, J. Restrepo, and L. Mignard-Debise, “Principles of light field imaging: Briefly revisiting 25 years of research,” IEEE Signal Process. Mag. 33(5), 59–69 (2016). [CrossRef]
3. C. Birklbauer and O. Bimber, “Panorama light-field imaging,” Comput. Graph. Forum 33(2), 43–52 (2014). [CrossRef]
4. Z. Xue, L. Baboulaz, P. Prandoni, and M. Vetterli, “Light field panorama by a plenoptic camera,” Proc. SPIE 9020, 90200S (2014). [CrossRef]
5. O. Johannsen, K. Honauer, B. Goldluecke, A. Alperovich, F. Battisti, Y. Bok, M. Brizzi, M. Carli, G. Choe, M. Diebold, M. Gutsche, H.-G. Jeon, I. So Kweon, J. Park, J. Park, H. Schilling, H. Sheng, L. Si, M. Strecke, A. Sulc, Y.-W. Tai, Q. Wang, T.-C. Wang, S. Wanner, Z. Xiong, J. Yu, S. Zhang, and H. Zhu, “A taxonomy and evaluation of dense light field depth estimation algorithms,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 1795–1812.
6. G. Wetzstein, I. Ihrke, D. Lanman, and W. Heidrich, “Computational plenoptic imaging,” Comput. Graph. Forum 30(8), 2397–2426 (2011). [CrossRef]
7. G. Wetzstein, I. Ihrke, and W. Heidrich, “On plenoptic multiplexing and reconstruction,” Int. J. Comput. Vis. 101(2), 384–400 (2013). [CrossRef]
8. K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar, “Compressive light field photography using overcomplete dictionaries and optimized projections,” ACM Trans. Graph. 32(4), 1–12 (2013). [CrossRef]
9. M. Hirsch, S. Sivaramakrishnan, S. Jayasuriya, A. Wang, A. Molnar, R. Raskar, and G. Wetzstein, “A switchable light field camera architecture with angle sensitive pixels and dictionary-based sparse coding,” in Proc. IEEE International Conference on Computational Photography, (IEEE, 2014), pp. 1–10.
10. I. Stamenov, I. P. Agurok, and J. E. Ford, “Optimization of two-glass monocentric lenses for compact panoramic imagers: general aberration analysis and specific designs,” Appl. Opt. 51(31), 7648–7661 (2012). [CrossRef]
11. I. Stamenov, I. Agurok, and J. E. Ford, “Optimization of high-performance monocentric lenses,” Appl. Opt. 52(34), 8287–8304 (2013). [CrossRef]
12. I. Stamenov, A. Arianpour, S. J. Olivas, I. P. Agurok, A. R. Johnson, R. A. Stack, R. L. Morrison, and J. E. Ford, “Panoramic monocentric imaging using fiber-coupled focal planes,” Opt. Express 22(26), 31708–31721 (2014). [CrossRef]
13. E. J. Tremblay, D. L. Marks, D. J. Brady, and J. E. Ford, “Design and scaling of monocentric multiscale imagers,” Appl. Opt. 51(20), 4691–4702 (2012). [CrossRef]
14. J. Ford, I. Stamenov, S. J. Olivas, G. Schuster, N. Motamedi, I. P. Agurok, R. Stack, A. Johnson, and R. Morrison, “Fiber-coupled monocentric lens imaging,” in Imaging and Applied Optics (Optical Society of America, 2013), p. CW4C.2.
15. B. Guenter, N. Joshi, R. Stoakley, A. Keefe, K. Geary, R. Freeman, J. Hundley, P. Patterson, D. Hammon, G. Herrera, E. Sherman, A. Nowak, R. Schubert, P. Brewer, L. Yang, R. Mott, and G. McKnight, “Highly curved image sensors: a practical approach for improved optical performance,” Opt. Express 25(12), 13010–13023 (2017). [CrossRef]
16. B. Chambion, C. Gaschet, T. Behaghel, A. Vandeneynde, S. Caplet, S. Gétin, D. Henry, E. Hugot, W. Jahn, S. Lombardo, and M. Ferrari, “Curved sensors for compact high-resolution wide-field designs: prototype demonstration and optical characterization,” Proc. SPIE 10539, 1053913 (2018). [CrossRef]
17. K. Itonaga, T. Arimura, K. Matsumoto, G. Kondo, K. Terahata, S. Makimoto, M. Baba, Y. Honda, S. Bori, T. Kai, K. Kasahara, M. Nagano, M. Kimura, Y. Kinoshita, E. Kishida, T. Baba, S. Baba, Y. Nomura, N. Tanabe, N. Kimizuka, Y. Matoba, T. Takachi, E. Takagi, T. Haruta, N. Ikebe, K. Matsuda, T. Niimi, T. Ezaki, and T. Hirayama, “A novel curved cmos image sensor integrated with imaging system,” in Proc. IEEE Symposium on VLSI Technology, (IEEE, 2014), pp. 1–2.
18. D. G. Dansereau, G. Schuster, J. Ford, and G. Wetzstein, “A wide-field-of-view monocentric light field camera,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 5048–5057.
19. G. M. Schuster, I. P. Agurok, J. E. Ford, D. G. Dansereau, and G. Wetzstein, “Panoramic monocentric light field camera,” in Optical Design and Fabrication (Optical Society of America, 2017), p. ITh4A.5.
20. S. Karbasi, I. Stamenov, N. Motamedi, A. Arianpour, A. R. Johnson, R. A. Stack, C. LaReau, R. Tenill, R. Morrison, I. P. Agurok, and J. E. Ford, “Curved fiber bundles for monocentric lens imaging,” Proc. SPIE 9579, 95790G (2015). [CrossRef]
21. A. Lumsdaine and T. Georgiev, “The focused plenoptic camera,” in Proc. IEEE International Conference on Computational Photography, (IEEE, 2009), pp. 1–8.
22. T. G. Georgiev and A. Lumsdaine, “Focused plenoptic camera and rendering,” J. Electron. Imaging 19(2), 021106 (2010). [CrossRef]
23. D. G. Dansereau, “LFCamExplore,” https://github.com/doda42/LFCamExplore (2017).
24. D. G. Dansereau, O. Pizarro, and S. B. Williams, “Decoding, calibration and rectification for lenselet-based plenoptic cameras,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2013), pp. 1027–1034.
25. E. H. Adelson and J. Y. A. Wang, “Single lens stereo with a plenoptic camera,” IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 99–106 (1992). [CrossRef]
26. I. P. Agurok and J. E. Ford, “Angle-invariant imaging using a total internal reflection virtual aperture,” Appl. Opt. 55(20), 5345–5352 (2016). [CrossRef]
27. H. S. Son, D. L. Marks, J. Hahn, J. Kim, and D. J. Brady, “Design of a spherical focal surface using close-packed relay optics,” Opt. Express 19(17), 16132–16138 (2011). [CrossRef]
28. H. Gross, F. Blechinger, and B. Achtner, Handbook of Optical Systems, vol. 4 (Wiley Online Library, 2008).
29. D. Shechtman, J. Shallo-Hoffmann, J. Rumsey, P. Riordan-Eva, and P. Hardigan, “Maximum angle of ocular duction during visual fixation as a function of age,” Strabismus 13(1), 21–26 (2005). [CrossRef]
30. R. Navarro, P. Artal, and D. R. Williams, “Modulation transfer of the human eye as a function of retinal eccentricity,” J. Opt. Soc. Am. A 10(2), 201–212 (1993). [CrossRef]
