Open Access
Add to BibSonomyAbstract
We propose an ultra-thin imaging system which is based on the neural superposition compound eye of insects. Multiple light sensitive pixels in the footprint of each lenslet of this multi-channel configuration enable the parallel imaging of the individual object points. Together with the digital superposition of related signals this multiple sampling enables advanced functionalities for artificial compound eyes. Using this technique, color imaging and a circumvention for the trade-off between resolution and sensitivity of ultra-compact camera devices have been demonstrated in this article. The optical design and layout of such a system is discussed in detail. Experimental results are shown which indicate the attractiveness of microoptical artificial compound eyes for applications in the field of machine vision, surveillance or automotive imaging.
©2007 Optical Society of America
1. Introduction
The enduring miniaturization of opto-electronic multimedia devices led to an extensive shrinkage of either the sensor pixel size and optical components so that these parts are now reaching physical limits. Diffraction effects become noticeable and decreasing light sensitivity causes image degradation due to noise as the pixel size is reduced. Although these effects might be tolerable in consumer’s digital photography, they affect the performance of subsequent image processing and analysis techniques in machine vision and industrial inspection.
When we look for compact vision systems in nature we find that the compound eyes of insects are the smallest known vision sensors which have been working successfully for over 350 million years. They exhibit lowest possible volume and weight as well as low power consumption. However, they have a low spatial resolution which is partly compensated by high temporal imaging rate and parallel signal processing [1].
For these reasons, compound eyes are the perfect archetypes for the development of a highly miniaturized vision sensor such as the artificial apposition compound eye camera [2]. Such a device consists of a microlens array (MLA, lens diameter D, focal length f, pitch p L) on top of a glass substrate with a thickness of less than 0.5mm (Fig. 1(a)). An optoelectronic sensor array with a different pitch p K is placed in the focal plane of the microlenses to pickup the image. If required, the size of the sensor pixels can be narrowed down by a pinhole array on the substrate backside in order to increase resolution. The pitch difference between the microlens and sensor pixel array Δp=p L-p K causes different viewing directions of the individual optical channels. Therefore, a large field-of-view (FOV) can be sampled with constant angular resolution depending on the sensor size and aberration control [3].
Fig. 1. (a): Schematic section of an ultra-thin artificial apposition compound eye camera. The object space is sampled by Δϕ which is the angle between two adjacent optical axes (sampling angle). The acceptance angle of one channel is denoted by Δφ. (b): Trade-off between resolution (red, dotted) and sensitivity (black line) in dependency on the pinhole diameter.
A known problem of artificial apposition compound eyes is the inverse relationship between resolution and sensitivity [2]. Keeping all other parameters fixed, the optical power in one pinhole (or the light-sensitive cell) is increased along with its diameter whereas the resolution is decreased (Fig. 1(b)).
Nature’s solution to this problem is the superposition compound eye. There are three types of these eyes: the reflective, the refractive and the neural superposition eye [1]. In the complex optical setup of the reflective and refractive type, light that originated from one point of a distant object propagates through several ommatidia and is superposed optically on the rhabdomere tips [1]. In the neural type, each object point is imaged by multiple ommatidia separately and the related signals are superposed in the first neural layer of the eye (Fig. 2). Insects use such a redundant sampling to increase light sensitivity [4].
Fig. 2. (a): Working principle of a neural superposition eye as found in nature. Note the neural wiring of different photo-receptors with parallel optical axes. (b): Its artificial counterpart - an artificial apposition compound eye with multiple pixels in each channel.
In Section 2 we explain the basic setup of an artificial neural superposition eye and the principle of redundant sampling which is used to increase the light sensitivity. In Section 3 we give a short overview of how color is sensed in digital imaging devices followed by the description of our solution for multi-channel imaging systems. Section 4 will deal with the optical design, fabrication and integration of these devices, whereas the characterization and results follow in next section. Finally, we will draw a conclusion in Section 6.
2. Increasing the sensitivity - suppression of temporal noise
The technical counterpart of the neural superposition eye exhibits N x N pixels in each channel (Fig. 2(b)). Thus, two sampling angles are defined. The first one is the offset angle between the optical axes of adjacent channels
Its size is determined by the difference between the pitch of the microlens array p L and the pitch of the pinhole group array p K which is analog to artificial apposition compound eyes (compare Fig. 1 with Fig. 2(b)). The second is the difference between the viewing angles of adjacent pixels in each group which is given by
With the focal length of the individual microlens f this angle is fixed for a specific image sensor with pixel pitch p px. Redundant sampling occurs when the sampling angle between channels is chosen that way that the ratio of both angles is an integer
If this condition is fulfilled, N 2 pixels from different channels observe the same point on a distant object. We sum up the output of the pixels that have parallel optical axes which increases the sensitivity of the eye by a factor of N 2, just as in the natural archetype.
Together with the sensitivity we have to consider noise influences. CMOS image sensors are dominated by thermal noise for low, and photodiode shot noise for high illumination conditions [5]. Usually, they also exhibit a non-uniform response to an uniformly illuminated scene (fixed-pattern noise or FPN). These deviations cause an offset between the response curves of the individual pixels and different gain factors [6]. We assume a simple composition of a temporal noise component (n i,k due to the thermal and shot noise) and a spatial noise component (o i,k due to the FPN) for the output (A i,k) of one pixel with indexes i,k in the array
Here, S i,k is the signal which is proportional to the number of incident photons. To simplify the problem, we assume that n and o are signal-independent and the source as well as the photon noise are small compared to n and may therefore be neglected. For a specific illumination, gain and temperature the terms of Eq. (4) are fixed, except the noise term (n) which is statistically distributed. For the case of dominating thermal or shot noise the distribution is described by a Gaussian with a mean value of zero. Hence, the value of n is spread around its zero-mean according to the standard deviation σ.
Due to redundant sampling in the artificial neural superposition compound eye, N 2 pixels will record the power that belongs to a common point in object space. We sum up the output of those pixels observing one object point a,b (offset term o subtracted)
The variance of the sumof all noise terms is equal to the sumof the individual variances because the noise processes of the individual pixels are uncorrelated [7].
Here, we assumed that the variances of the individual pixels’ noise processes are equal. For the standard deviation of the sum of noise components σtot we may therefore write
Thus, during summation the signal S increases proportional to m whereas the standard deviation of noise increases proportional to √m. Hence, the signal-to-noise ratio (SNR) which is defined by
is increased proportional to √m. For example, if we use m=9 pixels which observe the same point in object space, the SNR of the sum is increased by a factor of √m=3.
It should be noted that even with the fixed-pattern noise component o the average will lead to an increased SNR but the amount of increase is difficult to predict because the FPN of spatially separated pixels is not uncorrelated Gaussian distributed [6]. Nevertheless, the image is smoother because the offset between adjacent pixels in the final image is reduced after the averaging process.
3. Artificial neural superposition eye for color imaging
For conventional digital color vision a spectral, spatial or temporal split acquisition of the color signal has to be applied. For example, a color filter array which is applied directly to the sensor pixels, the so-called Bayer pattern [8], is most widely used in digital still cameras (Fig. 3(a)). In order to maintain the full physical resolution for the whole color image, subtle interpolation methods between the different single color images are used. Furthermore, color aliasing due to undersampling has to be suppressed at the expense of image sharpness [9]. These problems are solved by a sensor which applies stacked photodiodes for the different spectral components in each pixel [10]. However, the complexity of the sensor increases its fabrication costs and causes higher noise characteristics. Other methods like filter wheels, prisms and liquid crystal tunable filters are excluded here because they seem to be less adequate for a miniaturized camera system due to size and costs.
The idea of integrating color imaging in an artificial compound eye is not totally new. Tanida et. al. simulated such a system by acquiring three images of the same scene each with a different color filter in front of a thin observation module. Lacking a permanent, spatial distributed color filter array, they processed a RGB image from the sequence of single color images [11]. Furthermore, they extended the spectral resolution by using seven narrow-band interference filters [12]. The main difference to our approach is that they used a setup for close-up imaging where the distribution of observation points for each color changes according to the object distance. For the first time, we realized an artificial compound eye for color imaging of scenes of arbitrary depth without interpolation or further complex image processing.
Again we use the setup of an artificial neural superposition eye but with an additional color filter in each channel which is shown in Fig. 3(b).
Fig. 3. (a): Bayer color filter array on image sensor (adapted from [8]). (b): Layout of the artificial neural superposition eye with integrated polymer color filters to acquire color images. The case of N=3 is shown here. Each point in object space is imaged through a red, green and blue color filter in different channels. Note that axes with the same line style are parallel.
The sampling angles are chosen according to Eq. (3) so that redundant imaging is achieved. Each pixel in the overall color image results from mapping those pixels that share a common viewing direction through a red, green and blue color filter onto the specific location in the individual color plane. Due to the setup these pixels are situated within different channels. In our approach no interpolation is needed and the resolution is conserved.
4. Optical design, fabrication and integration
In the artificial neural superposition eye the pitch between the pixel groups is large compared to the sensor pixel pitch. Here, we used a conventional 2/3 inch CMOS sensor with a pixel pitch of p px=10.6µm. We optimized the optical parameters of the compound eye using the raytracing software ZEMAX. A pinhole array of matching pitch covers the sensor array in order to decrease the active area of each photodiode which leads to an increased angular resolution. The pitch between the central pinholes of each group is a multiple of the sensor pixel pitch. The pinhole array lies in the focal plane of the microlens array. To achieve redundant sampling, the pitch difference between the microlenses and central pinholes ∆p K is chosen according to Eq. (3). This condition assures that starting from a channel l, there is another pinhole sampling the same point of a distant object in channel l+m 0 and also in l-m0 for each row of the microlens array.
The fabrication techniques for the artificial neural superposition eye are very similar to those used for artificial apposition compound eyes presented in Ref. [2]. The aperture, pinhole (Fig. 4 right) and (optional) the color filter arrays are structured on a thin glass substrate by UV photo lithography. The microlens array is replicated on top using a reflow master. The individual objectives are then diced out and integrated with an optoelectronic sensor.
The color filters could alternatively be applied pixel-wise on the backside of the objective. We decided for the channel-wise solution because due to its large scale it is easier to fabricate. The filter material was chosen because it can be structured by UV-lithography and its transmission properties are suitable for color imaging applications (Fig. 4).
Fig. 4. Left: Transmission curves of the used polymer color filter array for a thickness of 1.5 µm (source: Brewer Science Inc.). Right: Microscopic view on the pinhole group array with front side illumination. Distance between two pinholes of one group and diameters are 10.6 µm and 3 µm, respectively.
An example of an integrated camera system is shown in the left part of Fig. 5. The optics is only 450µm thick and therefore vanishes under the rim of the ceramic package of the CMOS image sensor.
5. Characterization and results
The permanent integration of an artificial neural superposition eye is achieved by adhering the objective to the CMOS image sensor. Alternatively, we use a setup shown in Fig. 5 (right part) to characterize several objectives with the same sensor. The image sensor is aligned horizontally on a stage allowing for translation and rotation in three axes. A vacuum holder in combination with a three axes piezo-electric actuator is used to position the thin compound eye objective.
Fig. 5. Left: Artificial neural superposition eye objective for color imaging (red box) directly attached to CMOS image sensor (model Saentis, ZMD). The microlens array with integrated color filters is shown in detail (black box). Right: Experimental setup for the assembly and characterization of the artificial neural superposition eye.
During an active alignment we observe the image of an homogeneously illuminated white plane which is projected in the field-of-view of the system. The homogeneity and brightness of the image determine when the objective is well aligned with respect to the sensor. The system parameters and alignment tolerances are presented in Tab. 1.
Table 1. Parameters of selected demonstration systems and alignment tolerances.
5.1. Testing sensitivity increase and noise suppression
The superposition or in other words the relation between pixels with a common viewing direction (pixel rearrangement or averaging of gray values) is implemented in a software interface. To improve the image quality, the dark signal non-uniformity is subtracted and fixed-pattern-noise (FPN) is reduced using a single calibration. Thus, the individual pixel values of each frame are multiplied by correction factors which were calculated from a series of frames under homogeneous illumination and a fixed integration time T int.
For increasing the sensitivity, each pixel in the final image is an average of nine pixels with a common viewing direction. In Fig. 6 we compare some acquired test images qualitatively in case that the camera is used with and without this artificial neural superposition.
Fig. 6. Images taken with the artificial neural superposition eye: Left: Siemens radial star target, Middle: an image of Carl Zeiss and Right: “Image Processing Lena”. (a),(b) and (c): Each image pixel recorded from one pixel per channel. (d), (e) and (f): Each image pixel is an average of nine pixels out of different channels. In (b) and (e) the fixed-pattern noise (FPN) has not been subtracted. The image resolution is 70×53 pixels without and 65×43 pixels with digital superposition.
The number of image pixels of the artificial neural superposition eye is decreased due to missing of signals for superposition from at least one side at the borders (note the difference in the dimensions of Fig. 6(b) and 6(e)). Besides that the superposed results look smoother than the original images, there is no visible effect when looking at still images. The profit is measured in dynamic scenes and its quantity is the signal-to-noise ratio (SNR). Figure 7 visualizes the way to derive such a parameter for a 2D image sequence.
The whole algorithm can also be used for a small region in the images to reduce the influence of a spatially varying illumination (e.g. shading) on the SNR. Special care must be taken in selecting a light source for the noise measurements. First, the photon noise of the light source should be less dominant compared to the thermal and shot noise in the CMOS sensor. Second, variations of the light output on a time scale of several ms or even seconds are unwanted because they cause a shift of the average gray value of individual frames in the sequence. Also, a surging amplitude in the illumination due to conflicts of the source refresh rate and CMOS integration time has to be prevented. Last but not least, spatial variations of the illumination on the sensor array have to be corrected as good as possible because they strongly influence the measurement of a global image SNR. We used a temperature controlled LED source with collimation and homogenization optics to illuminate a paper-sized white screen. The distance to the screen was adjusted in order to fill the FOV of the camera at about 1.3 m.
In Fig. 8(a) we compare the measured SNRs for different integration times for the standard read out and the superposition mode where nine corresponding pixels of the raw image are averaged to give one pixel of the final image. To quantify the difference, we plotted the increase of the SNR, which is given by
in Fig. 8(b). The measured SNR increase ranges from 2.5 to 4 which is in good agreement to the predicted factor of 3. Deviations are due to residual fixed-pattern noise and illumination inhomogeneity.
Fig. 8. (a): The measured signal-to-noise ratio (SNR) as a function of the integration time. (b): The ratio of the SNR for nine pixels per channel and for one pixel per channel gives the increase of the SNR as a function of integration time.
To check whether the resolution stays constant when the system runs in the superposition mode, we compared the MTF for both modes (Fig. 9).
Fig. 9. Measured and simulated MTF (polychromatic, spatial frequency normalized to Nyquist frequency ν n=1.28 cycles/degree) for either one pixel per channel and nine pixels per channel mode. Both measured curves result from a measurement of an average MTF across all channels which approaches the tangential off-axis simulation rather than the paraxial one. Note that there is no severe difference between both MTFs.
The difference between both curves is marginal. A variation of the pinhole diameter and radius of curvature of the lenslets causes a deviation of the acceptance angles of each pixel. In other words, each pixel images a field of slightly different size leading to a blur in the superimposed image. We measured variations of 5 % and 1.3 % for the pinhole diameter and radius of curvature respectively. However, the lateral pitches of the different layers show sub-micron precision due to the high accuracy of photo lithography processes.
5.2. Investigating color imaging properties
We used the same experimental setup as in section 5.1 with a slightly modified software interface. In case of the color imaging artificial neural superposition eye, white balancing is carried out together with the FPN correction. We tested the color imaging abilities with different scenes in Fig. 10.
Fig. 10. Images taken with the color imaging artificial neural superposition eye: From left to right: Macbeth color checker, the principle author, “Image Processing Lena”, a color mixing circle and a Siemens radial star pattern. The image resolution is 52×43 pixels except the first image which has 70×39 pixels.
The artifacts in the corners of the images are caused by the vacuum holders that cover some optical channels. They disappear when the objective is fixed on the sensor and the holders are removed (Fig. 10 first image). Also, the number of pixels in the color image is slightly lower than the number of optical channels because at least one corresponding signal is missing at the border which results in false colors after the superposition.
For a quantitative comparison, the modulation transfer function (MTF) of the camera system is examined (Fig. 11(c)). Due to the use of a microlens array with spherical lenslets, the image quality suffers from aberrations like astigmatism and field curvature for large angles of incidence. Thus theMTF is reduced in the off-axis case. Contrasting to the simulation the image acquisition for one color channel (R, G or B) is not monochromatic due to the spectral distribution of the light source and the transmission curve of the color filter array (Fig. 4). Hence, the simulated and measured curves do not coincide totally.
Fig. 11. Comparison of measured and simulated MTF for the artificial compound eye color camera (demo 1 with a Nyquist frequency of ν n=1.28 cyc/deg). The simulations are done for the three wavelengths: λ 1=455nm for blue, λ 2=530nm for green and λ 3=680nm for red. The off-axis simulation was carried out for a field angle of 13.6 degrees tangential.
6. Conclusion and outlook
We report on the layout, microoptical fabrication and characterization of an multi-channel imaging system. Inspired by the neural superposition eye of insects, we implement redundant sampling to achieve an increased sensitivity with suppression of temporal noise and color imaging. In our solution each object point is observed through multiple pixels from different optical channels in parallel. The corresponding pixels are then superimposed digitally according to the wanted functionality. Besides the demonstrated increase of sensitivity and the ability for color imaging also polarization or multi-spectral imaging is feasible with this approach. The presented solution to ultra-compact color imaging offers the advantages that no interpolation between single color planes is needed and the resolution of the device is unchanged compared to grayscale imaging.
The ability of the artificial neural superposition eye to increase sensitivity and to suppress noise is demonstrated quantitatively by measurements of the signal-to-noise ratio. In the presented system the SNR is three times larger than in the conventional system because each pixel of the final image results from an average of nine pixels observing the same point in object space. Using redundant sampling with multiple pixels per channel allows to break the inverse relationship of resolution and sensitivity of artificial apposition compound eyes. Hence, we increase the photo-active area and therefore the sensitivity without decreasing the angular resolution.
Color contrast is a source of additional information which can be sensed with an artificial compound eye camera and by increasing the sensitivity the integration time may be reduced to allow for higher frame rates. The demonstrated modifications enhance the abilities of these multi-channel imaging devices and partly compensate for their characteristic low image resolution. The use of a customized image sensor would have the advantage that the superposition could be implemented into the electronic circuits which reduces digitalization noise and enhances speed. To further raise the sensitivity, the number of pixels per channel could be increased which requires a sensor with a smaller pixel pitch. However, in this case a channel-wise suppression of crosstalk becomes indispensable. The demonstrated devices are not seen as a replacement for conventional optics but they are useful in niche applications where the reduction in thickness below one millimeter is crucial. Among others, surveillance, machine and robot vision as well as imaging sensors for automobiles are promising fields of application for the artificial neural superposition eye.
Acknowledgments
The work presented here was partly funded by the German Federal Ministry of Education and Research (BMBF) within the project “Extremely compact imaging systems for automotive applications” (FKZ: 13N8796). Furthermore, the authors like to express their gratitude to Sylke Kleinle and Antje Oelschläger for their contributions to the fabrication of the artificial neural superposition eyes.
References and links
1. M. F. Land and D.-E. Nilsson, Animal Eyes, Oxford Animal Biology Series (Oxford University Press, Oxford, 2002).
2. J. Duparre, P. Dannberg, P. Schreiber, A. Bruer, and A. Tünnermann, “Thin compound-eye camera,” Appl. Opt . 44, 2949–2956 (2005). [CrossRef] [PubMed]
3. J. Duparré, F. Wippermann, P. Dannberg, and A. Reimann, “Chirped arrays of refractive ellipsoidal microlenses for aberration correction under oblique incidence,” Opt. Express 13, 10539–10551 (2005). [CrossRef] [PubMed]
4. K. Kirschfeld and N. Franceschini, “Optical characteristics of ommatidia in the complex eye of Musca,” Kybernetik 5, 47–52 (1968). [CrossRef] [PubMed]
5. H. Tian, B. Fowler, and A. E. Gamal, “Analysis of temporal noise in CMOS photodiode active pixel sensor,” IEEE J. Solid-State Circuits 36, 92–101 (2001). [CrossRef]
6. A. E. Gamal, B. Fowler, H. Min, and X. Liu, “Modeling and estimation of FPN components in CMOS image sensors,” in Solid State Sensor Arrays: Development and Applications II, M. M. Blouke, ed., Proc. SPIE 3301, 168–177 (1998).
7. I. N. Bronstein, K. A. Semendjajew, G. Musiol, and H. Mühlig, “Taschenbuch der Mathematik,” (Harri Deutsch, Frankfurt am Main, 2001).
8. B. E. Bayer, “Color Imaging Array,” U.S. Patent 3,971,065, 1976.
9. J. Adams, K. Parulski, and K. Spaulding, “Color processing in digital cameras,” IEEE Micro 18, 20–30 (1998). [CrossRef]
10. R. F. Lyon and P. M. Hubel, “Eyeing the camera: into the next century,” in Proc. IS&T/TSID 10th Color Imaging Conf., (Scottsdale, AZ, USA, 2002), pp. 349–355.
11. J. Tanida, R. Shogenji, Y. Kitamura, K. Yamada, M. Miyamoto, and S. Miyatake, “Color imaging with an integrated compound imaging system,” Opt. Express 11, 2109–2117 (2003). [CrossRef] [PubMed]
12. R. Shogenji, Y. Kitamura, K. Yamada, S. Miyatake, and J. Tanida, “Multispectral imaging using compact compound optics,” Opt. Express 12, 1643–1655 (2004). [CrossRef] [PubMed]
