Open Access
Add to BibSonomyAbstract
Current stereo eye-tracking methods model the cornea as a sphere with one refractive surface. However, the human cornea is slightly aspheric and has two refractive surfaces. Here we used ray-tracing and the Navarro eye-model to study how these optical properties affect the accuracy of different stereo eye-tracking methods. We found that pupil size, gaze direction and head position all influence the reconstruction of gaze. Resulting errors range between ± 1.0 degrees at best. This shows that stereo eye-tracking may be an option if reliable calibration is not possible, but the applied eye-model should account for the actual optics of the cornea.
© 2017 Optical Society of America
1. Introduction
Eye-tracking methods are used extensively in a wide range of research fields, such as studies on attention, visual search, reading and human-computer interaction [1–3]. Different types of eye-tracking systems exist (see e.g [4,5] for extensive overviews). The scleral search coil technique measures electromagnetic induction in a copper coil which is embedded in a contact lens [6,7]. Electro-oculography measures the electrical potential of the cornea-retinal dipole using electrodes placed on the skin [8]. However these methods are either relatively invasive and uncomfortable, or noisy and prone to drift, which makes them less suitable to use in clinical populations or in children. Therefore, there is a strong preference to use non-invasive eye-trackers. Currently, the majority of non-invasive eye-tracking systems are video-based. Images of the eye are captured by one or two video-cameras, and fast image-processing techniques calculate the gaze direction. Most systems estimate the point-of-gaze from images of the pupil, together with one or more corneal reflections, also called Purkinje images or glints, produced by external infrared (IR) light sources [9–12]. The different approaches to estimate the point-of-gaze, or the direction of gaze, can be categorized into 2D interpolation-based methods and 3D model-based techniques [13]. 2D interpolation methods estimate gaze by relating image features to 2D gaze coordinates on a screen by means of empirical mapping functions, whereas 3D model-based approaches estimate gaze direction from a geometrical model of the eye.
Traditionally, video-based eye-trackers require an in vivo calibration procedure to determine the mapping between image features and gaze in 2D systems, and to estimate parameters of the eye, such as the radius of corneal curvature, in 3D systems. This calibration procedure involves the fixation of several small visual targets at known locations by the test subject. However, certain participants, for example, infants and people with oculomotor problems, poor visual acuity, or cerebral visual impairment, are unable to perform such a fixation task reliably. To cope with this problem, researchers have developed alternative methods that make the calibration less dependent on such individual procedures. Several methods have been suggested to simplify the calibration process. Shih and colleagues were the first to propose a 3D method in which two cameras and two IR light sources are used to compute the optical axis of the eye from estimates of the location of the center of the pupil and the center of the corneal curvature [14]. Subsequently, Guestrin and Eizenman generalized the geometrical model to fully calibrated set-ups [9]. In these set-ups the internal camera parameters such as the focal length are known, as well as the position and orientation of the cameras and light sources. Other researchers have proposed similar 3D methods to compute the optical axis of the eye from the pupil center and corneal reflections (e.g [15], see [13] for an overview of the different approaches). Alternative methods have been proposed that estimate the optical axis from the shape of the limbus [16], ellipse descriptions of the pupil contour combined with conic algebra [17], or corneal reflections and pupil center combined with pupil-contour data [18]. The main advantage of these stereo eye-tracking methods compared to other video-based eye-tracking methods is that, in principle, only one calibration point is needed to estimate the deviation between the optical and visual axes.
Important for all methods using pupil data is that the actual pupil is not observed, but a virtual image of the pupil (entrance pupil) due to the refractive power of the cornea. In general, it is assumed that the entrance pupil lies in front of the actual pupil, that the entrance pupil is slightly larger, and that the optical axis goes through the center of the actual pupil and the entrance pupil. Chen et al. investigated this assumption through ray-tracing for a stereo-camera set-up and found small deviations of the virtual pupil, with a resulting error in simulated gaze estimates of 0.08 degrees on average [15]. However, they applied a simplified geometrical model of the eye, in which the cornea was modelled as a convex sphere with only one refractive surface. However, the actual cornea has two refractive surfaces: the anterior surface and the posterior surface, each having a different radius and center. As a result, light is refracted differently depending on the position on the cornea. Additionally, the anterior surface of the cornea is not a perfect sphere, but is slightly aspheric. Models that account for this aspect of the anatomy of the eye do exist, such as the Navarro eye model [19]. Fedtke et al. used this model to estimate the location and shape of the entrance pupil as a function of viewing angle. The results revealed that the entrance pupil moves forward, tilts and curves towards the observer as the viewing angle increases, and the geometric mid-point of the entrance pupil departs from the optical axis. Moreover, as pupil size increases, the deviation from the optical axis increases [20]. In addition, previous research has already revealed the influence of pupil size on the accuracy of gaze estimations in video-based eye-tracking [21] and the influence of gaze position on the estimates of pupil size [22].
However, the implications of these previous findings for the accuracy of stereo eye-trackers are unclear. As both cameras observe a different entrance pupil, the 3D reconstruction of the entrance pupil might not align with the actual pupil. Moreover, the effect of more complicated and anatomically more accurate models of the eye on the reconstruction of gaze using stereo eye-trackers has not been investigated either. Therefore, the first aim of our present study was to estimate the effect of the anatomy of the eye on the accuracy of 3D methods to reconstruct the direction of gaze using video-based stereo eye-trackers. The entrance pupil for each camera also depends on the viewing angle and pupil diameter. As a result, head translations, gaze angles as well as changes in pupil diameter may all influence the accuracy of stereo eye-trackers. The second aim is to quantify the effect of head translation and pupil diameter on gaze estimations.
2. Methods
2.1 Ray-tracing model
The Navarro schematic eye model (Table 1) was used to simulate the virtual pupil. Optical modelling was done by ray-tracing in Matlab (version R2013b, The MathWorks Inc, Natick, MA, USA). To model a stereo eye-tracker set-up, two virtual cameras with 2048 × 1088 pixels and a focal length of 3000 pixels (16.5 mm with a pixel size of 0.0055 mm) were used in a right handed coordinate system with the origin between the nodal points of the cameras. Figure 1 shows a top view of this set-up.
Table 1. Parameters of the Navarro schematic eye model . The surface of the anterior cornea is described by the formula , where Q is the conic constant and R is the radius of curvature.
Fig. 1 Top view of the simulated stereo eye-tracking set-up. Two virtual cameras were placed 120 mm apart, with their nodal points 60 mm left and right from the origin in a right-handed coordinate system. The optical axis of both cameras intersect at 400 mm from the origin in the horizontal plane. The simulated eye was placed at different spatial locations (X∈[0, 30, 60] mm, Y∈[0, 30, 60] mm and Z = 400 mm).
The eye model was placed at nine different 3D positions to simulate head translations in the fronto-parallel plane (X∈ [0, 30, 60] mm, Y ∈ [0, 30, 60] mm, Z = 400 mm), and 81 different gaze directions (horizontal and vertical angles of −15, −10, −5, −2.5, 0, 2.5, 5, 10 and 15 degrees). In addition, six pupil diameters ranging from 1.0 to 6.0 mm in 1.0 mm steps were analyzed. To isolate the effects of pupil translation and pupil rotation, the actual pupil was rotated around the center of the pupil instead of the center of rotation of the eye. This prevented translations of the pupil due to the rotation of the eye.
First, we created a general ray-tracing model of the pupil (Fig. 2). The model contained 32 equally spaced points on the pupil boundary to calculate the position and orientation of the entrance pupil. For each point on the pupil boundary, 125751 rays were aimed from that pupil point to the posterior surface of the cornea, and subsequently the refracted rays were computed using Snell’s law. This process was repeated for the anterior cornea. To calculate the corresponding point on the entrance pupil, it had to be determined which of these refracted rays would intersect the nodal point of each camera. Towards that end, the rays were first rotated according to the gaze direction. Subsequently, for each camera the minimal distance between the refracted rays and the nodal point of the camera was calculated and the ray with the smallest distance to the nodal point was used in an optimization procedure to find the optimal refracted ray. The optimal refracted rays for both cameras were triangulated to obtain the image point of the virtual pupil (Fig. 2).
Fig. 2 3D reconstruction of the virtual pupil. For each point on the pupil boundary the refracted ray that passes through the nodal point of the camera was determined. Subsequently, the refracted rays from both cameras were triangulated to obtain the 3D coordinates of each corresponding point of the virtual pupil. Parameters of the Navarro eye model are indicated on the left-hand side.
After calculating all the virtual pupil points in 3D, we determined the location and orientation of the entrance pupil by fitting a plane through the 32 virtual image points and taking the orientation of the normal vector of this plane.
2.2 Gaze reconstruction
To test the accuracy of gaze reconstructions when using an anatomically accurate model of the human cornea in stereo eye-tracking, we compared two different stereo eye-tracking methods. For both methods, the 3D points of the virtual pupil were first projected onto a 2D image for each camera using the Camera Calibration toolbox of Bouguet [23]. This procedure corresponds with the normal setting in which stereo eye-trackers are used to reconstruct the direction of gaze from two 2D images of the eye. The 2D images of the pupil were not pixelated in order to mimic the optimal situation without image noise. This ensures that a potential difference between the reconstructed gaze direction and the actual gaze direction is caused only by the optics of the eye, and is not confounded by additional image noise. A least-squares ellipse fit [24] was then applied to the pupil boundary points in the respective 2D images.
The optical axis of the eye passes through both the center of the actual pupil and the center of curvature of the cornea (Fig. 3, grey dashed line). Previously, it has been assumed that one can calculate the center of corneal curvature (CC) from the glint locations on the cornea, and that the center of the virtual pupil (VP) lies on the optical axis. If true, the VP-CC vector can be used to determine the orientation of the optical axis of the eye. However, this assumption is based on a simplified eye model. The first gaze reconstruction method uses the VP-CC vector (Fig. 3, light blue arrow), to estimate the gaze direction (see, e.g [9,15].).
Fig. 3 Estimation of the center of corneal curvature. A. The ray-trace model to estimate the center of corneal curvature based on a spherical anterior cornea, L1 and L2 are the light sources, C1 and C2 are the cameras. The incident ray is reflected at the anterior corneal surface. The angle of incidence θi is equal to the angle of reflection θr. The normal vectors intersect at the center of corneal curvature. B. The intersecting normal vectors in case of a spherical corneal curvature. C. The normal vectors at the surface of an aspherical cornea.
In the Navarro eye model, the center of the anterior corneal curvature is located 4.12 mm behind the center of the actual pupil on the optical axis (Fig. 2). If a spherical model of the anterior cornea is used, the center of corneal curvature can be estimated using the reflections of two IR light sources (e.g., [9]). The optics behind this is displayed in Fig. 3(A). Incident rays perpendicular to the surface, and thus coinciding with the normal vector at the surface, are reflected back along their own paths. For all other rays the angle between the normal vector and incident ray is equal to the angle between the normal vector and refracted ray (Fig. 3(A)). If the cornea had a perfect spherical surface, then all the normal vectors at the cornea surface intersect at the center of corneal curvature (Fig. 3(B)). However, if the corneal surface is aspherical, the normal vectors at the corneal surface will not intersect at one point (Fig. 3(C)). In fact, it depends entirely on the point of reflection of the light sources on the cornea whether the corresponding normal vectors intersect on or off the optical axis of the eye, or whether they intersect at all. Therefore, algorithms which approximate the center of corneal curvature by converging into a single point, may give values that are not on the optical axis of the eye.
To determine the size of the estimation error, an algorithm with two simulated light sources was used to determine the location of the center of corneal curvature. The light sources were located 50 mm left from the left camera and 50 mm right from the right camera and 50 mm above the cameras. Figure 3 illustrates the reflection of light source L1 on camera C1 and the reflection of light source L2 on camera C2. In reality, there are four points of reflection, which include also the reflection of light source L1 on camera C2 and the reflection of light source L2 on camera C1. We used ray-tracing to determine the point of reflection of each light source for both cameras. The location at the cornea where the difference between the angle of incidence and the angle of reflection was minimal was estimated using an optimization procedure. Subsequently, the reflections were projected onto the 2D image plane of the cameras. These 2D image coordinates were then used to calculate the apparent location of the center of corneal curvature following the method proposed by [9]. This method assumes that the anterior cornea acts as a spherical mirror and that the center of curvature belongs to the plane defined by the camera C, the light source L and the corresponding image point U of the corneal reflection (Fig. 3(A)). Because three coplanar vectors satisfy the constraint the center of corneal curvature can be estimated by finding the least-squares fit of Eq. (1) for the four reflection points.
To obtain the center of the virtual pupil, the centers of the fitted ellipses to the pupil data from each camera were triangulated using the same internal and external camera parameters that were used to project the entrance pupils on the 2D image plane. Next, the horizontal (or pan) angle (ϕ) and the vertical (or tilt) angle (θ) were calculated from Eq. (2), with p the estimated center of the virtual pupil, and c the location of the estimated center of corneal curvature.
These results were compared to a hypothetical situation, in which the center of corneal curvature is known (Fig. 3(B)). This fixed center of corneal curvature (fCC) was positioned 4.12 mm behind the center of the actual pupil on the optical axis, as defined by the Navarro eye model (Table 1). This simulation isolates the effect of deviations of the center of the virtual pupil from the eye’s optical axis. The comparison of the results of the VP-CC vector method and the VP-fCC vector method shows how errors in estimating the center of corneal curvature influence the overall accuracy of the VP-CC method.
The second gaze reconstruction method applies conic algebra and a parametric description of the pupil boundary to estimate the gaze direction (Fig. 4). Although it has been demonstrated that the image of the pupil is not a perfect ellipse [20], this method uses ellipse fits to define a cone through the nodal point of the camera and the virtual pupil image. By intersecting both cones the orientation and location of the virtual pupil was estimated as previously described by [17]. The orientation of the virtual pupil is the normal vector (n) of the virtual pupil surface, and Eq. (3) was used to calculate the pan and tilt angles.
Fig. 4 Gaze reconstruction using conic algebra. For each camera a cone can be projected through the projection of the virtual pupil at the image plane. The virtual pupil lies at the intersection of these cones.
The results of the VP-CC method and the conic algebra method were compared. Moreover, to test the assumption that the distance between the entrance pupil and the estimated center of corneal curvature remains stable, we calculated this distance for the different locations and orientations of the eye as well as for each pupil diameter.
3. Results
Figure 5 illustrates the 3D shape, orientation and location of the virtual pupil derived from the ray-tracing model for two different gaze directions. As expected from previous work [20], the virtual pupil (blue) lies in front of the actual pupil (black), and is larger than the actual pupil. Additionally, the 3D shape of the virtual pupil is slightly distorted compared with the actual pupil, its orientation (red arrow) is tilted compared to the actual pupil and its geometric center (VP) deviates slightly from the optical axis of the eye (dashed gray line). These effects are larger for the horizontal rotation (Fig. 5(A)) than for the vertical rotation (Fig. 5(B)) due to the geometry of the simulated set-up. Gaze reconstructions obtained with the VP-CC vector method are illustrated as well. The accuracy of this method is affected by the concentric gradient in refractive power of the cornea, which influences the estimated location of the center of the virtual pupil, as well as the aspherical shape of the anterior cornea, which influences the estimated location of the center of corneal curvature (Methods). In this example both points lie almost on the optical axis, resulting in a small gaze error. However, in general, both points can deviate significantly from the optical axis, resulting in larger gaze errors. Figure 6 shows the orientation of the virtual pupil as obtained through ray-tracing (black crosses) and the results of the different gaze reconstruction methods (cyan, blue and red dots) for two different head positions, and for a series of different orientations of the actual pupil (open circles) with a diameter of 4 mm. Note, that the orientation of the virtual pupil, which was determined from a plane fitted through the virtual pupil boundary points (Fig. 5), can deviate significantly from the orientation of the actual pupil. This difference depends systematically on the orientation of the actual pupil as well as on the translation of the head. If the pupil is located at the central position between both cameras (X = 0, Y = 0, Z = 400 mm, Fig. 6(A)), a symmetrical pattern is observed with increasing differences as the gaze angles increase in the horizontal and vertical direction. The discrepancies range up to 2.88 degrees for horizontal gaze angles of 15 degrees (~19%), and up to 0.96 degrees (~6.5%) for vertical gaze angles of 15 degrees. When the actual pupil is translated 6 cm to the right and 6 cm down (Fig. 6(B)) the pattern changes, with the smallest errors now occurring for a horizontal gaze angle of ~10 degrees to the right, and a vertical gaze angle of ~10 degrees down. The maximum differences increase to 4.47 degrees horizontally and 1.61 degrees vertically.
Fig. 5 Location, orientation and shape of the virtual pupil obtained through ray tracing compared to the actual pupil for two different cases. A. The actual pupil is rotated 15 degrees to the right. Top view of the simulation. B. The actual pupil is rotated 15 degrees up. Side view of the simulation. The optical axis (OA) always goes through the fixed center of corneal curvature (fCC) and the center of the actual pupil (AP). The 3D shape of the virtual pupil boundary varied between conditions. A plane was fit through the boundary points of the virtual pupil (VPplane). The normal vector of this plane (VPnorm) was used to describe the orientation of the virtual pupil. The VP-CC vector is defined by the estimated center of corneal curvature (CC) and the center of the virtual pupil (VP).
Fig. 6 Reconstruction of gaze direction with stereo eye-tracking methods compared to the orientation of the actual and virtual pupil. A. The actual pupil was positioned at the central location between both cameras (X = 0mm, Y = 0mm, Z = 400mm). B. The actual pupil was shifted 6 cm to the left and 6 cm downwards, simulating a translation of the head.
Gaze reconstructions obtained with the conic algebra method correspond closely to the orientation of the virtual pupil (Fig. 6, red circles). Within the range of simulated gaze angles and head translations for a pupil diameter of 4 mm, the maximum difference between the orientation of the virtual pupil and the conic algebra method was only 0.12 degrees.
The VP-fCC method resulted in significantly smaller errors (Fig. 6, cyan circles). For this method, the maximum horizontal and vertical errors were 0.08 and 0.09 degrees, respectively, when the pupil was located at the central position, and 0.12 and 0.18 degrees when the head was translated 6 cm to the left and 6 cm downwards. Note that these errors were only due to shifts in the center of the virtual pupil away from the eye’s optical axis, because the VP-fCC method assumes that one would be able to obtain error-free estimates of the center of corneal curvature (Methods). In practice, however, the center of corneal curvature must be estimated from the corneal reflections of the IR light sources. The VP-CC method includes this estimation procedure (Methods).
Errors obtained with the VP-CC method (Fig. 6, dark blue circles) increased compared with the theoretical VP-fCC method, but were still small compared with the conic algebra method. In the central position the VP-CC method resulted in maximum errors of 0.22 degrees horizontally and 0.47 degrees vertically. When the actual pupil was translated 6 cm up and to the right these errors increased, ranging up to 1.16 degrees horizontally and 1.3 degrees vertically for 15 degrees pupil rotations.
Figure 7 quantifies the systematic effect of pupil orientation and (head) translation on the gaze reconstruction errors for the different reconstruction methods. Each curve shows the results obtained with the eye model placed at a given location in the fronto-parallel plane. Left-hand panels plot the horizontal gaze errors as a function of horizontal pupil orientation, with data averaged across the nine different vertical pupil orientations. Error bars denote the range of horizontal gaze errors across those vertical pupil orientations. Right-hand panels plot vertical gaze errors as a function of vertical pupil orientation, with data averaged across the nine different horizontal pupil orientations. Error bars denote the range of vertical gaze errors across those horizontal pupil orientations. Colors identify the magnitude of the horizontal or vertical translation from the central position. Note that a 3 cm horizontal translation from the median plane corresponds closely to the natural location of the human eye. For symmetry reasons, we only translated the eye model in 1 quadrant of the fronto-parallel plane (down and to the left). For each of the nine simulated positions, gaze errors depended systematically on the orientation of the actual pupil. In general, the average horizontal and vertical gaze errors increased monotonically with increasing rotation of the actual pupil, but note that cross-talk between the horizontal and vertical components was quite limited. The latter can be inferred from the size of the error bars. Horizontal and vertical translations also had a significant influence on the estimated gaze direction, but note that the errors in the horizontal and vertical direction of gaze were remarkably invariant to vertical and horizontal head translations, respectively. Especially for the conic algebra method, this type of cross-talk is quite small, which is why the three curves of a given color overlap. In fact, gaze errors observed for the conical algebra method show a remarkably linear pattern; component errors increase at a fixed gain with the corresponding gaze angle and head translations add a fixed, position-dependent bias to these errors. For the simulated set-up, horizontal errors were approximately 3 times large than vertical errors.
Fig. 7 Accuracy of the different gaze reconstruction methods for a fixed pupil diameter of 4 mm. A. the conic algebra method. B. the VP-fCC method. C. the VP-CC method. Each plot shows the horizontal/vertical gaze reconstruction errors as a function of the horizontal/vertical orientation of the actual pupil for each of the nine different pupil/head translations. Data are averaged either across the nine different vertical pupil orientations (left-hand plots) or the nine different horizontal pupil orientations (right-hand plots). Colors identify the magnitude of the horizontal (left-hand plots) or vertical (right-hand plots) translation. Note the scaling differences between plots A, B and C.
For the VP-fCC method the gaze reconstruction errors also showed a near-linear dependency on translations and rotations of the actual pupil, but the resulting errors were in the opposite direction and much smaller. More specifically, the influence of horizontal pupil orientation was nearly 80 times smaller, and the effect of horizontal pupil translation was about 25 times smaller compared to the conic algebra method. This difference was smaller for errors in the vertical component, but still significant. There is also a slight position-dependent cross-talk between the horizontal and vertical components (see also Fig. 5).
For the VP-CC method the relationship between translation and rotation of the actual pupil and the error in estimated gaze direction clearly deviated from the other two methods. It showed smooth nonlinearities causing the average component error to increase with an increasing gain as a function of the actual pupil orientation. Cross-talk also increased with increasing pupil rotation and both effects were amplified by translations of the pupil away from the central position. Compared to the VP-fCC method the errors were on average between five and ten times larger. This shows that the aspherical shape of the anterior cornea has a significant impact on mislocalization of the center of corneal curvature. The largest errors (up to 1.5 degrees) occurred for 15 degrees rotations combined with a 6 cm translation.
The effect of pupil diameter on the different gaze reconstruction methods is presented in Fig. 8. The results reveal a strong effect of pupil size on the conic algebra method. The gaze error decreases as pupil size increases for all head positions. The maximum difference in gaze error for a pupil with a diameter of one millimeter and a pupil with a diameter of six millimeter reaches 1.3 degrees. The effect of pupil size on the VP-fCC and VP-CC methods is smaller with maximum differences between the gaze error of the smallest and largest pupil diameter of 0.14 and 0.18 degrees for the two methods respectively. Note that in contrast to the conic algebra method, the gaze error increases as pupil size increases for the VP-fCC and VP-CC method.
Fig. 8 The effect of pupil size on stereo eye-tracking methods. A. the conic algebra method. B. the VP-fCC method. C. the VP-CC method. The mean and range of the gaze errors is plotted for all pupil diameters at two head positions. The dashed lines indicate the results for a pupil diameter of 4 mm. For clarity of the graph, the actual pupil was only translated horizontally (left-hand plots) or vertically (right-hand plots). Note the scaling differences between the horizontal and vertical errors.
In Fig. 9 we show the results of calculations that tested how the distance between the center of corneal curvature and the center of the virtual pupil depended on pupil diameter, head position, and orientation of the virtual pupil. The distance increased for larger pupil sizes and larger gaze angles if the actual pupil is located at the central location (Fig. 9(A)). Both the horizontal and vertical orientation of the actual pupil have an influence, although the horizontal angle had a stronger effect. Additionally, the position of the actual pupil changed this pattern (Fig. 9(B)) into an asymmetrical function.
Fig. 9 The influence of size, orientation and position of the actual pupil on the distance between the estimation of the center of corneal curvature and the center of the virtual pupil. The color coding in each square represents the distance between VP and CC in mm for one simulated gaze orientation. The left panels show the results for a pupil size of 1 mm and the right panels the results for a pupil size of 6 mm. A. The actual pupil was located at the central position. B. The actual pupil was shifted 6 cm to the left, and 6 cm down.
4. Discussion
The results of the ray tracing model demonstrate that the reconstructed entrance pupil based on the images from two (synchronized) cameras does not have the same orientation as the actual pupil. The position of the eye and the gaze angle both influence the difference in orientation. In general, larger gaze angles result in bigger differences between the orientation of actual pupil and the virtual pupil. This is in line with previous research on the orientation of the entrance pupil [20] and demonstrates that the assumption that the virtual pupil is positioned in parallel with the actual pupil is not correct. Additionally, in contrast to previous arguments that stereo eye-tracking methods are invariant to head movements (cf [9]), we found significant effects of head translations on the accuracy of these methods.
Furthermore, the results of gaze reconstructions using two typical stereo eye-tracking methods revealed that the results from both methods are affected by using a more accurate eye model. However, the magnitude of the errors differed substantially between the two methods. Reconstructions based on conic algebra, which use the entire pupil boundary, resulted in relatively large errors. By comparing the conic algebra method with the orientation of the virtual pupil, we demonstrated that the former method essentially reconstructs the orientation of the entrance pupil, rather than the actual pupil. This is a logical consequence of using the images of the entire pupil: both cameras observe a different virtual pupil, and conic algebra assumes that both cameras are looking at the same object. In addition, because the image of the virtual pupil is not a perfect ellipse [20], the use of fitted ellipses might have contributed to larger gaze errors too. The horizontal errors could exceed 4 degrees. The vertical errors were smaller, which can be explained by the location and orientation of the simulated cameras. They were positioned at the same height, while only their horizontal positions differed. As a result, the virtual pupil observed by either camera differed especially in the horizontal direction, and less in the vertical direction. The large errors of the conic algebra method indicate that this method cannot be readily applied for stereo eye-trackers. However, as the errors appeared to be systematically related to the direction of gaze and to head position, it might be possible to correct for these errors. Note, however, that such corrections will be complicated by the influence of the pupil diameter. Further research is necessary to investigate the efficacy of possible methods to correct for these errors.
On the other hand, the gaze errors resulting from the VP-CC method were smaller, remaining within about 1.3 degrees. The largest gaze errors were found for large translations and large rotations (15 degrees) of the actual pupil. This effect is mainly caused by the error in estimating the center of corneal curvature. The gaze errors of the VP-fCC method, where only the center of the virtual pupil is estimated, were on average between five and ten times smaller, with a maximum of only 0.2 degrees within the applied range of eye locations, gaze angles and pupil sizes. The difference between the VP-CC method and the VP-fCC method is caused by the error in estimation of the center of corneal curvature. Additionally, the gaze error is in the opposite direction for the VP-fCC method. With this method the gaze direction is overestimated, whereas the other methods underestimate the gaze direction. The error in the estimation of the center of corneal curvature is caused by the aspherical anterior cornea. When the eye is translated and rotated, the rays from the light sources are reflected further away from the apex of the cornea. Especially the normal vectors at these reflection points will then intersect further away from the center of the cornea. This leads to larger errors in the estimation of the center of curvature and therefore to larger gaze errors. The gaze errors found in our simulations are similar to the errors found in different prototypes that have been developed. Most systems have an average accuracy of ~1 degree [15,25,26].
Previous studies [9,15] assumed that the distance between the center of corneal curvature and the center of the virtual pupil remains constant. This was based on simple eye models with only one spherical refractive surface of the cornea. We used a more realistic eye model and demonstrate that this distance depends on the orientation, position and diameter of the actual pupil (Fig. 9). As described above, errors in the estimation of the center of curvature and in the estimation of the virtual pupil resulted from the assumptions regarding the shape of the cornea. As a result of different optical geometries, the distance between these points varies. Chen et al. [15] proposed to reduce the noise during stereo eye-tracking by correcting the distance between the center of corneal curvature and the center of the virtual pupil to a fixed distance. However, we demonstrated that this assumption is not valid. The effect of such a correction on the accuracy of these methods has to be determined. It might result in additional errors when the pupil size changes. Furthermore, the effect of pupil diameter on the gaze reconstructions and the distance between the center of curvature and the center of the virtual pupil is in line with findings in eye-tracking methods that use one camera, where pupil size systematically influences the outcome of eye-tracking [21]. Changes in pupil size are common during experiments, for example due to differences in cognitive load, in arousal or to differences in luminance on the screen.
The set-up used in this article is a realistic set-up for remote stereo eye-trackers. However, as the spatial location of the eye influenced the results, the results may differ in set-ups where the distance between the eye and the cameras is different. Similarly, the distance between the cameras and orientation of the cameras relative to each other may influence the results, as well as the location of the IR light sources. The different results for the nine spatial locations of the eye indicate that the orientation of the eye relative to the cameras, i.e., a combination of the gaze orientation and the angle between the optical axis of the camera and the position of the eye, cause differences in the orientation of the virtual pupil. Additionally, the position of the IR light sources influences the estimation of the center of corneal curvature. Further research is necessary to investigate whether the set-up can be optimized.
5. Conclusion
We demonstrated that the shape of the cornea has a significant influence on the accuracy of stereo eye-tracking methods. Pupil size, gaze direction and head position all influence the accuracy of eye-tracking methods. The gaze reconstruction that uses the center of the pupil and reflections of IR light sources is more accurate than the conic algebra method, which uses the entire pupil. However, the gaze errors of the conic algebra method appear to be systematic, and therefore a correction could result in more accurate gaze reconstructions. In conclusion, stereo eye-tracking methods that assume a spherical cornea with one refractive surface can be an option in situations where reliable calibration is not possible. However, more accurate measurements require the use of a more elaborate model of the eye geometry in which the optics of the cornea are better taken into account.
Funding
Radboud Universitair Medisch Centrum; ODAS Stichting 2012-35.
References and links
1. A. T. Duchowski, “A breadth-first survey of eye-tracking applications,” Behav. Res. Methods Instrum. Comput. 34(4), 455–470 (2002). [CrossRef] [PubMed]
2. A. Poole and L. J. Ball, “Eye Tracking in Human-Computer Interaction and Usability Research: Current Status and Future Prospects,” in Encyclopedia of Human-Computer Interaction, C. Ghaoui, ed. (Idea Group Reference, 2005), pp. 211–219.
3. K. Rayner, “Eye movements in reading and information processing: 20 years of research,” Psychol. Bull. 124(3), 372–422 (1998). [CrossRef] [PubMed]
4. N. Wade and B. Tatler, The Moving Tablet of the Eye: The Origins of Modern Eye Movement Research (Oxford University Press, 2005).
5. K. Holmqvist, M. Nyström, R. Andersson, R. Dewhurst, H. Jarodzka, and J. Van De Weijer, Eye Tracking: A Comprehensive Guide to Methods and Measures (Oxford University Press, 2011).
6. D. A. Robinson, “Movement using a scieral search in a magnetic field,” IEEE Trans. Bio-med. Electron. 10(4), 137–145 (1963). [CrossRef]
7. H. Collewijn, F. van der Mark, and T. C. Jansen, “Precise recording of human eye movements,” Vision Res. 15(3), 447–450 (1975). [CrossRef] [PubMed]
8. B. Shackel, “Pilot Study in Electro-oculography,” Br. J. Ophthalmol. 44(2), 89–113 (1960). [CrossRef] [PubMed]
9. E. D. Guestrin and M. Eizenman, “General theory of remote gaze estimation using the pupil center and corneal reflections,” IEEE Trans. Biomed. Eng. 53(6), 1124–1133 (2006). [CrossRef] [PubMed]
10. Z. Zhu and Q. Ji, “Eye Gaze Tracking Under Natural Head Movements,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) (IEEE, 2005), pp. 918–923.
11. C. A. Hennessey and P. D. Lawrence, “Improving the accuracy and reliability of remote system-calibration-free eye-gaze tracking,” IEEE Trans. Biomed. Eng. 56(7), 1891–1900 (2009). [CrossRef] [PubMed]
12. T. N. Cornsweet and H. D. Crane, “Accurate two-dimensional eye tracker using first and fourth Purkinje images,” J. Opt. Soc. Am. 63(8), 921–928 (1973). [CrossRef] [PubMed]
13. D. W. Hansen and Q. Ji, “In the eye of the beholder: a survey of models for eyes and gaze,” IEEE Trans. Pattern Anal. Mach. Intell. 32(3), 478–500 (2010). [CrossRef] [PubMed]
14. S.-W. Shih, Y.-T. Wu, and J. Liu, “A calibration-free gaze tracking technique,” in Proceedings 15th International Conference on Pattern Recognition. ICPR-2000 (IEEE, 2000), pp. 201–204. [CrossRef]
15. J. Chen, Y. Tong, W. Gray, and Q. Ji, “A Robust 3D Eye Gaze Tracking System using Noise Reduction,” in Proceedings of the 2008 Symposium on Eye Tracking Research & Applications - ETRA ’08 (ACM, 2008), pp. 189–196. [CrossRef]
16. E. Trucco, T. Anderson, M. Razeto, and S. Ivekovic, “Robust correspondenceless 3-D iris location for immersive environments,” in International Conference on Image Analysis and Processing, F. Roli and S. Vitulano, eds. (Springer-Verlag Berlin Heidelberg, 2005), pp. 123–130. [CrossRef]
17. S. Kohlbecher, S. Bardins, K. Bartl, E. Schneider, T. Poitschke, and M. Ablassmeier, “Calibration-free eye tracking by reconstruction of the pupil ellipse in 3D space,” in Proceedings of the 2008 Symposium on Eye Tracking Research & Applications - ETRA ’08 (ACM, 2008), pp. 135–138. [CrossRef]
18. C. Lai, S. Shih, and Y. Hung, “Hybrid Method for 3-D Gaze Tracking Using Glint and Contour Features,” IEEE Trans. Circ. Syst. Video Tech. 25(1), 24–37 (2015). [CrossRef]
19. R. Navarro, J. Santamaría, and J. Bescós, “Accommodation-dependent model of the human eye with aspherics,” J. Opt. Soc. Am. A 2(8), 1273–1281 (1985). [CrossRef] [PubMed]
20. C. Fedtke, F. Manns, and A. Ho, “The entrance pupil of the human eye: a three-dimensional model as a function of viewing angle,” Opt. Express 18(21), 22364–22376 (2010). [CrossRef] [PubMed]
21. H. J. Wyatt, “The human pupil and the use of video-based eyetrackers,” Vision Res. 50(19), 1982–1988 (2010). [CrossRef] [PubMed]
22. B. Gagl, S. Hawelka, and F. Hutzler, “Systematic influence of gaze position on pupil size measurement: analysis and correction,” Behav. Res. Methods 43(4), 1171–1181 (2011). [CrossRef] [PubMed]
23. J.-Y. Bouguet, “Complete Camera Calibration Toolbox for Matlab,” http://www.vision.caltech.edu/bouguetj/.
24. A. Fitzgibbon, M. Pilu, and R. B. Fisher, “Direct least square fitting of ellipses,” IEEE Trans. Pattern Anal. Mach. Intell. 21(5), 476–480 (1999). [CrossRef]
25. S.-W. Shih and J. Liu, “A novel approach to 3-D gaze tracking using stereo cameras,” IEEE Trans. Syst. Man Cybern. B Cybern. 34(1), 234–245 (2004). [CrossRef] [PubMed]
26. T. Nagamatsu, J. Kamahara, T. Iko, and N. Tanaka, “One-point calibration gaze tracking based on eyeball kinematics using stereo cameras,” in Proceedings of the 2008 Symposium on Eye Tracking Research & Applications - ETRA ’08 (ACM, 2008), pp. 95–98. [CrossRef]
