Multi-sensor image registration based on algebraic projective invariants

Bin Li; Wei Wang; Hao Ye

doi:10.1364/OE.21.009824

1. Introduction

Image registration is a process to align two images which were acquired in the same scene, while from different sensors at different times [1]. The two images are often referred to as the reference image and the sensed image, respectively [1]. The purpose of registration is to estimate the coordinate transformation model, according to which the sensed image can be well registered to the reference image. Since the images captured through different sensors will involve distinct characteristics, fused images delivering more information than the original images may be expected after performing the image registration. This constitutes the main reason that automatic registration of multi-sensor images has become an indispensable step in many practical applications including change detection, medical diagnosis, and synthetic vision [2–4].

However, compared with fruitful results reported on the image registration for single-sensor images, the effective registration algorithms for multi-sensor images are still limited mainly due to the following two challenges. The first one is the gray scale discrepancy between corresponding pixels of the reference and sensed images. As two different sensors have different operation wavelengths, there is no strict one-to-one correspondence between the intensities of the two sensed images. The second challenge is geometric deformation between the image pairs. Note that the translation, scaling, rotation, affine and projective transformations are some representative deformation models. Clearly, it will be rather difficult to design multi-sensor registration algorithms if the deformation models are complex.

The existing multi-sensor image registration schemes can roughly be classified into two categories: the area-based and the feature-based methods [1]. In area-based registration methods, similarity measures, such as Mutual information (MI) based [5–7] and correlation based [8,9] similarity measures, are usually introduced. By searching the parameter space, the estimated transformation model parameters obtained by maximizing the similarity measures are often accepted as the final results. Since the parameter searching space will expand exponentially with the deformation complexity increases, this type of methods may cost extremely long time to find satisfactory results in the cases of affine and projective deformations. In contrast to these, salient features are extracted from reference and sensed images separately in feature-based registration methods. By matching the extracted features, a set of corresponding points can be obtained, which will then be utilized to estimate the parameters in the transformation models directly. Since parameter searching is not required in the feature-based methods, feature-based methods have advantages in computational efficiency over the area-based methods, especially when the images with complex deformations are dealt with.

It is worth mentioning that the contour features are often adopted in the feature-based multi-sensor image registration for its good robustness against gray scale discrepancy [10–12]. However, the contour-based methods proposed in [10–12] are not applicable if severe affine or projective deformations exist in the image pairs. This is because that these complex geometric deformations may significantly destroy the structure information of the contours, which will affect the accuracies of the registration results. In addition to the contour features, some registration methods based on other features have been proposed for the multi-sensor images recently. Wong et.al [13] adopted the local maximum moment of the phase congruency as an image feature, which is shown to be invariant to the gray scale discrepancy and affine deformation. Song et al. [14,15] introduced the affine-invariance property of Triangle Area Representation (TAR) [16] into two representative feature-based single-sensor image registration methods, i.e. the Scale-Invariant Feature Transform -SIFT [17] and the Speeded Up Robust Features -SURF [18], to deal with multi-sensor images with affine deformation. Nevertheless, the aforementioned methods cannot handle the image pairs with projective deformations. To the best of the authors’ knowledge, very few results in feature-based image registration are available to deal with multi-sensor images in the presence of projective deformations.

In this paper, a feature-based registration method will be proposed for the multi-sensor images with projective geometric deformation. Considering the fact that the contour-based features are robust to gray scale discrepancy of the images and the corners on a contour will not vanish after arbitrary projective transformations, we firstly select the corners on the extracted contours as basic feature points. It should be emphasized that if the framework of traditional feature-based methods is adopted, i.e. to design descriptor for the feature points directly [1], it is difficult to ensure that the designed descriptor can keep invariant when the projective deformation and gray scale discrepancy exist in the multi-sensor image pair simultaneously. To overcome this difficulty, instead of describing the detected corners directly, an intermediate step of constructing a new feature, called Five Sequential Corners (FSC), is introduced in this paper. Then a descriptor is designed for each FSC by employing the algebraic projective invariants [19]. Based on these, the set of corresponding FSC pairs can be obtained and the set of corresponding corner pairs will be determined in further. It will be shown that our proposed feature-based registration scheme has good robustness against both projective deformation and gray scale discrepancy between the image pairs.

2. The proposed image registration algorithm

2.1 Overview

The framework of the proposed registration algorithm is shown in Fig. 1. The brief introductions to all the steps involved in Fig. 1 are given as follows.

Fig. 1 The framework of the proposed method.

Download Full Size | PDF

Step 1: Detect edges by employing Canny [20] edge detector to obtain a binary edge map from each input image, and extract contours by the Curvature Scale Space (CSS) algorithm introduced in [21].

Step 2: Detect the corners from each extracted contour by adopting the contour-based corner detector proposed by He et al. [22].

Step 3: Construct the newly introduced feature FSCs on the basis of the corners. Each FSC is determined not only by the coordinates of the five corners chosen, but also by the sequence that the five corners are traversed.

Step 4: Describe each FSC using a ten-dimensional vector, in which each element is an algebraic projective invariant.

Step 5: Match the FSCs by comparing their descriptors, based on which the correspondences of the corners can be obtained.

Step 6: Estimate the parameters of transformation model using the RANSAC algorithm [23] based on the set of corresponding corners.

In the frameworks of the most existing feature-based methods, which generally consist of feature detection, feature description, feature matching and transformation estimation [1], the descriptors are designed directly for the feature points. Different from this traditional framework, the feature points (i.e. the corners detected in Step 2) are used to construct FSCs firstly in our proposed algorithm (see Step 3). A descriptor will then be designed for each FSC in Step 4 by introducing the algebraic projective invariants, which will be shown to be robust against both the projective deformation and gray scale discrepancy between the image pairs.

It is discussed in Step 3 that a FSC cannot be determined solely by the coordinates of the five chosen corners. In fact, if an FSC can be determined once the five corners are specified, the algebraic projective invariants can still be introduced to design the feature descriptor for the FSC. However, only the correspondences of the FSCs can be obtained by comparing their descriptors rather than the correspondences of the corners. Obviously, the collection of the former correspondences is not sufficient to estimate the final transformation model. To address this problem, the sequence information of the five corners selected is also included in defining an FSC as indicated in Step 3. By doing this, we can obtain the matched corners according to the matched FSCs finally. More detailed illustration about this issue will be given in Sec. 2.2.

2.2 Construction of the FSC

In this section, we will give a detailed introduction to the strategy of constructing the new feature FSC. As shown in Fig.1, a set of FSCs will be constructed from the reference and sensed images respectively in Step 3 of the proposed method. Note that the two constructed sets of the FSCs should contain sufficiently large number of pairs of corresponding elements, so that we can obtain enough matched points which will be used to estimate the final parameters of the transformation model in later steps of the proposed method.

In this paper, the definitions of an FSC and a pair corresponding FSCs are given as follows.

Definition 1: For a given contour, an FSC is defined as a group of five corners on the contour that are traversed in a sequential order. The FSC can be denoted as

g^{i} = {p_{1}^{i}, p_{2}^{i}, p_{3}^{i}, p_{4}^{i}, p_{5}^{i}}

where _$i$ is the sequence number of the FSC,

p_{1}^{i} = (x_{1}^{i}, y_{1}^{i})

,

p_{2}^{i} = (x_{2}^{i}, y_{2}^{i})

, …,

p_{5}^{i} = (x_{5}^{i}, y_{5}^{i})

are the five corners with

(x_{k}^{i}, y_{k}^{i})

representing the coordinate of the corner _{$p_{1}^{i}$}.

Definition 2: Two FSCs $g_{r e f}$ and $g_{s e n}$ (constructed from the reference and sensed image, respectively) are regarded as a pair of corresponding FSCs, if and only if the following conditions are satisfied:(1): _{$g_{r e f}$} and $g_{s e n}$ are constructed from five pairs of the corresponding corners.(2): The corresponding corners are recorded in the same sequence in $g_{r e f}$ and $g_{s e n}$ , respectively.

Remark 1: From Definition 1, it can be seen that an FSC is determined not only by the coordinates, but also the sequence in which the five corners are traversed. For example, $g = {p_{a}^{}, p_{b}^{}, p_{c}^{}, p_{d}^{}, p_{e}^{}}$ and $\tilde{g} = {p_{b}^{}, p_{a}^{}, p_{c}^{}, p_{d}^{}, p_{e}^{}}$ represent two different FSCs although they are defined from the same five corners. As will be discussed in Sec. 2.4, such definition of the FSC can guarantee that five pairs of corresponding corners can be determined from one pair of corresponding FSCs.

Based on the above definitions, the FSCs are constructed by the following procedure. Firstly, a subset of five corners is selected from the set of detected corners. Secondly, the FSCs are constructed from the five corners in the chosen subset by specifying different recorded sequences of them. Therefore, the number of the FSCs depends on the number of chosen subsets as well as the number of FSCs that a subset can construct. For example, suppose we get _$N$ corners after the step of corner detection (namely Step 2 of Fig. 1), if we randomly choose the subset of five corners and use all the possible sequences of the five corners to construct FSCs, then $(\begin{matrix} 5 \\ N \end{matrix}) = N! / [5! (N - 5)!]$ subsets and $5!$ FSCs from each subset will be obtained. Thus $N! / (N - 5)!$ FSCs will be constructed in total. Obviously, a larger _$N$ will result in a larger number of FSCs and a heavy computational load of the proposed method. To avoid this, we shall propose a construction strategy to reduce the number of FSCs while guaranteeing that enough pairs of the corresponding FSCs can be constructed from the reference and sensed images. Since the contours extracted in Step 1 of the proposed method can be classified into two types, i.e. the closed contour and the open contour, a construction strategy will be designed for each type of contours respectively.

In this paper, the set of corners detected from the ._. extracted contour is denoted as:

c^{j} = {p_{1}^{j}, p_{2}^{j}, ..., p_{n_{j}}^{j}}

where

{p_{k}^{j}}_{k = 1, 2, \dots, n_{j}}

are the corners on the _{$j' t h$} extracted contour, of which the total number is _{$n_{j}$}. The corners

{p_{k}^{j}}_{k = 1, 2, \dots, n_{j}}

are recorded in the following way. For the case of open contour, either terminal corner is recorded first, from which the rest corners are recorded in a sequential order along the contour. For the case of closed contour, the first corner to be recorded is selected arbitrarily, from which the rest corners are recorded in a clockwise direction along the contour. For example, for the open contour as shown in Fig. 2(a), if the top terminal corner

p_{a}

is recorded first, then the sequence of the corners recorded is

{p_{a}, p_{b}, p_{c}}

; for the closed contour as shown in Fig. 2(b), if the corner

{\bar{p}}_{b}

is recorded first, then the sequence of the corners recorded is

{{\bar{p}}_{b}, {\bar{p}}_{c}, {\bar{p}}_{d}, {\bar{p}}_{e}, {\bar{p}}_{a}}

.From Eq. (2), we can divide the entire set of corners detected from an image (denoted as

C

) into several subsets based on different contour information, i.e.,

C = {c^{1}, c^{2}, ..., c^{N_{c}}}

where _{$N_{c}$} is the total number of contours extracted from the image.

Fig. 2 (a) An open contour. (b) A closed contour.

Download Full Size | PDF

The detailed procedure to construct the FSCs is formally presented in Algorithm 1.Algorithm 1: FSC constructionInput: the whole set of corners $C = {c^{1}, c^{2}, ..., c^{N_{c}}}$ ;Output: a set of FSCs, named $G$ ;1. for $i = 1 : N_{c}$ do2. if the _{$i' t h$} contour is open and $n_{i} \geq 5$ do3. for $j = 1 : (n_{i} - 4)$ do4. record ${p_{j}^{i}, p_{j + 1}^{i}, p_{j + 2}^{i}, p_{j + 3}^{i}, p_{j + 4}^{i}}$ and ${p_{j + 4}^{i}, p_{j + 3}^{i}, p_{j + 2}^{i}, p_{j + 1}^{i}, p_{j}^{i}}$ into $G$ ;5. end for6. elseif the $i' t h$ contour is closed and $n_{i} \geq 5$ do7. for $j = 1 : n_{i}$ do8. record ${p_{j}^{i}, p_{j + 1}^{i}, p_{j + 2}^{i}, p_{j + 3}^{i}, p_{j + 4}^{i}}$ and ${p_{j + 4}^{i}, p_{j + 3}^{i}, p_{j + 2}^{i}, p_{j + 1}^{i}, p_{j}^{i}}$ into $G$ ;9. end for10. end if11. end for

Remark 2: In Line 2 and Line 6 of Algorithm 1, _{$n_{i}$}denotes the number of corners on the _{$i' t h$} extracted contour, for $i \in {1, 2, \dots, N_{c}}$ . From Line 8, if the value of the subscript $(j + k)$ for $k \in {1, 2, 3, 4}$ is larger than $n_{i}$ , $p_{j + k}^{i}$ will be replaced with $p_{j + k - n_{i}}^{i}$ when Algorithm 1 is under execution.

We shall use an example to better illustrate the idea of Algorithm 1. Given a reference image as shown in Fig. 3(a), $p_{1}$ , $p_{2}$ , $p_{3}$ , $p_{4}$ and $p_{5}$ are five corners detected from a contour. Suppose that there is a sensed image in Fig. 3(b), where $p_{1}'$ , $p_{2}'$ , $p_{3}'$ , $p_{4}'$ , $p_{5}'$ are the five corners detected from the contour. Clearly, ${p_{i}, p_{i}'}$ for _{$i = 1, 2, \dots, 5$} are the five pairs of corresponding corners. It can also be seen that $p_{1}$ , $p_{2}$ , $p_{3}$ , $p_{4}$ , $p_{5}$ and $p_{1}'$ , $p_{2}'$ , $p_{3}'$ , $p_{4}'$ , $p_{5}'$ are arranged in the same order on the contours in the two images. This is because the sequence of the five points on a contour can keep invariant after arbitrary projective transformations between the reference image and the sensed image.According to Line 2-4 of Algorithm 1, for a group of five successive corners on an open contour, only two FSCs indicating opposite traversal directions will be constructed by Algorithm 1. For example, $g^{1} = {p_{1}^{}, p_{2}^{}, p_{3}^{}, p_{4}^{}, p_{5}^{}},$ _{$g^{2} = {p_{5}^{}, p_{4}^{}, p_{3}^{}, p_{2}^{}, p_{1}^{}}$ and _{${\bar{g}}^{1} = {p_{1}^{}', p_{2}^{}', p_{3}^{}', p_{4}^{}', p_{5}^{}'},$ ${\bar{g}}^{2} = {p_{5}^{}', p_{4}^{}', p_{3}^{}', p_{2}^{}', p_{1}^{}'}$} are the two constructed FSCs for the reference image and the sensed image in Fig. 3, respectively. Moreover, ${g^{1}, {\bar{g}}^{1}}$ and _{${g^{2}, {\bar{g}}^{2}}$} are two pairs of corresponding FSCs obtained from the image pair in Fig. 3 according to Definition 2.}

Fig. 3 An example of five pairs of corresponding corners between the reference and sensed images.

Download Full Size | PDF

Note that if only one traversal direction had been considered in constructing the FSCs, it cannot be ensured that sufficiently large number of corresponding FSC pairs will be obtained since the first traversed corners for the group of corresponding five corners may not be matched. To be more concrete, if $p_{1}$ , $p_{2}$ , $p_{3}$ , $p_{4}$ and $p_{5}$ are sequentially traversed once for Fig. 3(a), only the FSC $g^{1}$ is constructed for the reference image. However, if $p_{5}'$ , $p_{4}'$ , $p_{3}'$ , $p_{2}'$ , $p_{1}'$ are sequentially traversed once for Fig. 3(b), only the FSC ${\bar{g}}^{2}$ is constructed for the sensed image. Therefore, no pair of corresponding FSCs is obtained from the image pair in Fig. 3.

Apart from these, the number of the FSCs to be constructed will be $2 (n - 4)$ , if there is an open contour with _$n$ corners (_{$n \geq 5$}). If the contour is closed, such number will be_{$2 n$}. Suppose that the total number of the corners in C is $N$ , i.e. $n_{1} + n_{2} + \dots + n_{N c} = N$ , then at most $2 (n_{1} + n_{2} + \dots + n_{N c}) = 2 N$ FSCs can be constructed by Algorithm 1. Obviously, compared with the aforementioned randomly choosing strategy with which $N! / (N - 5)!$ FSCs will be constructed, Algorithm 1 can effectively reduce the number of the FSCs to be constructed.

2.3 Design of descriptors for the FSC

As mentioned in previous section, two sets of the FSCs have been constructed after Step 2 of the proposed registration method. In order to obtain the corresponding elements from these two sets, we need to design a descriptor for each FSC.

Generally, a good feature descriptor should satisfy the following two requirements:

(1) it is robust against the differences such as projective geometric deformation and gray scale discrepancy between the reference and sensed images.

(2) the descriptors for a pair of non-corresponding features should be distinctly different from each other.

In this paper, the algebraic projective invariants [19] will be utilized to ensure that the proposed descriptor satisfies the above two requirements.

For a group of five co-plane points, $p_{i} = (x_{i}, y_{i})$ , with no three of which being co-linear, two functionally independent projective invariants can be defined as [19]:

{\begin{cases} I N V_{1} (p_{1}, p_{2}, p_{3}, p_{4}, p_{5}) = \frac{d e t (m_{431}) d e t (m_{521})}{d e t (m_{421}) d e t (m_{531})} \\ I N V_{2} (p_{1}, p_{2}, p_{3}, p_{4}, p_{5}) = \frac{d e t (m_{421}) d e t (m_{531})}{d e t (m_{432}) d e t (m_{521})} \end{cases}

where

d e t (m)

denotes the determinant of

m

and

m_{i j k} = (q_{i}, q_{j}, q_{k})

with

q_{i} = {(x_{i}, y_{i}, 1)}^{T}

.

If ${\bar{p}}_{i} = (x_{i}, y_{i})$ is the corresponding point of $p_{i} = (x_{i}, y_{i})$ after an arbitrary projective transform, the following equations hold [19],

I N V_{i} (p_{1}, p_{2}, p_{3}, p_{4}, p_{5}) = I N V_{i} ({\bar{p}}_{1}, {\bar{p}}_{2}, {\bar{p}}_{3}, {\bar{p}}_{4}, {\bar{p}}_{5}), i = 1, 2

Based on the introduced projective invariants, we adopt a ten-dimensional vector to describe each FSC. Take the FSC,

g = {p_{a}^{}, p_{b}^{}, p_{c}^{}, p_{d}^{}, p_{e}^{}}

, for example, its descriptor is given as:

d e s (g) = [\begin{array}{l} \begin{matrix} I N V_{1} (p_{a}^{}, p_{b}^{}, p_{c}^{}, p_{d}^{}, p_{e}^{}) \\ I N V_{2} (p_{a}^{}, p_{b}^{}, p_{c}^{}, p_{d}^{}, p_{e}^{}) \end{matrix} \\ \begin{matrix} I N V_{1} (p_{b}^{}, p_{c}^{}, p_{d}^{}, p_{e}^{}, p_{a}^{}) \\ I N V_{2} (p_{b}^{}, p_{c}^{}, p_{d}^{}, p_{e}^{}, p_{a}^{}) \end{matrix} \\ \begin{matrix} I N V_{1} (p_{c}^{}, p_{d}^{}, p_{e}^{}, p_{a}^{}, p_{b}^{}) \\ I N V_{2} (p_{c}^{}, p_{d}^{}, p_{e}^{}, p_{a}^{}, p_{b}^{}) \end{matrix} \\ \begin{matrix} I N V_{1} (p_{d}^{}, p_{e}^{}, p_{a}^{}, p_{b}^{}, p_{c}^{}) \\ I N V_{2} (p_{d}^{}, p_{e}^{}, p_{a}^{}, p_{b}^{}, p_{c}^{}) \end{matrix} \\ \begin{matrix} I N V_{1} (p_{e}^{}, p_{a}^{}, p_{b}^{}, p_{c}^{}, p_{d}^{}) \\ I N V_{2} (p_{e}^{}, p_{a}^{}, p_{b}^{}, p_{c}^{}, p_{d}^{}) \end{matrix} \end{array}]

Remark 3: In fact, increasing the dimensions of a descriptor will improve its ability to identify the pair of corresponding features in the step of feature matching. However, it will also increase the computational load. According to the experimental results on various image pairs, a ten-dimensional descriptor is selected as a good compromise.

The properties of the descriptor proposed above are summarized as follows:

(1) It is composed of ten projective invariants, which can maintain unchanged after arbitrary projective deformations according to Eq. (5). Thus the proposed descriptor is robust against projective deformations.

(2) Since no gray scale information is used in calculating the proposed descriptor, it is robust against gray scale discrepancy between the corresponding pixels of the image pairs.

(3) From the definition of the invariants in Eq. (4), the values of the invariants will be changed once the coordinates or the recording sequence of the five corners change. Therefore, different FSCs will have different descriptors, which enable us to identify the corresponding FSCs by comparing their descriptors.

2.4 Feature matching

The purpose of the feature matching step (i.e. Step 5 of Fig. 1) is to obtain a set of matched corners which can be used to estimate the parameters of the transformation model (i.e. Step 6 of Fig. 1). In this section, we first get the matched FSCs by comparing their descriptors, then obtain five matched corners from each pair of the matched FSCs. To this end, we define the following distance measure to indicate the similarity of two descriptors.

D_{d e s (g_{1}), d e s (g_{2})} = \sqrt{\sum_{i = 1}^{10} \frac{{(a_{i} - b_{i})}^{2}}{a_{i}^{2} + b_{i}^{2}}}

where $d e s (g_{1}) = {[\begin{matrix} a_{1} & a_{2} & \dots & a_{10} \end{matrix}]}^{T}$ and $d e s (g_{2}) = {[\begin{matrix} b_{1} & b_{2} & \dots & b_{10} \end{matrix}]}^{T}$ denote two arbitrary descriptors. In Eq. (7), each component represents the difference between _{$d e s (g_{1})$} and _{$d e s (g_{2})$} on a certain direction.

The detailed feature matching procedure is described in Algorithm 2:Algorithm 2: Feature MatchingInput: two sets of FSCs $G_{1} = {g^{1}, g^{2}, \dots, g^{N_{1}}}$ , $G_{2} = {{\bar{g}}^{1}, {\bar{g}}^{2}, \dots, {\bar{g}}^{N_{2}}}$ and their descriptors;Output: a set of matched corners _$F$;1. for $i = 1 : N_{1}$ do2. find $g^{i}$ ’s nearest and second nearest neighbors from $G_{2}$ , and record them as _{${\bar{g}}^{j}$}and _{${\bar{g}}^{k}$} respectively;3. calculate $d r = D_{d e s (g^{i}), d e s ({\bar{g}}^{j})} / D_{d e s (g^{i}), d e s ({\bar{g}}^{k})}$ ;4. if $d r < σ$ and _$g^{i}$ is also _{${\bar{g}}^{j}$}’s nearest neighbors from $G_{1}$ do5. record $(p_{1}^{i}, {\bar{p}}_{1}^{j})$ , $(p_{2}^{i}, {\bar{p}}_{2}^{j})$ , $(p_{3}^{i}, {\bar{p}}_{3}^{j})$ , $(p_{4}^{i}, {\bar{p}}_{4}^{j})$ and $(p_{5}^{i}, {\bar{p}}_{5}^{j})$ into $F$ ;6. end if7. end for

Remark 4: In Line 4 of Algorithm 2, $σ$ is a threshold belonging to the interval [0,1]. Sec. 3.2 will be devoted to discuss how to choose $σ$ properly.

3. Analysis of accuracy rate and repetition rate of the proposed algorithm

For a feature matching algorithm, the purpose is to extract the corresponding features exactly from two sets of features. However, some wrong matches will inevitably exist in its output which is a set of matched features. Besides, not all the corresponding pairs can be covered by the input sets. In this regard, the accuracy rate and repetition rate are two standard indexes introduced to indicate whether a feature matching result is good or not [24]. The accuracy rate is defined as the ratio of the number of corresponding pairs in the output to the number of the matched pairs in the output, i.e.

a c c u r a c y r a t e = \frac{# c o r r e s p o n d i n g p a i r s i n t h e o u t p u t}{# m a t c h e s i n t h e o u t p u t}

The repetition rate is defined as the ratio of the number of corresponding pairs in the output to the number of corresponding ones in the input, i.e.

r e p e t i t i o n r a t e = \frac{# c o r r e s p o n d i n g p a i r s i n o u t p u t}{# c o r r e s p o n d i n g p a i r s i n i n p u t}

In this section, the following simulation model is considered to generate the testing data

{\begin{cases} \bar{x} = \frac{a x + b y + c}{g x + h y + 1} + α R \cos (2 π β) \\ \bar{y} = = \frac{d x + e y + f}{g x + h y + 1} + α R \sin (2 π β) \end{cases}

where

H = [\begin{matrix} a & b & c \\ d & e & f \\ g & h & 1 \end{matrix}]

is a projective model,

(x, y)

and

(\bar{x,} \bar{y})

denote a chosen point from reference image and its corresponding point in the sensed image.

(α R \cos (2 π β), α R \sin (2 π β))

represents the influence of a positioning error which is the distance between the extracted corner and the real corner on the image in real applications. R is a constant,

α

and

β

are two random variables with uniform distributions

[0, 1]

In practical applications, the two input FSC sets may contain different percentages of the corresponding pairs due to various reasons. Thus we use three different percentage levels of the corresponding pairs, denoted as $λ$ , to generate the testing data in the simulation (i.e. $λ$ is chosen as 0.3, 0.5 and 0.7, respectively). For each $λ$ , the testing data involve two sets of 1000 FSCs. We employ Algorithm 2 to deal with the generated testing data by setting different values of the threshold $σ$ . Note that for either type of rate, a higher value indicates a better performance of Algorithm 2. We expect that there exist a threshold $σ$ which can make both of the two rates reach their maximum values simultaneously.However, from the results on the accuracy rate and repetition rate of each output of Algorithm 2 with respect to different $σ$ as given in Fig. 4, it can be seen that the two type of rates present inconsistent varying trends with $σ$ increases. To be concrete, a larger $σ$ leads to a higher repetition rate but a lower accuracy rate. This implies that specifying the threshold _$σ$ is a compromise between these two rates. As the repetition rate changes slowly when $σ > 0.5$ , we suggest the value of $σ$ better be chosen from [0.3,0.5] and we set $σ = 0.4$ in this paper.

Fig. 4 Accuracy rates and repetition rates of Algorithm 2 with different threshold $σ$ .

Download Full Size | PDF

4. Experiments

In this section, an experiment is firstly conducted to validate the robustness of the proposed registration method against projective deformation and gray scale discrepancy. Then comparative results of the proposed method and two existing representative feature-based image registration methods, i.e. SIFT and SURF algorithms [17,18], are provided.

4.1 Robustness Validation

The test images in Fig. 5 are adopted as the experimental database to show the robustness of the proposed method. Fig. 5(a) and 5(b) are a pair of well registered thermal infrared and visual images. Fig. 5(c) and 5(d) are the transformed images of Fig. 5(a) and 5(b), respectively, based on different projective models, which are known. As indicated in Fig. 5, three test sets are constructed from Fig. 5(a)-(d):• Test Set 1: Fig. 5(a) and 5(c). Only projective deformation exists in the image pair of this set;• Test Set 2: Fig. 5(a) and 5(b). Only gray scale discrepancy exists, in the image pair of this set;• Test Set 3: Fig. 5(a) and 5(d). Both projective deformation and gray scale discrepancy exists in the image pair of this test set.

Fig. 5 Test images for robustness validation experiment.

Download Full Size | PDF

The experiment results are provided in Table 1. It is worth mentioning that since Fig. 5(a), 5(b) are well registered, and the projective models corresponding to Fig. 5(c), 5(d) are both known, the number of corresponding (corner) pairs in both the input and output of the algorithm for each test set can be counted. Thus the accuracy rates and the repetition rates as defined in Eqs. (8) and (9) of the registration results for all three test sets can be explicitly calculated. Besides, the corresponding pairs in the matched results for each test set are shown in Fig. 6.According to Table 1, the following conclusions can be drawn.

Table 1. Feature Matching Results of Test Sets 1-3 by the Proposed Algorithm

View Table | View all tables in this article

Fig. 6 The corresponding pairs in the matched results by the proposed method for each test set.

Download Full Size | PDF

i) It is known that four pairs of corners can determine a projective model with eight parameters involved. It can be seen that nearly 10 corresponding pairs of corners are obtained after feature matching for the three test sets. Thus the projective transformation models corresponding to all the test sets can be estimated effectively;
ii) The accuracy rates and repetition rates for all the three test tests are over 0.4 and 0.5, respectively. Although different degrees of projective deformation and gray scale discrepancy exist in the image pairs, similar registration results in terms of the accuracy rates and repetition rates can be observed. Based on this, satisfying robustness of the proposed registration method against projective deformation and gray scale discrepancy is demonstrated.

4.2 Performance Comparison with Existing Representative Algorithms

Three pairs of multi-sensor images as shown in Figs. 7, 8, and 9 are used as the test data to compare the performance of the proposed method with the well-known SIFT and SURF algorithms [17,18]. Note that the source codes of SIFT and SURF algorithms used in conducting the comparative experiment can be found on “www.cs.ubc.ca/~lowe/keypoints/” and “www.chrisevansdev.com/computer-visionopensurf.html”, respectively. In contrast to the robustness validation experiment presented above, the actual projective model corresponding to the image pairs in Fig. 7-9 are not known. Thus 10 pairs of the corresponding points are manually selected from each image pair and the Root-Mean-Squared Error (RMSE) is utilized to evaluate the accuracies of the registration results.Observing Fig. 7-9, both projective deformation and gray scale discrepancy are involved in the three test sets. In Test Set 4, the image pair is visual and near infrared images, respectively. The gray scale discrepancy for this set is much weaker than those for Test Sets 5 and 6. The registration performances of SIFT, SURF and our proposed algorithm for Test Sets 4-6 are given in Table 2, including the number of obtained matches in the outputs and RMSE of the registration results.

Fig. 7 Test Set 4. (a) Reference image. (b) Sensed image.

Download Full Size | PDF

Fig. 8 Test Set 5. (a) Reference image. (b) Sensed image.

Download Full Size | PDF

Fig. 9 Test Set 6. (a) Reference image. (b) Sensed image.

Download Full Size | PDF

Table 2. Performance Comparison of SIFT, SURF and the Proposed Algorithms

View Table | View all tables in this article

It can be seen that SIFT and SURF can obtain enough matches to estimate the projective model in Test Set 4, however, the RMSE of the registration results by using these two algorithms are too large to be acceptable. This is because that the matches obtained by SIFT and SURF contain not enough corresponding corners (correct matches). For Test Sets 5 and 6, both SIFT and SURF deliver none matches in the outputs. Therefore, these two algorithms failed to provide valid registration results due to the effects of much serious gray scale discrepancy between the image pairs. In contrast to these, our proposed method can provide good registration results for all the three test sets with RMSE less than one. The aligned images by the proposed method are provided in Fig. 10.

Fig. 10 Aligned images by the proposed method. (a) Test set 1. (b) Test set 2. (c) Test set 3.

Download Full Size | PDF

5. Conclusion

In this paper, a new feature-based multi-sensor image registration algorithm is proposed based on contour feature extraction and algebraic projective invariants. Since the designed feature descriptor depends on the coordinate information of the corners only and has no relation with the gray scale information, the proposed method has good robustness against the gray scale discrepancy between the corresponding pixels. Moreover, the algebraic projective invariants guarantee that the feature descriptor is also robust against the projective deformation. The experiment results are given to show the effectiveness of the proposed scheme

Acknowledgment

This work was supported in part by the 973 Program of China under Grant 2010CB731800 and in part by the National Natural Science Foundation of China under Grants 61290324, 60974059, 60736026, 61021063 and 61203068.

References and links

1. B. Zitova and J. Flusser, “Image registration methods: a survey,” Image Vis. Comput. 21(11), 977–1000 (2003).

2. J. J. Arthur, L. J. Kramer, and R. E. Bailey, “Flight test comparison between enhanced vision (FLIR) and synthetic vision systems,” (SPIE-INT SOC Optical Engineering, BELLINGHAM, 2005), pp. 25–36.

3. R. J. Radke, S. Andra, O. Al-Kofahi, and B. Roysam, “Image change detection algorithms: a systematic survey,” IEEE Trans. Image Process. 14(3), 294–307 (2005).

4. J. P. Pluim, J. B. Maintz, and M. A. Viergever, “Mutual-information-based registration of medical images: a survey,” IEEE Trans. Med. Imaging 22(8), 986–1004 (2003).

5. H. Luan, F. Qi, Z. Xue, L. Chen, and D. Shen, “Multimodality image registration by maximization of quantitative-qualitative measure of mutual information,” Pattern Recognit. 41(1), 285–298 (2008).

6. F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens, “Multimodality image registration by maximization of mutual information,” IEEE Trans. Med. Imaging 16(2), 187–198 (1997).

7. A. A. Cole-Rhodes, K. L. Johnson, J. LeMoigne, and I. Zavorin, “Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient,” IEEE Trans. Image Process. 12(12), 1495–1511 (2003).

8. J. P. Heather and M. I. Smith, “Multimodal image registration with applications to image fusion,” 2005 7th International Conference on Information Fusion (FUSION), Vols 1 and 2, 372–379 (2005).

9. M. Irani and P. Anandan, “Robust multi-sensor image alignment,” Sixth International Conference on Computer Vision, 959–966 (1998).

10. Z. H. Zhang, C. H. Pan, and S. D. Ma, “An automatic method of coarse registration between multi-source satellite images,” Proceedings of the 2004 Intelligent Sensors, Sensor Networks & Information Processing Conference, 205–209 (2004).

11. M. A. Ali and D. A. Clausi, “Automatic registration of SAR and visible band remote sensing images,” 2002 IEEE International Geoscience and Remote Sensing Symposium. 24th Canadian Symposium on Remote Sensing. Proceedings (Cat. No.02CH37380), 1331–1333 (2002).

12. H. Li, B. S. Manjunath, and S. K. Mitra, “A contour-based approach to multisensor image registration,” IEEE Trans. Image Process. 4(3), 320–334 (1995).

13. S. Dawn, V. Saxena, and B. Sharma, “Remote Sensing Image Registration Techniques: A Survey,” Image and Signal Processing. Proceedings 4th International Conference, ICISP 2010, 103–112 (2010).

14. Z. L. Song and J. P. Zhang, “Remote Sensing Image Registration Based on Retrofitted SURF Algorithm and Trajectories Generated From Lissajous Figures,” IEEE Geosci. Remote Sens. Lett. 7(3), 491–495 (2010).

15. Z. L. Song, S. Li, and T. F. George, “Remote sensing image registration approach based on a retrofitted SIFT algorithm and Lissajous-curve trajectories,” Opt. Express 18(2), 513–522 (2010).

16. N. Alajlan, I. El Rube, M. S. Kamel, and G. Freeman, “Shape retrieval using triangle-area representation and dynamic space warping,” Pattern Recognit. 40(7), 1911–1920 (2007).

17. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).

18. H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-Up Robust Features (SURF),” Comput. Vis. Image Underst. 110(3), 346–359 (2008).

19. J. L. Mundy and A. Zisserman, Geometric Invariance in Computer Vision (the MIT Press, 1992).

20. J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986).

21. F. Mokhtarian and R. Suomela, “Robust image corner detection through curvature scale space,” IEEE Trans. Pattern Anal 20(12), 1376–1381 (1998).

22. X. C. He and N. H. C. Yung, “Corner detector based on global and local curvature properties,” Opt. Eng. 47, 057008 (2008).

23. M. A. Fischler and R. C. Bolles, “Random Sample Consensus - A Paradigm For Model-Fitting With Applications To Image-Analysis And Automated Cartography,” Commun. ACM 24, 381–395 (1981).

24. K. Mikolajczyk and C. Schmid, “Performance evaluation of local descriptors,” IEEE Trans. Pattern Anal. Mach. Intell. 27(10), 1615–1630 (2005).

Test Set	1		2		3
Image Pairs	Fig. 5(a)	Fig. 5(c)	Fig. 5(a)	Fig. 5(b)	Fig. 5(a)	Fig. 5(d)
Image Types	Thermal infrared	Thermal infrared	Thermal infrared	Visual	Thermal infrared	Visual
Image Sizes (pixels×pixels)	200×160	236×161	200×160	200×160	200×160	270×225
# corresponding pairs in the output	11		10		11
# obtained matches in the output	20		25		26
# corresponding pairs in the input	19		19		19
Accuracy Rates	0.550		0.400		0.423
Repetition Rates	0.579		0.526		0.579

Test Set		4		5		6
Image Pairs		Fig. 7(a)	Fig. 7(b)	Fig. 8(a)	Fig. 8(b)	Fig. 9(a)	Fig. 9(b)
Image Types		Visual	Near infrared	Thermal infrared	Visual	Visual	Thermal infrared
Image Sizes (pixels×pixels)		648×536	605×462	552×588	390×544	354×318	319×241
# obtained matches in the output	SIFT	11		0		0
	SURF	8		0		0
	Proposed method	60		43		72
RMSE (pixels)	SIFT	43.996		–¹		–¹
	SURF	61.867		–¹		–¹
	Proposed method	0.357		0.898		0.522

Test Set	1		2		3
Image Pairs	Fig. 5(a)	Fig. 5(c)	Fig. 5(a)	Fig. 5(b)	Fig. 5(a)	Fig. 5(d)
Image Types	Thermal infrared	Thermal infrared	Thermal infrared	Visual	Thermal infrared	Visual
Image Sizes (pixels×pixels)	200×160	236×161	200×160	200×160	200×160	270×225
# corresponding pairs in the output	11		10		11
# obtained matches in the output	20		25		26
# corresponding pairs in the input	19		19		19
Accuracy Rates	0.550		0.400		0.423
Repetition Rates	0.579		0.526		0.579

Test Set		4		5		6
Image Pairs		Fig. 7(a)	Fig. 7(b)	Fig. 8(a)	Fig. 8(b)	Fig. 9(a)	Fig. 9(b)
Image Types		Visual	Near infrared	Thermal infrared	Visual	Visual	Thermal infrared
Image Sizes (pixels×pixels)		648×536	605×462	552×588	390×544	354×318	319×241
# obtained matches in the output	SIFT	11		0		0
	SURF	8		0		0
	Proposed method	60		43		72
RMSE (pixels)	SIFT	43.996		–¹		–¹
	SURF	61.867		–¹		–¹
	Proposed method	0.357		0.898		0.522

Multi-sensor image registration based on algebraic projective invariants

Abstract

1. Introduction

2. The proposed image registration algorithm

2.1 Overview

2.2 Construction of the FSC

2.3 Design of descriptors for the FSC

2.4 Feature matching

3. Analysis of accuracy rate and repetition rate of the proposed algorithm

4. Experiments

4.1 Robustness Validation

4.2 Performance Comparison with Existing Representative Algorithms

5. Conclusion

Acknowledgment

References and links

Cited By

Figures (10)

Tables (2)

Equations (11)

Optics Express