EN中文

地球信息科学学报 >

2017 , Vol. 19 >Issue 4: 528 - 539

DOI: https://doi.org/10.3724/SP.J.1047.2017.00528

遥感科学与应用技术

激光点云与光学影像配准：现状与趋势

  • 张靖 ,
  • 江万寿 , *
展开
  • 武汉大学 测绘遥感信息工程国家重点实验室,武汉 430079
*通讯作者：江万寿（1967-）,男,博士,研究员,研究方向为数字摄影测量。E-mail:

作者简介：张 靖（1982-）,男,博士,讲师,研究方向为点云数据处理。E-mail:

收稿日期: 2016-05-30

  要求修回日期: 2016-12-20

  网络出版日期: 2017-04-20

基金资助

国家自然科学基金青年科学基金项目(41301367)

高等学校博士点专项科研基金(20130141120066)

地理空间信息工程国家测绘地理信息局重点实验室开放研究基金项目(201309)

收起

Registration between Laser Scanning Point Cloud and Optical Images: Status and Trends

  • ZHANG Jing ,
  • JIANG Wanshou , *
Expand
  • State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China
*Corresponding author: JIANG Wanshou, E-mail:

Received date: 2016-05-30

  Request revised date: 2016-12-20

  Online published: 2017-04-20

Copyright

《地球信息科学学报》编辑部 所有
Fold

摘要

激光点云与光学影像是2种重要的遥感数据源,二者的融合能够实现优势互补,具有应用价值。点云与影像的配准是实现二者集成应用的基础,虽然经历了多年的发展仍存在许多问题有待解决。本文首先通过建立点云与影像配准问题的数学范式,将整个配准问题划分为观测值提取、配准模型选择和参数优化3部分,深入分析各部分所面临的难点与挑战;然后对现有的点云与影像配准方法进行回顾与总结,对比分析各类方法的优缺点及适用范围;最后展望了今后的发展方向进行了展望,为后续的研究提供参考。

关键词： 光学影像; 激光探测与测距; 点云; 配准; 综述

本文引用格式

张靖 , 江万寿 . 激光点云与光学影像配准：现状与趋势[J]. 地球信息科学学报, 2017 , 19(4) : 528 -539 . DOI: 10.3724/SP.J.1047.2017.00528

Abstract

LiDAR point cloud and optical imagery are different types of remote sensing data source. They have some unique merits, respectively, that are complementary to each other. Integrating these two dataset has significant value in many applications. However, as the existence of various error sources, point cloud and optical imagery are usually misaligned. For the purpose of further integrated processing, the registeration of point cloud and imagery is a preliminary step which will align them into a unified geo-reference frame. Although after decades of research, this registration problem is far from solved. This paper gave a detailed survey of registration between point cloud and optical images. To obtain thorough understanding of this problem, a general mathematical paradigm for the registration was established firstly. By analyzing the mathematical paradigm, we indicated three main difficulties in this registration problem, and then definitely divided the whole workflow of registration into three key parts which are named: （1） the acquisition of corresponding observations, （2） the selection of transformation models; （3） the optimization of unknowns. Afterwards, we reviewed a series of representative registration methods from the above three aspects. In the acquisition of corresponding observations, the existing methods were classified into area-based method, feature-based method and multiple-view geometry based method. In the stage of transformation models selection, frequently-used models were classified into sensor-based models and empirical models. In the unknowns’ optimization part, two principal optimization methods termed local optimization and global optimization were introduced and the general usages of these optimization in registration were described. Furthermore, we summarized the mentioned registration methods and gave a detailed comparison and analysis including the advantages / shortcomings and the applicable scope. At last, the trends of registration development were forecasted.

Key words： optical image; LiDAR; point cloud; registration; literature review

1 引言

激光探测与测距LiDAR（Light Detection And Ranging）技术在过去20年获得了巨大发展,作为一种主动遥感技术,LiDAR通过发射激光脉冲并接收目标反射信号来直接确定目标的空间位置,具有数据采集速度快、几何定位精度高等优点。而传统的光学影像能获得丰富的地物光谱信息和纹理细节,将2种数据结合可以充分发挥各自的优势,并广泛应用于数字城市、灾害评估、精准农业和林业等领域,取得了巨大的社会经济效益。
但是激光点云和光学影像的几何参考框架不同,二者之间往往不能直接精确对准。为了实现二者的有效融合与应用,必须首先解决二者之间的几何配准问题。与传统的图像配准问题相比,激光点云与光学影像是二种跨模态（cross-modal）异源数据,二者之间的巨大差异给配准问题带来了很大困难。学者们对点云与影像配准问题开展了多年研究,并提出了一系列的算法,但在配准精度、鲁棒性以及自动化水平等方面还存在局限性,有必要对现有的研究进展进行梳理。
目前关于激光点云与影像配准的综述性文章[1-3]多关注于某个具体领域的应用问题,缺乏对点云与影像配准问题的整体而全面的分析和总结。本文通过对各类点云与影像配准问题进行分析,建立了二者配准的通用数学范式,并从观测值获取、配准模型选择和参数优化3个方面对现有的文献进行分析和梳理,对比分析各类方法的优缺点及适用范围,并对其发展方向进行预测,以便为后续的研究提供参考。

2 点云影像配准关键问题

从数据获取平台来看,目前的点云与影像数据主要有机载、车载和地面固定平台3种。不同平台获取的数据特点不同,其中机载数据的分辨率最低,但覆盖范围最大;而地面固定平台的数据分辨率最高,但覆盖范围最小。由于数据特点的差异,不同平台获取的数据的主要应用也不同。例如,机载数据主要用于地形制图和大范围城市场景建模,侧重建筑物顶面的几何与纹理重建;车载数据主要用于城市街景建模,侧重于建筑物立面的几何与纹理重建;而地面固定平台数据主要用于某类特定目标的精细建模,如文化遗产、古迹遗址、雕塑等。由于不同平台的点云与影像数据差异较大,本文仅讨论同类平台的数据配准问题。
对于机载、车载或地面固定平台获取的点云与影像数据,配准的目标都是实现二者的空间对准,因此尝试建立一般化的配准范式。假设对于某个场景S,由相机采集的光学影像记为I（S）,由LiDAR采集的点云记为D（S）,二者之间具有足够的重叠。由于影像是二维数据而点云是三维数据,设影像的坐标参考框架为 Ω I ,（ Ω I ⊂ R 2 ）,点云的坐标参考框架为 Ω D ,（ Ω D ⊂ R 3 ）,设A,B分别为影像和点云上的一组同名对应集合,其中 A ⊂ Ω I , B ⊂ Ω D 。配准过程可以理解为寻找一种合适的转换函数T（X）（X为转换函数的参数向量）,将二者转换到同一个参考框架下,并实现同名对应集合之间的精确几何对准。用数学形式表达如下：
T ( X ) : T ( X ) ⋅ B ⇔ A （1）
假设转换函数T的形式确定后,配准问题实际上是对函数的参数向量X的最优估计问题,即式（2）,其中 ϕ 为优化的目标函数。
X ˜ = argmin ϕ ( T ( X ) ⋅ B - A ) （2）
式（2）为点云与影像配准的数学范式,从中可见整个配准过程涉及3个方面的问题：确定2种数据之间的同名观测值A和B;选择合适的转换函数T（X）;对模型参数X的优化求解。下面将从这3个方面入手,对近年来提出的点云与影像配准算法进行分析和评述。

3 同名观测值获取

与传统的光学影像配准不同,点云和影像之间的巨大差异给同名观测值的确定造成了很大困难。二者之间的数据差异可概括为3个方面：① 二者的物理特性不同,点云反映的是目标对激光波束的后向散射特性以及场景的三维几何特征;而影像则记录目标对太阳光的反射,反映的是目标的物理和材质属性。② 二者的几何模型不同,点云是三维数据,采用的是基于测角和测距的直接定位模型;而影像是二维数据,采用的是基于共线方程的小孔成像模型。③ 二者的采样方式不同,点云是典型的离散采样,其数据分布受激光发射频率和系统扫描频率制约;而影像一般是面阵或线阵成像,是一种连续采样。
为了克服数据之间的差异,自动提取稳定可靠的同名观测值,学者们提出了许多方法。根据观测值的类型,本文将配准算法划分为3类：基于区域 的方法、基于特征的方法和基于多视几何的配准 方法。

3.1 基于区域的方法

基于区域的方法可以看成是一种传统的模板匹配方法,直接对给定模板窗口内的图像灰度信息进行处理,不需要提取特征,配准时计算影像对应窗口内的像元相似性,并将窗口中心点作为同名对应点。在处理影像与点云配准时,需要预先将点云转换为强度影像、DSM或距离图像（Range Image）。基于区域的配准方法可进一步分为空间域方法、频率域方法和统计方法。
3.1.1 空间域方法
空间域的配准方法通常也被称为基于影像灰度的配准方法（Intensity-based Method）[1,4],此类方法在空间域上直接比较影像窗口内灰度分布的相似程度。算法执行时,首先在参考影像上选择一个窗口,统计窗口内的灰度信息,然后在待配准影像上搜索,找到与参考影像窗口最相似的区域,将2个区域的中心点作为同名对应点,这类算法的关键问题在于相似性度量函数的设计。传统影像配准中常用的相似性度量函数有差绝对值SAD（Sum of Absolute Differences）、差平方和SSD（Sum of Squared Differences）、归一化相关系数NCC（Normalized Cross Correlation）、等级相关RC（Rank Correlation）、相关比CR（Correlation Ration）等[5]。由于光学影像和由点云产生的强度影像或DSM在灰度映射模式不同,二者在灰度值上存在很大相同,因此直接使用上述相似性测度难以获得稳定可靠的结果,Umeda等[6]和Duraisamy等[7]从光学影像和DSM上提取梯度信息后进行配准,而在文献[8]中通过结合局部自相似性特征（LSS）和NCC设计了一种形状相似性测度LSCC来克服影像之间非线性的灰度差异。基于区域的方法不需要预先对影像上的特征进行提取和定位,而是侧重于比较影像对应区域内所有像素点之间的相关性。
3.1.2 频率域方法
频率域配准方法是基于傅里叶变换原理,利用影像的相位信息进行配准的一类方法。最早是由Kuglin等[9]提出的,其处理过程是通过对图像进行快速傅里叶变换,将图像变换到频率域,再利用它们互功率谱中的相位信息进行配准。频率域配准算法最初只能解决仅存在平移的图像配准问题,Reddy等[10]对该方法进行了扩展,根据相位相关准则,可以得到图像间的平移、旋转和缩放等几何变换因子。由于相位相关方法对影像之间的全局灰度差异具有一定的鲁棒性,且对噪声不敏感,被一些学者用来解决光学影像和激光点云的配准问题：如Shorter等[11]分别从光学影像和DSM中检测建筑物区域构造掩膜图像,然后利用相位相关进行配准。Wong等[12]先对光学影像进行局部正规化并提取Harris角点,然后在点云生成的距离影像上进行同名点匹配,考虑到光学影像和点云距离影像之间的灰度差异,采用局部线性变换的方法对二者之间的灰度差异建模,并利用快速傅里叶变换的方法对图像之间的相关搜索进行加速。近年来,基于相位相关的高精度配准方法得到进一步发展,其配准精度已能达到了子像元级别[13],一些学者开始利用这类方法解决高精度的配准问题[14]。
3.1.3 统计方法
由于光学影像与激光点云是由不同类型的传感器获得的数据,影像灰度值与点云强度值、高程值之间存在较大的差异,但对于同一个场景目标,其影像灰度和点云强度或高程存在一定的统计相似性,因此可以通过估计2幅图像像元灰度之间的联合概率分布来判断图像之间的相关性。互信息MI（Mutual Information）是信息论中的一种信息度量,最初被用来衡量2个随机变量之间的统计相关性,在医学影像[15]和多源遥感影像[16-18]的配准问题上已经取得了巨大的成功。近年来,互信息也被用于光学影像与激光点云的自动配准。邓非等[19]直接利用互信息作为相似性测度来衡量光学影像和点云强度影像之间灰度分布的相似性,并采用梯度下降法进行迭代优化。然而仅利用灰度互信息的配准方法忽视了图像上灰度特征本身所具有的空间位置信息,王蕾[20]用两张影像的梯度方向夹角对灰度互信息进行加权,改善了配准精度。闫利等[21]利用互信息对车载激光点云和全景影像进行配准。Mastin等[22]为了提高配准效率,采用最小化联合熵（Joint Entropy）对机载点云与影像进行配准。Wang等[23]指出车载LiDAR数据的联合熵易受小扰动的影像,而互信息更适合车载数据的配准。Parmehr等[24]则同时用激光点云的强度信息和高程信息与光学影像配准,采用归一化联合互信息NCMI（Normalized Combined Mutual Information）来度量三者之间的统计相关性。

3.2 基于特征的方法

基于特征的方法是从原始影像和点云数据中提取显著的、可区分性的特征（如角点、边缘线、同质性区域等）,并利用这些对灰度和几何变形不敏感的特征进行配准。根据特征类型的不同,可分为点特征、线特征、面特征和多特征融合4类。
3.2.1 点特征
点特征是影像配准中最常用到的特征,在早期的遥感影像的配准中,常采用手工选取明显的地物点（如房屋角点、道路交叉点、水域的重心点等）作为控制点,这些都属于点特征。广义的点特征包括角点（corner）和斑点（blob）2类。角点是影像上边缘的交叉点,即在角点所处的局部区域内,影像梯度具有2个或2个以上的主方向;斑点是与周围区域存在灰度差异的图斑的中心点,是区域检测的一个特例[25]。影像配准中常用的角点特征有Moravec、Förstner、Harris等,斑点特征有Hession、LoG、DoG、MSER等[26]。角点和斑点代表了不同类型局部特征,在影像与点云配准问题中,研究人员会根据数据情况选择合适类型的点特征。为解决城市区域光学影像和激光点云的配准,Ding等[27]利用建筑物上丰富的水平和垂直线,从影像上提取水平线和垂直线交点,在点云DSM上提取屋顶边缘的角点,将这2个点集进行特征匹配;而张永军等[28]直接从激光点云中提取建筑物边缘,将正交的边缘线的交点作为特征点投影到影像上寻找对应点。Wong等[12]则从卫星影像上提取Harris角点,在DSM中寻找匹配点。Li等[29]为了解决沙漠地区影像与点云配准问题,同时在光学影像和激光点云中检测出沙漠草丛,并将图斑的中心点作为特征点进行配准。在地面摄影测量中,González-Aguilera等[30]从地面扫描的距离影像和光学影像中提取Förstner特征点进行配准;Altuntas等[31]用SIFT特征对室内激光扫描数据与影像进行配准,并通过实验对比分析了强度信息和高程信息对配准精度的影响。
基于特征点的配准中,特征描述是另一个关键问题,一些学者直接利用SIFT（Scale Invariant Feature Transform）[32]对光学影像与点云强度影像上的特征点进行描述,这种方法对地面激光扫描数据处理效果较好,但对机载数据却不理想[33]。Ju等[34]认为主要原因是地面激光点云与影像之间分辨率较为接近,而机载点云与影像间存在较大的尺度差异。此外点云强度值、高程值与影像灰度值之间的非线性差异,也会造成SIFT描述子失效。针对这些问题,Ding等[27]从几何关系着手,利用特征点所属的两条边缘线的夹角,设计了一种角度特征描述子;Bodensteiner等[35]和叶沅鑫等[8]利用邻域内像素灰度值之间的关系,设计了一种自相似性特征描述;Palenichka等[36]综合考虑了特征点的平面位置属性、邻域形状特征以及强度特征,设计了一种Salient Image Disk（SID）描述,取得了较为理想的效果。总的来看,点特征算法发展较为成熟,且在纹理丰富的区域效果较好。但由于点特征可区分性不高,在弱纹理和重复纹理较多的场景中,这一方法往往会失败,需要进行特殊的设计和改进[25]。
3.2.2 线特征
线特征是对场景中线状几何结构的反映,对影像的辐射差异、成像角度变化更为鲁棒。Habib等[37]和马洪超等[38]将线特征作为配准基元,建立了二维影像和三维点云之间的几何平差模型。Yao等[39]从点云中提取屋脊线和屋檐线,与光学影像上的灰度边缘进行配准。徐景中等[40]对提取的直线按照距离、角度等几何关系进行筛选和聚类,进一步构造出结构特征,并利用这些线特征进行匹配。Wang等[41]为了减少对线特征的漏检,设计了一种新的检测策略,然后基于检测到的特征线构造了具有更好区分性和重复率的结构特征3CS（3 Connected Segments）用于配准。基于线特征的配准当前的研究主要聚焦在如何合理的提取特征线,如何构造具有更好区分性的结构特征上。在特征描述方面与点特征略有不同,主要是基于角度、距离等几何信息进行描述,特征匹配时利用RANSAC[41]或Hough[27]变换等优化算法,根据给定的几何约束条件搜索可能的匹配结果。
3.2.3 面特征
面特征在影像与点云配准中用得较少,与点、线特征相比,面特征的优点和缺点都比较明显。在激光点云特别是城市场景的点云中,存在大量的几何平面,从点云中提取平面特征较为容易,且基于共面点集拟合的平面参数也更可靠。但是在成像过程中,空间几何信息发生退化,在光学影像上提取平面区域比较困难,且如何确定三维面片与二维影像平面的精确对应也是一个难题。Kwak等[42]采用局部极大值滤波器从点云中提取建筑物屋顶平面,用canny算子从光学影像中检测建筑物屋顶边界,最后将二者的中心进行配准。Habib等[43]和Armenakis等[44]通过摄影测量方法,匹配出同名点的三维空间信息,然后与激光点云中检测的平面进行配准。这一配准思路实际上是将影像与点云这种异源数据的配准转化成了三维点集之间的配准,属于三维空间配准的范畴。Roux等[45]采用图像分割方法从光学影像上检测同质区域,并假设这些区域在几何空间中是平面,然后将激光点云投影到影像上,用RANSAC方法拟合平面,最后用启发式的搜索算法Nelder-Mead对配准参数进行优化。
3.2.4 多特征融合
多特征融合并不是一类独立的方法,而是一种利用多种类型特征的综合框架。由于基于特征的配准需要有一定数量且分布均匀的特征,而在一些情况下采用单一类型的特征很难满足要求,因此一些学者提出了融合多种类型特征的配准思路。Habib等[46]推导了基于点、线、面特征进行影像与点云配准时的几何平差模型。张良等[47]同时利用SIFT点特征和边缘线特征进行配准,并根据局部空间中点-线相似不变性,用匹配点的相似变换参数来辅助线匹配,最后对匹配点和线特征进行联合平差求解配准参数。Zhang等[48]则综合利用Harris角点和BCF（Building Corner Feature）特征,首先在影像上提取Harris角点与点云生成的强度影像匹配,然后在原始点云和光学影像上检测BCF特征并进行配准。从上述公开发表的成果中看,多特征融合的方法,可以显著提高匹配特征的数量。

3.3 基于多视几何的配准方法

为了克服三维激光点云和二维光学影像之间特征差异大这一问题,一部分学者考虑对配准的数据源进行转换,利用多视几何原理从影像序列中恢复出三维信息,从而将三维激光点和二维影像的配准转化为2个三维点集的空间配准问题。Leberl等[49]对比分析了影像匹配点云和激光扫描点云,指出二者之间具有很多相似之处,相比于影像与激光点云的直接配准,2个三维点集之间的配准更容易实现。Postolov等[50]从影像中手工选择建筑物屋顶区域并匹配出屋顶的三维坐标,然后从激光点云中选择对应的屋顶点,采用共面约束进行迭代最小二乘配准。Stamos等[51]利用SIFT特征对影像序列进行匹配,然后用SFM（Structure From Motion）恢复稀疏的三维点云,最后将影像点云和激光点云配准。Zheng等[52]采用类似的思路首先对序列影像进行光束法平差,然后将平差得到的三维连接点坐标与激光点云按照ICP（Iterative Closest Point）方法进行匹配,最后用匹配出的同名激光点坐标来优化光学影像的内外方位元素。为了提高点集匹配时的稳健性,Li等[53]用PCA方法从扫描点云中检测平面,用匹配点到对应平面的距离代替ICP中最近点距离进行迭代平差;陈驰等[54]和Yang等[55]则采用相对运动阈值对ICP搜索对应点的过程进行约束。

4 几何转换模型

点云与影像之间几何参考框架的差异以及数据采集过程中各种误差因素的影响是造成二者之间无法对准的根本原因。因此在进行配准时,需要选择合适的几何转换模型对点云与影像之间的几何变形建模。由于几何转换模型和原始数据的采集方式和数据特点密切相关,在机载、车载和地面固定扫描等不同的应用领域,常常会选择不同的几何转换模型。本文对常用的模型进行总结,并分为传感器严格模型和经验模型2大类。

4.1 传感器严格模型

传感器严格模型是根据激光点云和影像的物理定位原理[56-57],直接对各类误差源进行参数化建模,能够有效描述数据采集过程中GPS、IMU、镜头畸变、激光轴偏心、扫描测距和时间同步等各种系统误差因素对影像、点云定位精度的影响,且模型参数具有严格的物理含义。但这类模型一般较复杂,参数多且需要原始传感器参数的支持。
4.1.1 共线方程
共线方程是表达影像成像过程的一种经典数学模型,该模型基于小孔成像原理,描述了摄影瞬间物点、摄影中心和对应像点必须共线。其基本形式如式（3）所示,其中（X, Y, Z）是物点坐标,（x, y）是对应像点坐标,（x0, y0, f）为相机的主点坐标和主距,合称为内方位元素,（Xs, Ys, Zs）为像片曝光时刻的位置,R为像片姿态角 （ φ , ω , κ ） 构成的旋转矩阵, λ 为归一化比例尺因子。
x - x 0 y - y 0 - f = 1 λ R T X - X S Y - Y S Z - Z S （3）
利用共线方程进行配准时,一般以激光点云作为参考,在共线条件约束下,对影像的内外方位元素进行纠正。例如,Gneeniss等用激光点云作为控制对光学影像配准并对相机的内方位元素进行检校,取得了良好的效果[58]。由于共线方程中的旋转矩阵是角元素的非线性函数,陈为民等[59]利用罗德里格矩阵代替旋转矩阵,实现点云与影像的配准。此外,由于3个欧拉角 （ φ , ω , κ ） 无法表示欧式空间中的连续转动,一些学者采用四元数来描述旋转关系,提高了参数求解的稳定性[60-61]。
4.1.2 直接线性变换
直接线性变换（Direct linear transformation,DLT）是通过对共线方程的简化,直接建立像点坐标和物点坐标的线性对应关系,其基本形式如 式（4）所示,其中（l1, …, l11）为直接线性变换参数。
x + l 1 X + l 2 Y + l 3 Z + l 4 l 9 X + l 10 Y + l 11 Z + 1 = 0 y + l 5 X + l 6 Y + l 7 Z + l 8 l 9 X + l 10 Y + l 11 Z + 1 = 0 （4）
DLT不需要像片的内外方位元素,在缺少传感器信息时也能进行严格配准,此外DLT是线性模型,不需要初值即可直接求解。但是DLT也存在一些明显的局限,DLT中的未知数为11个,由于未知数包含了像片的内外方位元素信息,未知数之间并不独立,在特定配置条件下会造成模型退化[62]。宋恒嘉采用DLT实现了单张像片与车载点云的配准[63]。
4.1.3 鱼眼相机模型
由于普通相机的视场角较小,在城市街景测绘中,为了获得更大的视场,常常采用鱼眼相机。与传统镜头相比,鱼眼镜头采用非线性光学结构设计,不能用传统的共线方程来描述其成像几何关系。鱼眼相机的成像几何可以简单的看成是用不同类型的曲面去替换传统共线方程中的像平面[64],因此从物方到相方的构象方程也不固定,与具体的镜头性质有关。为了对鱼眼影像进行配准,常用的方法有2种：① 根据选定的构象方程对鱼眼影像进行图像校正,然后对校正后的影像利用传统的共线方程进行配准;② 利用鱼眼相机模型将点云采样成鱼眼视角下的距离影像,然后再进行二者的配准[65]。
4.1.4 全景成像模型
全景影像是当前街景和室内制图中常用的一种数据源,通过将多个相机同步采集的图像进行拼接融合,获得反应360°场景信息的全景图。全景影像的成像模型并不是原始的物理模型,而是通过投影变换得到的一种“二次模型”,根据投影方式的不同,常用的全景模型有柱面模型、球面模型等[66]。由于各个相机的摄影中心不能完全重合,使全景图生成过程中不可避免会存在拼接误差,因此部分学者[21,67]采用一种间接配准方式,首先根据单独相机的物理成像模型,利用点云与原始影像配准,然后根据原始影像与全景图的转换关系,实现点云与全景影像的配准,而另一部分研究则直接基于全景图的成像数学模型进行单张全景图与点云的配准[23,68]。Swart等[69]提出一种全景图序列与点云的整体配准方法,直接利用全景成像模型对多张全景图进行SfM,然后用得到的稀疏点云与激光点云进行ICP。

4.2 经验模型

与传感器严格模型相比,经验模型实际上人们根据对实际应用问题的观察和实践经验设计的一种数学转换模型,对2组数据之间的几何变形进行近似的纠正。如果数据之间存在局部几何变形导致整体拟合效果较差时,还常常采用分块策略进行局部拟合。与严格模型相比,经验模型缺乏整体的约束性,且分块配准精度完全依赖于分块内同名匹配点的正确性及其分布,对匹配误差非常敏感。但在某些特定的应用场合,经验模型仍发挥不可替代的作用。
4.2.1 多项式模型
多项式模型是一种常用的数学变换模型,该模型将数据之间的几何差异看成是平移、旋转、缩放、仿射、偏扭、弯曲以及更高层次的基本变形综合作用的结果,回避了图像的真实成像机理和点云的定位原理。常用的有二阶多项式和三阶多项式,更高的阶数可以模拟更复杂的变形,但容易造成对应点处的过拟合（Overfitting）,因此并不常用。多项式模型是卫星影像与DEM配准时常采用的模型。
4.2.2 薄板样条模型
薄板样条模型是一种非刚性的几何变换模型,薄板样条函数描述了一个连续表面在控制点约束下发生的柔性几何变形,既可以实现局部区域的高精度配准,又能够保证整体的连续性。理论上说,只要有足够多的控制点,薄板样条模型都都能够达到理想的配准精度。薄板样条模型常常用于单张像片的高精度配准,如数字文化保护中的高精度纹理映射[70]。

5 参数优化方法

当点云与影像之间的同名观测值和几何转换模型确定以后,需要利用优化方法获得模型参数的最优估计。配准中的参数优化问题一般可以表示为式（5）,其中Ai, Bi为第i对同名对应点,将对应点坐标差值的平方和最小作为优化目标。
X ˜ = argmin ∑ i = 1 n ( A i - T ( X ) ∙ B i ) 2 （5）
如果将同名观测值提取也看成是一个优化估计问题,我们可以将同名观测值的确定与配准参数估计放在同一个优化过程中,如式（6）锁死,其中 Sim 是相似性度量函数。优化的目标是实现精确配准时所有同名点的相似度达到最大。
X ˜ = argmax ∑ i = 1 n Sim ( A i , T ( X ) ∙ B i ) （6）
对于配准问题的参数优化方法,可以分成局部优化和全局优化2类。

5.1 局部优化

等式是一个经典的最小二乘优化问题,常用的优化方法有梯度下降法、高斯牛顿法和Levenberg-Marquardt算法[71]（简称LM）等。其中,LM算法用得最广,这是因为LM算法实质上是在高斯牛顿法中加入了自适应的尺度因子μ,当μ较大时求解相当于梯度下降法,当μ较小时相当于高斯牛顿法。这样既可以保证远离极值时,算法能够快速收敛,又能保证在靠近极值时,能够以较小的步长去逼近,而不出现在极值附近跳跃的情况。
然而,由于转换模型T（X）一般是非线性函数,需要进行线性化后迭代求解,其求解过程易受初值的影响。当初值不理想时,常常会陷入局部最优而无法实现全局最优。
另外,在实际应用中,得到的同名观测值中难以避免会混入一些粗差点（Outliers）。这些粗差对结果影响极大,常常造成求解过程无法收敛,因此研究者常常在每次迭代完成后,增加一个稳健估计的步骤,对同名观测值进行检查。常用的稳健估计方法有RANdom SAmple Consensus（RANSAC）算法和选权迭代法。
RANSAC算法[72]采用一种“采样-投票”的方式来提取符合模型的内点（Inliers）,而选前迭代法[73]则是利用验后方差对观测值赋权,进行粗差点的探测。这2种稳健估计方法都是为了保证参与求解观测值的可靠性,并不能完全解决初值问题,整个优化方案仍属于局部优化。

5.2 全局优化

全局优化方法则是直接在待求未知数的解空间上进行全局搜索,以获得全局最优解。最基本的全局优化方法是穷举法（Brute-Force）。假设待求的转换参数是像片的3个线元素和3个角元素,它们的取值方位都是 0,1 ,如果采用0.1的间隔对取值区间进行量化,总共有需要搜索106种组合。可见穷举效率较低,而且解算精度和量化间隔相关,减小量化间隔又会极大的降低求解效率。
在当前的配准应用中,常常采用的是一些更高效的全局搜索算法,如Nelder-Mead算法[22-23]、遗传算法[74]和Branch-and-Bound（BnB）算法[75]等。这类算法不仅能求解类型的优化问题,而且可以解决类型的优化问题。
在利用互信息或ICP进行配准时,真正的同名观测值开始并不知道,需要进行一个迭代的搜索-配准参数求解过程。整个问题变成了一个无导优化问题（Derivative-free optimization,DFO）,传统的最小二乘优化无法求解,部分学者利用全局搜索的优化方法,实现了观测值搜索和转换参数估计的整体优化。

6 方法比较

设计一个完整的配准算法需要综合考虑观测值、转换模型和优化方法3个方面,下面从数据源、应用目的、适用场合等方面,对现有的方法进行深入比较和分析（表1-3）,有助于根据自己的应用需求进行方法选择和改进。
Tab.1 The comparison of correspondence extractions

表1 观测值获取方法比较

方法 数据源 特征提取 特征描述 相似性测度 鲁棒性 配准精度 备注
基于区域的方法 空间域方法 光学影像,强度影像或DSM 不需要 不需要 SAD,SSD,NCC,LSCC 较差,对灰度差异敏感 高 速度较慢,适用于机载数据
频率域方法 光学影像,强度影像或DSM 不需要 不需要 相位相关 对噪声和灰度差异具有一定鲁棒性 很高 计算速度快,适用于机载数据
统计方法 光学影像,强度影像或DSM 不需要 不需要 互信息 对灰度鲁棒性好 高 需要较好的初值,适用于机载、车载和地面固定站
基于特征的方法 点特征 光学影像,强度影像或DSM 检测角点或斑点 基于邻域内的灰度或几何结构的描述子 KNN或NCC 对灰度差异和几何变形鲁棒 高 自然场景和人工场景内都存在丰富的点特征,适应范围广
线特征 光学影像,强度影像或DSM 检测边缘,边缘编组 几何特征描述子 基于距离、角度等几何关系的判断 对灰度差异和几何变形鲁棒 高 适合城市区域,常用于车载数据和地面固定站数据
面特征 光学影像,激光点云或DSM 光学影像上检测纹理均质区域,点云中检测平面 不需要 基于距离判断 好 低 适合进行粗配准
多视几何配准方法 光学影像序列,激光点云 光学影像序列中恢复三维信息 不需要 基于ICP的对应点提取 好 高 需要较好的初值,常于车载数据
Tab.2 The comparison of transformation models (NA for not available, * for available )

表2 几何转换模型比较（NA表示不需要或不适用,*表示适用）

模型 所需附加数据 精度 适用平台 适用领域
机载 车载 地面
共线方程 相机参数 高 * * * 大部分应用领域
直接线性变换 NA 较高 * * * 城市场景
鱼眼相机模型 相机参数 高 NA * * 城市或室内场景
全景成像模型 全景模型参数（某些方法需要相机参数） 较高 NA * * 城市或室内场景
多项式模型 NA 一般,和对应点数量和分布相关 NA NA * 对精度要求不高的单片配准应用
薄板样条模型 NA 高,但需要大量对应点 NA NA * 基于地面平台的高精度纹理映射
Tab.3 The comparison of optimization methods

表3 优化方法比较

优化方法 适用问题 优化能力 计算效率
局部优化 适用于同名观测值已确定,仅需要对转换函数进行优化求解的情况 能够得到局部最优解,易受初值的影响 计算速度快
全局优化 能够同时优化同名观测值提取和转换函数估计 能够得到全局最优解 计算速度较慢

7 结论与展望

光学影像与激光点云的配准是摄影测量学中的一个重要研究课题,是实现二维、三维数据融合与集成应用的基础,本文将整个配准课题划分为同名观测值的获取、几何转换模型的选择和参数优化方法3个部分,并对现有的算法分别展开综述,并分析其优缺点及适用范围。
通过上述分析可见,光学影像和激光点云的配准虽然经历了长期发展,但仍存在很多难点有待突破,将在以下3个方面有较大的发展空间。
（1）同名观测值的获取是配准问题的难点,也是今后研究的重点。为了提取稳定可靠的同名观测值,需要克服影像与点云之间的几何与辐射差异,对于二者的几何差异,一种有效的思路是利用影像和点云之间的近似位置关系进行初步的几何纠正,一般采用虚拟成像技术,将点云投影到指定像平面,构造出中心投影的虚拟影像,利用虚拟影像进行配准即可初步消除二者之间的几何变形和尺度差异。对于点云与影像之间的辐射差异,主要是由于激光点云的强度值及高程值与光学影像的灰度值之间存在不同的灰度映射模型,造成二者之间的非线性差异。目前的解决思路是采用分块线性转换[12]或MTM（Matching by Tone Mapping）非线性灰度映射技术[76]来对辐射差异进行建模,并在配准过程中同时求解几何变换参数和辐射参数。此外,为了获得数量丰富且分布均匀的同名观测值,综合利用点、线、面等多类特征的方法也越来越受重视,且采用分块方式对不同区域的特征数量进行平衡也称为一种重要的策略。在特征描述方面,邻域内的梯度方向信息越来越受到重视,结合灰度信息和方向信息的描述子比传统方法具有更好的可区分性;在相似性测度方面,互信息成为一种主流的测度,对互信息的稳健性、收敛性问题的研究逐渐深入,在概率密度函数估计、互信息加权等方面,都出现了一些针对性的改进方法[77];在多视几何配准方面,研究主要集中在三维点集之间的ICP配准改进方法上,通过选取具有几何稳定性的特征点[78-79],改进对应点搜索策略[80],以及对ICP进行全局优化求解[75]来提高算法的稳定性和计算效率。
（2）基于传感器的严格配准模型受到关注,将会推动系统检校和联合平差的发展。虽然经验模型不需要附加的传感器信息,在很多应用中发挥重要的作用,但基于传感器的严格模型能够对数据采集过程中的各种误差源进行直接建模,通常具有更好的稳定性和配准精度,受到研究省的广泛关注。近年来,利用一种数据源作为控制,根据传感器严格模型对另一种数据源进行系统检校的研究越来越多,但是考虑到控制数据本身也会受传感器误差源的影响,将会严重影响系统检校的精度和可靠性。因此,对2种数据的误差源进行同时建模和联合平差的方法将是后续研究的重点。对传感器严格配准模型的研究,不仅有助于提高点云与影像的配准精度,还可以进一步推动系统检校和联合平差领域的发展。
（3）根据特定应用问题,选择多种优化方法进行合理组合。基于搜索的全局优化算法具有理论优势,但在实际应用中还存在计算效率低,约束条件多,解算不稳定等问题;而LM算法等局部优化算法在转换参数估计问题上,表现出稳健高效等优点。将2种优化进行有效组合,将是今后配准研究和应用中的一个发展方向。目前,RANSAC等稳健估计方法已经成为LM优化过程中的一个标准配置,在今后的配准研究中,学者们会根据应用需求,将稳健高效的参数求解方法与全局搜索策略结合,实现观测值提取和转换参数估计的整体优化,进一步提高算法的稳定性和实用性。

The authors have declared that no competing interests exist.

参考文献

原文顺序 | 文献年度倒序 | 文中引用次数倒序
[1]
张帆,黄先锋,李德仁.激光扫描与光学影像数据配准的研究进展[J].测绘通报,2008(2):7-10.激光扫描是近年来快速发展的一种新型而重要的测量技术.激光扫描点云数据和光学影像配准是融合处理的基础步骤,具有重要意义,并得到较多的关注和研究.从配准的基本问题、配准的方法两个方面出发,总结国内外在激光扫描数据与影像数据配准方面的研究进展,并进行展望,为后续的相关研究提供参考.

[Zhang F, Huang X F, Li D R. A review of registration of laser scanner data and optical image[J]. Bulletin of Surveying and Mapping, 2008(2):7-10. ]

[2]
Mishra R K, Zhang Y.A review of optical imagery and airborne LiDAR data registration methods[J]. The Open Remote Sensing Journal, 2012,6(5):54-63.ABSTRACT Representing a scene completely from remote sensing data requires both spectral and 3-D-surface information. Integration of spectral information from optical images and 3-D-surface information from LiDAR is important in a number of remote sensing applications such as feature extraction, image classification, image analysis, building extraction, 3-D city modelling, canopy modelling etc. Therefore, numerous methods have been developed in the last decade to align both data sets into a common reference frame to effectively utilize their complementary characteristics. However, due to the significantly different characteristics between optical image and LiDAR data, there are a number of technical challenges in the alignment of both data sets. Different research papers introduced different strategy or methodology to overcome the challenges, reaching different alignment/registration results. This paper presents a review of classical and up to date optical-LiDAR registration methods with the emphasis on control point detection and matching. The aim of this paper is to provide readers with an overview of existing methods, identify their advantages and limitations, and give readers the overall information on what will be useful for researchers and practitioners to realistically select proper method for their application.

DOI

[3]
Stamos I.Automated registration of 3D-range with 2D-color images: an overview[C]. Information Sciences and Systems (CISS), 2010 44th Annual Conference on, 2010:1-6.

[4]
G. Sindhu Madhuri.Classification of image registration techniques and algorithms in digital image processing: A research survey[J]. International Journal of Computer Trends and Technology (IJCTT), 2014,15(2):78-82.Image Registration (IR) occupied a dominant role in the digital Image processing in general and Image analysis in particular. Image registration is a process of transforming different sets of data into one coordinate system, and data may be from - (a) multiple photographs, (b) different sensors and both (a) (b) vary from different (i) times, (ii) depths, and (iii) viewpoints, and thus aligning to monitor the subtle differences between two or more images. The development of IR techniques and algorithms is highly complex because it is required to find spatial correspondences among images, and have vast applications in - Computer Vision, Medical Imaging, Image Mosaicking, Biological Imaging and Brain Mapping, Remote Sensing, Military, Satellite communication, Criminology, and Optimization, etc. Image registration techniques are not only required but also essentially necessary to compare data images obtained from different measurements based on their application requirements. Due to its high potential requirement for research, there is a need to carryout a research survey on Image Registration techniques in order to understand the phenomenon of Image Registration and its implementation methodologies. This survey emphasizes Image Registration as the most essential part of panoramic image generation creation, where applications and uses are unimaginable for researchers longing to invent implement alternative image registration methods from general to specific to complex applications.

DOI

[5]
Gendrin C, Markelj P, Pawiro S A, et al.Validation for 2D/3D registration II: The comparison of intensity- and gradient-based merit functions using a new gold standard data set[J]. Medical physics, 2011,38(3):1491-1502.Purpose: A new gold standard data set for validation of 2D/3D registration based on a porcine cadaver head with attached fiducial markers was presented in the first part of this article. The advantage of this new phantom is the large amount of soft tissue, which simulates realistic conditions for registration. This article tests the performance of intensity- and gradient-based algorithms for 2D/3D registration using the new phantom data set.<br/>Methods: Intensity-based methods with four merit functions, namely, cross correlation, rank correlation, correlation ratio, and mutual information (MI), and two gradient-based algorithms, the backprojection gradient-based (BGB) registration method and the reconstruction gradient-based (RGB) registration method, were compared. Four volumes consisting of CBCT with two fields of view, 64 slice multidetector CT, and magnetic resonance-T1 weighted images were registered to a pair of kV x-ray images and a pair of MV images. A standardized evaluation methodology was employed. Targets were evenly spread over the volumes and 250 starting positions of the 3D volumes with initial displacements of up to 25 mm from the gold standard position were calculated. After the registration, the displacement from the gold standard was retrieved and the root mean square (RMS), mean, and standard deviation mean target registration errors (mTREs) over 250 registrations were derived. Additionally, the following merit properties were computed: Accuracy, capture range, number of minima, risk of nonconvergence, and distinctiveness of optimum for better comparison of the robustness of each merit.<br/>Results: Among the merit functions used for the intensity-based method, MI reached the best accuracy with an RMS mTRE down to 1.30 mm. Furthermore, it was the only merit function that could accurately register the CT to the kV x rays with the presence of tissue deformation. As for the gradient-based methods, BGB and RGB methods achieved subvoxel accuracy (RMS mTRE down to 0.56 and 0.70 mm, respectively). Overall, gradient-based similarity measures were found to be substantially more accurate than intensity-based methods and could cope with soft tissue deformation and enabled also accurate registrations of the MR-T1 volume to the kV x-ray image.<br/>Conclusions: In this article, the authors demonstrate the usefulness of a new phantom image data set for the evaluation of 2D/3D registration methods, which featured soft tissue deformation. The author's evaluation shows that gradient-based methods are more accurate than intensity-based methods, especially when soft tissue deformation is present. However, the current nonoptimized implementations make them prohibitively slow for practical applications. On the other hand, the speed of the intensity-based method renders these more suitable for clinical use, while the accuracy is still competitive. (C) 2011 American Association of Physicists in Medicine. [DOI: 10.1118/1.3553403]

DOI

[6]
Umeda K, Godin G, Rioux M.Registration of range and color images using gradient constraints and range intensity images[C]. Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17th International Conference on, 2004,13(3):12-15.

[7]
Duraisamy P, Belkhouche Y, Jackson S, et al. Automated two-dimensional-threedimensional registration using intensity gradients for three-dimensional reconstruction[J]. Journal of Applied Remote Sensing, 2012,6(1):063517(1-13).We develop a robust framework for the registration of light detection and ranging (LiDAR) images with 2-D visual images using a method based on intensity gradients. Our proposed algorithm consists of two steps. In the first step, we extract lines from the digital surface model (DSM) given by the LiDAR image, then we use intensity gradients to register the extracted lines from the LiDAR image onto the visual image to roughly estimate the extrinsic parameters of the calibrated camera. In our approach, we overcome some of the limitations of 3-D reconstruction methods based on the matching of features between the two images. Our algorithm achieves an accuracy for the camera pose recovery of about 98% for the synthetic images tested, and an accuracy of about 95% for the real-world images we tested, which were from the downtown New Orleans area.

DOI

[8]
叶沅鑫,单杰,彭剑威,等.利用局部自相似进行多光谱遥感图像自动配准[J].测绘学报,2014,43(3):268-275.<p>为解决多光谱遥感图像间非线性灰度差异造成的配准困难问题，提出了一种基于局部自相似的自动配准方法。该方法首先引入局部自相似(local self-similarity, LSS)特征，并在对数极坐标系下建立反映图像内在几何布局和形状属性的特征描述子。之后结合LSS和归一化相关系数构建一种形状相似性测度，并通过模板匹配的策略实现图像的配准。试验结果表明，该方法能够较好地抵抗图像间的非线性灰度差异，并获得了可靠的配准精度，而且相比于传统方法，在顾及计算效率的同时获得了更高的匹配正确率。</p>

[Ye Y X, Shan J, Peng J W.Automated multispectral remote sensing image registration based on local self-similarity[J]. Acta Geodaetica et Cartographica Sinica, 2014,43(3):268-275. ]

[9]
Kuglin C D, Hines D C.The phase correlation image alignment method[C]. Proc. IEEE 1975 Inf. Conf. Cybernet. Society, New York, 1975:163-165.

[10]
Reddy B S, Chatterji B N.An FFT-based technique for translation, rotation, and scale-invariant image registration[J]. Image Processing, IEEE Transactions on, 1996,5(8):1266-1271.

[11]
Shorter N, Kasparis T.Autonomous registration of LiDAR data to single aerial image[C]. Geoscience and Remote Sensing Symposium, 2008. IGARSS 2008. IEEE International, IEEE, 2008,5:216-219.

[12]
Wong A, Orchard J.Efficient FFT-Accelerated approach to invariant optical-LIDAR registration[J]. Geoscience and Remote Sensing, IEEE Transactions on, 2008,46(11):3917-3925.This paper presents a fast Fourier transform (FFT)-accelerated approach designed to handle many of the difficulties associated with the registration of optical and light detection and ranging (LIDAR) images. The proposed algorithm utilizes an exhaustive region correspondence search technique to determine the correspondence between regions of interest from the optical image with the LIDAR image over all translations for various rotations. The computational cost associated with exhaustive search is greatly reduced by exploiting the FFT. The substantial differences in intensity mappings between optical and LIDAR images are addressed through local feature mapping transformation optimization. Geometric distortions in the underlying images are dealt with through a geometric transformation estimation process that handles various transformations such as translation, rotation, scaling, shear, and perspective transformations. To account for mismatches caused by factors such as severe contrast differences, the proposed algorithm attempts to prune such outliers using the random sample consensus technique to improve registration accuracy. The proposed algorithm has been tested using various optical and LIDAR images and evaluated based on its registration accuracy. The results indicate that the proposed algorithm is suitable for the multimodal invariant registration of optical and LIDAR images.

DOI

[13]
Foroosh H, Zerubia J B, Berthod M.Extension of phase correlation to subpixel registration[J]. Image Processing, IEEE Transactions on, 2002,11(3):188-200.In this paper, we have derived analytic expressions for the phase correlation of downsampled images. We have shown that for downsampled images the signal power in the phase correlation is not concentrated in a single peak, but rather in several coherent peaks mostly adjacent to each other. These coherent peaks correspond to the polyphase transform of a filtered unit impulse centered at the point of registration. The analytic results provide a closed-form solution to subpixel translation estimation, and are used for detailed error analysis. Excellent results have been obtained for subpixel translation estimation of images of different nature and across different spectral bands

DOI PMID

[14]
Yan H, Liu J G.Robust phase correlation based feature matching for image co-registration and DEM generation[J]. Remote Sensing and Spatial Information Sciences, 2008,37:1751-1756.Abstract This paper presents a robust phase correlation based sub-pixel feature matching technique and its application in motion flow estimation, pixel-to-pixel image-co-registration and DEM generation. We propose to use a phase fringe filter and a highly robust technique in the direct Fourier-based phase correlation algorithm for translational shift estimation in sub-pixel accuracy. Noting the problem that local phase correlation based feature matching may fail in areas either featureless or with significant spectral differences between an image pair, a direct frequency based motion estimation assessment technique and a novel motion flow refinement scheme are designed to improve the unreliable local motion estimates around these areas. With the robust phase correlation based local matching algorithm, we are able to derive accurate pixel-to-pixel image co-registration and disparity mapping for DEM generation in most synthetic and real images from different sensor platforms or different spectral bands.

[15]
Pluim J P, Maintz J A, Viergever M A.Mutual-information-based registration of medical images: A survey[J]. Medical Imaging, IEEE Transactions on, 2003,22(8):986-1004.An overview is presented of the medical image processing literature on mutual-information-based registration. The aim of the survey is threefold: an introduction for those new to the field, an overview for those working in the field, and a reference ...

DOI PMID

[16]
Chen H M, Varshney P K, Arora M K.Performance of mutual information similarity measure for registration of multitemporal remote sensing images[J]. Geoscience and Remote Sensing, IEEE Transactions on, 2003,41(11):2445-2454.ABSTRACT Accurate registration of multitemporal remote sensing images is essential for various change detection applications. Mutual information has recently been used as a similarity measure for registration of medical images because of its generality and high accuracy. Its application in remote sensing is relatively new. There are a number of algorithms for the estimation of joint histograms to compute mutual information, but they may suffer from interpolation-induced artifacts under certain conditions. In this paper, we investigate the use of a new joint histogram estimation algorithm called generalized partial volume estimation (GPVE) for computing mutual information to register multitemporal remote sensing images. The experimental results show that higher order GPVE algorithms have the ability to significantly reduce interpolation-induced artifacts. In addition, mutual-information-based image registration performed using the GPVE algorithm produces better registration consistency than the other two popular similarity measures, namely, mean squared difference (MSD) and normalized cross correlation (NCC), used for the registration of multitemporal remote sensing images.

DOI

[17]
Cole-Rhodes A, Johnson K L, LeMoigne J, et al. Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient[J]. Image Processing, IEEE Transactions on, 2003,12(12):1495-1511.Image registration is the process by which we determine a transformation that provides the most accurate match between two images. The search for the matching transformation can be automated with the use of a suitable metric, but it can be very time-consuming and tedious. We introduce a registration algorithm that combines a simple yet powerful search strategy based on a stochastic gradient with two similarity measures, correlation and mutual information, together with a wavelet-based multiresolution pyramid. We limit our study to pairs of images, which are misaligned by rotation and/or translation, and present two main results. First, we demonstrate that, in our application, mutual information may be better suited for sub-pixel registration as it produces consistently sharper optimum peaks than correlation. Then, we show that the stochastic gradient search combined with either measure produces accurate results when applied to synthetic data, as well as to multitemporal or multisensor collections of satellite data. Mutual information is generally found to optimize with one-third the number of iterations required by correlation. Results also show that a multiresolution implementation of the algorithm yields significant improvements in terms of both speed and robustness over a single-resolution implementation.

DOI PMID

[18]
Suri S, Reinartz P.Mutual-information-based registration of TerraSAR-X and Ikonos imagery in urban areas[J]. Geoscience and Remote Sensing, IEEE Transactions on, 2010,48(2):939-949.Abstract The launch of high-resolution remote sensing satellites like TerraSAR-X, WorldView, and Ikonos has benefited the combined application of synthetic aperture radar (SAR) and optical imageries tremendously. Specifically, in case of natural calamities or disasters, decision makers can now easily use an old archived optical with a newly acquired (postdisaster) SAR image. Although the latest satellites provide the end user already georeferenced and orthorectified data products, still, registration differences exist between different data sets. These differences need to be taken care of through quick automated registration techniques before using the images in different applications. Specifically, mutual information (MI) has been utilized for the intricate SAR-optical registration problem. The computation of this metric involves estimating the joint histogram directly from image intensity values, which might have been generated from different sensor geometries and/or modalities (e.g., SAR and optical). Satellites carrying high-resolution remote sensing sensors like TerraSAR-X and Ikonos generate enormous data volume along with fine Earth observation details that might lead to failure of MI to detect correct registration parameters. In this paper, a solely histogram-based method to achieve automatic registration within TerraSAR-X and Ikonos images acquired specifically over urban areas is analyzed. Taking future sensors into a perspective, techniques like compression and segmentation for handling the enormous data volume and incompatible radiometry generated due to different SAR-optical image acquisition characteristics have been rightfully analyzed. The findings indicate that the proposed method is successful in estimating large global shifts followed by a fine refinement of registration parameters for high-resolution images acquired over dense urban areas.

DOI

[19]
邓非,杨海关,李丽英.基于互信息的LIDAR与光学影像配准方法[J].测绘科学,2009,34(6):51-52.本章论述了基于互信息的是单幅 影像与LIDAR数据的配准方法。本方案是基于两幅图像灰度值的配准方法,即利用LIDAR数据生成的强度图像,实现与航空影像的配准。虽然两种传感器得 到图像的灰度值有较大差异,且不成线性关系,通过引入互信息,可以将穷举的搜索问题变成基于相似性测度函数的优化问题。实验表明互信息准则对于航空影像与 LiDAR数据之间是有效的。

[Deng F, Yang H G, Li L Y.Mutual information based registration between lidar and optical image[J]. Science of Surveying and Mapping, 2009,34(6):51-52. ]

[20]
王蕾,林立文.基于梯度互信息的光学影像和LIDAR强度图配准[J].地理空间信息,2010,8(3):56-58.LIDAR（激光雷达）数据与其相应的光学影像数据具有诸多互补性,通过配准将这两种影像数据纳入到统一的坐标系下,是实现多源数据融合处理的前提。提出一种顾及两影像空间梯度信息的互信息配准方法。利用高斯核函数构成梯度向量,并结合互信息作为相似性测度,使用粒子群优化算法作为搜索策略;设计在相同投影方式下的LIDAR强度图和航空光学影像的局部配准试验,验证了配准方法的有效性。

DOI

[Wang L, Lin L W.Registration of optical image and lidar intensity map by mutual information based on the gradient[J]. Geospatial Information, 2010,8(3):56-58. ]

[21]
闫利,索一凡,曹亮.车载激光雷达点云与全景影像的自动配准[J].测绘科学,2016,41(4):113-117,123.针对激光雷达点云数据缺乏纹理信息的问题,该文提出一种基于互信息的车载激光雷达点云与全景影像配准方法。该方法使用统一的球面全景成像模型,引入互信息作为相似性测度,将车载激光雷达点云生成的深度图与信息提取后的全景影像进行配准,实现配准参数的自动、高精度解算。同时,对车载激光雷达点云与全景影像配准的精度进行评定与分析。实验结果表明,车载点云与全景影像的配准方案是可行的,具有较高的配准精度。

DOI

[Yan L, Suo Y F, Cao L.Registration of vehicle-borne lidar point cloud and panoramic image[J]. Science of Surveying and Mapping, 2016,41(4):113-117,123. ]

[22]
Mastin A, Kepner J, Fisher J.Automatic registration of LIDAR and optical images of urban scenes[J]. Cvpr: 2009 Ieee Conference on Computer Vision and Pattern Recognition, Vols 1-4, 2009:2631-2638.ABSTRACT Fusion of 3D laser radar (LIDAR) imagery and aerial optical imagery is an efficient method for constructing 3D virtual reality models. One difficult aspect of creating such models is registering the optical image with the LIDAR point cloud, which is characterized as a camera pose estimation problem. We propose a novel application of mutual information registration methods, which exploits the statistical dependency in urban scenes of optical appearance with measured LIDAR elevation. We utilize the well known downhill simplex optimization to infer camera pose parameters. We discuss three methods for measuring mutual information between LIDAR imagery and optical imagery. Utilization of OpenGL and graphics hardware in the optimization process yields registration times dramatically lower than previous methods. Using an initial registration comparable to GPS/INS accuracy, we demonstrate the utility of our algorithm with a collection of urban images and present 3D models created with the fused imagery.

DOI

[23]
Wang R, Ferrie F P, Macfarlane J.Automatic registration of mobile LiDAR and spherical panoramas[C]. Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on, IEEE, 2012:33-40.

[24]
Parmehr E G, Fraser C S, Zhang C, et al.Automatic registration of optical imagery with 3D LiDAR data using statistical similarity[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014,88:28-40.The development of robust and accurate methods for automatic registration of optical imagery and 3D LiDAR data continues to be a challenge for a variety of applications in photogrammetry, computer vision and remote sensing. This paper proposes a new approach for the registration of optical imagery with LiDAR data based on the theory of Mutual Information (MI), which exploits the statistical dependency between same- and multi-modal datasets to achieve accurate registration. The MI-based similarity measures quantify dependencies between aerial imagery, and both LiDAR intensity data and 3D point cloud data. The needs for specific physical feature correspondences, which are not always attainable in the registration of imagery with 3D point clouds, are avoided. Current methods for registering 2D imagery to 3D point clouds are first reviewed, after which the mutual MI approach is presented. Particular attention is given to adoption of the Normalised Combined Mutual Information (NCMI) approach as a means to produce a similarity measure that exploits the inherently registered LiDAR intensity and point cloud data so as to improve the robustness of registration between optical imagery and LiDAR data. The effectiveness of local versus global similarity measures is also investigated, as are the transformation models involved in the registration process. An experimental program conducted to evaluate MI-based methods for registering aerial imagery to LiDAR data is reported and the results obtained in two areas with differing terrain and land cover, and with aerial imagery of different resolution and LiDAR data with different point density are discussed. These results demonstrate the potential of the MI and especially the CMI methods for registration of imagery and 3D point clouds, and they highlight the feasibility and robustness of the presented MI-based approach to automated registration of multi-sensor, multi-temporal and multi-resolution remote sensing data for a wide range of applications. (C) 2013 International Society for Photogrammetry and Remote Sensing, Inc. (ISPRS) Published by Elsevier B.V. All rights reserved.

DOI

[25]
Mikolajczyk K, Tuytelaars T.Local invariant feature detectors: A survey[J]. Foundations and Trends® in Computer Graphics and Vision, 2007,3(3):177-280.ABSTRACT In this survey, we give an overview of invariant interest point detectors, how they evolved over time, how they work, and what their respective strengths and weaknesses are. We begin with defining the properties of the ideal local feature detector. This is followed by an overview of the literature over the past four decades organized in different categories of feature extraction methods. We then provide a more detailed analysis of a selection of methods which had a particularly significant impact on the research field. We conclude with a summary and promising future research directions.

DOI

[26]
Aanæs H, Dahl A, Pedersen K S.Interesting interest points[J]. International Journal of Computer Vision, 2012,97(1):18-35.Not all interest points are equally interesting. The most valuable interest points lead to optimal performance of the computer vision method in which they are employed. But a measure of this kind will be dependent on the chosen vision application. We propose a more general performance measure based on spatial invariance of interest points under changing acquisition parameters by measuring the spatial recall rate. The scope of this paper is to investigate the performance of a number of existing well-established interest point detection methods. Automatic performance evaluation of interest points is hard because the true correspondence is generally unknown. We overcome this by providing an extensive data set with known spatial correspondence. The data is acquired with a camera mounted on a 6-axis industrial robot providing very accurate camera positioning. Furthermore the scene is scanned with a structured light scanner resulting in precise 3D surface information. In total 60 scenes are depicted ranging from model houses, building material, fruit and vegetables, fabric, printed media and more. Each scene is depicted from 119 camera positions and 19 individual LED illuminations are used for each position. The LED illumination provides the option for artificially relighting the scene from a range of light directions. This data set has given us the ability to systematically evaluate the performance of a number of interest point detectors. The highlights of the conclusions are that the fixed scale Harris corner detector performs overall best followed by the Hessian based detectors and the difference of Gaussian (DoG). The methods based on scale space features have an overall better performance than other methods especially when varying the distance to the scene, where especially FAST corner detector, Edge Based Regions (EBR) and Intensity Based Regions (IBR) have a poor performance. The performance of Maximally Stable Extremal Regions (MSER) is moderate. We observe a relatively large decline in performance with both changes in viewpoint and light direction. Some of our observations support previous findings while others contradict these findings.

DOI

[27]
Ding M, Lyngbaek K, Zakhor A.Automatic registration of aerial imagery with untextured 3D LiDAR models[C]. IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, 2008:2486-2493.

[28]
张永军,熊小东,沈翔.城区机载LiDAR数据与航空影像的自动配准[J].遥感学报,2012,16(3):579-595.为解决机载LiDAR数据与航空影像集成应用中二者的配准问题,提出了一种机载LiDAR数据与航空影像配准的方法。首先,直接在LiDAR点云中提取建筑物3维轮廓线,通过将轮廓线规则化得到由两条相互垂直的直线段组成的建筑物角特征,并在航空影像上提取直线特征;然后,根据影像初始外方位元素将建筑物角特征投影到航空影像上,并采用一定的相似性测度在影像上寻找同名的影像角特征;最后,将角特征的角点当作控制点,利用传统的摄影测量光束法区域网平差解求影像新的外方位元素。解算过程中采用循环迭代策略。本方法的主要特点是,直接从LiDAR点云中提取线特征,避免了常规方法从距离图(或强度图)中提取线特征所产生的内插误差。通过与现有基于点云强度图的配准方法的对比实验表明,在低精度初始外方位元素的辅助下,本文方法能够达到较高的配准精度。

DOI

[Zhang Y J, Xiong X D, Shen X.Automatic registration of urban aerial imagery with airborne LiDAR data[J]. Journal of Remote Sensing, 2012,16(3):579-595. ]

[29]
Li N, Huang X, Zhang F, et al.Registration of aerial imagery and lidar data in desert areas using the centroids of bushes as control information[M]. Bethesda, MD, ETATS-UNIS: Photogrammetric Engineering & Remote Sensing, 2013:743-752.

[30]
González-Aguilera D, Rodríguez-Gonzálvez P, Gómez-Lahoz J.An automatic procedure for co-registration of terrestrial laser scanners and digital cameras[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2009,64(3):308-316.<h2 class="secHeading" id="section_abstract">Abstract</h2><p id="">Driven by progress in sensor technology, algorithms and data processing capabilities, close range photogrammetry has found a wide range of new application fields over the past two decades. Particularly, the emergence of terrestrial laser scanner has contributed to the close range photogrammetry &ldquo;popularization&rdquo; through many promising new applications. Nevertheless, a central issue in many of these developments is the integration of sensor technology with reliable data processing schemes to generate highly automated photogrammetric measurements systems.</p><p id="">This paper presents a flexible approach for the automatic co-registration of terrestrial laser scanners (TLS) and amateur digital cameras (DC) to be used effectively in practice. Particularly, the developed approach deals with two different images: a camera image acquired with a DC and a range image obtained with a TLS. To this end, an open-source tool &ldquo;USAlign&rdquo; has been developed for testing the different experiments.</p>

DOI

[31]
Altuntas C.An Experimental study on registration three-dimensional range images using range and intensity data[J]. ISPRS-International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2011,3822:115-118.

[32]
Lowe D G.Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004,60(2):91-110.<a name="Abs1"></a>This paper presents a method for extracting distinctive invariant features from images that can be used to perform reliable matching between different views of an object or scene. The features are invariant to image scale and rotation, and are shown to provide robust matching across a substantial range of affine distortion, change in 3D viewpoint, addition of noise, and change in illumination. The features are highly distinctive, in the sense that a single feature can be correctly matched with high probability against a large database of features from many images. This paper also describes an approach to using these features for object recognition. The recognition proceeds by matching individual features to a database of features from known objects using a fast nearest-neighbor algorithm, followed by a Hough transform to identify clusters belonging to a single object, and finally performing verification through least-squares solution for consistent pose parameters. This approach to recognition can robustly identify objects among clutter and occlusion while achieving near real-time performance.

DOI

[33]
Toth C K, Ju H, Grejner-Brzezinska D A. Experience with using SIFT for Multiple Image Domain Matching[C]. Annual Conference, Apr 2010, American Society for Photogrammetry and Remote Sensing, San Diego, CA, 2010.

[34]
Ju H, Toth C K, Grejner-Brzezinska D A. Evaluation of Mutlipe-domain imagery matching based on different feature spaces[C]. 2011 ASPRS Fall Conference, Herndon, VA, USA, 2011.

[35]
Bodensteiner C, Huebner W, Juengling K, et al.Local multi-modal image matching based on self-similarity[C]. Image Processing (ICIP), 2010 17th IEEE International Conference on, 2010:937-940.

[36]
Palenichka R M, Zaremba M B.Automatic extraction of control points for the registration of optical satellite and LiDAR images[J]. Geoscience and Remote Sensing, IEEE Transactions on, 2010,48(7):2864-2879.A novel method for automatic extraction of control points for the registration of optical images with Light Detection And Ranging (LiDAR) data is proposed. It is based on transformation-invariant detection of salient image disks (SIDs), which determine the location of control points as the centers of the corresponding image fragments. The SID is described by a feature vector, which, in addition to the coordinates and diameter, includes intensity descriptors and region shape characteristics of the image fragment. SIDs are effectively extracted using multiscale isotropic matched filtering-a visual attention operator that indicates image locations with high-intensity contrast, homogeneity, and local shape saliency. This paper discusses the extraction of control points from both natural landscapes and structured scenes with man-made objects. Registration experiments conducted on QuickBird imagery with corresponding LiDAR data validated the proposed approach.

DOI

[37]
Habib A, Ghanma M, Morgan M, et al.Photogrammetric and LiDAR data registration using linear features[J]. Photogrammetric Engineering & Remote Sensing, 2005,71(6):699-707.The enormous increase in the volume of datasets acquired by lidar systems is leading towards their extensive exploitation in a variety of applications, such as, surface reconstruction, city modeling, and generation of perspective views. Though being a fairly new technology, lidar has been influenced by and had a significant impact on photogrammetry. Such an influence or impact can be attributed to the complementary nature of the information provided by the two systems. For example, photogrammetric processing of imagery produces accurate information regarding object space break lines (discontinuities). On the other hand, lidar provides accurate information describing homogeneous physical surfaces. Hence, it proves logical to combine data from the two sensors to arrive at a more robust and complete reconstruction of 3D objects. This paper introduces alternative approaches for the registration of data captured by photogrammetric and lidar systems to a common reference frame. The first approach incorporates lidar features as control for establishing the datum in the photogrammetric bundle adjustment. The second approach starts by manipulating the photogrammetric imagery to produce a 3D model, including a set of linear features along object space discontinuities, relative to an arbitrarily chosen coordinate system. Afterwards, conjugate photogrammetric and lidar straight-line features are used to establish the transformation between the arbitrarily chosen photogrammetric coordinate system and the lidar reference frame. The second approach (bundle adjustment, followed by similarity transformation) is general enough to be applied for the co-registration of multiple three-dimensional datasets regardless of their origin (e.g., adjacent lidar strips, surfaces in GIS databases, and temporal elevation data). The registration procedure would allow for the identification of inconsistencies between the surfaces in question. Such inconsistencies might arise from changes taking place within the object space or inaccurate calibration of the internal characteristics of the lidar and the photogrammetric systems. Therefore, the proposed methodology is useful for change detection and system calibration applications. Experimental results from aerial and terrestrial datasets proved the feasibility of the suggested methodologies.

DOI

[38]
马洪超,姚春静,邬建伟.利用线特征进行高分辨率影像与LiDAR点云的配准[J].武汉大学学报·信息科学版, 2012,37(2):136-140,159.试图从离散点云数据中寻找影像的同名点是非常困难的,因此传统的基于同名特征点的配准方法难以使用。应用共线方程作为严格配准模型,利用lidar点云空间中的线特征替代传统配准模型中的点特征,取得了高精度的配准结果,同时对点云密度和影像分辨率之间的尺度关系进行了半定量分析。

[Ma H C, Yao C J, Wu J W.Registration of lidar point clouds and high resolution images based on linear features[J]. Geomatics and Information Science of Wuhan University, 2012,37(2):136-140,159. ]

[39]
Yao C, Guang G.The direct registration of LiDAR point clouds and high resolution image based on linear feature by introducing an unknown parameter[C]. Int. Arch. Photogramm. Remote Sensing and Spatial Information Sciences, 2012:403-408.

[40]
徐景中,寇媛,袁芳,等.基于结构特征的机载LiDAR数据与航空影像自动配准[J].红外与激光工程,2013(12):3502-3508.针对现有机载LiDAR数据与航空影像配准方法对匹配特征具有较强的依赖性，易受数据等影响的问题，提出了一种基于结构特征的自动配准方法。该方法首先提取LiDAR距离图像与对应影像的结构特征，利用初始姿态参数将LiDAR结构特征投影至影像坐标系下，根据结构特征的几何约束条件获取初始匹配点集，完成粗匹配；接着利用粗匹配结果计算直接变换模型（DLT）参数，并以此为初值引入双点几何约束，采用循环迭代的匹配策略，不断剔除错误匹配，获得一组新的匹配点集，完成精匹配；最后根据精匹配结果，采用基于单位四元数的空间后方交会方法解算航空影像的姿态参数，实现机载LiDAR数据与航空影像的自动配准。实验证明，该方法受噪声影响小，能实现机载LiDAR数据与航空影像的自动配准。

DOI

[Xu J Z, Kou Y, Yuan F, et al.Auto-registration of aerial imagery and airborne lidar data based on structure feature[J]. Infrared and Laser Engineering, 2013,12:3502-3508. ]

[41]
Wang L, Neumann U.A robust approach for automatic registration of aerial images with untextured aerial LiDAR data[C]. IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, 2009:2623-2630.

[42]
Kwak T S, Kim Y I, Yu K Y, et al.Registration of aerial imagery and aerial LiDAR data using centroids of plane roof surfaces as control information[J]. KSCE Journal of Civil Engineering September, 2006,10(5):365-370.This study proposes using centroid of the plane roof surface of a building as control information for registering the aerial im agery relative to the aerial LiDAR data. A majority of the roofs in South Korea are plane roofs. The centroid of the plane roof is extracted from aerial imagery by using the Canny Edge Detector and from aerial LiDAR data by using Local Maxima Filtering. These extracted centroids from LiDAR data are used as control information and exterior orientation parameters of aerial imagery are estimated. Also, exterior orientation parameters of aerial imagery are estimated by using GCPs and the accuracy of registration is evaluated. From the experimental results, the positional accuracy satisfied the error range of 1/5,000 digital map which was prescribed by National Geographic Information Institute of South Korea. From this study, it is found that centroid could be useful source of control information.

DOI

[43]
Habib A, Schenk T.A new approach for matching surfaces from laser scanners and optical scanners[J]. International Archives of Photogrammetry and Remote Sensing, 1999,32(3/W14):55-61.

[44]
Armenakis C, Gao Y, Sohn G. Co-Registration of Lidar and Photogrammetric Data for Updating Building Databases[C]."Core Spatial Databases - Updating, Maintenance and Services - from Theory to Practice", Haifa, Israel, 2010, ISPRS Archive Vol. 38, Part 4-8-3-W9: 96-100.

[45]
Michel R.Registration of airborne laser data with one aerial image[C]. ISPRS XXth congress, Istanbul, 2004,35:1682-1750.

[46]
Habib A, Aldelgawya M.Alternative procedures for the incorporation of LiDAR-derived linear and areal features for photogrammetric geo- referencing[C]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Vol. XXXVII. Part B1. Beijing 2008.

[47]
张良,马洪超,高广,等.点、线相似不变性的城区航空影像与机载激光雷达点云自动配准[J].测绘学报,2014,43(4):372-379.提出一种基于点、线相似不变性的城区航空影像与机载LiDAR点云自动配准算法。首先通过SIFT算子提取点特征并进行粗配准,同时分别基于影像和LiDAR点云提取直线特征;然后利用局部区域点特征与线特征的相似不变性,通过匹配点对搜索匹配直线对;最后采用基于扩展共线方程的2D-3D严密配准模型实现航空影像与LiDAR点云的精配准。本方法的特点是:采取了由粗到精的配准策略,通过点、线相似不变性,将基于强度的配准算法和基于线特征的配准算法有机结合,在较高的自动化程度下实现了影像与点云的精确配准。试验证明,与基于点云强度影像的自动配准算法相比,本文的算法在城市地区能够取得较好的配准结果。

DOI

[Zhang L, Ma H C, Gao G, et al.Automatic registration of urban aerial images with airborne lidar points based on line-point similarity invariants[J]. Acta Geodaetica et Cartographica Sinica, 2014,43(4):372-379. ]

[48]
Zhang Y, Xiong X, Zheng M, et al.LiDAR strip adjustment using multifeatures matched with aerial images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015,53(2):976-987.ABSTRACT Airborne light detecting and ranging (LiDAR) systems have been widely used for the fast acquisition of dense topographic data. Regrettably, coordinate errors always exist in LiDAR-acquired points. The errors are attributable to several sources, such as laser ranging errors, sensor mounting errors, and position and orientation system (POS) systematic errors, among others. LiDAR strip adjustment (LSA) is the solution to eliminating the errors, but most state-of-the-art LSA methods neglect the influence from POS systematic errors by assuming that the POS is precise enough. Unfortunately, many of the LiDAR systems used in China are equipped with a low-precision POS due to cost considerations. Subsequently, POS systematic errors should be also considered in the LSA. This paper presents an aerotriangulation-aided LSA (AT-aided LSA) method whose major task is eliminating position and angular errors of the laser scanner caused by boresight angular errors and POS systematic errors. The aerial images, which cover the same area with LiDAR strips, are aerotriangulated and serve as the reference data for LSA. Two types of conjugate features are adopted as control elements (i.e., the conjugate points matched between the LiDAR intensity images and the aerial images and the conjugate corner features matched between LiDAR point clouds and aerial images). Experiments using the AT-aided LSA method are conducted using a real data set, and a comparison with the three-dimensional similarity transformation (TDST) LSA method is also performed. Experimental results support the feasibility of the proposed AT-aided LSA method and its superiority over the TDST LSA method.

DOI

[49]
Leberl F, Irschara A, Pock T, et al.Point clouds: lidar versus 3D vision[J]. Photogrammetric Engineering & Remote Sensing, 2010,76(10):1123.Abstract Novel automated photogrammetry is based on four innovations. First is the cost-free increase of overlap between images when sensing digitally. Second is an improved radiometry. Third is multi-view matching. Fourth is the Graphics Processing Unit (GPU), making complex algorithms for image matching very practical. These innovations lead to improved automation of the photogrammetric workflow so that point clouds are created at sub-pixel accuracy, at very dense intervals, and in near real-time, thereby eroding the unique selling proposition of lidar scanners. Two test projects compare point clouds from aerial and street-side lidar systems with those created from images. We show that the photogrammetric accuracy compares well with the lidar-method, yet the density of surface points is much higher from images. and the throughput is commensurate with a fully automated all-digital approach. Beyond this density, we identify 15 additional advantages of the photogrammetric approach.

DOI

[50]
Postolov Y, Krupnik A, McIntosh K. Registration of airborne laser data to surfaces generated by Photogrammetric means[J]. International Archives of Photogrammetry and Remote Sensing, 1999,32(3/W14):95-99.

[51]
Stamos I, Liu L Y, Chen C, et al.Integrating automated range registration with multiview geometry for the photorealistic modeling of large-scale scenes[J]. International Journal of Computer Vision, 2008,78(2-3):237-260.The photorealistic modeling of large-scale scenes, such as urban structures, requires a fusion of range sensing technology and traditional digital photography. This paper presents a system that integrates automated 3D-to-3D and 2D-to-3D registration techniques, with multiview geometry for the photorealistic modeling of urban scenes. The 3D range scans are registered using our automated 3D-to-3D registration method that matches 3D features (linear or circular) in the range images. A subset of the 2D photographs are then aligned with the 3D model using our automated 2D-to-3D registration algorithm that matches linear features between the range scans and the photographs. Finally, the 2D photographs are used to generate a second 3D model of the scene that consists of a sparse 3D point cloud, produced by applying a multiview geometry (structure-from-motion) algorithm directly on a sequence of 2D photographs. The last part of this paper introduces a novel algorithm for automatically recovering the rotation, scale, and translation that best aligns the dense and sparse models. This alignment is necessary to enable the photographs to be optimally texture mapped onto the dense model. The contribution of this work is that it merges the benefits of multiview geometry with automated registration of 3D range scans to produce photorealistic models with minimal human interaction. We present results from experiments in large-scale urban scenes.

DOI

[52]
Zheng S, Huang R, Zhou Y.Registration of optical images with lidar data and its accuracy assessment[J]. Photogrammetric Engineering & Remote Sensing, 2013,79(8):731-741.Photogrammetry and lidar are two technologies complementary for 3D reconstruction. However, the problem is that the current registration methods of optical images with lidar data cannot satisfy all the requirements for the fusion of the above two technologies, especially for close-range photogrammetry and terrestrial lidar. In this paper, we propose a novel method for registration of optical images with terrestrial lidar data, which is implemented by minimizing the distances from the photogrammetric matching points to terrestrial lidar data surface, with the collinearity equation as the basic mathematical model. One advantage of this method is that it requires no feature extraction and segmentation from the lidar data. Another advantage is that non-rigid deformation caused by lens distortion can be eliminated through the use of bundle adjustment similar to self-calibration. In addition, experiments with two different data sets show that this method cannot only eliminate the influence of certain gross errors, but also offer a high accuracy of 3 mm to 5 mm. Therefore, the proposed registration method is proved to be more effective, accurate, and reliable.

DOI

[53]
Li Y, Low K L.Automatic registration of color images to 3D geometry[C]. Proceedings of the 2009 Computer Graphics International Conference, Victoria, British Columbia, Canada, ACM, 2009:21-28.

[54]
陈驰,杨必胜,彭向阳.低空UAV激光点云和序列影像的自动配准方法[J].测绘学报,2015,44(5):518-525.提出了一种低空无人机(unmanned aerial vehicle,UAV)序列影像与激光点云自动配准的方法。首先分别基于多标记点过程与局部显著区域检测对激光点云和序列影像的建筑物顶部轮廓进行提取,并依据反投影临近性匹配提取的顶面特征。然后利用匹配的建筑物角点对,线性解算序列影像外方位元素,再使用建筑物边线对的共面条件进行条件平差获得优化解。最后,为消除错误提取与匹配特征对整体配准结果的影响,使用多视立体密集匹配点集与激光点集进行带相对运动阈值约束的 ICP(迭代最临近点)计算,整体优化序列影像外方位元素解。试验结果表明本文方法能实现低空序列影像与激光点云像素级精度的自动配准,联合制作 DOM 精度满足现行无人机产品1∶500比例尺标准。

DOI

[Chen C, Yang B S, Peng X Y.Automatic registration of low altitude uav sequent images and laser point clouds[J]. Acta Geodaetica et Cartographica Sinica, 2015,44(5):518-525. ]

[55]
Yang B, Chen C.Automatic registration of UAV-borne sequent images and LiDAR data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015,101:262-274.Abstract Use of direct geo-referencing data leads to registration failure between sequent images and LiDAR data captured by mini-UAV platforms because of low-cost sensors. This paper therefore proposes a novel automatic registration method for sequent images and LiDAR data captured by mini-UAVs. First, the proposed method extracts building outlines from LiDAR data and images and estimates the exterior orientation parameters (EoPs) of the images with building objects in the LiDAR data coordinate framework based on corresponding corner points derived indirectly by using linear features. Second, the EoPs of the sequent images in the image coordinate framework are recovered using a structure from motion (SfM) technique, and the transformation matrices between the LiDAR coordinate and image coordinate frameworks are calculated using corresponding EoPs, resulting in a coarse registration between the images and the LiDAR data. Finally, 3D points are generated from sequent images by multi-view stereo (MVS) algorithms. Then the EoPs of the sequent images are further refined by registering the LiDAR data and the 3D points using an iterative closest-point (ICP) algorithm with the initial results from coarse registration, resulting in a fine registration between sequent images and LiDAR data. Experiments were performed to check the validity and effectiveness of the proposed method. The results show that the proposed method achieves high-precision robust co-registration of sequent images and LiDAR data captured by mini-UAVs.

DOI

[56]
Baltsavias E P.Airborne laser scanning: basic relations and formulas[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 1999,54(2):199-214.ABSTRACT An overview of basic relations and formulas concerning airborne laser scanning is given. They are divided into two main parts, the first treating lasers and laser ranging, and the second one referring to airborne laser scanning. A separate discussion is devoted to the accuracy of 3D positioning and the factors influencing it. Examples are given for most relations, using typical values for ALS and assuming an airplane platform. The relations refer mostly to pulse lasers, but CW lasers are also treated. Different scan patterns, especially parallel lines, are treated. Due to the complexity of the relations, some formulas represent approximations or are based on assumptions like constant flying speed, vertical scan, etc. q 1999 Elsevier Science B.V. All rights reserved. Keywords: Airborne laser scanning; Terminology; Basic relations; Formulas; 3D accuracy analysis 1. Introduction In this article, some basic relations and formulas Z. Z. concerning a laser ranging, and b airborne laser sc...

DOI

[57]
McGlone J C. Manual of photogrammetry, 6th edition[M]. Bethesda: American Society for Photogrammetry and Remote Sensing Publishing, 2013.

[58]
Gneeniss A, Mills J, Miller P.In-flight photogrammetric camera calibration and validation via complementary lidar[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2015,100:3-13.

[59]
陈为民,聂倩,林昀. 基于罗德里格矩阵的车载激光点云与全景影像配准研究[J].测绘通报,2013(11):21-24.提出一种基于罗德里格矩阵的车载激光点云与全景影像的配准算法,该方法利用车载GPS/IMU获取全景影像投影中心的初始位置,并采用共线条件方程描述全景投影中心、全景影像像点和同名激光点云间的几何关系,最后基于罗德里格矩阵变化法实现配准参数的解算。试验结果表明,本文提出的车载点云与全景影像的配准方法计算简单,且具有较高的配准精度。

[ Chen W M, Nie Q, Lin Y.Research on registration of vehicle-borne laser point clouds and panoramic images based on lodrigues matrix transformation[J]. Bulletin of Surveying and Mapping, 2013,11:21-24. ]

[60]
盛庆红,陈姝文,费利佳,等.基于Plücker直线的机载LiDAR点云与航空影像的配准[J].测绘学报,2015,44(7):761-767.机载LiDAR点云与航空影像的配准是地物提取的关键。本文提出了基于Plücker直线的机载LiDAR点云与航空影像的配准模型。基于三维线空间线变换Plücker直线方程,确定点云和影像中的同名直线间的相对位姿关系,建立Plücker直线共面条件方程,通过影像上的Plücker直线在三维线空间中的螺旋运动,实现像点到其对应LiDAR点的坐标变换。结果表明,Plücker直线共面条件配准模型简洁,将旋转和平移统一处理,避免了两者间的耦合误差,提高了配准精度,为获取高质量的地球三维空间信息提供了技术支持。

DOI

[Sheng Q H, Chen S W, Fei L J, et al.Registration of aerial image with airborne lidar data based on plücker line[J]. Acta Geodaetica et Cartographica Sinica, 2015,44(7):761-767. ]

[61]
王永波,杨化超,刘燕华,等.线状特征约束下基于四元数描述的LiDAR点云配准方法[J].武汉大学学报·信息科学版,2013,39(9):1057-1062.针对经典的基于同名点状特征匹配的lidar点云配准算法存在计算量大,点状特征提取精度低,以及基于七参数转换模型的lidar点云配准算法中方程线性化过程对配准精度影响较大的特点,提出了以线状特征作为lidar点云配准的基元,利用四元数法来表达旋转矩阵,进而形成线状特征约束下基于四元数描述的lidar点云配准方法,给出了线状特征约束下三维相似变换的相似性测度表达方法,推导并论证了以线状特征作为配准基元时同名线状特征需要满足的条件。根据四元数与旋转变换矩阵之间的对应关系,求解了基于四元数法的旋转矩阵,并根据旋转矩阵求解了平移及缩放系数。

[Wang Y B, Yang H C, Liu Y H, et al.Linear-feature-constrained registration of lidar point cloud via quaternion[J]. Geomatics and Information Science of Wuhan University, 2013,38(9):1057-1062. ]

[62]
Hatze H.High-precision three-dimensional photogrammetric calibration and object space reconstruction using a modified DLT-approach[J]. Journal of Biomechanics, 1988,21(7):533-538.Two modified DLT algorithms are presented that improve the accuracy of three-dimensional object space reconstruction by almost an order of magnitude when compared with conventional methods. The improvement in the linear modified DLT (MDLT) algorithm is achieved by satisfying certain orthogonality conditions in the form of a non-linear constraint, thereby effectively eliminating a redundant DLT parameter. In the non-linear MDLT algorithm, the improvement and computational stability results from the appropriate elimination of implicit variables from one side of the approximating relations and the corresponding reformulation of the objective function to be minimized. The highest reconstruction accuracy of 0.733 mm rms mean error was obtained with the non-linear MDLT algorithm. This corresponds to a spatial resolution of about one part in 2860 or 0.035% overall accuracy. The accuracy obtainable with the linear MDLT was found to be slightly less and about 0.041% (0.833 mm rms mean error).

DOI PMID

[63]
宋恒嘉. 车载激光点云和光学影像的配准方法研究[J].测绘通报,2014(1):67-70.首先总结国内外激光点云和光学影像配准的研究现状;然后针对单张影像提出一种基于直接线性变换的车载激光点云和光学影像的配准方法,针对车载序列影像提出一种基于SIFT角点提取、影像匹配、光束法平差、密集点云生成、密集点云和激光点云自动配准并生成对应三维彩色点云的方法;最后以VC++ 6.0为开发平台,利用Optech公司提供的车载激光点云和序列影像数据设计并实现了车载激光点云和光学影像的配准,并验证了算法的有效性.

DOI

[Song H J.Registration of vehicle-borne laser point cloud and optical image[J]. Bulletin of Surveying and Mapping, 2014,1: 67-70. ]

[64]
李根,费章君,杨仕友.基于几何成像模型的鱼眼镜头图像校正算法和技术研究[J].机电工程,2013,30(10):1268-1272.鱼眼镜头是一种具有广视场角的摄像机镜头,广泛应用于大场景视频监控领域,但其成像存在严重的非线性径向畸变,违反人类的视觉习惯,且为测量和模式识别等应用带来了不便.为了实现鱼眼镜头图像的校正,提出了一种基于几何成像模型的鱼眼镜头校正算法和技术,通过研究鱼眼镜头成像模型,根据鱼眼镜头类型采用了相应的几何模型校正图像;以“等距投影”模型为例提出了校正公式和实现方法.最后针对工程应用的实时性要求,提出了基于缓存的快速优化方法以及在保证一定精度的前提下快速实现双线性插值的优化方法.研究结果表明,这种新方法可以高效地完成鱼眼镜头图像校正,比传统方法适用面更广,更易实现.

DOI

[Li G, Fei Z J, Yang S Y.Fish-eye image correction method and technique based on geometrical imaging model[J]. Journal of Mechanical & Electrical Engineering, 2013,30(10):1268-1272. ]

[65]
熊友谊. 地面激光扫描点云与鱼眼全景影像的配准与整合研究[D].北京:中国矿业大学(北京),2014.

[Xiong Y Y. Research on data registration and integration of terrestrial laser scanning point clouds and fisheye panoramic images[D]. Beijing: China University of Mining and Technology(Beijing), 2014. ]

[66]
Szeliski R.Image alignment and stitching: A tutorial[J]. Foundations and Trends® in Computer Graphics and Vision, 2006,2(1):1-104.

[67]
聂倩,蔡元波,林昀,等.车载激光点云与全景影像的配准研究[J].遥感信息,2014,29(1):15-18.针对全球定位系统(GPS)、惯性测量单元(IMU)、激光扫描仪(LS)和全景相机集成的车载测量系统,本文提出一种车载激光点云与全景影像配准的方法.该方法利用POS数据和各个传感器间的相对空间位置信息建立激光点云与单张面阵CCD影像间的对应关系,并根据全景影像与单张面阵CCD影像间的映射关系,从而实现车载点云与全景影像的高精度配准.试验结果表明,本文提出的车载点云与全景影像的配准方法计算简单,具有较高的配准精度,对城市三维测量与重建具有重要的应用价值.

DOI

[Nei Q, Cai Y B, Lin Y, et al.Registration of vehicle-borne laser point clouds and panoramic images[J]. Remote Sensing Information, 2014,29(1):15-18. ]

[68]
刘全海,鲍秀武,李楼.激光点云与全景影像配准方法研究[J].现代测绘,2016,39(1):21-25.针对全景成像系统获取的全景影像的内外方位元素未知的问题,充分利用激光点云的可量测优势和全景影像的360°视场角浏览查看优点,提出并实现了一种基于点特征的激光点云和全景影像的配准算法。该算法只需在激光点云和全景影像上各提取少量同名特征点便可进行配准。试验表明,算法能够很好地实现全景影像与激光点云的完美配准。

DOI

[Liu Q H, Bao X W, Li L.Research of registration based on panoramic images and laser point cloud[J]. Modern Surveying and Mapping, 2016,39(1):21-25. ]

[69]
Swart A, Broere J, Veltkamp R, et al.Refined non-rigid registration of a panoramic image sequence to a LiDAR point cloud[C]. Photogrammetric Image Analysis, Springer, 2011:73-84.

[70]
张帆. 激光扫描数据三维建模中高保真度纹理重建研究[D].武汉:武汉大学,2008.

[Zhang F.High fidelity texture reconstruction in 3d modeling from laser scanning data[D]. Wuhan: Wuhan University, 2008. ]

[71]
Levenberg K.A method for the solution of certain non-linear problems in least squares[J]. Quarterly of applied mathematics, 1944,2(2):164-168.date-added={2012-09-25 09:54:31 +0200}, date-modified={2012-09-25 09:54:31 +0200}, optannote={.}, optkey={.}, optmonth={.}, optnote={.}, optvolume={.}, project={fremdliteratur},

DOI

[72]
Fischler M A, Bolles R C.Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM, 1981,24(6):381-395.In this paper, the authors introduce a new paradigm, Random Sample Consensus (RANSAC), for fitting a model to experimental data. RANSAC is capable of interpreting/smoothing data containing a significant percentage of gross errors, and thus is ideally suited for applications in automated image analysis where interpretation is based on the data provided by error-prone feature detectors. A major portion of the paper describes the application of RANSAC to the Location Determination Problem (LDP): Given an image depicting a set of landmarks with known locations, determine that point in space from which the image was obtained. In response to a RANSAC requirement, the authors derive new results on the minimum number of landmarks needed to obtain a solution, and present algorithms for computing these minimum-landmark solutions in closed form. These results provide the basis for an automatic system that can solve the LDP under difficult viewing and analysis conditions. Implementation details and computational examples also are presented.

DOI

[73]
李德仁. 利用选择权迭代法进行粗差定位[J].武汉大学学报·信息科学版,1984,9(1):46-68.选择权迭代法是一种可望用于粗差定位的有效方法。本文回顾该方法中几种主要的不同权函数后,从最小二乘法验后方差估计原理推导出选择权迭代法进行粗差定位的权函数的和统计检验量。通过对不同形式权函数的对比试验证实了所提方法之优越性。将其引入丹麦法中亦获得同样好的结果。

[Li D R.Gross error location by means of the iteration method with variable weights[J]. Geomatics and Information Science of Wuhan University, 1984,9(1):46-68.]

[74]
Santamaría J, Cordón O, Damas S.A comparative study of state-of-the-art evolutionary image registration methods for 3D modeling[J]. Computer Vision and Image Understanding, 2011,115(9):1340-1354.ABSTRACT Image registration (IR) aims to find a transformation between two or more images acquired under different conditions. This problem has been established as a very active research field in computer vision during the last few decades. IR has been applied to a high number of real-world problems ranging from remote sensing to medical imaging, artificial vision, and computer-aided design. Recently, there is an increasing interest on the application of the evolutionary computation paradigm to this field in order to solve the ever recurrent drawbacks of traditional image registration methods as the iterated closest point algorithm. Specially, evolutionary image registration methods have demonstrated their ability as robust approaches to the problem. Unlike classical IR methods, they show the advantage of not requiring a good initial estimation of the image alignment to proceed. In this contribution, we aim to review the state-of-the-art image registration methods that lay their foundations on evolutionary computation. Moreover, we aim to analyze the performance of some of the latter approaches when tackle a challenging real-world application in forensic anthropology, the 3D modeling of forensic objects.

DOI

[75]
Yang J, Li H, Campbell D, et al.Go-ICP: A globally optimal solution to 3D ICP point-set registration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016,38(11):2241-2254.The Iterative Closest Point (ICP) algorithm is one of the most widely used methods for point-set registration. However, being based on local iterative optimization, ICP is known to be susceptible to local minima. Its performance critically relies on the quality of the initialization and only local optimality is guaranteed. This paper presents the first globally optimal algorithm, named Go-ICP, for Euclidean (rigid) registration of two 3D point-sets under the <inline-formula><tex-math notation

DOI PMID

[76]
Hel-Or Y, Hel-Or H, David E.Matching by tone mapping: Photometric invariant template matching[J]. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2014,36(2):317-330.Abstract A fast pattern matching scheme termed matching by tone mapping (MTM) is introduced which allows matching under nonlinear tone mappings. We show that, when tone mapping is approximated by a piecewise constant/linear function, a fast computational scheme is possible requiring computational time similar to the fast implementation of normalized cross correlation (NCC). In fact, the MTM measure can be viewed as a generalization of the NCC for nonlinear mappings and actually reduces to NCC when mappings are restricted to be linear. We empirically show that the MTM is highly discriminative and robust to noise with comparable performance capability to that of the well performing mutual information, but on par with NCC in terms of computation time.

DOI PMID

[77]
Tagare H D, Rao M.Why does mutual-information work for image registration? A deterministic explanation[J]. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 2015,37(6):1286-1296.Abstract This paper proposes a deterministic explanation for mutual-information-based image registration (MI registration). The explanation is that MI registration works because it aligns certain image partitions. This notion of aligning partitions is new, and is shown to be related to Schur- and quasi-convexity. The partition-alignment theory of this paper goes beyond explaining mutual- information. It suggests other objective functions for registering images. Some of these newer objective functions are not entropy-based. Simulations with noisy images show that the newer objective functions work well for registration, lending support to the theory. The theory proposed in this paper opens a number of directions for further research in image registration. These directions are also discussed.

DOI PMID

[78]
Gelfand N, Ikemoto L, Rusinkiewicz S, et al.Geometrically stable sampling for the ICP algorithm[C]. Fourth International Conference on 3-D Digital Imaging and Modeling, Proceedings, 2003:260-267.

[79]
Gressin A, Mallet C, Demantké J, et al.Towards 3D lidar point cloud registration improvement using optimal neighborhood knowledge[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2013,79(I-3):240-251.Automatic 3D point cloud registration is a main issue in computer vision and remote sensing. One of the most commonly adopted solution is the well-known Iterative Closest Point (ICP) algorithm. This standard approach performs a fine registration of two overlapping point clouds by iteratively estimating the transformation parameters, assuming good a priori alignment is provided. A large body of literature has proposed many variations in order to improve each step of the process (namely selecting, matching, rejecting, weighting and minimizing). The aim of this paper is to demonstrate how the knowledge of the shape that best fits the local geometry of each 3D point neighborhood can improve the speed and the accuracy of each of these steps. First we present the geometrical features that form the basis of this work. These low-level attributes indeed describe the neighborhood shape around each 3D point. They allow to retrieve the optimal size to analyze the neighborhoods at various scales as well as the privileged local dimension (linear, planar, or volumetric). Several variations of each step of the ICP process are then proposed and analyzed by introducing these features. Such variants are compared on real datasets with the original algorithm in order to retrieve the most efficient algorithm for the whole process. Therefore, the method is successfully applied to various 3D lidar point clouds from airborne, terrestrial, and mobile mapping systems. Improvement for two ICP steps has been noted, and we conclude that our features may not be relevant for very dissimilar object samplings.

DOI

[80]
Rusinkiewicz S., Levoy M.Efficient variants of the ICP algorithm[C]. Third International Conference on 3-D Digital Imaging and Modeling, Proceedings, 2001:145-152.

Options
文章导航

/

〈 〉