利用视角转换的跨视角影像匹配方法

doi:10.12082/dqxxkx.2023.220312

地球信息科学学报 ›› 2023, Vol. 25 ›› Issue (2): 368-379.doi: 10.12082/dqxxkx.2023.220312

利用视角转换的跨视角影像匹配方法

饶子昱(), 卢俊(), 郭海涛, 余东行, 侯青峰

战略支援部队信息工程大学地理空间信息学院，郑州 450001

收稿日期:2022-05-16 修回日期:2022-06-29 出版日期:2023-02-25 发布日期:2023-04-19
通讯作者: *卢俊（1981— ），男，湖北武汉人，博士，副教授，硕士生导师，主要从事摄影测量与遥感方面的研究。 E-mail: ljhb45@126.com
作者简介:饶子昱（1998— ），男，河南新乡人，硕士生，主要从事多尺度跨视角遥感影像场景匹配与定位方面的研究。E-mail: ziyu769@163.com。
基金资助:
国家自然科学基金项目(41601507);基础加强计划技术领域基金项目(2019-JCJQ-JJ-126)

A Cross-View Image Matching Method with Viewpoint Conversion

RAO Ziyu(), LU Jun(), GUO Haitao, YU Donghang, HOU Qingfeng

Institute of Geospatial Information, Strategic Support Force Information Engineering University, Zhengzhou 450001, China

Received:2022-05-16 Revised:2022-06-29 Online:2023-02-25 Published:2023-04-19
Contact: LU Jun
Supported by:
National Natural Science Foundation of China(41601507);Foundation Strengthening Program Technology Field Fundation(2019-JCJQ-JJ-126)

摘要/Abstract

摘要：

目前遥感影像跨视角匹配技术无法直接使用大幅卫星影像进行匹配，难以满足大范围复杂场景匹配的任务需求，且依赖大规模数据集，不具备良好的泛化能力。针对上述问题，本文在质量感知模板匹配方法的基础上结合多尺度特征融合算法，提出一种基于视角转换的跨视角遥感影像匹配方法。该方法首先利用手持摄影设备采集地面多视影像，经密集匹配生成点云数据，利用主成分分析法拟合最佳地平面并进行投影变换，以实现地面侧视视角到空视视角的转换；然后设计了特征融合模块对VGG19网络从遥感影像中提取的低、中、高尺度特征进行融合，以获取遥感影像丰富的空间信息和语义信息；最后利用质量感知模板匹配方法将从视角转换后的地面影像上提取的特征与遥感影像的融合特征进行匹配，获取匹配的软排名结果，并采用非极大值抑制算法从中筛选出高质量的匹配结果。实验结果表明，在不需要大规模数据集的情况下本文方法具有较高的准确性和较强的泛化能力，平均匹配成功率为64.6%，平均中心点偏移量为5.9像素，匹配结果准确完整，可为大场景跨视角影像匹配任务提供一种新的解决方案。

关键词: 遥感影像, 跨视角匹配, 视角转换, 多尺度特征, 特征融合, 模板匹配, 泛化性

Abstract:

At present, the cross-view matching technology of remote sensing images cannot directly use large-scale satellite images for matching, which is difficult to meet the requirements of large-scale complex scene matching tasks and relies on large-scale datasets, thus lacking a good generalization ability. Aiming at the above problems, this paper proposes a cross-view remote sensing image matching method based on visual transformation using the quality-aware template matching method combined with the multi-scale feature fusion algorithm. In this method, the ground multi-view images are collected by using handheld photographic equipment. The portability and flexibility of the handheld photographic equipment can make it easier for us to collect multi-view images covering the target area. The acquired images are densely matched to generate point cloud data, and principal component analysis is used to fit the best ground plane and perform projection transformation to realize the conversion from the ground side view to the aerial view. Then, a feature fusion module is designed for the VGG19 network. The low, medium and high-level features extracted from remote sensing images are fused to obtain rich spatial and semantic information of remote sensing images. The fusion features of semantic information and spatial information can resist large-scale differences. Finally, the quality-aware template matching method is used. The features extracted from the ground images are matched with the fusion features of the remote sensing images. The matching soft ranking results are obtained, and the non-maximum suppression algorithm is used to select high-quality matching results. The experimental results show that the method proposed in this paper has a high accuracy and strong generalization ability without the need of large-scale datasets. The average matching success rate is 64.6%, and the average center point offset is 5.9 pixels. The matching results are accurate and complete, which provide a new solution for the task of cross-view image matching in large scenes.

Key words: remote sensing image, cross-view matching, viewpoint transformation, multi-scale feature, feature fusion, template matching, generalization

饶子昱, 卢俊, 郭海涛, 余东行, 侯青峰. 利用视角转换的跨视角影像匹配方法[J]. 地球信息科学学报, 2023, 25(2): 368-379.DOI:10.12082/dqxxkx.2023.220312

RAO Ziyu, LU Jun, GUO Haitao, YU Donghang, HOU Qingfeng. A Cross-View Image Matching Method with Viewpoint Conversion[J]. Journal of Geo-information Science, 2023, 25(2): 368-379.DOI:10.12082/dqxxkx.2023.220312

图/表 13

图1

图2

图3

表1

图4

图5

图6

图7

图8

图9

图10

表2

表3

参考文献 29

[1]	Choi J F G, Choi J, Friedland G. Multimodal location estimation of videos and images[M]. Cham: Springer, [2014] DOI:10.1007/978-3-319-09861-6 doi: 10.1007/978-3-319-09861-6
[2]	Hays J, Efros A A. IM2GPS: estimating geographic information from a single image[C]. IEEE Conference on Computer Vision and Pattern Recognition, 2008:1-8. DOI:10.1109/CVPR.2008.4587784 doi: 10.1109/CVPR.2008.4587784
[3]	Siagian C, Itti L. Biologically inspired mobile robot vision localization[J]. IEEE Transactions on Robotics, 2009, 25(4):861-873. DOI:10.1109/TRO.2009.2022424 doi: 10.1109/TRO.2009.2022424
[4]	Arandjelović R, Gronat P, Torii A, et al. NetVLAD: CNN architecture for weakly supervised place recognition[C]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017:1437-1451. DOI:10.1109/TPAMI.2017.2711011 doi: 10.1109/TPAMI.2017.2711011
[5]	Lin T Y, Cui Y, Belongie S, et al. Learning deep representations for ground-to-aerial geolocalization[C]. IEEE Conference on Computer Vision and Pattern Recognition, 2015:5007-5015. DOI:10.1109/CVPR.2015.7299135 doi: 10.1109/CVPR.2015.7299135
[6]	Hu S X, Feng M D, Nguyen R M H, et al. CVM-net: Cross-view matching network for image-based ground-to-aerial geo-localization[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018:7258-7267. DOI:10.1109/CVPR.2018.00758 doi: 10.1109/CVPR.2018.00758
[7]	Shi Y J, Yu X, Liu L, et al. Optimal feature transport for cross-view image geo-localization[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7):11990-11997. DOI:10.1609/aaai.v34i07.6875 doi: 10.1609/aaai.v34i07.6875
[8]	Liu L, Li H D. Lending orientation to neural networks for cross-view geo-localization[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019:5617-5626. DOI:10.1109/CVPR.2019.00577 doi: 10.1109/CVPR.2019.00577
[9]	Zhu S J, Yang T, Chen C. VIGOR: cross-view image geo-localization beyond one-to-one retrieval[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021:5316-5325. DOI:10.1109/CVPR46437.2021.00364 doi: 10.1109/CVPR46437.2021.00364
[10]	Zhu S J, Yang T, Chen C. Revisiting street-to-aerial view image geo-localization and orientation estimation[C]. IEEE Winter Conference on Applications of Computer Vision, 2021:756-765. DOI:10.1109/WACV48630.2021.00080 doi: 10.1109/WACV48630.2021.00080
[11]	Ge Y X, Wang H B, Zhu F, et al. Self-supervising fine-grained region similarities for large-scale image localization[C]. European Conference on Computer Vision, 2020:369-386. DOI:10.1007/978-3-030-58548-8_22 doi: 10.1007/978-3-030-58548-8_22
[12]	Shi Y J, Yu X, Campbell D, et al.Where Am I looking at? joint location and orientation estimation by cross-view matching[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020:4063-4071. DOI:10.1109/CVPR42600.2020.00412 doi: 10.1109/CVPR42600.2020.00412
[13]	Toker A, Zhou Q J, Maximov M, et al. Coming down to earth: Satellite-to-street view synthesis for geo-localization[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021:6484-6493. DOI:10.1109/CVPR46437.2021.00642 doi: 10.1109/CVPR46437.2021.00642
[14]	Mousavian A, Kosecka J. Semantic image based geolocation given a map[EB/OL]. 2016: arXiv: 1609.00278. https://arxiv.org/abs/1609.00278.
[15]	Wu S, Tang H, Jing X Y, et al. Cross-View Panorama Image Synthesis[J]. IEEE Transactions on Multimedia, 2022. DOI:10.1109/tmm.2022.3162474 doi: 10.1109/tmm.2022.3162474
[16]	Chen Z, Sun K, Yang F, et al. SC2-PCR: A Second Order Spatial Compatibility for Efficient and Robust Point Cloud Registration[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 13221-13231. DOI:10.5220/0005750606300637 doi: 10.5220/0005750606300637
[17]	Regmi K, Shah M. Bridging the domain gap for ground-to-aerial image matching[C]. IEEE/CVF International Conference on Computer Vision, 2019:470-479. DOI:10.1109/ICCV.2019.00056 doi: 10.1109/ICCV.2019.00056
[18]	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[EB/OL]. 2014: arXiv: 1409.1556. https://arxiv.org/abs/1409.1556
[19]	Cheng J X, Wu Y, AbdAlmageed W, et al. QATM: quality-aware template matching for deep learning[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019:11545-11554. DOI:10.1109/CVPR.2019.01182 doi: 10.1109/CVPR.2019.01182
[20]	闫利, 费亮, 陈长海, 等. 利用网络图进行高分辨率航空多视影像密集匹配[J]. 测绘学报, 2016, 45(10):1171-1181.
	[ Yan L, Fei L, Chen C H, et al. A multi-view dense matching algorithm of high resolution aerial images based on graph network[J]. Acta Geodaetica et Cartographica Sinica, 2016, 45(10):1171-1181. ] DOI:10.11947/j.AGCS.2016.20160068 doi: 10.11947/j.AGCS.2016.20160068
[21]	郭复胜, 高伟. 基于辅助信息的无人机图像批处理三维重建方法[J]. 自动化学报, 2013, 39(6):834-845.
	[ Guo F S, Gao W. Batch reconstruction from UAV images with prior information[J]. Acta Automatica Sinica, 2013, 39(6):834-845. ] DOI:10.3724/SP.J.1004.2013.00834 doi: 10.3724/SP.J.1004.2013.00834
[22]	郭复胜, 高伟, 胡占义. 无人机图像全自动生成大比例尺真正射影像方法[J]. 中国科学:信息科学, 2013, 43(11):1383-1397.
	[ Gao W, Hu Z Y. Automatic generation of large scale true ortho-image from UAV images[J]. Scientia Sinica (Informationis), 2013, 43(11):1383-1397. ] DOI: 10.1360/112012-591 doi: 10.1360/112012-591
[23]	Fan J H, Bocus M J, Hosking B, et al. Multi-scale feature fusion: Learning better semantic segmentation for road pothole detection[C]. IEEE International Conference on Autonomous Systems, 2021:1-5. DOI: 10.1109/ICAS49788.2021.9551165 doi: 10.1109/ICAS49788.2021.9551165
[24]	Ma C, Huang J B, Yang X K, et al. Hierarchical convolutional features for visual tracking[C]. IEEE International Conference on Computer Vision, 2015:3074-3082. DOI:10.1109/ICCV.2015.352 doi: 10.1109/ICCV.2015.352
[25]	Lin T Y, Dollár P, Girshick R, et al. Feature pyramid networks for object detection[C]. IEEE Conference on Computer Vision and Pattern Recognition, 2017:936-944. DOI:10.1109/CVPR.2017.106 doi: 10.1109/CVPR.2017.106
[26]	Workman S, Souvenir R, Jacobs N. Wide-area image geolocalization with aerial reference imagery[C]. Proceedings of the IEEE International Conference on Computer Vision, 2015: 3961-3969. DOI: 10.1109/iccv.2015.451 doi: 10.1109/iccv.2015.451
[27]	Vo N N, Hays J. Localizing and Orienting Street Views Using Overhead Imagery[C]. European Conference on Computer Vision, 2016:494-509. DOI:10.1007/978-3-319-46448-0_30 doi: 10.1007/978-3-319-46448-0_30
[28]	Zheng Z D, Wei Y C, Yang Y. University-1652: A multi-view multi-source benchmark for drone-based geo-localization[C]. Proceedings of the 28th ACM International Conference on Multimedia, 2020:1395-1403. DOI:10.1145/3394171.3413896 doi: 10.1145/3394171.3413896
[29]	Talmi I, Mechrez R, Zelnik-Manor L. Template matching with deformable diversity similarity[C]. IEEE Conference on Computer Vision and Pattern Recognition, 2017:1311-1319. DOI:10.1109/CVPR.2017.144 doi: 10.1109/CVPR.2017.144

数据集名称	CVUSA	Vo and Hays	郑州地区Google Earth影像	University1652
空间分辨率/m	亚米级	亚米级	0.45~0.6	亚米级
辅助信息(GPS)	有	有	无	无
地区	美国各地	波士顿、休斯顿	郑州	世界各地
数据类型	城市、郊区	城市	城市、乡村	大学

	匹配成功率/%		平均多余误匹配数/个
	良好影像	瑕疵影像	良好影像	瑕疵影像
城市	66.9	62.1	2.2	3.8
乡村	62.3	60.3	4.5	5.1

conv3-4	conv4-4	conv5-4	匹配成功率 /%			中心点偏移量/像素
conv3-4	conv4-4	conv5-4	视角转换影像	未视角转换影像	瑕疵影像	中心点偏移量/像素
√	×	×	44.6	-	40.1	10.8
×	√	×	41.7	-	34.9	13.6
×	×	√	57.4	-	51.9	8.8
√	×	√	61.3	-	56.1	7.6
×	√	√	56.8	-	53.7	8.4
√	√	√	64.6	-	61.2	5.9

利用视角转换的跨视角影像匹配方法

A Cross-View Image Matching Method with Viewpoint Conversion

RichHTML

PDF (PC)

赞

可视化

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 29

相关文章 15

Metrics

本文评价

推荐阅读 0

[1]	张俊瑶, 杨晓梅, 王志华, 杨海坤, 张博淳, 万庆, 雷梅. 绿色发展理念视角下内蒙古煤矿区格局演变分析[J]. 地球信息科学学报, 2023, 25(8): 1655-1668.
[2]	林娜, 何静, 王斌, 唐菲菲, 周俊宇, 郭江. 结合植被光谱特征与Sep-UNet的城市植被信息智能提取方法[J]. 地球信息科学学报, 2023, 25(8): 1717-1729.
[3]	吴敏, 张明达, 李盼盼, 张勇健. 面向多源遥感影像数据的溯源模型研究[J]. 地球信息科学学报, 2023, 25(7): 1325-1335.
[4]	张金雷, 陈奕洁, Panchamy Krishnakumari, 金广垠, 王骋程, 杨立兴. 基于注意力机制的城市轨道交通网络级多步短时客流时空综合预测模型[J]. 地球信息科学学报, 2023, 25(4): 698-713.
[5]	衡雪彪, 许捍卫, 唐璐, 汤恒, 许怡蕾. 基于改进全卷积神经网络模型的土地覆盖分类方法研究[J]. 地球信息科学学报, 2023, 25(3): 495-509.
[6]	高鹏飞, 曹雪峰, 李科, 游雄. 融合多元稀疏特征与阶层深度特征的遥感影像目标检测[J]. 地球信息科学学报, 2023, 25(3): 638-653.
[7]	陈凯, 雷少华, 代文, 王春, 刘爱利, 李敏. 基于开源数据和条件生成对抗网络的地形重建方法[J]. 地球信息科学学报, 2023, 25(2): 252-264.
[8]	王龙号, 蓝朝桢, 姚富山, 侯慧太, 武蓓蓓. 多源遥感影像深度特征融合匹配算法[J]. 地球信息科学学报, 2023, 25(2): 380-395.
[9]	张建兵, 严泽枭, 马淑芳. 用于遥感影像建筑物变化检测的多尺度交叉对偶注意力网络[J]. 地球信息科学学报, 2023, 25(12): 2487-2500.
[10]	谭敏, 林惠晶, 郝明. 先验知识引导深度学习的耕地范围建筑物和大棚房监测方法[J]. 地球信息科学学报, 2023, 25(11): 2293-2302.
[11]	蒋伟杰, 张春菊, 徐兵, 罗晨晨, 周晗, 周康. AED-Net：滑坡灾害遥感影像语义分割模型[J]. 地球信息科学学报, 2023, 25(10): 2012-2025.
[12]	于明洋, 陈肖娴, 张文焯, 刘耀辉. 融合网格注意力阀门和特征金字塔结构的高分辨率遥感影像建筑物提取[J]. 地球信息科学学报, 2022, 24(9): 1785-1802.
[13]	闵杰, 张永生, 于英, 吕可枫, 王自全, 张磊. 增强型遥感影像SRGAN算法及其在三维重建精度提升中的应用[J]. 地球信息科学学报, 2022, 24(8): 1631-1644.
[14]	张华, 郑祥成, 郑南山, 史文中. 基于MAEU-CNN的高分辨率遥感影像建筑物提取[J]. 地球信息科学学报, 2022, 24(6): 1189-1203.
[15]	岳振宇, 范大昭, 董杨, 纪松, 李东子. 一种星载平台轻量化快速影像匹配方法[J]. 地球信息科学学报, 2022, 24(5): 925-939.