增强型遥感影像SRGAN算法及其在三维重建精度提升中的应用

doi:10.12082/dqxxkx.2022.210766

摘要/Abstract

摘要：

遥感影像是地形测绘、三维重建等任务的主要数据源之一,分辨率影响着被测目标在影像上的表示能力,对后期三维模型的定位精度及重建效果起着重要作用。针对遥感影像像幅较大且目标特征表现复杂、细节丰富的特点,结合实景三维模型重建的需求,提出了一种增强型遥感影像SRGAN算法。克服了传统方法进行超分重建时易出现边缘效应、产生模糊重建的情况,改进了简单卷积网络仅能提取影像中较为浅层的特征信息,无法在提高分辨率的同时保留影像丰富细节的局限。本文所提算法在生成模型中使用密集剩余残差块进行深层特征提取,在判别模型中引入多尺度判别思想,从而保证遥感影像重建时特征纹理、细节信息、高频目标的完整与精确。实验构建不同时间、不同类型区域的遥感影像数据集,在此基础上将本文算法与Bicubic、SRGAN、ESRGAN算法进行对比分析,在超分重建中PSNR较对比算法提升约3个单位,渗透指数PI更趋向且稳定于1,SSIM与清晰度指标Q同样得到较好改善;在三维重建中影像密集匹配点数量得到提升,同时误差减少,模型精细程度和定位精度得到提高。结果表明,本文算法适用于遥感影像超分辨率重建问题,并在实景三维模型重建中对精度的提升表现较好。

关键词: 超分辨率重建, 三维重建, 深度学习, 生成对抗网络, 多尺度相对判别, 高分辨率遥感, 遥感影像处理, 定位精度

Abstract:

Remote sensing images are important data sources for terrain mapping, 3D reconstruction, and other tasks. The spatial resolution of remote sensing images determines the representation ability of the measured object on the image and plays an important role in the positioning accuracy and reconstruction effect of 3D model in the later stage. In view of the characteristics of high resolution remote sensing images including large scale, complex target features, and rich details, an enhanced SRGAN algorithm for remote sensing image reconstruction is proposed to meet the needs of 3D model reconstruction. The proposed algorithm overcomes the problems of edge effect and fuzzy reconstruction using traditional methods for super-resolution reconstruction. In traditional methods, there is limitation that simple convolutional networks can only extract the shallow feature information of the image and cannot retain the rich details of the image with the increasing resolution. The proposed algorithm is based on the generative adversarial networks using deep learning, in which dense residual blocks are used to extract deep features, and multi-scale discrimination is introduced into the discriminant model. In the training, the generation model and the discrimination model learn features together and are optimized to finally obtain a super-resolution reconstruction model suitable for remote sensing image application. This model can improve the resolution and image quality of remote sensing images, and ensure the integrity and accuracy of feature texture, detail information, and high-frequency target. In our study, the proposed algorithm is compared with the Bicubic, SRGAN, and ESRGAN algorithms. Our results show that the PSNR of the proposed algorithm is improved by about three units, the Penetration Index (PI) is stable and closer to one, and the SSIM and clarity index Q are also improved. In 3D reconstruction, the number of image dense matching points is increased, and the error is reduced. The measured point values of the model are closer to the measured point values from the field. The visual perception of the model is also more real and delicate, which indicates that the precision and positioning accuracy of the 3D model can be significantly improved using the remote sensing images constructed by the proposed algorithm. The results demonstrate the proposed algorithm that considers the characteristics of remote sensing images performs better than other algorithms for the super-resolution reconstruction, and the geometric accuracy and visual accuracy of the real 3D models based on the constructed images are also significantly improved.

Key words: super-resolution reconstruction, 3D reconstruction, deep learning, SRGAN, multi-scale relative discrimination, high-resolution remote sensing, image processing, positioning accuracy

闵杰, 张永生, 于英, 吕可枫, 王自全, 张磊. 增强型遥感影像SRGAN算法及其在三维重建精度提升中的应用[J]. 地球信息科学学报, 2022, 24(8): 1631-1644.DOI:10.12082/dqxxkx.2022.210766

MIN Jie, ZHANG Yongsheng, YU Ying, LV Kefeng, WANG Ziquan, ZHANG Lei. Enhanced Remote Sensing Image SRGAN Algorithm and Its Application in Improving the Accuracy of 3D Reconstruction[J]. Journal of Geo-information Science, 2022, 24(8): 1631-1644.DOI:10.12082/dqxxkx.2022.210766

图/表 13

图1

图2

图3

图4

表1

表2

图5

表3

图6

表4

图7

图8

表5

参考文献 21

[1]	张春森, 张卫龙, 郭丙轩, 等. 倾斜影像的三维纹理快速重建[J]. 测绘学报, 2015, 44(7):782-790.
	[ Zhang C S, Zhang W L, Guo B X, et al. Rapidly 3D texture reconstruction based on oblique photography[J]. Acta Geodaetica et Cartographica Sinica, 2015, 44(7):782-790. ] DOI: 10.11947/j.AGCS.2015.20140341 doi: 10.11947/j.AGCS.2015.20140341
[2]	王冬, 丁奥, 施银迪, 等. 基于倾斜摄影测量的油库精细三维建模方法研究[J]. 测绘与空间地理信息, 2021, 44(5):8-11,16.
	[ Wang D, Ding A, Shi Y D, et al. Research on precise three-dimensional modeling method of oil depot based on tilt photogrammetry[J]. Geomatics & Spatial Information Technology, 2021, 44(5):8-11,16. ] DOI: 10.3969/j.issn.1672-5867.2021.05.003 doi: 10.3969/j.issn.1672-5867.2021.05.003
[3]	Leonardo G, Olga R P B and Luciano S. 3D reconstruction methods for digital preservation of cultural heritage: A survey[J]. Pattern Recognition Letters, 2014, 50:3-14. DOI: 10.1016/j.patrec.2014.03.023 doi: 10.1016/j.patrec.2014.03.023
[4]	李德仁, 肖雄武, 郭丙轩, 等. 倾斜影像自动空三及其在城市真三维模型重建中的应用[J]. 武汉大学学报·信息科学版, 2016, 41(6):711-721.
	[ Li D R, Xiao X W, Guo B X, et al. Oblique image based automatic aerotriangulation and its application in 3D city model reconstruction[J]. Geomatics and Information Science of Wuhan University, 2016, 41(6):711-721. ] DOI: 10.13203/j.whugis20160099 doi: 10.13203/j.whugis20160099
[5]	Hu H, Zhu Q, Du Z, et al. Reliable Spatial Relationship Constrained Feature Point Matching of Oblique Aerial Images[J]. Photogrammetric Engineering&Remote Sensing, 2015, 81(1):49-58. DOI: 10.14358/PERS.81.1.49 doi: 10.14358/PERS.81.1.49
[6]	Hartley R, Zisserman A. Multiple view geometry in computer vision: Two-view geometry[J]. 2004,(PART II):237-238. DOI: 10.1017/CBO9780511811685.013 doi: 10.1017/CBO9780511811685.013
[7]	肖雄武, 李德仁, 郭丙轩, 等. 一种具有视点不变性的倾斜影像快速匹配方法[J]. 武汉大学学报·信息科学版, 2016, 41(9):1151-1159.
	[ Xiao X W, Li D R, Guo B X, et al. A robust and rapid viewpoint-invariant matching method for oblique images[J]. Geomatics and Information Science of Wuhan University, 2016, 41(9):1151-1159. ] DOI: 10.13203/j.whugis20140405 doi: 10.13203/j.whugis20140405
[8]	李德仁, 袁修孝. 误差处理与可靠性理论:第2版[M]武汉大学出版社, 2012:44-62.
	[ Li D R, Yuan X X. Error Processing and Reliability Theory: Second Edition[M]. Wuhan University Press, 2012:44-62. ]
[9]	Tsai R Y, Huang T S. Multi-frame image restoration and registration[J]. Advances in Computer Vision and Image Processing, 1984, 1:317-339.
[10]	Harris J L. Diffraction and resolving Powerp[J]. Journal of the Optical Society of America, 1964, 54(7):931-933. DOI: 10.1364/josa.54.000931 doi: 10.1364/josa.54.000931
[11]	Goodman J W, Gustafson S C, Reviewer. Introduction to fourier optics, second edition[J]. Optical Engineering, 1996, 35(5):1513-1515. DOI: 10.1117/1.601121 doi: 10.1117/1.601121
[12]	李欣, 韦宏卫, 张洪群. 结合深度学习的单幅遥感图像超分辨率重建[J]. 中国图象图形学报, 2018, 23(2):209-218.
	[ Li X, Wei H W, Zhang H Q. Super-resolution reconstruction of single remote sensing image combined with deep learning[J]. Journal of Image and Graphics, 2018, 23(2):209-218. ] DOI: 10.11834/jig.170194 doi: 10.11834/jig.170194
[13]	Dong C, Loy C C, He K, et al. Learning a deep convolutional network for image super-resolution[A].European Conference on Computer Vision[C]. Springer, Cham, 2014:184-199. DOI: 10.1007/978-3-319-10593-2_13 doi: 10.1007/978-3-319-10593-2_13
[14]	唐艳秋, 潘泓, 朱亚平, 等. 图像超分辨率重建研究综述[J]. 电子学报, 2020, 48(7):1407-1420. doi: 10.3969/j.issn.0372-2112.2020.07.022
	[ Tang Y Q, Pan H, Zhu Y P, et al. A survey of image super-resolution reconstruction[J]. Acta Electronica Sinica, 2020, 48(7):1407-1420. ] DOI: 10.3969/j.issn.0372-2112.2020.07.022 doi: 10.3969/j.issn.0372-2112.2020.07.022
[15]	Ledig C, Theis L, F Huszar, et al. Photo- realistic single image super-resolution using a generative adversarial Network[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York:IEEE, 2017:105-114. DOI: 10.1109/CVPR.2017.19 doi: 10.1109/CVPR.2017.19
[16]	Wang X, Yu K, Wu S, et al. ESRGAN: Enhanced super-resolution generative adversarial networks[C]. Computer Vision -- ECCV 2018 Workshops. Cham: Springer International Publishing, 2018:63-79. DOI: 10.1007/978-3-030-11021-5_5 doi: 10.1007/978-3-030-11021-5_5
[17]	毕晓君, 潘梦迪. 基于生成对抗网络的机载遥感图像超分辨率重建[J]. 智能系统学报, 2020, 15(1):74-83.
	[ Bi X J, Pan M D. Super-resolution reconstruction of airborne remote sensing images based on the generative adversarial networks[J]. CAAI Transactions on Intelligent Systems, 2020, 15(1):74-83. ] DOI: CNKI:SUN:ZNXT.0.2020-01-014 doi: CNKI:SUN:ZNXT.0.2020-01-014
[18]	Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks[J]. Computer ence, 2015. DOI: 10.48550/arXiv.1511.06434 doi: 10.48550/arXiv.1511.06434
[19]	Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4):600-612. DOI: 10.1016/j.jvcir.2019.102655 doi: 10.1016/j.jvcir.2019.102655
[20]	范冲, 龚健雅, 朱建军, 等. ALOS-PRISM遥感影像超分辨率重建[J]. 遥感学报, 2009, 13(1):75-82.
	[ Fan C, Gong J Y, Zhu J J, et al. Super-resolution reconstruction of ALOS-PRISM remote sensing images[J]. National Remote Sensing Bulletin, 2009, 13(1):75-82. ] DOI: 10.3321/j.issn:1007-4619.2009.01.010 doi: 10.3321/j.issn:1007-4619.2009.01.010
[21]	Keys R G. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics, Speech, and Signal Processing, 2003, 29(6):1153-1160. DOI: 10.1109/TASSP.1981.1163711 doi: 10.1109/TASSP.1981.1163711

软硬件名称	参数及信息
CPU	Intel® Core™i9 1100K @3.70GHz
GPU	Intel(R) UHD Graphics (Driver: 27.20.100.9168)NVIDIA GeForce RTX 3080 (Driver: 27.21.14.6192)
RAM	32 GB
系统配置	Ubuntu 18.04 LTS Windows 10
程序语言	Python 3.7.0
底层框架	PyTorch 1.4.0
模型重建软件	Pix4D, Context Capture

数据集	指标	内容
数据集1	时间	2014年夏季5月
	覆盖区域	登封嵩山遥感卫星定标场
	影像数量/幅	176
	飞行航高/m	200
	镜头焦距/mm	15
	传感器大小/mm	23.52×15.68
	影像分辨率（尺寸）/像素	6000×4000
	影像像素大小/um	3.92
数据集2	时间	2017年夏季9月
	覆盖区域	登封雪沟及其周边地区
	影像数量/幅	191
	飞行航高/m	1220
	镜头焦距/mm	35
	传感器大小/mm	35.00×23.35
	影像分辨率（尺寸）/像素	7952×5304
	影像像素大小/um	4.40
数据集3	时间	2017年冬季12月
	覆盖区域	登封雪沟及其周边地区
	影像数量/幅	206
	飞行航高/m	1255
	镜头焦距/mm	35
	传感器大小/mm	35.00×23.35
	影像分辨率（尺寸）/像素	7952×5304
	影像像素大小/um	4.40
数据集4	时间	2018年夏季5月
	覆盖区域	郑州市某校区
	影像数量/幅	55
	飞行航高/m	90
	镜头焦距/mm	35
	传感器大小/mm	36×24
	影像分辨率（尺寸）/像素	6000×4000
	影像像素大小/um	6.00
数据集5	时间	2016年夏季9月
	覆盖区域	郑州市某独立建筑
	影像数量/幅	496
	飞行航高/m	90
	镜头焦距/mm	3.5
	传感器大小/mm	6.32×4.74
	影像分辨率（尺寸）/像素	4000×3000
	影像像素大小/um	1.58
数据集6	时间	2021年秋季10月
	覆盖区域	沈阳市某校区
	影像数量/幅	390×5
	飞行航高/m	153
	镜头焦距/mm	下视25,前后左右侧视35
	传感器大小/mm	下视30.78×20.52,前后左右侧视42.81×28.54
	影像分辨率（尺寸）/像素	6000×4000
	影像像素大小/um	下视5.13,前后左右侧视7.14

算法		Bicubic^[20]			SRGAN^[14]			ESRGAN^[15]			增强型SRGAN
评价指标	放大因子	DF1000	DB_x	DB_q	DF1000	DB_x	DB_q	DF1000	DB_x	DB_q	DF1000	DB_x	DB_q
PSNR	×2	20.32	21.07	20.97	25.47	29.17	26.64	25.39	28.01	28.13	25.43	28.14	29.43
PSNR	×4	23.14	26.21	24.32	31.02	29.95	28.46	30.34	34.96	34.11	30.78	35.07	34.11
SSIM	×2	0.53	0.48	0.43	0.51	0.63	0.52	0.62	0.69	0.70	0.69	0.72	0.71
SSIM	×4	0.56	0.66	0.70	0.90	0.78	0.78	0.86	0.87	0.84	0.89	0.90	0.89
PI	×2	23.87	22.67	23.71	1.48	0.67	0.55	1.36	1.14	1.09	1.31	1.07	1.11
PI	×4	25.56	23.47	27.32	1.66	0.48	0.61	1.23	1.27	1.10	1.19	1.21	1.09
Q	×2	4.13	4.07	4.00	5.44	5.37	5.80	5.89	6.12	5.88	5.91	6.18	5.92
Q	×4	4.34	3.98	4.03	5.69	5.72	5.75	6.00	6.03	6.11	6.12	6.29	6.22

指标	LR	Bicubic	SRGAN	ESRGAN	增强型SRGAN
连接点的数量/个	183 348	198 627	313 914	353 418	340 024
每个连接点影像覆盖中位数	3	3	3	3	3
每幅影像上匹配连接点中位数	396	429	678	791	734
重投影误差中值	1.68	1.44	0.72	0.891	0.41
重投影均方误差	1.60	1.53	1.03	1.00	0.62
投影射线的均方误差	0.01380	0.00921	0.00748	0.00803	0.00247

	HR	Bicubic			SRGAN			ESRGAN			增强型SRGAN
P1	(391.44, 437.50, 54.74)	(387.41, 429.37, 51.09)			(388.37, 435.42, 50.97)			(392.49, 435.55, 51.92)			(390.47, 436.01, 55.91)
P2	(414.69, 681.50, 61.82)	(416.78, 688.79, 58.79)			(419.03, 678.69, 57.64)			(413.04, 683.93, 63.87)			(415.81, 682.47, 60.01)
P3	(267.69, 381.00, 57.34)	(263.14, 377.35, 60.19)			(264.83, 385.01, 60.03)			(265.97, 383.71, 55.46)			(266.03, 382.64, 59.43)
P4	(293.63, 541.52, 54.28)	(296.19, 537.99, 47.39)			(297.97, 543.979, 50.97)			(291.68, 539.98, 51.49)			(294.84, 543.87, 51.86)
P5	(352.19, 466.13, 118.53)	(353.46, 469.37, 108.42)			(350.29, 462.18, 112.49)			(354.40, 464.71, 115.51)			(350.97, 467.71, 120.03)
		△X	△Y	△Z	△X	△Y	△Z	△X	△Y	△Z	△X	△Y	△Z
平均误差		2.90	5.17	5.31	3.30	3.06	4.00	1.72	2.01	2.51	1.24	1.61	1.80
最大误差		4.03	8.13	10.11	4.34	4.01	6.04	2.21	2.71	3.02	1.67	2.35	2.42