基于多任务学习的高分辨率遥感影像建筑提取

doi:10.12082/dqxxkx.2021.190805

地球信息科学学报 ›› 2021, Vol. 23 ›› Issue (3): 514-523.doi: 10.12082/dqxxkx.2021.190805

基于多任务学习的高分辨率遥感影像建筑提取

朱盼盼¹^,²(), 李帅朋¹^,², 张立强¹^,²^,^*(), 李洋¹^,²

1.北京师范大学地理科学学部, 北京 100875
2.北京师范大学环境遥感与数字城市北京市重点实验室, 北京 100875

收稿日期:2019-12-26 修回日期:2020-04-23 出版日期:2021-03-25 发布日期:2021-05-25
通讯作者: 张立强 E-mail:zlyxbmsl@163.com;zhanglq@bnu.edu.cn
作者简介:朱盼盼（1989- ）,女,河南周口人,博士生,主要从高分辨率光学遥感影像信息提取研究。E-mail: zlyxbmsl@163.com
基金资助:
国家自然科学基金项目(41371324)

Multitask Learning-based Building Extraction from High-Resolution Remote Sensing Images

ZHU Panpan¹^,²(), LI Shuaipeng¹^,², ZHANG Liqiang¹^,²^,^*(), LI Yang¹^,²

1. Faculty of Geographical Science, Beijing Normal University, Beijing 100875, China
2. Beijing Key Laboratory of Environmental Remote Sensing and Digital City, Beijing Normal University, Beijing 100875, China

Received:2019-12-26 Revised:2020-04-23 Online:2021-03-25 Published:2021-05-25
Contact: ZHANG Liqiang E-mail:zlyxbmsl@163.com;zhanglq@bnu.edu.cn
Supported by:
National Natural Science Foundation of China(41371324)

摘要/Abstract

摘要：

建筑物的自动提取对城市发展与规划、防灾预警等意义重大。当前的建筑物提取研究取得了很好的成果,但现有研究多把建筑提取当成语义分割问题来处理,不能区分不同的建筑个体,且在提取精度方面仍然存在提升的空间。近年来,基于多任务学习的深度学习方法已在计算机视觉领域得到广泛应用,但其在高分辨率遥感影像自动解译任务上的应用还有待进一步发展。本研究借鉴经典的实例分割算法Mask R-CNN和语义分割算法U-Net的思想,设计了一种将语义分割模块植入实例分割框架的深度神经网络结构,利用多种任务之间的信息互补性来提升模型的泛化性能。自底向上的路径增强结构缩短了低层细节信息向上传递的路径。自适应的特征池化使得实例分割网络可以充分利用多尺度信息。在多任务训练模式下完成了对遥感影像中建筑物的自动分割,并在经典的遥感影像数据集SpaceNet上对该方法进行验证。结果表明,本文提出的基于多任务学习的建筑提取方法在巴黎数据集上建筑实例分割精度达到58.8%,在喀土穆数据集上建筑实例分割精度达到60.7%,相比Mask R-CNN和U-Net提升1%~2%。

关键词: 深度学习, 多任务学习, 语义分割, 实例分割, 遥感影像, 建筑提取, Mask R-CNN, U-Net

Abstract:

Automatic extraction of buildings is of great significance to urban development and planning, and disaster prevention and early warning. Current researches on building extraction have achieved good results, but the existing research methods often take building extraction as a semantic segmentation problem and cannot distinguish different building individuals. Thus, there is still room of improvement in extraction accuracy. In recent years, deep learning methods based on multitask learning have been widely used in the field of computer vision, but its application in automatic interpretation of high-resolution remote sensing images has not yet further developed. The instance segmentation branch of Mask R-CNN is built on the basis of target detection, and can predict segmentation masks on each region of interest. However, some spatial details and the contextual information of the edge pixels of the region of interest will be lost inevitably. The semantic segmentation task can introduce more contextual information to the network. Therefore, the integration of semantic segmentation and instance segmentation tasks can improve the generalization performance of the whole network. Based on the classic instance segmentation method (Mask R-CNN) and a typical semantic segmentation method (U-Net), this research designs a deep neural network structure which embeds the semantic segmentation module into the instance segmentation framework, and improves the generalization performance of the model by using the information complementarity between various tasks. The bottom-up path augmentation structure shortens the path of lower layers’ information to pass up. The adaptive feature pooling makes it possible for instance segmentation network to make full use of multi-scale information. The automatic building segmentation of remote sensing images is performed in the multi-task training mode and the proposed method is verified on the classic remote sensing image data set (SpaceNet). The result shows that the building instance segmentation accuracy of our proposed method is 58.8% in the Paris data set and 60.7% in the Khartoum data set, increased by 1%~2% compared to individual Mask R-CNN and U-Net. The disadvantages of the proposed method are shown in two aspects, one is that the false extraction and missing extraction of small buildings are relatively high, and the other is that the accuracy of building boundary extraction needs to be improved.

Key words: deep learning, multi-task learning, semantic segmentation, instance segmentation, remote sensing, building extraction, Mask R-CNN, U-Net

朱盼盼, 李帅朋, 张立强, 李洋. 基于多任务学习的高分辨率遥感影像建筑提取[J]. 地球信息科学学报, 2021, 23(3): 514-523.DOI:10.12082/dqxxkx.2021.190805

ZHU Panpan, LI Shuaipeng, ZHANG Liqiang, LI Yang. Multitask Learning-based Building Extraction from High-Resolution Remote Sensing Images[J]. Journal of Geo-information Science, 2021, 23(3): 514-523.DOI:10.12082/dqxxkx.2021.190805

图/表 11

图1

图2

图3

表1

图4

表2

表3

图5

图6

图7

图8

参考文献 37

[1]	王俊, 秦其明, 叶昕, 等. 高分辨率光学遥感图像建筑物提取研究进展[J]. 遥感技术与应用, 2016,31(4):653-662.
	[ Wang J, Qin Q M, Ye X, et al. A survey of building extraction methods from optical high resolution remote sensing imagery[J]. Remote Sensing Technology and Application, 2016,31(4):653-662. ]
[2]	刘帆. 一种高分辨率遥感影像建筑物提取方法研究[J]. 中国新技术新产品, 2018,13(7):18-19.
	[ Liu F. A method of building extraction from high resolution remote sensing image[J]. New Technology & New Products of China, 2018,13(7):18-19. ]
[3]	杨安妮, 许亚辉, 苏红军. 结合建筑指数的城市建筑用地提取与变化检测分析[J]. 测绘与空间地理信息, 2014,37(8):30-34.
	[ Yang A N, Xu Y H, Su H J. Urban built-up land extraction and change detection analysis using built -up indexes[J]. Geomatics & Spatial Information Technology, 2014,37(8):30-34. ]
[4]	Boonpook W, Tan Y M, Ye Y H, et al. A deep learning approach on building detection from unmanned aerial vehicle-based images in riverbank monitoring[J]. Sensors, 2018,18(11):3921.
[5]	Gavankar N L, Ghosh S K. Automatic building footprint extraction from high-resolution satellite image using mathematical morphology[J]. European Journal of Remote Sensing, 2018,51(1):182-193.
[6]	Huang X, Zhang L P. Morphological building/shadow index for building extraction from high-resolution imagery over urban areas[J]. IEEE Journal of Selected Topics in Applied Earth Observations & Remote Sensing, 2011,5:161-172.
[7]	Jabri S, Zhang Y, Alaeldin S. Stereo-based building detection in very high resolution satellite imagery using IHS color system[J]. Geoscience and Remote Sensing Symposium (IGARSS), 2014: 2301-2304.
[8]	Sirmacek B, Unsalan C. Urban-area and building detection using sift keypoints and graph theory[J]. IEEE Transactions on Geoscience & Remote Sensing, 2009,47(4):1156-1167.
[9]	Tournaire O, Brédif M, Boldo D, et al. An efficient stochastic approach for building footprint extraction from digital elevation models[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2010,65(4):317-327.
[10]	李青, 李玉, 王玉, 等. 利用格式塔的高分辨率遥感影像建筑物提取[J]. 中国图象图形学, 2017,22(8):1162-1174.
	[ Li Q, Li Y, Wang Y, et al. Building extraction from high resolution remote sensing image by using Gestalt[J]. Journal of Imageand Graphics, 2017,22(8):1162-1174. ]
[11]	Yuan J Y.Automatic building extraction in aerial scenes using convolutional networks[J]. arXiv Preprint arXiv: 1602. 06564, 2016.
[12]	Sohn G, Dowman I. Data fusion of high-resolution satellite imagery and LiDAR data for automatic building extraction[J]. ISPRS Journal of Photogram metry and Remote Sensing, 2007,62(1):43-63.
[13]	Zhai W, Shen H F, Huang C L, et al. Fusion of polarimetric and texture information for urban building extraction from fully polarimetric SAR imagery[J]. Remote Sensing Letters, 2016,7(1):31-40.
[14]	Maruyama Y, Tashiro A, Yamazaki F. Use of digital surface model constructed from digital aerial images to detect collapsed buildings during earthquake[J]. Procedia Engineering, 2011,14:552-558.
[15]	Pan X R, Gao L R, Marinoni A, et al. Semantic labeling of high resolution aerial imagery and Lidar data with fine segmentation network[J]. Remote Sensing, 2018,10(5):743.
[16]	Alshehhi R, Marpu P R, Woon W L, et al. Simultaneous extraction of roads and buildings in remote sensing imagery with convolutional neural networks[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017,130:139-149.
[17]	Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 3431-3440.
[18]	Maggiori E, Tarabalka Y, Charpiat G, et al. High-resolution aerial image labeling with convolutional neural networks[J]. IEEE Transactions on Geoscience and Remote Sensing. 2017,55(12):7092-7103.
[19]	Kaiser P, Wegner J D, Lucchi A, et al. Learning aerial image segmentation from online maps[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017,55(11):6054-6068.
[20]	Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation [C]//International Conference on Medical Image Computing and Computer-assisted Intervention. Springer, Cham, 2015: 234-241.
[21]	Wu G, Shao X, Guo Z, et al. Automatic building segmentation of aerial imagery using multi-constraint fully convolutional networks[J]. Remote Sensing, 2018,10(3):407.
[22]	惠健, 秦其明, 许伟, 等. 基于多任务学习的高分辨率遥感影像建筑实例分割[J]. 北京大学学报(自然科学版), 2019,55(6):1-12.
	[ Hui J, Qin Q M, Xu W, et al. Instance segmentation of buildings from high-resolution remote sensing images with multitask learning[J]. Acta Scientiarum Naturalium Universitatis Pekinensis, 2019,55(6):1-12. ]
[23]	Bischke B, Helber P, Folz J, et al. Multi-task learning for segmentation of building footprints with deep neural networks [C]//2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019: 1480-1484.
[24]	Lounici K, Pontil M, Tsybakov A B, et al. Taking advantage of sparsity in multi-task learning[J]. arXiv preprint arXiv: 0903. 1468, 2009.
[25]	Zhang Y, Yeung D Y. A convex formulation for learning task relationships in multi-task learning[J]. arXiv preprint arXiv: 1203. 3536, 2012.
[26]	Mou L C, Zhu X X. Vehicle instance segmentation from aerial image and video using a multitask learning residual fully convolutional network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018,56(11):6699-6711.
[27]	He K M, Gkioxari G, Dollár P, et al. Mask r-cnn [C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 2961-2969.
[28]	Ren S Q, He K M, Girshick R, et al. Faster r-cnn: Towards real-time object detection with region proposal networks [C]//Advances in Neural Information Processing Systems, 2015: 91-99.
[29]	Liu S, Qi L, Qin H F, et al. Path aggregation network for instance segmentation [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 8759-8768.
[30]	Lin T Y, Dollár P, Girshick R, et al. Feature pyramid networks for object detection [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2117-2125.
[31]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[32]	Sergey I, Christian S. Batch normalization: Accelerating deep network training by reducing internal covariate shift [C]// International Conference on International Conference on Machine Learning. JMLR.org, 2015.
[33]	Deng J, Dong W, Socher R, et al. Imagenet: A large-scale hierarchical image database [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009: 248-255.
[34]	Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks [C]//Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 2010: 249-256.
[35]	Lin T Y, Maire M, Belongie S, et al. Microsoft COCO: Common Objects in Context[J]. Lecture Notes in Computer Science, 2014: 740-755.
[36]	Adam V E, Dave L, Bacastow T. Spacenet: A remote sensing dataset and challenge series. 2019. arXiv preprint arXiv:1807.01232. http://explore.digitalglobe.com/spacenet.
[37]	Everingham M, Van G L, Williams C K I, et al. The pascal visual object classes (voc) challenge[J]. International Journal of Computer Vision, 2010,88(2):303-338.

地区	面积/km²	建筑物数目/个
巴黎	1030	23 816
喀土穆	765	35 503

	实验组	平均值	人	骑手	汽车	卡车	公交车	火车	摩托车	自行车
AP_DET	S	37.8	33.5	29.5	50.1	33.3	41.3	30.0	25.6	22.4
AP_DET	M	38.3	34.5	30.1	51.3	32.7	41.9	33.3	26.9	20.1
AP_MASK	S	34.7	31.4	28.7	49.0	30.6	39.9	29.2	23.1	17.9
AP_MASK	M	35.6	33.3	28.5	49.2	29.1	40.1	30.4	23.9	19.1

方法	巴黎			喀土穆
方法	AP_DET	AP_SEM	AP_I_NS	AP_DET	AP_SEM	AP_INS
U-Net	-	60.8	-	-	60.1	-
Mask R-CNN	88.2	-	57.9	88.0	-	58.9
本文的方法（S）	88.8	61.3	58.4	87.6	59.8	60.7
本文的方法（M）	90.1	62.9	58.8	89.0	61.3	59.0
ALL	91.8	63.4	60.7	90.6	62.7	61.5

基于多任务学习的高分辨率遥感影像建筑提取

Multitask Learning-based Building Extraction from High-Resolution Remote Sensing Images

RichHTML

PDF (PC)

赞

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 37

相关文章 15

Metrics

本文评价

推荐阅读 0

[1]	郭紫甜, 王春梅, 刘欣, 庞国伟, 朱梦阳, 王晋卿. 基于小流域抽样单元的中国FROM-GLC30数据精度评价[J]. 地球信息科学学报, 2021, 23(3): 524-535.
[2]	陈昂, 杨秀春, 徐斌, 金云翔, 张文博, 郭剑, 邢晓语, 杨东. 基于面向对象与深度学习的榆树疏林识别方法研究[J]. 地球信息科学学报, 2020, 22(9): 1897-1909.
[3]	王学文, 赵庆展, 韩峰, 马永建, 龙翔, 江萍. 机载多光谱影像语义分割模型在农田防护林提取中的应用[J]. 地球信息科学学报, 2020, 22(8): 1702-1713.
[4]	赵青松, 兰安军, 范泽孟, 杨青. 贵州省不同地貌形态类型土壤侵蚀强度变化的定量分析[J]. 地球信息科学学报, 2020, 22(7): 1555-1566.
[5]	崔成, 任红艳, 赵璐, 庄大方. 基于街景影像多特征融合的广州市越秀区街道空间品质评估[J]. 地球信息科学学报, 2020, 22(6): 1330-1338.
[6]	毛文山, 赵红莉, 孙凤娇, 蒋云钟, 姜倩, 朱彦儒. 基于Item2Vec负采样优化的专题地图产品个性化推荐方法研究[J]. 地球信息科学学报, 2020, 22(11): 2128-2139.
[7]	何红术, 黄晓霞, 李红旮, 倪凌佳, 王新歌, 陈崇, 柳泽. 基于改进U-Net网络的高分遥感影像水体提取[J]. 地球信息科学学报, 2020, 22(10): 2010-2022.
[8]	郭子慧, 刘伟. 深度学习和遥感影像支持的矢量图斑地类解译真实性检查方法[J]. 地球信息科学学报, 2020, 22(10): 2051-2061.
[9]	宋关福, 卢浩, 王晨亮, 胡辰璞, 黄科佳. 人工智能GIS软件技术体系初探[J]. 地球信息科学学报, 2020, 22(1): 76-87.
[10]	蔡博文,王树根,王磊,邵振峰. 基于深度学习模型的城市高分辨率遥感影像不透水面提取[J]. 地球信息科学学报, 2019, 21(9): 1420-1429.
[11]	王兴,康俊锋,刘学军,王美珍,张超. 设施农业典型地物改进Faster R-CNN识别方法[J]. 地球信息科学学报, 2019, 21(9): 1444-1454.
[12]	李泽宇,明冬萍,范莹琳,赵林峰,刘思民. 遥感影像监督分割评价指标比较与分析[J]. 地球信息科学学报, 2019, 21(8): 1265-1274.
[13]	蔡惠慧, 朱伟, 李孜轩, 刘园园, 李龙斌, 刘畅. 基于深度学习的钨钼找矿靶区预测方法研究[J]. 地球信息科学学报, 2019, 21(6): 928-936.
[14]	方颖, 李连发. 基于机器学习的高精度高分辨率气象因子时空估计[J]. 地球信息科学学报, 2019, 21(6): 799-813.
[15]	张刚, 刘文彬, 张男. 基于区域特征分割的密集匹配点云渐进形态学滤波[J]. 地球信息科学学报, 2019, 21(4): 615-622.