基于卫星影像全局和局部深度学习特征检索的无人机绝对定位方法

doi:10.12082/dqxxkx.2023.220827

Abstract

Abstract:

With the development of Unmanned Aerial Vehicle (UAV) technology, it has been applied to various tasks in different fields. The prerequisite for a UAV to perform successful aerial tasks is accurate localization of its own position. Generally, traditional UAV navigation relies on the Global Navigation Satellite System (GNSS) for localization. However, this system has disadvantages such as instability and susceptibility to interference, leading to situations where UAV cannot use GNSS for positioning, known as GNSS-denied environments. This study focuses on the navigation and positioning of UAV in GNSS-denied environments and proposes a UAV visual retrieval and positioning method that comprehensively utilizes local and global deep learning features of known satellite orthophotos. Specifically, ConvNeXt is used as the backbone network, combined with generalized mean pooling, to form a retrieval feature extraction algorithm for extracting global features of satellite and UAV images. A triplet loss function considering the overlapping area between images is designed for the retrieval and positioning tasks, and a corresponding training data set is established to train the feature extraction algorithm. Then, the satellite images within a certain range are retrieved according to the extracted global features, and the preliminary retrieval results are obtained. In order to further improve the accuracy of the retrieved target images, the LoFTR algorithm based on deep learning local features is used for matching and reordering. Since the LoFTR algorithm has many mismatches, RANSAC is used to screen the matching results. Experiments using the test datasets we established demonstrate that the proposed method obtains an average accuracy of 90.9% and an average time cost of 2.22 seconds for retrieving satellite images in different seasons from fully overlapped UAV simulated images. The accuracy of the UAV real image test is 87.5%, which can meet the UAV positioning requirements.

Key words: GNSS-denied, deep learning features, convolutional neural network, image retrieval, visual localization, Unmanned Aerial Vehicle, local feature, global feature

HOU Huitai, LAN Chaozhen, XU Qing. UAV Absolute Positioning Method based on Global and Local Deep Learning Feature Retrieval from Satellite Images[J].Journal of Geo-information Science, 2023, 25(5): 1064-1074.DOI:10.12082/dqxxkx.2023.220827

Figures/Tables 13

Fig. 1

Tab. 1

ConvNeXt - B Model Architecture

层名称	输出大小/像素	Convnext-B
conv1	56×56	4×4,128,步长4
conv2_x	56×56	$7 × 7 128 1 × 1 512 1 × 1 128 × 3$
conv3_x	28×28	$7 × 7 256 1 × 1 1024 1 × 1 256 × 3$
conv4_x	14×14	$7 × 7 512 1 × 1 2048 1 × 1 512 × 27$
conv5_x	7×7	$7 × 7 1024 1 × 1 4096 1 × 1 1024 × 3$

Tab. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Tab. 2

Fig. 6

Fig. 7

Tab. 3

Fig. 8

Fig. 9

Tab. 4

References 28

[1]	Conte G, Doherty P. An integrated UAV navigation system based on aerial image matching[C]// 2008 IEEE Aerospace Conference. IEEE, 2008:1-10. DOI:10.1109/AERO.2008.4526556 doi: 10.1109/AERO.2008.4526556
[2]	Viswanathan A, Pires B R, Huber D. Vision-based robot localization across seasons and in remote locations[C]// 2016 IEEE International Conference on Robotics and Automation (ICRA). 2016:4815-4821. DOI:10.1109/ICRA. 2016.7487685 doi: 10.1109/ICRA. 2016.7487685
[3]	Coopmans C. Architecture, inertial navigation, and payload designs for low-cost unmanned aerial vehicle-based personal remote sensing[M]. Utah State University, 2010.
[4]	Hou H T, Xu Q, Lan C Z, et al. UAV pose estimation in GNSS-denied environment assisted by satellite imagery deep learning features[J]. IEEE Access, 2020, 9:6358-6367. DOI:10.1109/ACCESS.2020.3048342 doi: 10.1109/ACCESS.2020.3048342
[5]	赵春晖, 周昳慧, 林钊, 等. 无人机景象匹配视觉导航技术综述[J]. 中国科学:信息科学, 2019, 49(5):507-519.
	[ Zhao C H, Zhou I, Lin Z, et al. Review of scene matching visual navigation for unmanned aerial vehicles[J]. Scientia Sinica (Informationis), 2019, 49(5):507-519. ]
[6]	Yol A, Delabarre B, Dame A, et al. Vision-based absolute localization for unmanned aerial vehicles[C]// 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2014:3429-3434. DOI:10.1109/IROS.2014.6943040 doi: 10.1109/IROS.2014.6943040
[7]	Goforth H, Lucey S. GPS-denied UAV localization using pre-existing satellite imagery[C]// 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019:2974-2980. DOI:10.1109/ICRA.2019.8793558 doi: 10.1109/ICRA.2019.8793558
[8]	Zheng Z D, Wei Y C, Yang Y.University-1652: A multi-view multi-source benchmark for drone-based geo-localization[C]// MM '20:Proceedings of the 28th ACM International Conference on Multimedia. 2020:1395-1403. DOI:10.1145/3394171.3413896 doi: 10.1145/3394171.3413896
[9]	秦剑琪, 蓝朝桢, 崔志祥, 等. 一种面向无人机绝对定位的基准卫星影像检索方法[J]. 武汉大学学报·信息科学版, 2021.
	[ Qin J, Lan C, Cui Z, et al. A Reference Satellite Image Retrieval Method for Drone Absolute Positioning[J]. Geomatics and Information Science of Wuhan University, 2021. ] DOI:10.13203/j.whugis20200229 doi: 10.13203/j.whugis20200229
[10]	Demir B, Bruzzone L. A novel active learning method in relevance feedback for content-based remote sensing image retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(5):2323-2334. DOI:10.1109/TGRS.2014.2358804 doi: 10.1109/TGRS.2014.2358804
[11]	Liu C, Ma J J, Tang X, et al. Deep hash learning for remote sensing image retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(4):3420-3443. DOI:10.1109/TGRS.2020.3007533 doi: 10.1109/TGRS.2020.3007533
[12]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2016:770-778. DOI:10.1109/CVPR.2016.90 doi: 10.1109/CVPR.2016.90
[13]	Zhao Z Q, Zheng P, Xu S T, et al. Object detection with deep learning: A review[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(11):3212-323 2. DOI:10.1109/TNNLS.2018.2876865 doi: 10.1109/TNNLS.2018.2876865
[14]	蓝朝桢, 卢万杰, 于君明, 等. 异源遥感影像特征匹配的深度学习算法[J]. 测绘学报, 2021, 50(2):189-202.
	[ Lan C Z, Lu W J, Yu J M, et al. Deep learning algorithm for feature matching of cross modality remote sensing images[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(2):189-202. ]
[15]	He K M, Gkioxari G, Dollár P, et al. Mask R-CNN[C]// 2017 IEEE International Conference on Computer Vision. IEEE, 2017:2980-2988. DOI:10.1109/ICCV.2017.322 doi: 10.1109/ICCV.2017.322
[16]	Babenko A, Slesarev A, Chigorin A, et al. Neural Codes for Image Retrieval[C]// European Conference on Computer Vision. Cham. Springer, 2014:584-599. DOI:10.1007/978-3-319-10590-1_38 doi: 10.1007/978-3-319-10590-1_38
[17]	Kalantidis Y, Mellina C, Osindero S. Cross-Dimensional weighting for aggregated deep convolutional features[C]// European Conference on Computer Vision. Cham. Springer, 2016:685-701. DOI:10.1007/978-3-319-46604-0_48 doi: 10.1007/978-3-319-46604-0_48
[18]	Noh H, Araujo A, Sim J, et al. Large-scale image retrieval with attentive deep local features[C]// 2017 IEEE International Conference on Computer Vision. IEEE, 2017:3476-3485. DOI:10.1109/ICCV.2017.374 doi: 10.1109/ICCV.2017.374
[19]	Liu Z, Mao H Z, Wu C Y, et al. A ConvNet for the 2020s[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022:11966-11976. DOI:10.1109/CVPR52688.2022.01167 doi: 10.1109/CVPR52688.2022.01167
[20]	Radenovic F, Tolias G, Chum O. Fine-tuning CNN image retrieval with no human annotation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(7):1655-1668. DOI:10.1109/tpami.2018.2846566 doi: 10.1109/TPAMI.2018.2846566 pmid: 29994246
[21]	Liu Z, Lin Y T, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]// 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021:9992-10002. DOI:10.1109/ICCV48922.2021.00986 doi: 10.1109/ICCV48922.2021.00986
[22]	Sun J M, Shen Z H, Wang Y A, et al. LoFTR: detector-free local feature matching with transformers[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021:8918-8927. DOI:10.1109/CVPR46437.2021.00881 doi: 10.1109/CVPR46437.2021.00881
[23]	Fischler M A, Bolles R C. Random sample consensus[J]. Communications of the ACM, 1981, 24(6):381-395. DOI:10.1145/358669.358692 doi: 10.1145/358669.358692
[24]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need;proceedings of the Advances in neural information processing systems[C]// 31st Conference on Neural Information Processing Systems (NIPS 2017). Long Beach, CA, USA, 2017:6000-6010. DOI:10.48550/arXiv.1706.03762 doi: 10.48550/arXiv.1706.03762
[25]	Yang Y, Newsam S. Bag-of-visual-words and spatial extensions for land-use classification[C]// GIS '10: Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems. 2010:270-279. DOI:10.1145/1869790.1869829 doi: 10.1145/1869790.1869829
[26]	Schönberger J L, Radenović F, Chum O, et al. From single image query to detailed 3D reconstruction[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2015:5126-5134. DOI:10.1109/CVPR.2015.7299148 doi: 10.1109/CVPR.2015.7299148
[27]	Brown M, Hua G, Winder S. Discriminative learning of local image descriptors[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(1):43-57. DOI:10.1109/TPAMI.2010.54 doi: 10.1109/TPAMI.2010.54 pmid: 21088318
[28]	Kingma D P, Ba J. Adam: A method for stochastic optimization[EB/OL]. 2014: arXiv: 1412.6980. https://arxiv.org/abs/1412.6980

重叠率	100	75	56	50	25
第一组	69.6	38.8	27.7	14.3	4.9
第二组	64.2	33.9	29.5	13.8	3.1
第三组	71.4	39.7	27.2	12.5	6.7

重叠率	100	75	56	50	25
第一组	92.9	75.4	62.1	32.6	12.1
第二组	89.7	69.6	56.7	33.0	9.8
第三组	90.2	68.3	55.8	27.7	9.8

无人机影像	第一组卫星影像		第二组卫星影像		第三组卫星影像
无人机影像	重排序前ToP1	重排序后ToP1	重排序前ToP1	重排序后ToP1	重排序前ToP1	重排序后ToP1
1-1	错误	正确	正确	正确	正确	正确
1-2	错误	正确	错误	错误	错误	正确
1-3	错误	错误	错误	正确	错误	正确
1-4	正确	正确	错误	错误	正确	正确
2-1	正确	正确	正确	正确	错误	正确
2-2	正确	正确	正确	正确	错误	正确
2-3	正确	正确	正确	正确	错误	正确
2-4	错误	正确	正确	正确	正确	正确

UAV Absolute Positioning Method based on Global and Local Deep Learning Feature Retrieval from Satellite Images

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 28

Related Articles 15

Metrics

Comments

Recommended 0

[1]	HUANG Gaoshuang, ZHOU Yang, HU Xiaofei, ZHAO Luying, ZHANG Chenglong. A Survey of the Research Progress in Image Geo-localization [J]. Journal of Geo-information Science, 2023, 25(7): 1336-1362.
[2]	LING Zhenfei, LIU Tao, DU Ping, ZHAO Dan, CHEN Puyi, MA Tianen. A Variational Graph Convolutional Auto-encoder Model Supporting Similarity Retrieval of Building Groups Patterns [J]. Journal of Geo-information Science, 2023, 25(7): 1405-1417.
[3]	LIU Xiao, LIU Zhi, LIN Yuzhun, WANG Shuxiang, ZUO Xibing. Class-centric Knowledge Distillation for RSI Scene Classification [J]. Journal of Geo-information Science, 2023, 25(5): 1050-1063.
[4]	CHEN Ke, GUAN Haiyan, LEI Xiangda, CAO Shuang. A Multispectral LiDAR Point Cloud Classification Method based on Enhanced Features Kernel Point Convolutional Network [J]. Journal of Geo-information Science, 2023, 25(5): 1075-1087.
[5]	HENG Xuebiao, XU Hanwei, TANG Lu, TANG Heng, XU Yilei. Research on Land Cover Classification Method based on Improved Fully Convolutional Neural Network Model [J]. Journal of Geo-information Science, 2023, 25(3): 495-509.
[6]	XU Fanshu, WANG Baoyun, HAN Jun. A Potential Gully Debris Flows Hazard Assessment Method: A CNN Model based on Multi-source Data Fusion [J]. Journal of Geo-information Science, 2023, 25(3): 588-605.
[7]	GAO Jianwen, GUAN Haiyan, PENG Daifeng, XU Zhengsen, KANG Jian, JI Yating, ZHAI Ruoxue. Local-global Semantic Feature Enhancement Model for Remote Sensing Imagery Change Detection [J]. Journal of Geo-information Science, 2023, 25(3): 625-637.
[8]	GAO Pengfei, CAO Xuefeng, LI Ke, YOU Xiong. Object Detection in Remote Sensing Images by Fusing Multi-neuron Sparse Features and Hierarchical Depth Features [J]. Journal of Geo-information Science, 2023, 25(3): 638-653.
[9]	HUANG Shuaiyuan, DONG Youfu, LI Haipeng. Establishment and Comparative Analysis of SRTM1 DEM Error Correction Model in the Loess Plateau [J]. Journal of Geo-information Science, 2023, 25(3): 669-681.
[10]	HUANG Xin, MAO Zhengyuan. Prediction of Passenger Demand for Online Car-hailing based on Spatio-temporal Multi-graph Convolution Network [J]. Journal of Geo-information Science, 2023, 25(2): 311-323.
[11]	LIU Yang, KANG Jian, GUAN Haiyan, WANG Hanyun. Road Extraction Model of High-resolution Remote Sensing Images based on Dual-attention Residual Network [J]. Journal of Geo-information Science, 2023, 25(2): 396-408.
[12]	HE Haiqing, ZHOU Fuyang, CHEN Min, CHEN Ting, GUAN Yunlan, ZENG Huaien, WEI Yan. Fruit Tree Canopy Segmentation by Unmanned Aerial Vehicle Photogrammetry Coupled on Convolutional Neural Network and Attention Mechanism [J]. Journal of Geo-information Science, 2023, 25(12): 2387-2401.
[13]	WU Xinhui, MAO Zhengyuan, WENG Qian, SHI Wenzao. A Neural Network based on Residual Multi-attention and ACON Activation Function for Extract Buildings [J]. Journal of Geo-information Science, 2022, 24(4): 792-801.
[14]	JIAO Yangyang, LIU Pingzhi, LIU Ailong, LIU Songlin. Map Building Shape Classification Method based on AlexNet [J]. Journal of Geo-information Science, 2022, 24(12): 2333-2341.
[15]	ZHU Yu, PAN Yaozhong, ZHANG Dujuan. An Agriculture Parcel Identification Method based on Convolutional Neural Network and Watershed Segmentation [J]. Journal of Geo-information Science, 2022, 24(12): 2389-2403.