基于自定义RDD的海量遥感图像并行镶嵌方法

doi:10.3724/SP.J.1047.2017.01346

摘要/Abstract

摘要：

图像镶嵌是遥感图像处理中的重要内容,在跨区域遥感图像分析中发挥重要作用。为了解决传统遥感图像并行算法中存在的计算节点利用率低、频繁数据I/O等问题,本文根据Spark分布式内存计算框架,充分利用Spark利于迭代数据处理的优势,提出了一种基于Spark自定义RDD（弹性分布式数据集）的并行镶嵌方法。该方法首先在集群的多个节点上通过相位相关法执行图像重叠区域估计操作,从而提高了图像重叠区域估计的多节点并行计算;然后,通过重写Spark中RDD的compute和getPartitions方法,自定义针对遥感图像处理的RDD,并将图像镶嵌中的重叠区域估计、图像配准和图像融合3个关键步骤作为自定义RDD的Transformation类型的操作算子;最后,通过隐式转换创建自定义RDD,并调用自定义RDD的操作算子实现图像镶嵌的并行处理。实验结果表明,与传统基于MPI的并行镶嵌算法相比,该方法在保证图像镶嵌效果的基础上,能够有效提高大数据量的图像镶嵌效率。

关键词: 遥感图像, 并行镶嵌, Spark, 相位相关法, 自定义RDD

Abstract:

Image mosaicking is an important part of remote sensing image processing. It plays a vital role in the analysis of trans-regional remote sensing images. In order to solve the problems of low utilization rates of the nodes and frequent data I/O in the traditional parallel algorithms of remote sensing images, we proposed a parallel mosaicking algorithms based on self-defined RDD (Resilient Distributed Datasets), in which the Spark distributed memory computing framework has been used. In this paper, we take full advantage of the Spark, which is conducive to the processing of iterative data, and build remote sensing images parallel mosaic processing model through the operation of the Spark RDD. Firstly, according to the logical separability and data independence of the Fourier transform and inverse Fourier transform in the phase correlation method, we improved the traditional phase correlation method by executing a single instruction on multiple nodes, which are executed parallel in the cluster. We did so to improve the image overlapping region estimation multi-node parallel computation in the algorithm. Then, we override the compute and getPartitions methods in RDD and self-define the RDD for remote sensing image processing. Meanwhile, we used the three key steps of the image mosaicking, including overlapping region estimation, image registration and image fusion, which are the transformation-type operators of the self-defined RDD. These transformation-type operators do not perform calculations in the process of parallel mosaicking, until the final mosaicking image is required to be written to disk or file system. Thus, reducing the time consumption in the process of image parallel mosaicking. Finally, the parallel processing of image mosaicking is realized by calling the operators of self-defined RDD with the method of implicit conversion, compared with the parallel mosaicking algorithm based on MPI. The experimental results show that the parallel mosaicking algorithm of massive remote sensing image based on self-defined RDD can effectively improve the image mosaicking efficiency of large data volume on the basis of guaranteeing the image mosaicking effects.

Key words: remote sensing images, parallel mosaicking, spark, phase correlation methods, self-defined RDD

景维鹏, 霍帅起. 基于自定义RDD的海量遥感图像并行镶嵌方法[J]. 地球信息科学学报, 2017, 19(10): 1346-1354.DOI:10.3724/SP.J.1047.2017.01346

JING Weipeng,HUO Shuaiqi. A Model of Parallel Mosaicking for Massive Remote Sensing Images Based on Self-defined RDD[J]. Journal of Geo-information Science, 2017, 19(10): 1346-1354.DOI:10.3724/SP.J.1047.2017.01346

图/表 9

图1

图2

算法1：

图3

图4

图5

图6

图7

图8

参考文献 26

[1]	Franklin S E, Wulder M A.Remote sensing methods in medium spatial resolution satellite data land cover classification of large areas[J]. Progress in Physical Geography, 2002,26(2):173-205. doi: 10.1191/0309133302pp332ra
[2]	Scheidt S, Ramsey M, Lancaster N.Radiometric normalization and image mosaic generation of ASTER thermal infrared data: An application to extensive sand sheets and dune fields[J]. Remote Sensing of Environment, 2008,112(3):920-933. doi: 10.1016/j.rse.2007.06.020
[3]	Bielski C, Grazzini J, Soille P.Automated morphological image composition for mosaicing large image data sets[C]. Geoscience and Remote Sensing Symposium, 2007. IGAR-SS 2007. IEEE International, 2007:4068-4071.
[4]	Liu Z, Gong P, Shi P J.Study on image automatic registration based on layered multi-template matching[J]. Computer Applications, 2005,25(2):193-203.
[5]	Szeliski R.Video mosaics for virtual environments[J]. IEEE Computer Graphics& Applications, 1996,16(2):22-30. doi: 10.1109/38.486677
[6]	Shum H Y, He L W.Rendering with concentric mosaics[C]. Proceeding of Siggraph'99, Los Angeles, California, 1999:8-13.
[7]	Lowe D G.Object recognition from local scale-invariant features[C]. Proceedings of the 7th International Conference on Computer Vision, 1999:1150-1157.
[8]	Brown M, Lowe D G.Recognizing panoramas[C]. Proceedings of IEEE International Conference on Computer Vision, Washington, 2003:1218-1225.
[9]	Afek Y, Brand A.Mosaicking of orthorectified aerial images[J]. Photogrammetric Engineering & Remote Sensing, 1998,64(2):115-125. doi: 10.1144/petgeo.4.1.95
[10]	朱述龙,钱曾波.遥感影像镶嵌时拼接缝的消除方法[J].遥感学报,2002,6(3):183-187. doi: 10.11834/jrs.20020305
	[ Zhu S L, Qian Z B.The seam-line removal under mosaicking of remotely sensed images[J]. Journal of Remote Sensing, 2002,6(3):183-187. ] doi: 10.11834/jrs.20020305
[11]	张剑清,柯涛,孙明伟.基于集群计算机的海量航空数码影像并行处理—并行计算在航空数字摄影测量中的应用[J]计算机工程与应用,2008,44(13):12-15. doi: 10.3778/j.issn.1002-8331.2008.13.004
	[ Zhang J Q, Ke T, Sun M W.Parallel processing of mass aerial digital images based on cluster computer-the application of parallel computing in aerial digital photogrammetry[J]. Computer Engineering and Applications, 2008,44(13):12-15. ] doi: 10.3778/j.issn.1002-8331.2008.13.004
[12]	安兴华,王小鸽,都志辉,等.一种适用于机群系统的细粒度遥感图像镶嵌并行算法[J].清华大学学报(自然科学版),2002,42(10):1389-1392. doi: 10.3321/j.issn:1000-0054.2002.10.031
	[ An X H, Wang X G, Du Z H, et al.Fine-grained parallel algorithm for remote sensing image mosaics for cluster system[J]. Journal of Tsinghua University (Science and Technology), 2002,42(10):1389-1392. ] doi: 10.3321/j.issn:1000-0054.2002.10.031
[13]	陈晨,谭毅华,李海涛,等.遥感图像快速镶嵌并行算法研究[J].微电子学与计算机,2011,28(3):59-62.
	[ Chen C, Tan Y H, Li H T, et al.A fast and automatic parallel algorithm of remote sensing image mosaic[J]. Microelectronics and Computer, 2011,28(3): 59-62. ]
[14]	Zhao W.Flexible image blending for image mosaicing with reduced artifacts[J]. International Journal of Pat-tern Recognition and Artificial Intelligence, 2006,20(4):609-628. doi: 10.1142/S0218001406004806
[15]	王妍颖,马艳,刘定生.一种基于动态分组策略和多线程并行IO的并行镶嵌算法优化[J].遥感信息,2012(2):3-8.
	[ Wang Y Y, Ma Y, Liu D S.Optimization of image mosaic algorithm based on parallel I/O and dynamic grouping[J]. Remote Sensing Information, 2012,2:3-8. ]
[16]	Ma Y, Wang L, Zomaya A Y, et al.Task-tree based large-scale mosaicking for massive remote sensed imageries with dynamic DAG scheduling[J]. IEEE Transactions on Parallel & Distributed Systems, 2014,25(8):2126-2137. doi: 10.1109/TPDS.2013.272
[17]	Tabbb Y, Medouri A, Tetouan M.Towards a next gereration of scientific computing in the cloud[J]. International of Computer Science, 2012,9(6):177-183.
[18]	夏俊鸾. Spark大数据处理技术[M].北京:电子工业出版社,2015.
	[ Xia J L.Big data processing technology with spark[M]. Beijing: Electronic Industry Press, 2015. ]
[19]	高彦杰. Spark大数据处理:技术、应用与性能优化[M].北京:机械工业出版社,2014.
	[ Gao Y J.Data processing with spark technology, application and performance optimization[M]. Beijing: China Machine Press, 2014. ]
[20]	Kuglin C D.The phase correlation image alignment method[J]. Proc.intl Conf.cybernetics & Society, 1975:163-165.
[21]	Zaharia M, Chowdhusry M, Das T, et al.Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing[C]// Usenix Conference on Networked Systems Design and Implementation. 2012:141-146.
[22]	Zaharia M, Das T, Li H, et al.Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters[C]//Proceedings of the 4th USENIX conference on Hot Topics in Cloud Computing. USENIX Association, 2012:10-10.
[23]	Engle C, Lupher A, Xin R, et al.Shark: fast data analysis using coarse-grained distributed memory[C]//Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. ACM. 2012:689-692.
[24]	Xin R S. Rosen J. Zaharia M, et al.Shark: SQL and rich analytics at scale[C]//Proceedings of the 2013 ACM SIGMOD International Conference on Management of data. ACM, 2013:13-24.
[25]	Ramirez-Gallego S, Garcia S, Mourino-Talin H, et al.Distributed entropy minimization discretizer for big data analysis under apache spark[C]// Trustcom/bigdatase/ispa. IEEE, 2015.
[26]	Han Z, Zhang Y.Spark: A big data processing platform based on memory computing[C]//International Symposiumon Parallel Architectures. IEEE, 2015:172-176.