地球信息科学学报 ›› 2017, Vol. 19 ›› Issue (10): 1346-1354.doi: 10.3724/SP.J.1047.2017.01346
收稿日期:
2017-02-28
修回日期:
2017-06-09
出版日期:
2017-10-20
发布日期:
2017-10-20
作者简介:
作者简介:景维鹏(1979-),男,博士,副教授,研究方向为并行计算、分布式计算、空间数据挖掘。E-mail:
基金资助:
Received:
2017-02-28
Revised:
2017-06-09
Online:
2017-10-20
Published:
2017-10-20
Contact:
JING Weipeng
E-mail:nefujwp@163.com
摘要:
图像镶嵌是遥感图像处理中的重要内容,在跨区域遥感图像分析中发挥重要作用。为了解决传统遥感图像并行算法中存在的计算节点利用率低、频繁数据I/O等问题,本文根据Spark分布式内存计算框架,充分利用Spark利于迭代数据处理的优势,提出了一种基于Spark自定义RDD(弹性分布式数据集)的并行镶嵌方法。该方法首先在集群的多个节点上通过相位相关法执行图像重叠区域估计操作,从而提高了图像重叠区域估计的多节点并行计算;然后,通过重写Spark中RDD的compute和getPartitions方法,自定义针对遥感图像处理的RDD,并将图像镶嵌中的重叠区域估计、图像配准和图像融合3个关键步骤作为自定义RDD的Transformation类型的操作算子;最后,通过隐式转换创建自定义RDD,并调用自定义RDD的操作算子实现图像镶嵌的并行处理。实验结果表明,与传统基于MPI的并行镶嵌算法相比,该方法在保证图像镶嵌效果的基础上,能够有效提高大数据量的图像镶嵌效率。
景维鹏, 霍帅起. 基于自定义RDD的海量遥感图像并行镶嵌方法[J]. 地球信息科学学报, 2017, 19(10): 1346-1354.DOI:10.3724/SP.J.1047.2017.01346
JING Weipeng,HUO Shuaiqi. A Model of Parallel Mosaicking for Massive Remote Sensing Images Based on Self-defined RDD[J]. Journal of Geo-information Science, 2017, 19(10): 1346-1354.DOI:10.3724/SP.J.1047.2017.01346
算法1:
创建针对遥感图像镶嵌的自定义RDD"
初始化:创建SparkConf对象conf,将conf作为SparkContext构造函数的参数创建SparkContext对象sc,调用sc的textFile 方法创建初始RDD |
阶段1:在自定义RDD中添加操作方法 Iterator[BufferedImage]←compute(split: Partition,context: TaskContext)//调用父RDD的iterator方法,返回一个内部 //元素类型为bufferImage的迭代器对象 Array[Partition]←firstParent[BufferedImage].partitions//调用父RDD的partitions方法,返回父RDD的分区 RDD[BufferedImage]←Image overlap region estimation//重叠区域估计方法 RDD[BufferedImage]←Image registration//图像配准方法 RDD[BufferedImage]←Image fusion//图像融合方法 阶段2:调用隐式转换的处理方法 self-definedRDD[rdd]←exchange(rdd:RDD[String])//转换类中的exchange方法由implicit关键字修饰,RDD为方法参数,//自定义RDD作为返回值 import RDDtoSelf-defiendRDD.exchange//在程序中导入声明的隐式转换的方法 阶段3:生成自定义RDD对象. imageRDD ←fileRDD.exchange//初始RDD调用exchange方法生成自定义RDD对象 |
[1] |
Franklin S E, Wulder M A.Remote sensing methods in medium spatial resolution satellite data land cover classification of large areas[J]. Progress in Physical Geography, 2002,26(2):173-205.
doi: 10.1191/0309133302pp332ra |
[2] |
Scheidt S, Ramsey M, Lancaster N.Radiometric normalization and image mosaic generation of ASTER thermal infrared data: An application to extensive sand sheets and dune fields[J]. Remote Sensing of Environment, 2008,112(3):920-933.
doi: 10.1016/j.rse.2007.06.020 |
[3] | Bielski C, Grazzini J, Soille P.Automated morphological image composition for mosaicing large image data sets[C]. Geoscience and Remote Sensing Symposium, 2007. IGAR-SS 2007. IEEE International, 2007:4068-4071. |
[4] | Liu Z, Gong P, Shi P J.Study on image automatic registration based on layered multi-template matching[J]. Computer Applications, 2005,25(2):193-203. |
[5] |
Szeliski R.Video mosaics for virtual environments[J]. IEEE Computer Graphics& Applications, 1996,16(2):22-30.
doi: 10.1109/38.486677 |
[6] | Shum H Y, He L W.Rendering with concentric mosaics[C]. Proceeding of Siggraph'99, Los Angeles, California, 1999:8-13. |
[7] | Lowe D G.Object recognition from local scale-invariant features[C]. Proceedings of the 7th International Conference on Computer Vision, 1999:1150-1157. |
[8] | Brown M, Lowe D G.Recognizing panoramas[C]. Proceedings of IEEE International Conference on Computer Vision, Washington, 2003:1218-1225. |
[9] |
Afek Y, Brand A.Mosaicking of orthorectified aerial images[J]. Photogrammetric Engineering & Remote Sensing, 1998,64(2):115-125.
doi: 10.1144/petgeo.4.1.95 |
[10] |
朱述龙,钱曾波.遥感影像镶嵌时拼接缝的消除方法[J].遥感学报,2002,6(3):183-187.
doi: 10.11834/jrs.20020305 |
[ Zhu S L, Qian Z B.The seam-line removal under mosaicking of remotely sensed images[J]. Journal of Remote Sensing, 2002,6(3):183-187. ]
doi: 10.11834/jrs.20020305 |
|
[11] |
张剑清,柯涛,孙明伟.基于集群计算机的海量航空数码影像并行处理—并行计算在航空数字摄影测量中的应用[J]计算机工程与应用,2008,44(13):12-15.
doi: 10.3778/j.issn.1002-8331.2008.13.004 |
[ Zhang J Q, Ke T, Sun M W.Parallel processing of mass aerial digital images based on cluster computer-the application of parallel computing in aerial digital photogrammetry[J]. Computer Engineering and Applications, 2008,44(13):12-15. ]
doi: 10.3778/j.issn.1002-8331.2008.13.004 |
|
[12] |
安兴华,王小鸽,都志辉,等.一种适用于机群系统的细粒度遥感图像镶嵌并行算法[J].清华大学学报(自然科学版),2002,42(10):1389-1392.
doi: 10.3321/j.issn:1000-0054.2002.10.031 |
[ An X H, Wang X G, Du Z H, et al.Fine-grained parallel algorithm for remote sensing image mosaics for cluster system[J]. Journal of Tsinghua University (Science and Technology), 2002,42(10):1389-1392. ]
doi: 10.3321/j.issn:1000-0054.2002.10.031 |
|
[13] | 陈晨,谭毅华,李海涛,等.遥感图像快速镶嵌并行算法研究[J].微电子学与计算机,2011,28(3):59-62. |
[ Chen C, Tan Y H, Li H T, et al.A fast and automatic parallel algorithm of remote sensing image mosaic[J]. Microelectronics and Computer, 2011,28(3): 59-62. ] | |
[14] |
Zhao W.Flexible image blending for image mosaicing with reduced artifacts[J]. International Journal of Pat-tern Recognition and Artificial Intelligence, 2006,20(4):609-628.
doi: 10.1142/S0218001406004806 |
[15] | 王妍颖,马艳,刘定生.一种基于动态分组策略和多线程并行IO的并行镶嵌算法优化[J].遥感信息,2012(2):3-8. |
[ Wang Y Y, Ma Y, Liu D S.Optimization of image mosaic algorithm based on parallel I/O and dynamic grouping[J]. Remote Sensing Information, 2012,2:3-8. ] | |
[16] |
Ma Y, Wang L, Zomaya A Y, et al.Task-tree based large-scale mosaicking for massive remote sensed imageries with dynamic DAG scheduling[J]. IEEE Transactions on Parallel & Distributed Systems, 2014,25(8):2126-2137.
doi: 10.1109/TPDS.2013.272 |
[17] | Tabbb Y, Medouri A, Tetouan M.Towards a next gereration of scientific computing in the cloud[J]. International of Computer Science, 2012,9(6):177-183. |
[18] | 夏俊鸾. Spark大数据处理技术[M].北京:电子工业出版社,2015. |
[ Xia J L.Big data processing technology with spark[M]. Beijing: Electronic Industry Press, 2015. ] | |
[19] | 高彦杰. Spark大数据处理:技术、应用与性能优化[M].北京:机械工业出版社,2014. |
[ Gao Y J.Data processing with spark technology, application and performance optimization[M]. Beijing: China Machine Press, 2014. ] | |
[20] | Kuglin C D.The phase correlation image alignment method[J]. Proc.intl Conf.cybernetics & Society, 1975:163-165. |
[21] | Zaharia M, Chowdhusry M, Das T, et al.Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing[C]// Usenix Conference on Networked Systems Design and Implementation. 2012:141-146. |
[22] | Zaharia M, Das T, Li H, et al.Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters[C]//Proceedings of the 4th USENIX conference on Hot Topics in Cloud Computing. USENIX Association, 2012:10-10. |
[23] | Engle C, Lupher A, Xin R, et al.Shark: fast data analysis using coarse-grained distributed memory[C]//Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. ACM. 2012:689-692. |
[24] | Xin R S. Rosen J. Zaharia M, et al.Shark: SQL and rich analytics at scale[C]//Proceedings of the 2013 ACM SIGMOD International Conference on Management of data. ACM, 2013:13-24. |
[25] | Ramirez-Gallego S, Garcia S, Mourino-Talin H, et al.Distributed entropy minimization discretizer for big data analysis under apache spark[C]// Trustcom/bigdatase/ispa. IEEE, 2015. |
[26] | Han Z, Zhang Y.Spark: A big data processing platform based on memory computing[C]//International Symposiumon Parallel Architectures. IEEE, 2015:172-176. |
[1] | 聂沛, 陈广胜, 景维鹏. 矢量瓦片并行构建与分布式存储模型研究[J]. 地球信息科学学报, 2020, 22(7): 1487-1496. |
[2] | 潘淼鑫, 林甲祥, 陈崇成, 叶晓燕. 基于C-SOM和Spark的并行空间离群挖掘方法及应用[J]. 地球信息科学学报, 2019, 21(1): 128-136. |
[3] | 刘文涛, 李世华, 覃驭楚. 基于全卷积神经网络的建筑物屋顶自动提取[J]. 地球信息科学学报, 2018, 20(11): 1562-1570. |
[4] | 韦兴旺, 张雪锋, 薛云. 基于光谱和形状的遥感图像分割质量评估方法[J]. 地球信息科学学报, 2018, 20(10): 1489-1499. |
[5] | 温馨, 罗侃, 陈荣国. 基于Shark/Spark的分布式空间数据分析框架[J]. 地球信息科学学报, 2015, 17(4): 401-407. |
[6] | 吴一全, 沈毅, 陶飞翔. 基于局部空间信息KFCM的遥感图像聚类算法[J]. 地球信息科学学报, 2014, 16(5): 769-775. |
[7] | 王旭红, 李飞, 张哲, 秦慧杰, 刘晓宁, 李钢. 遥感图像信息容量约束区间的选择与空间分异性[J]. 地球信息科学学报, 2014, 16(1): 108-116. |
[8] | 戴芹, 刘建波, 刘士彬. 综合多特征遥感图像智能检索方法的概念设计[J]. 地球信息科学学报, 2011, 13(3): 401-408. |
[9] | 李京忠, 刘咏梅. 古遗址区形迹特征的影像融合信息识别分析——以陕西省五陵原遗址为例[J]. 地球信息科学学报, 2010, 12(6): 886-891. |
[10] | 万其明, 汪闽, 张星月, 蒋圣, 谢玉林. 基于五叉树分解与多特征直方图匹配的高分辨遥感图像检索[J]. 地球信息科学学报, 2010, 12(2): 275-281. |
[11] | 李春华, 徐涵秋. 高分辨率遥感图像融合的光谱保真问题[J]. 地球信息科学学报, 2008, 10(4): 520-526. |
[12] | 慈慧, 张行南, 阮仁宗. 上海边滩资源彩红外遥感影像分析应用[J]. 地球信息科学学报, 2008, 10(2): 247-250. |
[13] | 吐热尼古丽·阿木提, 张晓帆. 自组织神经网络的土地覆盖图像识别分析应用[J]. 地球信息科学学报, 2007, 9(3): 128-131. |
[14] | 朱光良, 陆丽珍. 遥感图像检索系统的设计与实现[J]. 地球信息科学学报, 2005, 7(4): 15-19,24. |
[15] | 沈占锋, 骆剑承, 马伟峰, 明冬萍. 基于SVG的分布式遥感图像特征提取方法[J]. 地球信息科学学报, 2005, 7(3): 87-90. |
|