基于MapReduce的空间敏感性分析并行算法设计

doi:10.3724/SP.J.1047.2014.00874

地球信息科学学报 ›› 2014, Vol. 16 ›› Issue (6): 874-881.doi: 10.3724/SP.J.1047.2014.00874

基于MapReduce的空间敏感性分析并行算法设计

李帆^1,²(), 何洪林^1,^*(), 任小丽^1,², 张黎¹, 路倩倩^1,², 于贵瑞¹

1. 中国科学院地理科学与资源研究所生态系统网络观测与模拟重点实验室,北京 100101
2. 中国科学院大学,北京 100049

收稿日期:2013-12-28 修回日期:2014-02-24 出版日期:2014-11-10 发布日期:2014-11-01
通讯作者: 何洪林 E-mail:davidlee0408@gmail.com;hehl@igsnrr.ac.cn
作者简介:
作者简介：李帆(1987-),男,山西太原人,硕士生,研究方向为生态信息学。E-mail:davidlee0408@gmail.com
基金资助:
科学院先导专项“应对气候变化的碳收支认证及相关问题”(XDA05050600);国家生态系统观测评估技术系统集成研究与示范(2013BAC03B00);MSR-CNIC Windows Azure 合作计划项目“基于Windows Azure的陆地生态系统碳水通量估算与不确定性研究”;国家科技部环保公益性行业科研专项(gyh5031103)

Research on Spatial Sensitivity Analysis Using Parallel Algorithm Based on MapReduce

LI Fan^1,²(), HE Honglin^1,^*(), REN Xiaoli^1,², ZHANG Li¹, LU Qianqian^1,², YU Guirui¹

1. Key Laboratory of Ecosystem Network Observation and Modeling, Institute of Geographic Sciences and Natural Resources Research, CAS, Beijing 100101, China
2. University of Chinese Academy of Sciences, Beijing 100049, China

Received:2013-12-28 Revised:2014-02-24 Online:2014-11-10 Published:2014-11-01
Contact: HE Honglin E-mail:davidlee0408@gmail.com;hehl@igsnrr.ac.cn
About author:
*The author: CHEN Nan, E-mail:fjcn99@163.com

摘要/Abstract

摘要：

近年来,随着遥感空间数据广泛应用于生态系统,推动了区域尺度生态遥感参数模型的发展。敏感性分析对识别模型关键参数,降低模型不确定性和完善模型具有重要作用。区域尺度的生态遥感参数模型,在进行模型参数敏感性分析时,由于涉及到空间数据的复杂运算,单机环境无法满足快速分析的要求。为了提高生态遥感参数模型空间敏感性分析效率,本文以青藏高原为研究区域,利用植被光合模型VPM(Vegetation Photosynthesis Model)和开源云计算平台Hadoop,设计和实现了基于Sobol′的生态遥感参数模型空间敏感性分析并行算法,并在实验室集群环境下进行算法分析,验证了算法的有效性和适用性。该算法的核心是利用MapReduce并行编程技术,对空间敏感性分析中的地图抽样和模型迭代过程进行任务分割,将分割后的子任务分配至不同的计算节点进行并行计算。实验表明,本文提出的并行策略,能有效缩短地图抽样和模型迭代计算时间,相比于单机算法,并行算法的运行速度提高了14倍左右。

关键词: 遥感参数模型, 空间敏感性分析, Sobol′, MapReduce

Abstract:

In recent years, with the rapid development of remote sensing technology, the spatial data represented by remote sensing images is widely used in ecosystem modeling, which promoted the development of ecological remote sensing parametric model in the regional scale. Sensitivity analysis is a key step for ecosystem model uncertainty quantification. It can identify the dominant parameters, reduce the model calibration uncertainty, and enhance the model optimization efficiency. Due to the intensive computation of spatial data during the sensitivity analysis, the traditional stand-alone environment cannot meet the requirements of rapid analysis for the regional scale remote sensing parametric model. This study designed and realized a parallel algorithm of Sobol′ spatial sensitivity analysis utilizing Hadoop, which is an open source cloud computing platform, based on VPM (Vegetation Photosynthesis Model). In order to verify the efficiency of the algorithm, we designed a comparison experiment to compare the efficiency differences of the traditional serial algorithm and the parallel algorithm. The parallel programming technology we used in this research was MapReduce, which divided the processes of map sampling and the iterative calculation during the spatial sensitivity analysis into subtasks, and assigned them to multiple computing nodes for parallel computing. The numerical experiment showed that the parallel strategy proposed in this study effectively shortened the time of model iterative calculations and significantly improved the efficiency of spatial sensitivity analysis for ecological remote sensing parametric model. Compared with the serial algorithm, the computing efficiency of the parallel algorithm was enhanced by 14 times.

Key words: remote sensing parametric model, spatial sensitivity analysis, Sobol′, MapReduce

李帆, 何洪林, 任小丽, 张黎, 路倩倩, 于贵瑞. 基于MapReduce的空间敏感性分析并行算法设计[J]. 地球信息科学学报, 2014, 16(6): 874-881.DOI:10.3724/SP.J.1047.2014.00874

LI Fan,HE Honglin,REN Xiaoli,ZHANG Li,LU Qianqian,YU Guirui. Research on Spatial Sensitivity Analysis Using Parallel Algorithm Based on MapReduce[J]. Journal of Geo-information Science, 2014, 16(6): 874-881.DOI:10.3724/SP.J.1047.2014.00874

图/表 11

图1

图2

表1

图3

图4

图5

表2

表3

图6

图7

图8

参考文献 30

[1]	徐崇刚,胡远满,常禹,等.生态模型的灵敏度分析[J].应用生态学报,2004,15(6):1056-1062.
[2]	蔡毅,邢岩,胡丹,等.敏感性分析综述[J].北京师范大学学报(自然科学版),2008,44(1):9-16.
[3]	孔凡哲,宋晓猛,占车生,等.水文模型参数敏感性快速定量评估的RSMSobol方法[J].地理学报,2011,66(9):1270-1280.
[4]	Sobol I M.Sensitivity analysis for non-linear mathematical models[J]. Mathematical modeling & Computational Experiment, 1993,1:407-414.
[5]	Lilburne L, Tarantola S.Sensitivity analysis of spatial models[J]. International Journal of Geographical Information Science, 2009,23(2):151-168.
[6]	Saltelli A.Making best use of model evaluations to compute sensitivity indices[J]. Computer Physics Communications, 2002,145(2):280-297.
[7]	Tarantola S, Nardo M, Saisana M, et al.A new estimator for sensitivity analysis of model output: An application to the e-business readiness composite indicator[J]. Reliability Engineering & System Safety, 2006,91(10-11):1135-1141.
[8]	Apache Hadoop. Hadoop[EB/OL]. , .
[9]	Ghemawat S, Gobioff H, Leung S, et al.The Google file system[C]. Symposium on Operating Systems Principles, 2003.
[10]	Chang F, Dean J, Ghemawat S, et al.A distributed storage system for structured data[J]. ACM Transactions on Computer Systems, 2008,26(2):205-218.
[11]	Dean J, Ghemawat S.MapReduce: Simplified data processing on large cluster[J]. Communications of the ACM, 2008,51(1):107-113.
[12]	Dean J, Ghemawat S.MapReduce: A flexible data processing tool[J]. Communications of the ACM, 2010,53(1):72-77.
[13]	康超. 面向时空分析的海量日志处理关键技术研究及应用[D].北京:中国科学院计算机网络信息中心,2012.
[14]	冯敏,尹芳,诸云强,等.基于MapReduce的分布式地形数据计算研究[J].华中科技大学学报(自然科学版),2011,39(1):24-27.
[15]	赵青松,陈林,孙波,等.基于Hadoop的云环境下作物生长模型算法的实现与测试[J].农业工程学报, 2013,29(8):179-186.
[16]	任启伟,陈洋波,周浩澜,等.基于Sobol法的TOPMODEL[J].人民长江,2010,41(19):91-94.
[17]	Xiao X M, Hollinger D, Aber J D, et al.Satellite-based modeling of gross primary production in an evergreen needle leaf forest[J]. Remote Sensing of Environment, 2004,89:519-534.
[18]	Xiao X M, Zhang Q Y, Braswell B, et al.Modeling gross primary production of temperate deciduous broadleaf forest using satellite image and climate data[J]. Remote Sensing of Environment, 2004,91:256-270.
[19]	Xiao X M, Zhang Q Y, Hollinger D, et al.Modeling gross primary production of an evergreen needleaf forest using MODIS and climate data[J]. Ecological Applications, 2005,15:954-969.
[20]	Xiao X M, Zhang Q Y, Saleska S, et al.Satellite-based modeling of gross primary production in a seasonally moist tropical evergreen forest[J]. Remote Sensing of Environment, 2005,94:105-122.
[21]	刘敏. 基于RS和GIS的陆地生态系统生产力估算及不确定性研究[D].南京: 南京师范大学硕士学位论文, 2008.
[22]	Potter C S, Randerson J T, Field C B, et al.Terrestrial ecosystem production —— a process model-based on global satellite and surface data[J]. Global Biogeochemical Cycles, 1993,7(4), 811-841.
[23]	Wang H S, Jia G S, Fu C B, et al.Deriving maximal light use efficiency from coordinated flux measurements and satellite data for regional gross primary production modeling[J]. Remote Sens. Environ, 2010,114(10):2248-2258.
[24]	Li Z Q, Yu G R, Xiao X M, et al.Modeling gross primary production of alpine ecosystems in the Tibetan Plateau using MODIS images and climate data[J]. Remote Sens. Environ, 2007,107(3):510-519.
[25]	张镱锂,李炳元,郑度,等.论青藏高原范围与面积[J].地理研究, 2002,21(1):1-8.
[26]	Hutchinson M F. ANUSPLIN Version 4.2 User Guide[R]. Canberra: Centre for Resource and Environmental Studies, Australian National University, 2002.
[27]	于贵瑞,何洪林,刘新安,等.中国陆地生态信息空间化技术研究(Ⅰ)——气象/气候信息的空间化途径[J].自然资源学报,2004,19(4):537-544.
[28]	何洪林,于贵瑞,刘新安,等.陆地生态信息空间化技术研究(Ⅱ)——太阳辐射要素[J].自然资源学报,2004,19(5):679-687.
[29]	刘新安,于贵瑞,范辽生,等.陆地生态信息空间化技术研究(Ⅲ) ——温度、降水等气候要素[J].自然资源学报,2004,19(6):818-825.
[30]	曹晨. 基于MCMC方法的统计模型的参数估计[D].南京:南京航空航天大学,2007.

数据项	键(key)	值(value)
输入	驱动变量名	抽样结果编号
输出	空值(NullValue)	二进制抽样结果文件

数据项	键(Key)	值(Value)
Map输入	矩阵行索引	各驱动变量抽样编号列表
Map输出	矩阵行索引	模拟结果Y
Reduce输入	矩阵行索引	模拟结果Y
Reduce输出	空值(NullValue)	模拟结果矩阵

输入变量	一阶敏感度系数(S_i)	总敏感度系数(S_Ti)
ε₀	0.92270	1.00414
T	0.00508	0.04957
PAR	0.00042	0.00563
EVI	0.00184	0.02241
LSWI	0.00005	0.00070
总和	0.93009	1.08245

基于MapReduce的空间敏感性分析并行算法设计

Research on Spatial Sensitivity Analysis Using Parallel Algorithm Based on MapReduce

RichHTML

PDF (PC)

赞

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 30

相关文章 3

Metrics

本文评价

推荐阅读 0

[1]	邬群勇, 苏克云, 邹智杰. 基于MapReduce的海量公交乘客OD并行推算方法[J]. 地球信息科学学报, 2018, 20(5): 647-655.
[2]	唐梦梦, 吉根林, 赵斌. 利用MapReduce的异常轨迹检测并行算法[J]. 地球信息科学学报, 2015, 17(5): 523-530.
[3]	张栋海, 黄丽娜, 刘晖, 唐健. 基于MapReduce的多机并行DP算法与实验分析[J]. 地球信息科学学报, 2013, 15(1): 55-60.