pGTIOL:GeoTIFF数据并行I/O库
收稿日期: 2014-12-29
修回日期: 2015-02-11
网络出版日期: 2015-05-10
基金资助
教育部高等学校博士学科点专项科研基金(20130145120013)。
pGTIOL: A Parallel GeoTIFF I/O Library
Received date: 2014-12-29
Revised date: 2015-02-11
Online published: 2015-05-10
在地理栅格并行计算处理中,数据I/O 已成为制约计算性能的主要瓶颈之一。本文针对该问题,首先分析广泛应用于GIS 栅格数据存储的GeoTIFF 格式,重点研究数据的2 种存储模式(即条带存储与块状存储),并根据这2 种存储方式,分别构建了栅格数据从逻辑结构向物理存储结构的映射模型。然后,针对地理空间并行计算的需要,提出了栅格数据的并行读写框架,并利用MPI 并行I/O 技术的文件视图方法,实现了GeoTIFF 数据并行I/O库(pGTIOL)。结果表明,对比开源栅格空间数据转换库(GDAL)的主从I/O 模式,本文提出的pGTIOL 准确读写数据,具有更高的性能。该库隐藏了底层并行I/O 的细节,提供简单易用的并行读写GeoTIFF 栅格数据的接口,支持多数据类型和多种空间分割,实现了对条带存储与块状存储数据的异步并行读写,从而满足动态负载均衡的需求。
胡树坚, 关庆锋, 龚君芳, 刘洋, 范天恒, 云硕 . pGTIOL:GeoTIFF数据并行I/O库[J]. 地球信息科学学报, 2015 , 17(5) : 575 -582 . DOI: 10.3724/SP.J.1047.2015.00575
Data I/O has become one of the main bottlenecks for parallel geospatial computing. In this study, we firstly explore the data structure of a widely used GIS raster data format-GeoTIFF, particularly focusing on its storage modes (strip storage and tile storage). The transfer functions which map the logical structure of data to the physical storage structure were constructed for both storage modes.This article also designs a framework for parallel I/O of raster data and implementsa parallel GeoTIFF I/O library (pGTIOL) using the file-view technique of MPI-IO. Experimental results showed that pGTIOL effectively enhances the I/O performance in comparison with the master-worker I/O mode which uses the Geospatial Data Abstraction Library (GDAL). pGTIOL encapsulates the underlying parallel I/O routines, and provides easy-to-use interfaces for the parallel reading and writing of GeoTIFF data. Compared with other parallel raster I/O software packages, pGTIOL supports a wide range of data types, both the strip and tile data storage modes, and various domain decomposition methods. Most importantly, pGTIOL supports asynchronous parallel I/O, which allows multiple processes to read and write sub-domains of data on demand.Hence,it could facilitate dynamic load-balancing in application.
Key words： dynamic load-balancing; MPI; Parallel I/O; asynchronous; GeoTIFF
[1] 周建鑫,陈荦,熊伟,等.地理栅格数据并行I/O 的研究与实 现[J].地理信息世界,2013,20(6):62-65.
[2] Finn M P, Liu Y, Mattli D M, et al. pRasterBlaster: Highperformance small-scale raster map projection transformation using the Extreme Science and Engineering Discovery Environment[C]. Abstract presented at the XXII International Society for Photogrammetry & Remote Sensing Congress, Melbourne, Australia, 2012.
[3] Wang X, Li Z, Gao S. Parallel remote sensing image processing: taking image classification as an example[C]. In Li Z, Li X, Liu Y, and Cai Z (eds) Proceedings of the SixtInternational Symposium on Intelligence Computation and Applications, Wuhan, China, 27-28 October, Berlin, Springer-Verlag, 2012:159-69
[4] Guan Q, Clarke K C. A general-purpose parallel raster processing programming library test application using a geographic cellular automata model[J]. International Journal of Geographical Information Science, 2010,24(5):695-722.
[5] Guan Q, Zeng W, Gong J, et al. pRPL 2.0: Improving the parallel raster processing library[J]. Transactions in GIS, 2014,18(S1):25-52.
[6] Tarboton D G. TauDEM, terrain analysis using digital elevation models. Logan, UT. http://www.engineering.usu. edu/dtarb/. Computer program available at http://www.engineering. usu.edu/taudem/getinfo.htm. [9 August 2002].
[7] Tesfa T K, Tarboton D G, Watson D W, et al. Extraction of hydrological proximity measures from DEMs using parallel processing[J]. Environmental Modelling & Software, 2011,26(12):1696-1709.
[8] Qin C Z, Zhan L J, Zhu A. How to apply the Geospatial Data Abstraction Library (GDAL) properly to parallel geospatial raster I/O?[J]. Transactions in GIS, 2014,18(6):950-957.
[9] Li J, Liao W, Choudhary A, et al. Parallel netCDF: A highperformance scientific I/O interface[C]. Supercomputing, 2003 ACM/IEEE Conference, 2003:39-39.
[10] Ritter N, Ruth M. GeoTIFF format specification-specification version 1.8. 2[R]. GeoTIFFWorking Group, 2000.
[11] 陈端伟,束炯,王强,等.遥感图像格式GeoTIFF 解析[J].华 东师范大学学报:自然科学版,2006(2):18-26.
[12] 牛芩涛,盛业华.GeoTIFF 图像文件的数据存储格式及读 写[J].四川测绘,2005,27(3):105-108.
[13] 杨伟光,李文.使用MPI的并行I/O 实现及性能分析[J].计 算机工程与应用,2006,42(17):96-98.
[14] Thakur R, Lusk E, Gropp W. Users guide for ROMIO: A high-performance, portable MPI-IO implementation[R]. Technical Report ANL/MCS-TM-234, Mathematics and Computer Science Division.Argonne, IL: Argonne National Laboratory, 1997.
[15] 欧阳柳,熊伟,程果,等.地理栅格数据的并行访问方法研 究[J].计算机科学,2012,39(11):116-121.
[16] 杨典华,潘欣.一种面向大型地理栅格数据的并行处理框 架[J].国防科技大学学报,2013,35(6):152-156.
/
| 〈 |
|
〉 |