EN中文

Journal of Geo-information Science >

2015 , Vol. 17 >Issue 5: 575 - 582

DOI: https://doi.org/10.3724/SP.J.1047.2015.00575

pGTIOL: A Parallel GeoTIFF I/O Library

Expand
  • 1. National Engineering Research Center of GeographicInformation System, China University of Geosciences,Wuhan 430074, China;
    2. Faculty of Information Engineering, China University of Geosciences, Wuhan 430074, China

Received date: 2014-12-29

  Revised date: 2015-02-11

  Online published: 2015-05-10

Fold

Abstract

Data I/O has become one of the main bottlenecks for parallel geospatial computing. In this study, we firstly explore the data structure of a widely used GIS raster data format-GeoTIFF, particularly focusing on its storage modes (strip storage and tile storage). The transfer functions which map the logical structure of data to the physical storage structure were constructed for both storage modes.This article also designs a framework for parallel I/O of raster data and implementsa parallel GeoTIFF I/O library (pGTIOL) using the file-view technique of MPI-IO. Experimental results showed that pGTIOL effectively enhances the I/O performance in comparison with the master-worker I/O mode which uses the Geospatial Data Abstraction Library (GDAL). pGTIOL encapsulates the underlying parallel I/O routines, and provides easy-to-use interfaces for the parallel reading and writing of GeoTIFF data. Compared with other parallel raster I/O software packages, pGTIOL supports a wide range of data types, both the strip and tile data storage modes, and various domain decomposition methods. Most importantly, pGTIOL supports asynchronous parallel I/O, which allows multiple processes to read and write sub-domains of data on demand.Hence,it could facilitate dynamic load-balancing in application.

Key words： dynamic load-balancing; MPI; Parallel I/O; asynchronous; GeoTIFF

Cite this article

HU Shujian, GUAN Qingfeng, GONG Junfang, LIU Yang, FAN Tianheng, YUN shuo . pGTIOL: A Parallel GeoTIFF I/O Library[J]. Journal of Geo-information Science, 2015 , 17(5) : 575 -582 . DOI: 10.3724/SP.J.1047.2015.00575

References

[1] 周建鑫,陈荦,熊伟,等.地理栅格数据并行I/O 的研究与实 现[J].地理信息世界,2013,20(6):62-65.
[2] Finn M P, Liu Y, Mattli D M, et al. pRasterBlaster: Highperformance small-scale raster map projection transformation using the Extreme Science and Engineering Discovery Environment[C]. Abstract presented at the XXII International Society for Photogrammetry & Remote Sensing Congress, Melbourne, Australia, 2012.
[3] Wang X, Li Z, Gao S. Parallel remote sensing image processing: taking image classification as an example[C]. In Li Z, Li X, Liu Y, and Cai Z (eds) Proceedings of the SixtInternational Symposium on Intelligence Computation and Applications, Wuhan, China, 27-28 October, Berlin, Springer-Verlag, 2012:159-69
[4] Guan Q, Clarke K C. A general-purpose parallel raster processing programming library test application using a geographic cellular automata model[J]. International Journal of Geographical Information Science, 2010,24(5):695-722.
[5] Guan Q, Zeng W, Gong J, et al. pRPL 2.0: Improving the parallel raster processing library[J]. Transactions in GIS, 2014,18(S1):25-52.
[6] Tarboton D G. TauDEM, terrain analysis using digital elevation models. Logan, UT. http://www.engineering.usu. edu/dtarb/. Computer program available at http://www.engineering. usu.edu/taudem/getinfo.htm. [9 August 2002].
[7] Tesfa T K, Tarboton D G, Watson D W, et al. Extraction of hydrological proximity measures from DEMs using parallel processing[J]. Environmental Modelling & Software, 2011,26(12):1696-1709.
[8] Qin C Z, Zhan L J, Zhu A. How to apply the Geospatial Data Abstraction Library (GDAL) properly to parallel geospatial raster I/O?[J]. Transactions in GIS, 2014,18(6):950-957.
[9] Li J, Liao W, Choudhary A, et al. Parallel netCDF: A highperformance scientific I/O interface[C]. Supercomputing, 2003 ACM/IEEE Conference, 2003:39-39.
[10] Ritter N, Ruth M. GeoTIFF format specification-specification version 1.8. 2[R]. GeoTIFFWorking Group, 2000.
[11] 陈端伟,束炯,王强,等.遥感图像格式GeoTIFF 解析[J].华 东师范大学学报:自然科学版,2006(2):18-26.
[12] 牛芩涛,盛业华.GeoTIFF 图像文件的数据存储格式及读 写[J].四川测绘,2005,27(3):105-108.
[13] 杨伟光,李文.使用MPI的并行I/O 实现及性能分析[J].计 算机工程与应用,2006,42(17):96-98.
[14] Thakur R, Lusk E, Gropp W. Users guide for ROMIO: A high-performance, portable MPI-IO implementation[R]. Technical Report ANL/MCS-TM-234, Mathematics and Computer Science Division.Argonne, IL: Argonne National Laboratory, 1997.
[15] 欧阳柳,熊伟,程果,等.地理栅格数据的并行访问方法研 究[J].计算机科学,2012,39(11):116-121.
[16] 杨典华,潘欣.一种面向大型地理栅格数据的并行处理框 架[J].国防科技大学学报,2013,35(6):152-156.

Options
Outlines

/

〈 〉