分布式环境下矢栅一体化数据组织模型设计

doi:10.3724/SP.J.1047.2016.01588

摘要/Abstract

摘要：

随着地理信息技术与计算机网络技术的结合和发展,基于全球框架的地理信息服务对海量数据管理的要求越来越高,传统的单中心的关系数据库的管理模式已经不能满足要求。分布式文件系统、半结构化数据库和关系数据库技术优势互补,为海量数据高效管理提供了新的技术思路。本文提出了分布式环境下空间数据一体化存储管理架构,设计了矢量和栅格数据的数据逻辑组织和物理存储模型,通过统一的分层+分块的数据划分规则,实现了矢、栅数据分布式环境下的一体化管理。在该模型中,利用关系数据库和半结构化数据库的特点,对空间索引和实体数据分开管理,有效地提高了数据处理和访问效率。实验表明,该模型具有更高的数据管理能力,可为分布式环境下数据服务中心构建提供一个有效的解决方案。

关键词: 分布式, 关系数据库, 图层对象模型, 矢量数据模型, 栅格数据模型

Abstract:

With the combination and development of geographic information technology and computer network technology, geographic information services based on global framework demand for more efficient massive data management, the traditional single center relational database management mode is unable to meet the requirements. Since the distributed file systems, semi-structured databases and relational database technology have complementary advantages to each other, a new technical method for efficient management of massive data is developed. In order to achieve high effective geospatial data management, this paper presented an integrated architecture oriented to the storage and management of geospatial data in distributed environment, designed user - oriented massive geospatial data integration model and distributed storage organization model . In this model, a technical route combining the NoSQL database and relational database is adopted, and a layered + partitioned data model and multi-level index mechanism for the rapid access of massive data is designed, so it can realizes the integrated management of vector and raster data in distributed environment. Because the model has taken advantages of relational database and semi-structured database, structured geographic information, spatial index and entity data can be managed separately and the efficiency of data processing and access is improved effectively. Vector data and raster data is the largest and most widely used geospatial data, In this paper, an experiment system is set up in the experiment environment, which realizes vector data and raster data management model. TB-level data are used to conduct experiments of data loading, index (pyramid) creation and concurrent data access efficiency, compared with the traditional data model, the model in the data management capacity, processing speed and access efficiency have greatly improved. The results show that the model can support the parallel operation in distributed environment, with a higher data management capability will offer an effective solution to massive data management in distributed environment.

Key words: distributed, relational database, layer object model, vector data model, raster data model, raster data model

徐道柱, 罗斌, 周岩, 金澄. 分布式环境下矢栅一体化数据组织模型设计[J]. 地球信息科学学报, 2016, 18(12): 1588-1596.DOI:10.3724/SP.J.1047.2016.01588

XU Daozhu,LUO Bin,ZHOU Yan,JIN Cheng. Model Design of Integrated Vector and Raster Data Organization Under the Distributed Environment[J]. Journal of Geo-information Science, 2016, 18(12): 1588-1596.DOI:10.3724/SP.J.1047.2016.01588

图/表 12

图1

图2

图3

图4

图5

图6

表1

表2

表3

表4

表5

表6

参考文献 17

[1]	付青松. GIS矢栅数据结构及数据组织管理研究[J]. 绘与空间地理信息,2010,33(6):64-72. doi: 10.3969/j.issn.1672-5867.2010.06.019
	[ Fu Q S.The Research of GIS Raster Data Structure and Data Organization and Management[J].GEOMATICS & SPATIAL INFORMATION TECHNOLOGY, 2010,33(6):64-72. ] doi: 10.3969/j.issn.1672-5867.2010.06.019
[2]	李冬梅. 三维GIS中的矢栅一体化技术的研究与设计[D].沈阳:沈阳工业大学,2007:18-21.
	[ Li D M.RESEARCH AND DESIGN ON INTEGRATION OF VECTOR DATA AND RASTER DATA IN THREE-DIMENSION GIS[D] . Shenyang:Shenyang University of Technology, 2007:18-21. ]
[3]	王珂,张周威.矢栅一体化拓扑关系的度量描述研究[J].武汉大学学报信息科学版,2015,40(5):638-643. doi: 10.13203/j.whugis20130357
	[ Wang K,Zhang Z W.Quantitative Representation of Topology Relations Based on Integrative Data Model of Vector and Raster[J]. Geomatics and Information Science of Wuhan University, 2015,40(5):638-643. ] doi: 10.13203/j.whugis20130357
[4]	Kjenstad K.On the Integration of Object-Based Models and Field-based Models in GIS[J].International Journal of Geographical Information Science, 2006,20(5):491-509. doi: 10.1080/13658810600607329
[5]	Cova T, Goodchild M.Extending Geographical Representation to Include Fields of Spatial Objects[J].International Journal of Geographical Information Science, 2002,16(6):509-532. doi: 10.1080/13658810210137040
[6]	Goodchild M, Yuan M, Cova T.Towards a General Theory of Geographical Representation in GIS[J]. International Journal of Geographical Information Science, 2007,21(3):239-260. doi: 10.1080/13658810600965271
[7]	汪娜,岳丽华,金培权,等.基于矢栅一体化的多源地理空间信息关联模型[J].计算机研究与发展,2012,49(suppl.):134-139.
	[ Wang N, Yue L H, Jin P Q, et al. Multi-Source Geospatial Information Association Model Based on the Integration of Vector and Raster Data[J]. Journal of Computer Research and Development, 2012,49(suppl.):134-139. ]
[8]	Chang F, Dean J, Ghemawat S, et al. Bigtable: A distributed storage system for structured data[J]. ACM Transactions on Computer Systems(TOCS), 2008,26(2):1-26. doi: 10.1145/1365815.1365816
[9]	康俊锋. 云计算环境下高分辨率遥感影像存储与高效管理技术研究[D].杭州:浙江大学,2011:25-47.
	[ Kang J F.Technologies of Storage and Efficient Management on Cloud Computing for High Resolution Remote Sensing Image[D].Hangzhou: Zhejian University, 2011:25-47. ]
[10]	王旭东. 面向海量遥感影像数据的分布式文件系统管理技术研究[D].兰州:兰州交通大学,2012:19-26.
	[ Wang X D.Distributed File System Management Technology Research Based on The Massive Remote Sensing Image Data[D]. Lanzhou: Lanzhou Jiaotong University, 2012:19-26. ]
[11]	程付超. 面向海量遥感数据的分布式混合计算技术研究[D].成都:成都理工大学,2014:35-41.
	[ Chen F C.Research on Distributed Hybrid Computing Technologies for Massive Remote Sensing Data[D]. Chengdu: Chengdu University of Technology, 2014:35-41. ]
[12]	霍旭光. 基于云计算的大规模地形数据处理方法的研究[D].北京:中国地质大学(北京),2013:23-39.
	[ Huo X G.The Research to Massive Terrain Data Processing Method based on Cloud Computing[D].Beijing: China University of Geosciences, 2013:23-39. ]
[13]	杨文奇. 基于MapRedcue的大规模栅格数据空间分析算法并行化研究[D].赣州:江西理工大学,2013:26-35.
	[ Yang W Q.Research on Parallel Spatial Analysis Algorithm of Massive Raster Data Based on MapReduce[D]. Ganzhou: Jiangxi University of Technology, 2013:26-35. ]
[14]	刘灿由. 电子海图云服务关键技术研究与实践[D].郑州:信息工程大学,2013:69-87.
	[ Liu C Y.Research and Practice on Key Technologies of Electronic Chart Cloud Service[D].Zhengzhou: Information Engineering University,2013:69-87. ]
[15]	陈军. 基于G_S模式的空间分析云服务关键技术研究[D].成都:成都理工大学,2012:41-49.
	[ Chen J.The Study of Key Technologies of Spatial Analysis Services Based On G/S Model[D]. Chengdu: Chengdu University of Technology, 2012:41-49. ]
[16]	张广弟. 分布式环境下海量空间数据的存储和并行查询技术研究[D]. 赣州:江西理工大学,2012:18-24.
	[ Zhang G D.Research on Storage and Parallel Query of Massive Spatial Data in The Distributive Environment[D]. Ganzhou: Jiangxi University of Technology, 2012:18-24. ]
[17]	岳利群. 基于分布式存储的虚拟地理环境关键技术研究[D].郑州:信息工程大学,2011:106-116.
	[ Yue L Q.Research on Key Technologies for Virtual Geographic Environment Based on Distributed Storage[D]. Zhengzhou: Information Engineering University, 2011:106-116. ]

入库数据量/GB	并行客户端数/台	入库时间/h	入库速度/（GB/(小时×单台）)
1648	25	29.76	2.22
120	25	2.13	2.25

入库数据量/GB	并行客户端数/台	入库时间/h	入库速度/（GB/(小时×单台）)
6656	10	30.05	22.15
878	10	3.92	22.40

入库数据量/GB	并行客户端数/台	入库时间/h	入库速度/（GB/(小时×单台）)
88	10	1.97	4.47
37	10	1.60	2.31

试验内容	入库数据/GB	客户端数/台	入库时间/h	入库速度/（GB/(小时×单台）)
矢量数据入库、索引建立	300	1	85	3.45
影像数据入库、金字塔创建	500	1	50	10

并发访问用户个数	访问时间 /min	发送请求个数	系统成功处理请求个数	系统处理请求成功率/（%）	成功处理的请求的响应速度/（要素/秒）
并发访问用户个数	访问时间 /min	发送请求个数	系统成功处理请求个数	系统处理请求成功率/（%）	最小速度	最大速度	平均速度
25×20	5	624	621	99.51	1231.85	578000	44152.99
25×24	5	608	607	99.83	1189.83	578000	46118.62
25×28	5	700	697	99.58	4244.28	578000	31001.02
25×32	5	805	805	100	785.82	578000	43859.84
25×36	5	900	893	99.22	766.60	578000	39124.65
25×40	5	1000	990	99	2918.80	578000	26832.16
25×45	5	1125	1116	99.2	568.63	578000	9292.78