基于MPP架构的并行空间数据库原型系统的设计与实现

doi:10.3724/SP.J.1047.2016.00151

摘要/Abstract

摘要：

快速高效地查询信息是衡量当前空间数据库性能的重要指标之一。传统的单节点关系型空间数据管理方式难以满足大数据量空间数据查询的需求,特别是高性能的复杂空间多表连接任务需求。鉴此,本文设计并实现了基于Massive Parallel Processing（MPP）架构的并行空间数据库中间件原型系统。系统充分利用无共享（shared-nothing）架构的优势,特别是针对空间数据的特性,设计了并行空间数据划分与导入、并行空间多表连接、空间数据查询优化等算法与模型。首先介绍了近年来并行数据库系统的发展现状,接着阐述了基于MPP架构的并行空间数据库中间件系统的查询计划算法及其系统架构,最后作者对一些大规模数据量做查询实验及其查询结果分析。实验表明,在处理挖掘大规模数据量时,该系统有近似线性的加速比,相比于传统单节点数据库,它能充分提高海量空间数据的复杂查询的性能,解决了空间数据库并行化处理海量数据的问题。

关键词: MPP, 空间数据库, 并行, Shared Nothing

Abstract:

The efficiency for querying complex spatial information resources is an important indicator to evaluate the performance of current spatial databases. Traditional single node relation spatial data management is difficult to meet the demand of high-performance in querying large amounts of spatial data, especially for the complex join query on multi-table. In order to solve this problem, we design and implement a spatial database middleware prototype system. This system takes full advantages of the massive parallel processing (MPP) and shared-nothing architecture. In consideration of the characteristics of spatial data, we design the spatial data parallel import, multi-spatial-tables join strategy, spatial data query optimization and other algorithms and models. This paper firstly introduces the development status of parallel database systems in recent years, and then elaborates its MPP architecture and its organizational model, and the strategy of the join query on multi-spatial-table. Finally, we made some query experiments on massive spatial data and analyzed the results of these inquiries. The experimental results show that this system indicates a good performance (nearly linear speedup) in processing the complex query of massive spatial data. Compared with the tradition single node database, this system can fully improve the efficiency of complex querying for large spatial data, and it is a more efficient solution to solve the complex spatial data queries.

Key words: MPP, spatial database, parallel processing, shared-nothing

陈达伦, 陈荣国, 谢炯. 基于MPP架构的并行空间数据库原型系统的设计与实现[J]. 地球信息科学学报, 2016, 18(2): 151-159.DOI:10.3724/SP.J.1047.2016.00151

CHEN Dalun,CHEN Rongguo,XIE Jiong. Research of the Parallel Spatial Database Proto System Based on MPP Architecture[J]. Journal of Geo-information Science, 2016, 18(2): 151-159.DOI:10.3724/SP.J.1047.2016.00151

图/表 13

图1

图2

表1

图3

图4

图5

表2

图6

图7

图8

图9

表3

表4

参考文献 23

[1]	雷德龙,郭殿升,陈崇成,等.基于MongoDB 的矢量空间数据云存储与处理系统[J].地球信息科学学报,2014,16(4):507-516. doi: 10.3724/SP.J.1047.2014.00507
	[ Lei D L, Guo D S, Chen C C, et al.Vector spatial data cloud storage and processing based on MongoDB[J]. Journal of Geo-information Science, 2014,16(4):507-516. ] doi: 10.3724/SP.J.1047.2014.00507
[2]	龚健雅. 空间数据库管理统的概念与发展趋势[J].测绘科学,2001,26(3):4-9.
	[ Gong J Y.Concepts and development of spatial database management system[J]. Science of Surveying and Mapping, 2001,26(3):4-9. ]
[3]	Stonebraker M.SQL databases v. NoSQL databases[J]. Communications of the ACM, 2010,53(4):10-11. doi: 10.1145/1721654.1721659
[4]	范协裕,任应超,邓富亮,等.基于代理的并行空间查询语言[J].计算机工程,2013,39(11):61-64. doi: 10.3969/j.issn.1000-3428.2013.11.013
	[ Fan X Y, Ren Y C, Deng F L, et al.Parallel spatial query language based on proxy[J]. Computer Engineering, 2013,39(11):61-64. ] doi: 10.3969/j.issn.1000-3428.2013.11.013
[5]	柳锴,唐雨新,张云泉,等.基于机群架构的并行数据库实现技术研究[J].计算机工程与设计,2008,29(3):526-529.
	[ Liu K, Tang Y X, Zhan Y Q, et al.Research on implementation technology of cluster parallel database system[J]. Computer Engineering and Design, 2008,29(3):526-529. ]
[6]	汪永琳,郑若忠.并行数据库系统实现技术的研究[J].陕西工学院学报,2000,16(2):10-12. doi: 10.3969/j.issn.1673-2944.2000.02.003
	[ Wang Y L, Liu Q, Zheng R_Z. Study of implementation techniques for parallel database systems[J]. Journal of Shaanxi Institute of Technology, 2000,16(2):10-12. ] doi: 10.3969/j.issn.1673-2944.2000.02.003
[7]	陈虎,张云泉,柳锴,等.基于机群架构的并行数据库中间件系统改进研究[J].计算机研究与发展,2007(z3):142-146.
	[ Chen H, Zhang Y Q, Liu K,et al. Research on improvement of parallel middleware for SQL on beowulf cluster[J]. Journal of Computer Research and Development, 2007(z3):142-146. ]
[8]	贾婷,魏祖宽,唐曙光,等.一种面向并行空间查询的数据划分方法[J].计算机科学,2010,37(8):198-200. doi: 10.3969/j.issn.1002-137X.2010.08.043
	[ Jia T, Wei Z K, Tang S G, et al.New spatial data partition approach for spatial data query[J]. Computer Science, 2010,37(8):198-200. ] doi: 10.3969/j.issn.1002-137X.2010.08.043
[9]	周胜. 并行数据库系统 PBASE/3 数据缓冲技术研究[D].北京:北京中国科学院研究生院(计算技术研究所), 2000.
	[ Zhou S.The buffering techniques in parallel database system PBASE/3[D]. Beijing: Institute of Computing Technology, Chinese Academy of Sciences, 2000. ]
[10]	王珊,王会举,覃雄派.架构大数据:挑战,现状与展望[J].计算机学报,2011,34(10):1741-1752. doi: 10.3724/SP.J.1016.2011.01741
	[ Wang S, Wang H J, Qin X P, et al.Architecting big data: challenges, studies and forecasts[J]. Chinese Journal of Computers, 2011,34(10):1741-1752. ] doi: 10.3724/SP.J.1016.2011.01741
[11]	刘宇,孙莉.并行空间连接查询处理[J].上海交通大学学报,2002,36(4):512-515. doi: 10.3321/j.issn:1006-2467.2002.04.019
	[ Liu Y, Sun L, Tian Y Q.Parallel spatial join query processing[J]. Journal of Shanhai Jiaotong University, 2002,36(4):512-515. ] doi: 10.3321/j.issn:1006-2467.2002.04.019
[12]	蔡亚杰,赵政文,白松,等.并行数据库基于多连接的查询优化算法的研究[J].计算机应用与软件, 2012,12:52. doi: 10.3969/j.issn.1000-386x.2012.12.050
	[ Cai Y J, Zhao Z W, Bai S, et al. On query optimization algorithm based on multi-table join for parallel database[J]. Computer Applications and Software, 2012,12:52. ] doi: 10.3969/j.issn.1000-386x.2012.12.050
[13]	王永杰,洪新兰,孙伟.基于并行空间数据库和 GML 的 WebGIS 系统的研究与实现[J].测绘科学,2007,32(4):131-132. doi: 10.3771/j.issn.1009-2307.2007.04.048
	[ Wang Y J, Hong X L, Sun W.Research and realization of WebGIS based on parallel spatial database and GML[J]. Science of Surveying and Mapping, 2007,32(4):131-132. ] doi: 10.3771/j.issn.1009-2307.2007.04.048
[14]	文继荣,陈红.Shared-nothing 并行数据库系统查询优化技术[J].计算机学报,2000,23(1):28-38. doi: 10.3321/j.issn:0254-4164.2000.01.004
	[ Wen J R, Chen H, Wang S.Query optimization techniques of a shared-nothing parallel database system[J]. Chinese Journal of Computers, 2000,23(1):28-38. ] doi: 10.3321/j.issn:0254-4164.2000.01.004
[15]	许新华,唐胜群,宋振云,等.并行数据库查询优化技术的最新进展[J].计算机工程与设计,2009(16):3814-3819.
	[ Xu X H, Tang S Q, Song Z Y, et al.Latest developments on parallel database query optimization[J]. Computer Engineering and Design, 2009,16:3814-3819. ]
[16]	程昌秀. 空间数据库管理系统概论[M].科学出版社,2012.
	[ Cheng C X.Introduction to spatial database management systems[M]. Beijing: Science Press, 2012. ]
[17]	周光明. DB2并行数据库核心技术探究[J].电子制作,2013,9:69.
	[ Zhou G M.Research for core techniques of parallel spatial database[J]. Practical Electronics, 2013,9:69. ]
[18]	Li C, Yang J, Han J, et al.The distributed storage system based on MPP for mass data[C]. 2012 IEEE Asia-Pacific Services Computing Conference (APSCC), 2012:384-387.
[19]	Alam S R, Agarwal P K, Kuehn J A.Performance evaluation of a scalable molecular dynamics simulation framework on a massively-parallel system[C]. Proceedings of the 7th IEEE International Conference on Bioinformatics and Bioengineering, 2007:1459-1466.
[20]	颜勋,陈荣国,程昌秀,等.内嵌式空间数据库优化器代价评估框架及实现[J].武汉大学学报 (信息科学版),2011,36(6):726-730.
	[ Yan X, Chen R G, Cheng C X, et al.Optimizer cost estimation framework and implementation for spatially-enabled database[J]. Geomatics and Information Science of Wuhan University, 2011,36(6):726-730. ]
[21]	Welch M C, Kwan P W, Sajeev A S M. Applying GIS and high performance agent-based simulation for managing an Old World Screwworm fly invasion of Australia[J]. Acta tropica, 2014,138: S82-S93. doi: 10.1016/j.actatropica.2014.03.021 pmid: 24705073
[22]	高昂,陈荣国,赵彦庆,等.空间数据访问集成与分布式空间数据源对象查询[J].地球信息科学学报,2010,12(4):532-540.
	[ Gao A, Chen R G, Zhao Y Q, et al.Access and integration on geospatial data and its distributed query[J]. Journal of Geo-information Science, 2010,12(4):532-540. ]
[23]	范娟娟,海洋,高明星,等.基于分布式存储与并行计算架构的历史数据库分析解决方案[J].中国铁路,2014(7):60-63. doi: 10.3969/j.issn.1001-683X.2014.07.016
	[ Fan J J, Hai Y, Gao M X, et al.The solution for historical database based on distributed storage and parallel computing[J].Chinese Railways, 2014,7:60-63. ] doi: 10.3969/j.issn.1001-683X.2014.07.016

节点名	硬件配置	操作系统/JDK版本	数量/台	设备用途
协调节点	CPU：I7-3610 QM 内存：8 GB	CentOS 6.4/ JDK 1.7.0	1	管理后端支持节点,解析SQL语句,制定查询计划,归并查询结果
后端从节点	CPU：I3-3220 内存：4 GB	CentOS 6.4/ JDK 1.7.0	4	提供数据库计算功能,返回子查询结果集
物理元数据库节点	CPU：I5-2400 内存: 6 GB	CentOS 6.4/ JDK 1.7.0	1	提供数据库元数据信息。包括表、视图、约束等元数据
客户端节点	CPU：I7-3610 QM 内存: 8 GB	CentOS 6.4/ JDK 1.7.0	1	输入SQL查询语句,输出查询结果