面向数据传输的地理栅格数据快速压缩方法

doi:10.3724/SP.J.1047.2016.00894

摘要/Abstract

摘要：

随着对地观测技术的高速发展,高分辨率地理栅格数据已被广泛应用于地貌、环境、水文等领域,传输与存储海量数据亟需通过数据压缩来解决有限信道容量的制约。本文分析了地理栅格数据特征,并基于数据保真性和压缩即时性原则,提出了融合转换压缩和编码压缩的地理栅格数据两阶段压缩方法,并从精度和效率2个视角构建了两阶段压缩方法的评价方法。利用不同大小的规则格网DEM数据,在集群系统上对两阶段压缩方法的数据保真性和压缩性能进行了测试。实验结果表明,本文构建的两阶段压缩方法在数值和地表形态上均有较好的精度,数据保真性高。同时,其压缩率一般在50%以上,解/压速率达到实时层次,能够显著地减少数据传输时间消耗,提高网络传输效率。两阶段压缩方法具有较好的普适性,可为高性能地学并行计算等领域提供技术支撑。

关键词: 数据压缩, 两阶段压缩, 栅格数据, DEM, 并行计算

Abstract:

As the main form of representing the geographical information, geo-raster data contains abundant geographical knowledge. With the rapid development of earth observation technology, high-resolution geo-raster data has been widely applied to many research fields, such as landform, soil, environment and hydrology. With respect to this context, the contradiction between the saving and transferring of massive geo-raster data and a limited channel capacity has become increasingly prominent with regard to the intensive increase of data size. Data compression techniques provide the possibility to solve this problem. This paper studies the compression method of geo-raster data based on gridded DEMs for the purpose of realizing massive data’s online transmission. By analyzing the characteristics of geo-grid data, this paper proposes a new compression method named as the two-phase compression method, which combines the conversion compression and the coding compression based on the data fidelity and the real-time compression principle. Meanwhile, this paper establishes an assessment method of two-phase compression method from the perspectives of accuracy and efficiency. In order to test and verify the data fidelity and the compression performance of the two-phase compression method, this paper conducted several experiments on a 10-node server cluster under the Linux operating system by using different sizes of gridded DEMs. The experiment results showed that the proposed two-phase compression method has provided good data fidelity. It keeps the data accuracy on both of the numerical and the representation structure. At the same time, the compression ratio is generally above 50%, and the almost real-time decompression/compression efficiency also indicates that it has a good performance. The two-phase compression method can significantly reduce the time consumption of data transmission through network, and improve the efficiency of network transmission. In all, this two-phase compression method of geo-raster data presents a good universality, and it can provide a technical support to the application of geo-raster data, such as the high-performance geo-computation.

Key words: data compression, two-phase compression, geo-raster data, DEM, parallel computation

江岭, 王春, 赵明伟, 杨灿灿. 面向数据传输的地理栅格数据快速压缩方法[J]. 地球信息科学学报, 2016, 18(7): 894-901.

JIANG Ling,WANG Chun,ZHAO Mingwei,YANG Cancan. A Fast Compression Approach of Geo-raster Data for Network Transmission[J]. Journal of Geo-information Science, 2016, 18(7): 894-901.

图/表 8

图1

图2

表1

图3

表2

表3

图4

表4

参考文献 21

[1]	Xiong L ., Tang G ., Li F ., et al.Modeling the evolution of loess-covered landforms in the Loess Plateau of China using a DEM of underground bedrock surface[J]. Geomorphology, 2014,209:18-26. doi: 10.1016/j.geomorph.2013.12.009
[2]	De Donato ., Barres ., Sausse ., et al.Advances in 3-D infrared remote sensing gas monitoring: application to an urban atmospheric environment[J]. Remote Sensing of Environment, 2016,175:301-309. doi: 10.1016/j.rse.2015.12.045
[3]	Wulfa ., Bookhagena ., Scherlera D.Differentiating between rain, snow, and glacier contributions to river discharge in the western Himalaya using remote-sensing data and distributed hydrological modeling[J]. Advances in Water Resources, 2016,88:152-169.
[4]	宫鹏,黎夏,徐冰.高分辨率影像解译理论与应用方法中的一些研究问题[J].遥感学报,2006,10(1):1-5. doi: 10.3321/j.issn:1007-4619.2006.01.001
	[ Gong ., Li ., Xu B.Interpretation theory and application method development for information extraction from high resolution remotely sensed data[J]. Journal of Remote Sensing, 2006,10(1):1-5. ] doi: 10.3321/j.issn:1007-4619.2006.01.001
[5]	Kidner D ., Smith, D H.Advances in the data compression of digital elevation models[J]. Compression & Geoscience, 2003,29(4):985-1002. doi: 10.1016/S0098-3004(03)00097-9
[6]	Losasso ., Hoppe H.Geometry clip maps: terrain rendering using nested regular grids[J]. ACM Transaction on Graphics, 2004,23(3):769-776. doi: 10.1145/1186562.1015799
[7]	李艳红,庞小平,李海亭.网络环境下的遥感影像金字塔纹理压缩算法与实验[J].地球信息科学学报,2012,14(1):109-115. doi: 10.3724/SP.J.1047.2012.00109
	[ Li Y ., Pang X ., Li H T.Research of remote sensing pyramid texture compression in network[J]. Journal of Geo-information Science, 2012,14(1):109-115. ] doi: 10.3724/SP.J.1047.2012.00109
[8]	卫俊霞,相里斌,段晓峰,等.基于EZW的高光谱图像压缩技术研究[J].光谱学与光谱分析,2011,31(8):2283-2286. doi: 10.3964/j.issn.1000-0593(2011)08-2283-04
	[ Wei J ., Xiang L ., Duan X ., et al.Hyperspectral image compression technology research based on EZW[J]. Spectroscopy and Spectral Analysis, 2011,31(8):2283-2286. ] doi: 10.3964/j.issn.1000-0593(2011)08-2283-04
[9]	Ouafi ., Ahmed A ., Barria ., ZIitouni A.A modified embedded zero tree wavelet (MEZW) algorithm for image compression[J]. Journal of Mathematical Imaging and Vison, 2008,30(7):298-307. doi: 10.1007/s10851-007-0057-y
[10]	Tamilarasi ., Palanisamy V.A novel embedded coding for medical image compression using contourlet transform[J]. International Journal of Signal and Imaging Systems Engineering, 2012,5(3):204-212. doi: 10.1504/IJSISE.2012.049854
[11]	Jiang J.Image compression with neural networks - a survey[J]. Signal Processing: Image Communication, 1999,14(9):737-760. doi: 10.1016/S0923-5965(98)00041-1
[12]	Toledo ., Pinzolas ., Ibarrola J ., et al.Improvement of the neighborhood based Levenberg-Marquardt algorithm by local adaptation of the learning coefficient[J]. IEEE Transactions on Neural Networks, 2005,16(4):988-992. doi: 10.1109/TNN.2005.849849 pmid: 16121740
[13]	Gerstner T.Multiresolution compression and visualization of globaltopographic data[J]. Geoinformatica, 2003,7(1):7-32. doi: 10.1023/A:1022818126783
[14]	Platings ., Day A M.Compression of large-scale terrain data for realtime visualization using a tiled quad tree[J]. Computer Graphics Forum, 2004,23(4):41-759.
[15]	Yusov ., Shevtsov M.High-performance terrain rendering using hardware tessellation[J]. Journal of WSCG, 2011,19(23):85-92.
[16]	Zhao J ., Tang ., Tong R F.Connectivity-based segmentation for GPU-accelerated mesh decompression[J]. Journal of Computer Science and Technology, 2012,27(6):1110-1118. doi: 10.1007/s11390-012-1289-x
[17]	Durdevic D ., Tartalja I I.HFPAC: GPU friendly height field parallel compression[J].GeoInformatica,2013,17(1):207-233. doi: 10.1007/s10707-012-0171-x
[18]	陈志荣,尹天鹤,徐财江,等.面向移动用户的矢量地图数据压缩方法[J]. 浙江大学学报(理工版),2016,43(1):45-50. doi: 10.3785/j.issn.1008-9497.2016.01.008
	[ Chen Z ., Yin T ., Xu C ., et al.Vector data compression method for mobile users[J]. Journal of Zhejiang University (Science Edition), 2016,43(1):45-50. ] doi: 10.3785/j.issn.1008-9497.2016.01.008
[19]	蔡明,乔文孝,鞠晓东,等.一种新的数据无损压缩编码方法[J].电子与信息学报,2014,36(4):1008-1012. doi: 10.3724/SP.J.1146.2013.00863
	[ Cai ., Qiao W ., Ju X ., et al.A new coding method for lossless data compression[J]. Journal of Electronics & Information Technology, 2014,36(4):1008-1012. ] doi: 10.3724/SP.J.1146.2013.00863
[20]	汤国安,刘学军,闾国年,等.数字高程模型及地学分析的原理与方法[M].北京:科学出版社,2005.
	[ Tang G ., Liu X ., Lv G ., et al.The principle and methodology of DEM based Geo-analysis[M]. Beijing: Science Press, 2005. ]
[21]	袁玫,袁文.数据压缩技术及其应用[M].北京:电子工业出版社,1994.
	[ Yuan ., Yuan W.Application of data compression technique[M]. Beijing: Publishing House of Electronics Industry, 1994. ]

	数据组1（1821行×2134列）			数据组2（2001行×2285列）				数据组3（2645行×2759列）
	CE/(%)		CT/10^-2 s	UCT10^-2 s	CE/(%)	CT/10^-2 s	UCT/10^-2 s	CE/(%)	CT/10^-2 s	UCT/10^-2 s
LZO	-0.36(39.03)	0.35( 2.93)	0.27( 2.31)	13.87(46.11)	0.47( 3.10)	0.36( 2.35)		45.49(62.75)	0.58( 3.45)	0.51( 2.58)
QUICKLZ	0.00(48.68)	1.66( 3.50)	0.24( 2.97)	0.00(53.93)	2.24( 3.65)	0.33( 3.14)		40.35(67.43)	3.49( 4.20)	4.27( 3.71)
LZ4	0.08(38.91)	0.70( 3.64)	0.28( 1.05)	14.90(45.63)	0.92( 3.79)	0.34( 1.14)		46.49(61.80)	1.07( 4.10)	0.51( 1.31)
LZFX	-1.96(39.62)	6.42( 5.45)	0.61( 3.76)	13.08(46.04)	6.74( 5.79)	0.85( 3.86)		45.26(62.16)	7.14( 6.69)	1.91( 4.03)
SNAPPY	0.29(38.02)	0.44( 5.46)	0.30( 1.39)	14.33(44.36)	0.75( 5.53)	0.35( 1.49)		44.53(59.85)	0.80( 5.54)	0.61( 1.83)
FASTLZ	-1.66(32.40)	5.02( 5.69)	0.67( 3.10)	13.40(40.43)	5.22( 6.04)	0.73( 3.14)		45.73(59.44)	5.32( 6.96)	0.86( 3.31)
LZW	-39.79(34.14)	36.13(25.50)	7.55( 8.35)	-19.11(16.00)	37.52(30.91)	8.50( 8.80)		24.85(46.12)	36.92(37.56)	9.65(11.25)
RLE	-0.19( 0.08)	2.14( 2.28)	1.19( 1.36)	14.76( 0.07)	2.68( 2.67)	1.53( 1.55)		46.59( 0.05)	4.18( 4.01)	1.94( 2.26)
HUFFMAN	1.68( 7.58)	30.72(29.35)	32.30(29.17)	8.82(12.40)	33.60(33.15)	34.17(32.86)		35.28(33.00)	34.83(39.22)	35.69(31.96)
SFANO	1.22( 7.05)	31.26(28.56)	24.71(23.74)	8.50(12.02)	31.78(29.19)	25.69(26.54)		30.99(28.47)	36.99(37.34)	31.50(31.57)

	数据组1（1821行×2134列）			数据组2（2001行×2285列）			数据组3（2645行×2759列）
	浮点型NCV/(MB/s)		整型NCV/(MB/s)		浮点型NCV/(MB/s)	整型NCV/(MB/s)	浮点型NCV/(MB/s)	整型NCV/(MB/s)
LZO	-4.26	55.18		147.37	73.79		576.93	145.00
QUICKLZ	0.00	55.75		0.00	69.28		72.36	118.64
LZ4	0.62	61.48		103.15	80.80		409.46	158.93
LZFX	-2.06	31.90		15.02	41.63		69.61	80.76
SNAPPY	2.89	41.15		114.11	55.08		438.75	112.94
FASTLZ	-2.16	27.34		19.64	38.42		103.00	80.50
LZW	-6.75	7.47		-3.62	3.51		7.43	13.15
RLE	-0.42	0.17		30.54	0.14		105.92	0.10
HUFFMAN	0.20	0.96		1.14	1.64		6.96	6.45
SFANO	0.16	1.00		1.29	1.88		6.30	5.75

	CE/(%)	CT/10^-2 s	UCT/10^-2 s
数据组1 （1821行×2134列）	49.82 (69.51)	1.59 (3.85)	3.05 (4.65)
数据组2 （2001行×2285列）	56.93 (73.05)	1.56 (4.16)	2.83 (5.11)
数据组3 （2645行×2759列）	72.75 (81.38)	2.15 (5.21)	3.63 (6.92)

	进程数
	2	4	8	16	32	64
采样数据1 （13 225行×13 795列）	2.50	2.49	2.49	2.47	2.64	2.69
采样数据2 （33 063行×34 488列）	-	2.54	2.60	2.76	2.73	2.75