基于张量分解的AoT序列数据有损压缩方法

doi:10.12082/dqxxkx.2021.200425

地球信息科学学报 ›› 2021, Vol. 23 ›› Issue (1): 134-142.doi: 10.12082/dqxxkx.2021.200425

• 专栏："全空间信息建模分析方法与应用研究" • 上一篇下一篇

基于张量分解的AoT序列数据有损压缩方法

杨晨¹^,²(), 高鸿¹^,², 张丽莹¹^,², 胡旭¹^,², 俞肇元¹^,²^,³, 李冬双⁴^,⁵^,^*()

1.南京师范大学虚拟地理环境教育部重点实验室,南京 210023
2.江苏省地理环境演化国家重点实验室培育建设点,南京 210023
3.江苏省地理信息资源开发与利用协同创新中心,南京 210023
4.江苏省作物遗传生理国家重点实验室/江苏省作物栽培生理重点实验室,扬州大学农学院,扬州 225009
5.江苏省粮食作物现代产业技术协同创新中心,扬州大学,扬州 225009

收稿日期:2020-07-31 修回日期:2020-12-29 出版日期:2021-01-25 发布日期:2021-03-25
作者简介:杨晨（1997— ）,女,山西大同人,硕士生,主要从事地理空间数据处理与分析方面研究。E-mail: yangchen765@126.com
基金资助:
国家重点研发计划项目(2016YFB0502301);国家自然科学基金项目(42001320);国家自然科学基金项目(41976186)

A Lossy Compression Method for AoT Sequence Data based on Tensor Decomposition

YANG Chen¹^,²(), GAO Hong¹^,², ZHANG Liying¹^,², HU Xu¹^,², YU Zhaoyuan¹^,²^,³, LI Dongshuang⁴^,⁵^,^*()

1. Key Laboratory of Virtual Geographic Environment of The Ministry of Education (Nanjing Normal University), Nanjing 210023, China
2. Cultivation Base of State Key Laboratory of Geographical Environment Evolution, Jiangsu Province, Nanjing 210023, China
3. Jiangsu Provincial Center for Collaborative Innovation in Geographical Information Resource Development and Application, Nanjing 210023, China
4. Jiangsu Key Laboratory of Crop Genetics and Physiology/Jiangsu Key Laboratory of Crop Cultivation and Physiology, Agricultural College of Yangzhou University, Yangzhou 225009, China
5. Jiangsu Co-Innovation Center for Modern Production Technology of Grain Crops, Yangzhou University, Yangzhou 225009, China

Received:2020-07-31 Revised:2020-12-29 Online:2021-01-25 Published:2021-03-25
Contact: LI Dongshuang
Supported by:
National Key Research and Development Program of China(2016YFB0502301);National Natural Science Foundation of China(42001320);National Natural Science Foundation of China(41976186)

摘要/Abstract

摘要：

Array of Things (AoT)通过单一位置上的多传感器对城市系统进行连续动态观测。AoT观测数据量大且持续增长,使得如何利用有限的计算资源进行AoT序列数据的压缩传输成为其应用的关键瓶颈之一。本文提出了一种基于张量分解的AoT序列数据的有损压缩方法。面向其海量、高维且需在传感器端处理的需求,该方法首先将AoT序列数据组织成高维张量,利用算法复杂度较低的张量CANDECOMP/PARAFAC (CP)分解提取各维度上的特征主分量,而后利用张量重构实现特征保持的数据有损压缩。利用基于张量分解的有损压缩方法,针对美国芝加哥市区的24 h内感测的声光电磁数据进行了实验,讨论了不同压缩参数对压缩比、压缩误差、压缩精度、压缩时间、压缩过程运行内存占用和压缩结果内存占用之间的影响。实验结果表明该方法可实现AoT序列数据的有损压缩,其较小的内存占用能够支持传感器端数据压缩。并且与原始光场强度对比表明,压缩后的数据保持了原有时空分布特征。与传统矢量量化编码压缩方法相比,在相同压缩精度下,本文方法的压缩比约高27%~76%,压缩时间约节省46%~73%,压缩结果所占内存约节省17%~57%,因此本文方法具有更高的压缩比,更低的压缩时间和内存占用,可为AoT这一类数据的大规模有损压缩提供借鉴意义。

关键词: 传感器, 时空序列, AoT, 有损压缩, 多维张量, 张量分解, CP分解, 张量重构

Abstract:

Array of Things (AoT) provides continuous and dynamic observations of urban systems through multiple sensors at a single location. How to utilize the limited computing resources to compress and transmit AoT sequence data becomes one of the key bottlenecks of the AoT application. Considering that most AoT sequence data are massive, high-dimensional and needed to be processed at the sensor side, a tensor decomposition method is introduced to the lossy compression for AoT sequence data in this work. This method first organizes the AoT sequence data as a high-dimensional tensor to preserve the multidimensional coupling relationship among the different dimensions. The CANDECOMP/PARAFAC (CP) decomposition, which has simple parameter, relatively simple principle and low algorithm complexity, is then utilized to decompose and extract the principal feature components in each dimension of AoT sequence data. Since these principal feature components are obtained by absorbing the multidimensional coupling relationship, they can be further combined with tensor reconstruction to approximate the original data accurately. Considering that the data approximation is obtained by removing the redundant information, it can achieve the data lossy compression with the feature preservation. The simulation experiment is conducted based on the acousto-optic electromagnetic data sensed within 24 hours in the downtown area of Chicago in the United States. The influences of different compression parameters on compression ratio, compression error, compression accuracy, compression time, memory usage under the conditions of different compression parameter are discussed. The experimental results show that, with the increase of compression parameter, the compression error obviously decreases and the memory occupation weakly increases, which demonstrates that tensor-based method can achieve lossy compression of AoT sequence data and both the memory occupation during the running process and the memory occupation of the final results can support the data compression of sensor segments. Compared with the original intensity of the light field, the compressed data maintain the spatio-temporal distribution characteristics of original data that would not affect the further data analysis. In addition, compared with the traditional vector quantization coding compression method, the compression ratio of this method is higher about 27%~76%, the compression time is less about 46%~73%, and the memory occupation of compression result is smaller about 17%~57%. Therefore, the tensor-based method has a higher compression ratio, less compression time and smaller memory occupation under the same compression accuracy. The tensor-based method can also be applied to the data with multidimensional features, such as spatial dimensions with different locations, time dimensions at different time nodes, and attribute dimensions of different variables (temperature, humidity, etc.), which could provide a feasible idea for large-scale lossy compression of massive multidimensional geographic sensor sequence data represented by AoT sequence data.

Key words: sensor, spatio-temporal sequence, Array of Things (AoT), lossy compression, multidimensional tensor, tensor decomposition, CANDECOMP/PARAFAC decomposition, tensor reconstruction

杨晨, 高鸿, 张丽莹, 胡旭, 俞肇元, 李冬双. 基于张量分解的AoT序列数据有损压缩方法[J]. 地球信息科学学报, 2021, 23(1): 134-142.DOI:10.12082/dqxxkx.2021.200425

YANG Chen, GAO Hong, ZHANG Liying, HU Xu, YU Zhaoyuan, LI Dongshuang. A Lossy Compression Method for AoT Sequence Data based on Tensor Decomposition[J]. Journal of Geo-information Science, 2021, 23(1): 134-142.DOI:10.12082/dqxxkx.2021.200425

图/表 9

表1

图1

图2

图3

图4

表2

压缩方法评价指标

结果评价	评价指标	指标计算公式编号	参数描述
压缩效果	压缩比	$Compressionratio = A . dataSize A ? . dataSize$ （4）	$A . dataSize$ 表示原始数据内存大小 $A ? . dataSize$ 为压缩结果数据内存大小
压缩误差	均方根误差	$RMSE = 1 IJK ∑ i = 1 I ∑ j = 1 J ∑ k = 1 K (χ ijk - χ ? ijk) 2$ （5）	$χ ? ijk$ 为重构张量的对应位置上的元素 $χ ijk$ 为原始张量值对应位置上的元素 $χ ?$ 为原始张量的平均值 I、J、K分别为空间、属性、和时间维度上的数据个数
压缩精度	决定系数	$R - Squared = ∑ i = 1 I ∑ j = 1 J ∑ k = 1 K (χ ijk - χ ?) 2 - ∑ i = 1 I ∑ j = 1 J ∑ k = 1 K (χ ijk - χ ? ijk) 2 ∑ i = 1 I ∑ j = 1 J ∑ k = 1 K (χ ijk - χ ?) 2$ （6）

表2

图5

图6

图7

参考文献 23

[1]	Wang S, Hou Y, Gao F , et al. A reconfigurable smart interface based on IEEE 1451 and field programmable gate array for multiple internet of things devices[J]. International Journal of Distributed Sensor Networks, 2017,13(2):1-20.
[2]	Catlett C E, Beckman P H, Sankaran R , et al. Array of things: A scientific research instrument in the public way [C]// International Workshop. ACM, 2017: 26-33.
[3]	胡永利, 孙艳丰, 尹宝才 . 物联网信息感知与交互技术[J]. 计算机学报, 2012,35(6):75-91.
	[ Hu Y L, Sun Y F, Yin B C . Information sensing and interaction technology in internet of things[J]. Journal of Software, 2012,35(6):75-91.]
[4]	Kolo J G, Shanmugam S A, Lim D W G , et al. An adaptive lossless data compression scheme for wireless sensor networks[J]. Journal of Sensors, 2012,2012(1):1-20.
[5]	Shannon C E . A Mathematical Theory of Communication[J]. The Bell System Technical Journal, 1948,27(4):623-656. doi: 10.1002/bltj.1948.27.issue-4
[6]	Kasban H, Hashima S . Adaptive radiographic image compression technique using hierarchical vector quantization and huffman encoding[J]. Journal of Ambient Intelligence and Humanized Computing, 2018,10(7):2855-2867. doi: 10.1007/s12652-018-1016-8
[7]	Hale J C, Sellars H L . Historical data recording for process computers[J]. Chemical Engineering Progress, 1981,77(11):38-43. doi: 10.1016/j.cep.2014.01.003
[8]	Ganesan D, Estrin D, Heildemann J . Dimensions: Why do we need a new data handling architecture for Sensor Networks?[J]. Acm Sigcomm Computer Communication Review, 2002,33(1):143-148. doi: 10.1145/774763.774786
[9]	Beckman P, Sankaran R, Catlett C , et al. Waggle: An open sensor platform for edge computing [C]// 2016 IEEE SENSORS. IEEE, 2017: 1-3.
[10]	Kolda T G, Bader B W . Tensor decompositions and applications[J]. SIAM Review, 2009,51(3):455-500. doi: 10.1137/07070111X
[11]	Li D S, Yang L, Yu Z Y , et al. A tensor-based interpolation method for sparse spatio-temporal field data[J]. Journal of Spatial Science, 2018,65(2):307-325. doi: 10.1080/14498596.2018.1509740
[12]	Phan A H, Cichocki A . Tensor decompositions for feature extraction and classification of high dimensional datasets[J]. Nonlinear Theory and Its Applications, 2011,1(1):37-68.
[13]	Bengua J A, Ho P N, Tuan H D , et al. Matrix product state for higher-order tensor compression and classification[J]. IEEE Transactions on Signal Processing, 2017,65(15):4019-4030. doi: 10.1109/TSP.2017.2703882
[14]	Yuan L, Yu Z, Luo W , et al. A hierarchical tensor-based approach to compressing, updating and querying geospatial data[J]. IEEE Transactions on Knowledge and Data Engineering, 2015,27(2):312-325. doi: 10.1109/TKDE.69
[15]	王东方, 周激流, 何坤 , 等. 基于张量Tucker分解的彩色图像压缩[J]. 四川大学学报:自然科学版, 2010,47(2):287-92.
	[ Wang D F, Zhou J L, He K , et al. Compression of color images based on tucker-tensor[J]. Journal of Sichuan University (Natural Science Edition), 2010,47(2):287-292.]
[16]	张乐飞, 何发智 . 基于张量分解的超光谱图像降秩与压缩[J]. 武汉大学学报·信息科学版, 2017,42(2):193-197.
	[ Zhang L F, He F Z . Hyper-spectal image rank-reducing and compression based on tensor decomposition[J]. Geomatics and Information Science of Wuhan University, 2017,42(2):193-197.]
[17]	赵洪山, 马利波 . 基于张量Tucker分解的智能配电网大数据压缩[J]. 中国电机工程学报, 2019,39(16):4744-4752.
	[ Zhao H S, Ma L B . Big data compression of smart distribution systems based on tensor tucker decomposition[J]. Proceedings of the CSEE, 2019,39(16):4744-4752.]
[18]	Gujral E, Pasricha R, Yang T , et al. OCTEN: Online compression-based tensor decomposition [C]// 8th IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), 2019: 455-459.
[19]	Zhang Q, Yang L T, Chen Z , et al. High-order possibilistic c-means algorithms based on tensor decompositions for big data in IoT[J]. Information Fusion, 2018,39:72-80. doi: 10.1016/j.inffus.2017.04.002
[20]	Koyama S, Furuya K I, Uematsu H , et al. Real-time sound field transmission system by using wave field reconstruction filter and its evaluation[J]. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 2014, E97.A(9):1840-1848. doi: 10.1587/transfun.E97.A.1840
[21]	Xiong H, Pan Z, Ye X , et al. Sparse spatio-temporal representation with adaptive regularized dictionary learning for low bit-rate video coding[J]. IEEE Transactions on Circuits Systems for Video Technology, 2013,23(4):710-728. doi: 10.1109/TCSVT.2012.2221271
[22]	Nowakova J, Prilepok M, Snasel V . Medical image retrieval using vector quantization and fuzzy s-tree[J]. J Med Syst, 2017,41(2):18. doi: 10.1007/s10916-016-0659-2 pmid: 27981409
[23]	Zang M, Liu T, Lang L , et al. Dictionary learning for VQ feature extraction in ECG beats classification[J]. Expert Systems with Application, 2016,53:129-137. doi: 10.1016/j.eswa.2016.01.031

数据集名称	描述	数据集内容
data.csv	所有节点的感测值文件	数据采集时间戳、数据采集节点ID、传感器类型、传感器名称、感测类型、电子传感器原始测量值、经转换后的可读值（HRF值）
nodes.csv	数据集中的节点及其元数据	节点ID、项目ID、节点序列号（在物理机箱可见）、节点安装的街道地址、节点所在经度、节点所在纬度、节点构建和配置的更详细描述、节点安装开始时间戳、节点安装结束时间戳
sensors.csv	传感器及其元数据	数据采集内容、传感器类型、传感器名称、感测类型、转换值的物理单位、数据表的最小HPF值（用作范围过滤器的下限）、数据表的最大HPF值（用作范围过滤器的上限）、传感器数据表的参考网址
provenance.csv	整个数据集中的元数据	数据格式版本、项目ID、数据创建时间戳、数据结束时间戳、创建该摘要的时间戳、此摘要的URL地址

基于张量分解的AoT序列数据有损压缩方法

A Lossy Compression Method for AoT Sequence Data based on Tensor Decomposition

RichHTML

PDF (PC)

赞

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 23

相关文章 13

Metrics

本文评价

推荐阅读 0

[1]	胡校飞, 周杨, 蓝朝桢, 黄高爽, 赵璐颖. 结构语义辅助约束的数字孪生场景虚实配准方法[J]. 地球信息科学学报, 2023, 25(5): 883-895.
[2]	欧庚, 周良辰, 林冰仙, 汪洋, 周海洋, 闾国年. 塔基遥感视频监控有效性评估方法[J]. 地球信息科学学报, 2022, 24(1): 165-175.
[3]	袁林旺, 俞肇元, 罗文, 袁帅, 周春烨. PIR传感网数据的几何代数建模与行为分析[J]. 地球信息科学学报, 2020, 22(1): 21-29.
[4]	黄耀欢, 李中华, 朱海涛. 作物胁迫无人机遥感监测研究评述[J]. 地球信息科学学报, 2019, 21(4): 512-523.
[5]	段炼, 党兰学, 胡涛, 朱欣焰, 叶信岳. 融合历史犯罪数据的疑犯社会活动位置预测[J]. 地球信息科学学报, 2018, 20(7): 929-938.
[6]	刘菊, 许珺, 蔡玲, 孟斌, 裴韬. 基于出租车用户出行的功能区识别[J]. 地球信息科学学报, 2018, 20(11): 1550-1561.
[7]	周振超, 李贺, 黄翀, 刘庆生, 刘高焕, 何云, 于涵. 红树林遥感动态监测研究进展[J]. 地球信息科学学报, 2018, 20(11): 1631-1643.
[8]	刘涛, 张星, 李清泉, 方志祥, 李秋萍. 一种运动恢复结构和航位推算结合的室内行人视觉定位方法[J]. 地球信息科学学报, 2017, 19(6): 744-753.
[9]	唐炉亮, 段倩, 阚子涵, 李清泉. 出租车交接班行为识别与时空分布研究[J]. 地球信息科学学报, 2017, 19(2): 167-175.
[10]	胡楚丽, 钟翔, 关庆锋, 陈能成. 面向应急观测规划的对地观测传感器集成管理方法与应用[J]. 地球信息科学学报, 2015, 17(6): 631-367.
[11]	邬明权, 牛铮, 乔玉良, 吴朝阳. 基于MODIS数据的北京气溶胶类型特性与影响因素分析[J]. 地球信息科学学报, 2009, 11(4): 541-548.
[12]	郭继发, 刘臻, 崔伟宏. 基于时空特征分析技术的烤烟GIS系统[J]. 地球信息科学学报, 2009, 11(3): 370-375.
[13]	付民, 胡晓辉. 地下水分布式多参数实时监测系统[J]. 地球信息科学学报, 2004, 6(1): 72-75.