基于ODC的国产卫星影像存储与应用研究

doi:10.12082/dqxxkx.2020.190351

摘要/Abstract

摘要：

随着地球观测进入大数据时代,传统的数据管理技术已经难以适应大数据需求, Open Data Cube （ODC）作为新型的开源的地球观测数据管理与分析平台,适合进行时间序列数据的高性能计算和探索性数据分析,是亚大区域综合地球观测系统AOGEOSS的重要技术支撑平台。但当前ODC对国产卫星影像支持不友好,缺乏自动化管理和数据组织工具,使用ODC进行国产卫星影像数据管理的技术不成熟。因此,本文以高分一号卫星影像为例,通过开发ODC_GFTool中间件和自定义高分卫星影像产品格式探索ODC框架下国产影像数据自动化管理流程,最后以鄱阳湖为试验区,进行了基于ODC框架的水体提取应用实验,论证了ODC框架下国产卫星数据存取的可行性,研究结果表明相较于传统的数据处理工具ODC具有明显的效率优势,能够为AOGEOSS基础设施建设和国产卫星影像数据管理提供一定的参考。

关键词: Open Data Cube, ODC, 国产影像, 数据管理, 影像存储, 高分一号, 鄱阳湖

Abstract:

As earth observation enters the big data era, the traditional data management technology has often become insufficient in meeting research needs. As a new open source earth observation data management and analysis platform, Open Data Cube (ODC) is suitable for high-performance computing and exploratory data analysis of time series data. It's an important technical support platform for Asia-Oceania Global Earth Observation System of Systems (AOGEOSS). However, few attempts have been made on domestic satellite imagery data management using ODC. The current ODC lacks automated data organization tools for China's satellite imagery. Therefore, this paper took the domestic GF-1 imagery as an example to explore the automated management of domestic imagery data under the ODC framework by developing middleware called ODC_GFTool. The middleware's development environment is .NET Framework 4.0, which supports the current mainstream Windows 10 operating system. Based on GDAL, YamlDotNet and other components, ODC_GFTool achieves spectral information, band information, spatial reference, and other contents from high-resolution image files. In addition, with the strong matrix computing ability of the Matlab software, ODC_GFTool realizes high score image processing and visualization through mixed programming. The main processes of importing domestic satellite data into ODC using ODC_GFTool include defining GF-1 image products, building indexes, and ingesting data. For demenstration, the paper took Poyang Lake as the test area and did an experimental water extraction. Results show that ODC has obvious efficiency advantage compared with traditional data processing tools. The use of ODC for domestic satellite imagery management increased the storage space by 7.2% and reduced the processing time to one third of the traditional methods. In addition, compared to Google Earth Engine and EarthServer, ODC has the following advantages: (1) ODC supports localized database storage, considering the security of some satellite imagery, data storage and computing need to be local, not in the cloud. (b) As an open source earth observation data management framework, ODC facilitates integration and secondary development according to actual project requirements. Research on ODC data management and application for domestic satellite imagery not only can enrich new ways of domestic image data management in China and increase the practical application value of ODC in the world, but also promote international cooperation and respond to the national earth observation infrastructure construction.

Key words: Open Data Cube, ODC, domestic satellite images, data management, image storage, GF-1, Poyang Lake

李俊杰, 陈舒博, 张文, 余长慧, 张志远, 孟令奎. 基于ODC的国产卫星影像存储与应用研究[J]. 地球信息科学学报, 2020, 22(9): 1860-1867.DOI:10.12082/dqxxkx.2020.190351

LI Junjie, CHEN Shubo, ZHANG Wen, YU Changhui, ZHANG Zhiyuan, MENG Lingkui. Management and Application of Domestic Satellite Imagery based on Open Data Cube[J]. Journal of Geo-information Science, 2020, 22(9): 1860-1867.DOI:10.12082/dqxxkx.2020.190351

图/表 8

图1

图2

图3

图4

图5

图6

图7

表2

参考文献 26

[1]	李国庆, 庞禄申. 公众化驱动的地球观测发展新时代[J]. 中国科学:信息科学, 2017,47(2):193-206.
	[ Li G Q, Pang L S. A new age of public-oriented Earth observation development[J]. Scientia Sinica(Informationis), 2017,47(2):193-206. ]
[2]	Mattmann C A. Computing: A vision for data science[J]. Nature, 2013,493(7433):473-475.
[3]	李国庆, 张红月, 张连翀, 等. 地球观测数据共享的发展和趋势[J]. 遥感学报, 2016,20(5):979-990.
	[ Li G Q, Zhang H Y, Zhang L C, et al. Development and trend of Earth observation data sharing[J]. Journal of Remote Sensing, 2016,20(5):979-990. ]
[4]	Committee on Earth Observation Satellites (CEOS). The Open Data Cube Initiative[EB/OL]. [2019-04-30]. https://docs.wixstatic.com/ugd/f9d4ea_1aea90c5bb7149c8a7308 90c0f791496.pdf.
[5]	B Killough. Overview of the Open Data Cube Initiative[C]// 2018 International Geoscience and Remote Sensing Symposium. Valencia: IEEE, 2018: 8629-8632.
[6]	Matthew B J, Lewis A, Oliver S, et al. Unlocking the Australian landsat archive-from dark data to High Performance Data infrastructures[J]. GeoRes Journal, 2015,6:135-140.
[7]	Giuliani G, Chatenoux B, De B A, et al. Building an Earth Observations Data Cube: Lessons learned from the Swiss Data Cube (SDC) on generating Analysis Ready Data (ARD)[J]. Big Earth Data, 2017,1(2):100-117.
[8]	Gorelick N, Hancher M, Dixon M, et al. Google Earth Engine: Planetary-scale geospatial analysis for everyone[J]. Remote Sensing of Environment, 2017,202:18-27.
[9]	何昭欣, 张淼, 吴炳方, 等. Google Earth Engine支持下的江苏省夏收作物遥感提取[J]. 地球信息科学学报, 2019,21(5):752-766.
	[ He Z X, Zhang M, Wu B F, et al. Extraction of summer crop in Jiangsu based on Google Earth Engine[J]. Journal of Geo-information Science, 2019,21(5):752-766. ]
[10]	Baumann P, Dehmel A, Furtado P, et al. The multidimensional database system RasDaMan[J]. Acm Sigmod Record, 1998,27(2):575-577.
[11]	Baumann P, Mazzetti P, Ungar J, et al. Big data analytics for Earth sciences: The EarthServer approach[J]. International Journal of Digital Earth, 2016,9(1):3-29.
[12]	Lewis A, Oliver S, Lymburner L, et al. The Australian Geoscience Data Cube - Foundations and lessons learned[J]. Remote Sensing of Environment, 2017,202:276-292.
[13]	顾行发, 周翔, 张松梅, 等. 亚洲大洋洲区域综合地球观测系统计划进展[J]. 遥感学报, 2018,22(4):658-671.
	[ Gu X F, Zhou X, Zhang S M, et al. Asia-Oceania GEOSS: A GEO initiative to implement GEOSS for sustainable development in Asia-Oceania[J]. Journal of Remote Sensing, 2018,22(4):658-671. ]
[14]	Open Data Cube. Data Catalog: Getting Into the Cube[EB/OL]. [2019-05-02]. https://www.opendatacube.org/installation.
[15]	Lewis A, Lymburner L, Purss M B J, et al. Rapid, high-resolution detection of environmental change over continental scales from satellite data-the Earth Observation Data Cube[J]. International Journal of Digital Earth, 2016,9(1):106-111.
[16]	Sixsmith J, Oliver S, Lymburner L. A hybrid approach to automated landsat pixel quality[C] // Geoscience& Remote Sensing Symposium. Melbourne: IEEE, 2013: 4146-4149.
[17]	Dhu T, Dunn B, Lewis B, et al. Digital earth Australia- unlocking new value from earth observation data[J]. Big Earth Data, 2017,1(1):64-74. doi: 10.1080/20964471.2017.1402490
[18]	Dwyer J L, Roy D P, Sauer B, et al. Analysis ready data: Enabling analysis of the landsat archive[J]. Remote Sensing, 2018,10(9):1363.
[19]	O Ben-Kiki, C Evans, B Ingerson. YAML Ain't Markup Language[S/OL]. (2009-10-01) [2019-05-15]. https://yaml.org/spec/cvs/spec.pdf.
[20]	Rew R, Davis G. NetCDF: An interface for scientific data access[J]. IEEE Computer Graphics & Applications, 1990,10(4):76-82.
[21]	况润元, 谢佳, 张萌, 等. 基于实测数据的鄱阳湖悬浮泥沙粒度遥感反演模式分析[J]. 长江科学院院报, 2017,34(11):132-137.
	[ Kuang R Y, Xie J, Zhang M, et al. Models of remote sensing retrieval of suspended sediment particle size in the Poyang Lake based on measured spectral reflectance data[J]. Journal of Yangtze River Scientific Research Institute, 2017,34(11):132-137. ]
[22]	万松贤, 兰志春, 刘以珍, 等. 基于NDVI指数的鄱阳湖丰水期湿地植被覆盖对水文情势变化的响应[J]. 南昌大学学报(理科版), 2017,41(4):341-348.
	[ Wan S X, Lan Z C, Liu Y Z, et al. The response of wetland vegetation cover to hydrological regime change in Poyang Lake during the wet season based on NDVI index[J]. Journal of Nanchang University(Natural Science), 2017,41(4):341-348. ]
[23]	方朝阳, 邬浩, 陶长华, 等. 鄱阳湖南矶湿地景观信息高分辨率遥感提取[J]. 地球信息科学学报, 2016,18(6):847-856.
	[ Fang C Y, Wu H, Tao C H, et al. The wetland information extraction research of Nanji wetland in Poyang lake based on high resolution remote sensing image[J]. Journal of Geo-information Science, 2016,18(6):847-856. ]
[24]	McFEETERS S K. The use of the Normalized Difference Water Index(NDWI) in the delineation of open water features[J]. International Journal of Remote Sensing, 1996,17(7):1425-1432.
[25]	朱小强, 丁建丽, 夏楠, 等. 一种稳定阈值的湖泊水体信息提取方法[J]. 资源科学, 2019,41(4):790-802.
	[ Zhu X Q, Ding J L, Xia N, et al. Temperature vegetation water index: A novel stabilized threshold method for lake surface water mapping[J]. Resources Science, 2019,41(4):790-802. ]
[26]	李国庆, 黄震春. 遥感大数据的基础设施:集成、管理与按需服务[J]. 计算机研究与发展, 2017,54(2):267-283.
	[ Li G Q, Huang Z C. Data infrastructure for remote sensing big data:integration,management and on-demand service[J]. Journal of Computer Research and Development, 2017,54(2):267-283. ]

	处理时间/s	占用空间/GB	空间冗余率/%
ODC	79.06	15.05	7.2
IDL	227.55	14.04	0.0
ArcPy	245.41	14.04	0.0