Automatic data quality control for understanding extreme climate event


Contact
brenner.silva [ at ] awi.de

Abstract

The understanding of extreme events strongly depends on knowledge gained from data. Data integration of mul-tiple sources, scales and earth compartments is the fo-cus of the project Digital Earth, which also join efforts on the quality control of data. Automatic quality control is embedded in the ingest component of the O2A, the ob-servation-to-archive data flow framework of the Alfred-Wegener-Institute. In that framework, the O2A-Sensor provides observation properties to the O2A-Ingest, which delivers quality-flagged data to the O2A-dash-board. The automatic quality control currently follows a procedural approach, where modules are included to implement formulations found in the literature and other operational observatory networks. A set of plausibility tests including range, spike and gradient tests are cur-rently operational. The automatic quality control scans the ingesting data in near-real-time (NRT) format, builds a table of devices, and search - either by absolute or derivative values - for correctness and validity of obser-vations. The availability of observation properties, for in-stance tests parameters like physical or operation ranges, triggers the automatic quality control, which in turn iterates through the table of devices to set the qual-ity flag for each sample and observation. To date, the quality flags in use are sequential and qualitative, i.e. it describes a level of quality in the data. A new flagging system is under development to include a descriptive characteristic that will comprise technical and user inter-pretation. Within Digital Earth, data on flood and drought events along the Elbe River and methane emissions in the North Sea are to be reviewed using automatic qual-ity control. Fast and scalable automatic quality control will disentangle uncertainty raised by quality issues and thus improve our understanding of extreme events in those cases.



Item Type
Conference (Poster)
Authors
Divisions
Primary Division
Programs
Primary Topic
Research Networks
Publication Status
Published
Event Details
2nd International REKLIM Conference, 23 Sep 2019 - 26 Sep 2019, Berlin.
Eprint ID
51315
Cite as
Silva, B. , Koppe, R. , Haas, A. , Schäfer-Neth, C. , Fischer, P. , Immoor, S. , Gerchow, P. , Fritzsch, B. and Frickenhaus, S. (2019): Automatic data quality control for understanding extreme climate event , 2nd International REKLIM Conference, Berlin, 23 September 2019 - 26 September 2019 .


Download
[thumbnail of REKLIM_TP5P19_AWI_BSILVA_col.pdf]
Preview
PDF
REKLIM_TP5P19_AWI_BSILVA_col.pdf

Download (1MB) | Preview

Share
Add to AnyAdd to TwitterAdd to FacebookAdd to LinkedinAdd to PinterestAdd to Email

Geographical region
N/A

Research Platforms
N/A

Campaigns
N/A


Actions
Edit Item Edit Item