DASHBOARD.awi.de: Streaming and monitoring solutions for near real-time data


Contact
roland.koppe [ at ] awi.de

Abstract

DASHBOARD.awi.de is a component of our data flow framework designed to enable a semi-automated flow of sensor observations to archives (acronym O2A). The dramatic increase in the number and type of platforms and respective sensors operated by Alfred Wegener Institute along with complex project-driven requirements in terms of satellite communication, sensor monitoring, quality control and validation, processing pipelines, visualization, and archival under FAIR principles, led us to build a generic and cost-effective data flow framework. Most important, all components and services which make up this framework are extensible and exchangeble, were built using open access technologies (e.g. elastic search) and vocabularies (SeaVox NERC 2.0 vocabulary) and are compliant with various interoperability standards recommended by the international community. In this poster we illustrate the DASHBOARD.awi.de component which is a web-based monitoring environment for supporting scientists in the graphing, mapping and simple analysis of time series. With a set of fit-for-purpose widgets including data download, scientists are able to identify gaps and outliers in the streamed data. Morover, we are in the process of building alerting solutions for individual parameters using the parameter properties available from SENSOR.awi.de (e.g. min/max parameter range). The streaming services attached to this component are using the near real-time data transfered from remote field sites to local databases and storage systems. For this sake, we are participating in the RDA "Array Database Assessment" Working Group and "Big Data" Interest Group. The graphing solutions built within DASHBOARD.awi.de can be easily re-used in other context (e.g. project web pages) . Our solutions support the OGC standard Observations and Measurements (O&M), JSON and CSV as exchange data format. To date, ~ 150 individual parameters are being transfered in near real-time from remote sites to our onshore storage systems and monitored by scientists. In the week of the 11th RDA Plenary Berlin, we will be able to interact with live data from Polarstern crossing the Drake Passage, on its way to Antarctica. Scientific Discipline/Research Area: Data flow, standardized vocabulary, Visualisation, Streaming. Relevance/Link to RDA: The streaming services attached to this component are using the near real-time data transfered from remote field sites to local databases and storage systems. For this sake, we are participating in the RDA "Array DB Assesment" Working Group and "Big Data" Interest Group.



Item Type
Conference (Poster)
Authors
Divisions
Primary Division
Programs
Primary Topic
Peer revision
Not peer-reviewed
Publication Status
Published
Event Details
RDA plenary 2018, 21 Mar 2018 - 23 Mar 2018, Berlin.
Eprint ID
46854
Cite as
Koppe, R. , Gerchow, P. , Macario, A. , Haas, A. , Schäfer-Neth, C. , Rehmcke, S. , Walter, A. , Düde, T. , Weidinger, P. , Schäfer, A. and Pfeiffenberger, H. (2018): DASHBOARD.awi.de: Streaming and monitoring solutions for near real-time data , RDA plenary 2018, Berlin, 21 March 2018 - 23 March 2018 .


Download
[img]
Preview
PDF
O2A_dashboard_rda_2018_03_21.pdf

Download (755kB) | Preview

Share

Research Platforms
N/A

Campaigns


Actions
Edit Item Edit Item