Data import

The Data import module configures how hydro meteo data (water levels, currents, waves, wind) is retrieved from external sources, processed and stored. Data is periodically collected, parsed, transformed and saved. The stored data is shown on dashboards and used in tidal window calculations.

Data collection

Data collection is managed through a hierarchy of servers, interface types, data types and collectors.

The Data servers submodule manages the external sources and their authentication. The Interface types submodule defines the protocols and parameters used to access those servers. The Data types submodule defines the kinds of data that can be collected. A data type provides the link between collected data and the processor that handles it.

The actual retrieval of data is managed in the Collectors submodule. Each collector is defined for a data type. It specifies the data server and interface type to retrieve from, along with parameters such as the data location and model. A collector runs on a schedule, based on its configured polling interval. When it runs, it retrieves the data and stores it in the Raw data submodule, linked to its data type.

Analysis times

Some sources separate analysis times from their data series. For these, data collection can be split into two steps. The first collector polls for new analysis times. Through data processing, each new analysis time triggers a second collector that retrieves the full data series. The second collector's polling interval should be set to zero, to prevent it from polling data independently.

Manual import

Use the Manual data collection submodule to trigger a specific collector by hand. The response can be reviewed and optionally saved.

The File upload submodule lets you upload a data file for a specific data type.

Data processing

Data in the Raw data submodule still needs to be processed. There are two possible processing methods, depending on the data format.

Parsers / Data type processors

This method should be used for NetCDF files, CSV files and analysis times provided as plain text.

The Parsers submodule manages the logic for interpreting specific file formats. Each parser has a description, a type and specific parameters. A parser is then linked to a data type in the Data type processors submodule. The behavior depends on the type of data type processor:

A Hydro meteo: Analysis time processor activates a data collector.
A Hydro meteo: Data processor saves the processed data to the database, for the configured index (from the parser), series and location.

Processors

This method should be used for all other data formats.

Define a data processor at Data > Processors > Processors, choosing one of the predefined types. Then couple it to a data type at Data > Processors > Data types, where the unit and parameters are also set. The parameters can be used to trigger a data collector, or to add a conversion factor. A conversion factor can change the reference plane (for example from NAP to LAT) or the unit magnitude (for example from centimeters to meters).

Finally, configure how the processed data is stored, by coupling it to a data series and location at Settings > Data import > Processor-settings coupling.