Format and File Structure of Time Series Data

Introduction

In ESS systems like AirWare, RiskWare or WaterWare, regular time series data such as air quality measurements, meteorological observations, hydrological data, emission data can be stored in a simple ASCII file format, associated with the respective measurement station (object).

Alternatively, these data can be stored in, and retrieved from, any datbase management system that supports embedded SQL calls. To support legacy software and existing data bases, the connection to such an RDBMS will be customized on a case by case basis to provide a completely transparent link to existing data systems.

For the time series data, several display and analysis functions are provided. The data can be displayed (line graphs, histograms) and analysed (spatial analysis with interpolation, several statistical methods). The data are stored in an ASCII time series data file . Alternative file formats, or the coupling to an existing RDBMS (e.g., ORACLE) is possible.

Each time series data set is assigned to a specific OBJECT. This object describes the real-world entity that was the source for the time series data (i.e. an air quality measurement station, a river gauge etc). Each data object has distinct properties such as its geographical location, name, identification number etc.. All properties of an object are stored in an individual object data file (ASCII TEMPLATE format).

Each station object is a member of an object CLASS. Each object class has a name and a class id, and defines the available functionality (display, analysis) of its members. All available object classes are listed in a configuration file CONFIG (located in the ./data/objects directory). This file also defines the names, id numbers and location of object data files for each object of a class. The CONFIG file configures and drives the object class and object selectors.

Format of the Configuration (CONFIG) File

CLASS                                       
Messdaten:_NO2   CS                           
E AKH     CS2030 ./data/objects/air/akh_no2.dat   
E EXAMPLE CS333  ./data/objects/air/example.object
 .....
 .....
ENDCLASS                                         

CLASS                                           
Air_Quality_Stations:_SO2 CS                   
E Site_A CS01 ./data/objects/sites/site_A.dat
E Site_B CS02 ./data/objects/sites/site_B.dat
 .....
 .....
ENDCLASS

CLASS
Scenic_sites SS
E Gaza       SS01 ./data/objects/sites/Gaza.dat
E Ras_Karkar SS02 ./data/objects/sites/RasKarkar.dat 
ENDCLASS

A class is defined after the identifier CLASS in the configuration file with a name, here: Messdaten:_NO2, and a class ID, here: CS. This is followed by a listing of the members or elements of the class E, each identified by a name, and ID, and a file name that refers to the actual time series data. The listing of the class members is ended with the ENDCLASS identifier, after which another class can be defined. The sequence of classes and members as defined in the CONFIG file is used for the display in the class and object selectors, respectively.

The name is used in the display, the ID for internal reference purposes. each class has its own specific display and analysis functionality. Time series data use the CS (originally from climate station) identifier, which will trigger the time series analysis functionality.

Format of Object Data File


NA Station_name                # object name 
ID CS333                       # object id 
LA 42460                       # latitude (y coordinates)
LO 51070                       # longitude (x coordinates)
EL 190.                        # elevation (m)
PI file/name                   # description/picture of site 
HY OBJECTS/air/akh.html        # hypertext explanation for site 
TABLE
georeference                   # geographical framework
E  string_name                 # geographical unit
E  string_value                # unit name 
E  overlay                     # not used
E  attribute                   # not used
D  Land       Land_Wien        0 0
D  Gemeinde   Stadt_Wien       0 0
D  Bezirk     Bezirk_Doebling  0 0
D  Strasse    Hohe_Warte       0 0
END_TABLE
TS ./air/example.data      NO2 # location and parameter

The station object is defined by a header block, that includes the station name, following the keyword NA, as used in the display. The station ID is following the ID keyword, and must be consistent with the ID used in the CONFIG file. LA and LO define the map location of the station in the coordinate system used for the GIS of a given application. For urban applications, this is usually a Gauss-Krueger projection in meters. EL is followed by elevation above sea level in meters.

PI and HY refer to two alternative hypertext files that contain background information in hypertext format about the measurement station. Please note that the current implementation of the observation station objects does not use the PI link. Depending on the implementation, the hypertext format used is either a specific hypertext format used by the ACA ToolKit, or HTML.

Additional georeferencing uses between the keywords TABLE and END_TABLE an (optional) overlay name and id to associate the object to a geographical object (overlay) in the GIS; and a list of currently four hierarchically structured geographical units (for example: county, city, district, street). Please note that this georeferencing (to be used other than for display purposes in the station objects description) requires that the corresponding GIS coverages are available.

The reference to the actual time series data follows the TS keyword and consists of the file name of the data, and the name of the variable (in this example: NO2). The latter must be consistent with the respective Descriptor definition in the systems knowledge base, also used by the embedded expert system. The descriptor definition provides the name, unit, and allowable range of the variable, as well as meta information specifying allowable operations depending on its nature.

Format of Time Series Data File


ID za_no2                # TS id       
NA ZA_1995_NO2           # TS name 
ME Stickstoffdioxid      # parameter name
DE NO2                   # descriptor name
BE 199501010030          # begin (start date)
ST 30                    # time step in minutes 
NV 17520                 # total number of values
33 34 33 32 28 28 24 25  # data values
24 26 24 22 21  7  5  6 
 7 11 12  8  7  8         
 9  7  9 10  9  8  9 
10 9 7 6 7 8 6 
 .....
 .....

The time series has again an ID and a name (NA). The name of the variable ME is used for display purposes, the descriptor name DE provides the link to the expert system's knowledge base with the parameters unit und allowable range.

The data set is characterized by:

a start date BE in the format YYYMMDDHHMM, which translates in to January 1, 1995, 00:30 in the above example. Please note that time steps below one minute are not supported in this version.
The time step ST is again given in minutes (30 in the example).
The total number of values NV is followed by the list blank-separated of data values integer or float notation).