TeaFile Specification

Purpose

The TeaFile format provides a very efficient and simple way to persist time series data in flat files. Items of homogeneous time series are stored in raw binary format. The header holds optional descriptions of the binary layout of the items or the file content. This removes the opaqueness from the binary data stored, making TeaFiles a self contained transparent storage method that also serves as the data centric interface between applications.

Audience

TeaFiles can easily be read using the canonical file I/O functions. APIs will provide a more convenient and safe way to access such files and will be prefered most times. This document targets API writers and those accessing or analyzing TeaFiles directly. It assumes basic background in data representations and programming.

Goals and Forces

• The origin and focus of TeaFiles is best achievable performance of time series data access.
• Making binary formatted time series file self contained removing their opaqueness was the second goal: Someone who receives the file "x-series" by email, not knowing anything about its contents, can open it and read it.
• Such self contained files are suitable for data interchange between applications, a very common and important task in time series data analysis. To ease data exchange between arbitrary appplications, was the third principle that guided the file format design.
• Finally, the file format shall make no assumptions about hardware or operating system used.

Design

To achive best possible performance, time series data is stored in TeaFiles such that it can be mapped directly into memory. Data might also be written by traditional read/write IO functions, but ensuring the possibility to memory map the data puts a stronger constraint upon the file format, as it requires the data to be written in the form of the data item's binary memory footprint. The platform neutrality requirement therefore can only be achieved if the file has no fixed endianness specification. On a little endian machine, a TeaFile must be in little endian format to allow memory mapping, while on a big endian it must be in big endian format.

Platform compatibility (operating sysytem, cpu type or application) is approached by restriction to ubiquitious number formats, in particular the IEEE 754 floating point numbers. Other platform specific data types are still allowed. If used, they will affect portability.

To ease data exchange via TeaFiles, simple access to their data via primitive IO functions is provided. Raw access requires not more than reading of the first 4 mandatory values in the file. A useful API is therefore written within minutes and can then be incrementally extended to read also the optional sections inside the file that describe the item layout and content in the file.

Items

Items must be primitive types like integers or doubles or structures of such primitive types. Pointers to memory or strings can not be stored. While the design targeted time series storage, in which case each item has one value that holds a time value, the file format is also useful to generally store homogeneous collections of items (arrays) without time stamp.

File Format Progression

Thi file format specification is expected to change rarely. Upcoming needs for changes are supposed to be handled by a adding new sections or possibly accompanied by dropping others. If for instance the Item Layout Section should be changed, a new section (Item-Layout-2) with a different id would be proposed that has the new format.

Sample File

A sample TeaFile holding Time/Price/Volume data, a content description, name value description and a description of the time format used might have the following binary layout:

struct Tick
{
    public Time Time;
    public double Price;
    public long Volume;
}
TeaFile<Tick>.Create("lab.tea", "ACME prices", NameValueCollection.From("decimals", 2));

The C# code above creates this binary file:

File Format

Header + Data

TeaFiles start with a header followed by the item area holding the data in binary format.

Mandatory Header Fields

Magic Value

Magic value for byte order checking and file detections.

The first octet of a TeaFile are alays the magic number 0x0d0e0a0402080500 which can be memorized as follows:

0d  0e  0a  04  02  08  05  00
 T   e   a for two  at   5 :00

If an application does not find this sequence at the file begin, it should raise an exception. In such case, the file might have been written by a different endianness application. TeaFiles must have the endianness of the local machine so they can be mapped directly into memory. If files are transfered between systems, the file might need endian conversion before it can be read correctly or mapped into memory.

This magic number allows to

• check if the file's endianness matches the local endianness and to
• detect whether a file is a TeaFile.

An API can translate different endian-formatted files on the fly for read and write operations, for memory mapping purposess the file would need to be converted.

Item Area Start (ItemStart)

The absolute byte offset of the item area.

Setting the file pointer to this position allows reading of the first item in the file. If the first value is placed directly after the header, this value equals the size of the header. Often however, it will be larger: First, the data area should be aligned at the 8 byte boundary to avoid considerable performance penalties from unaligned double or long values. Second, if the description of the file is modified at a time it contains data, the header might increase in size and the change will thus require the item area to be moved. To avoid costly data moving, the file can be created with some headroom behind the header.

Item Area End (ItemEnd)

The absolute byte offset of the item area end.

If this value is 0, then the end of the physical file equals the end of the item area. Otherwise it specifies the absolute byte offset of the end of the item area from the begin of the file.

Leaving this value at 0 provides simple addition of new items to the file. Besides writing new items to the end of the file, no further updates are required. To avoid fragmentation, files might initially be preallocated at a larger size. To allow this, the ItemEnd value can be set to determine the logical end of file. The value of ItemEnd will always be in the range

Sections Count

The TeaFile header can optionally hold an arbitrary number of sections. This is the number of such sections.

Optional Header Fields - Sections

TeaFiles may contain sections describing their content. Each section is optional, if it occurs it must occur only once. This document defines several sections and leaves room for custom sections that allow storage of additional features. For each section (official or custom) the following fields are written:

Section Id

Each section has a unique ID. The id of the first section is always at position 32 in the file. Its value is one the of the section id values specified in this document or a custom section id above 0xffff. The IDs specified in this document are:

Next Section Offset

Sections are written into the file one after another. Users interested in specific sections only can read the Section Id, decide whether to read the section and optionally skip it by reading the next section offset and adding it to the file pointer to reach the next section id.

This value might equal the length of the section but might also be higher due to padding or because some headroom was reserved for the section such that modifications of the section can be done without affecting other sections or the file's item area.

For the last section, the next section offset shall be ignored by file readers.

Item Size

The size of the item. This is the space occupied by each item in the file, including any padding bytes.

Item Name

The name of the type, like "OHLCV", as UTF8 string. Strings are written length prefixed (int32) following by the text data in UTF8.

Field Count

The number of fields in the item. This value must be >= 1. If the Item Type of a TeaFile is a plain primitive value, the Item Name might be "Int" while the single field value might be "Value". In other words primitives are treated as structs with one field having the type of the primitive.

Fields

For each field, a Field description is written holding:

• Type of the field
• Offset of the field inside the item data structure
• Name

Field.Type

The types of fields inside a TeaFile should be available on as many platforms as possible to support versatile data access. At the same time, TeaFiles might be used in focused scenarios were such ubiquitous data access possibility is not required and where other formats are desirable. The file format foresees this possibility.

FieldType is either a value from 1 to 10, indicating a platform agnostic, ubiquitous type or a platform specific type which currently encompasses the .Net decimal. For fields holding other types, one can use value the values >= 0x1000.

    // platform agnostic
    Int8 = 1,
    Int16 = 2,
    Int32 = 3,
    Int64 = 4,

    UInt8 = 5,
    UInt16 = 6,
    UInt32 = 7,
    UInt64 = 8,

    Float = 9, // IEEE 754
    Double = 10, // IEEE 754

    // platform specific
    NetDecimal = 0x200,

    // private extensions must have integer identifiers above 0x1000.
    Custom = 0x1000

Field.Offset

The field's byte-offset inside the item.

Field.Name

The name of the field as UTF8 string. Strings are written length prefixed (int32) following by the text data in UTF8.

Content Description Section

Description

A string that describes the content of the file. Examples: "Daily average temperature Easy Village", "Prices of ACME at NYSE".

Name/Value Section

Name/Value Count

The number of name/Value pairs to follow.

Name/Value Pairs

For each Name/Value pair, the following values are written

• Name
• Kind
• Value

NameValue.Name

The name.

NameValue.Kind

One of the following integer values, indicating the type of the value to follow:

• 1 - Int32
• 2 - Double
• 3 - Text
• 4 - Uuid

NameValue.Value

Dependant on the previous value of Kind, the value will hold an int32, double, string or uuid value. A uuid value is a sequence of 16 bytes considered to be unique.

Time Section

This section describes how a number is interpreted as time and which fields in the item are to be interpreted as time.

Epoch

Time is specified as the number of "tick" intervals that elapsed since some origin. The epoch value gives this origin as the number of days that passed between since 1.1.0000 and the origin. For the .Net System.DateTime Type, that counts ticks since 1.1.0000 this value would be 0. For time systems counting from 1.1.1970, the epoch is 719162.

Ticks Per Day

This value specifies how many ticks are counted per day. Conversely this number specifies the length of the tick interval. A value of 1 would mean that days are counted, 86400 would mean seconds and 86400000 milliseconds.

For TeaFiles that shall be accessed from various platforms, the Java Time format is recommended, as it often sufficient resolution of milliseconds and due to its origin at 1.1.1970, its values remain much smaller than that of .Net System.DateTime that gives values hard to deal with in applications that cannot handle 64 bit integers like R.

Time Fields Count

For time series, the item will hold one field that is the event time, the time at which the observation or the event occured. If the file holds a plain collection of values and is not a time series, then no time field might exist. On the other hand items might have an event time and in addition ther time fields. In summary the number of time fields is between 0 and the number fo fields in the item.

If an item has more than one time field, the firstone is considered the event time. The values of this field must then be non-decreasing. In other words each event must have a time >= time of the previous event.

Time Fields

As many int32 values follow as [Time Fields Count] indicates. Each of these values holds the offset of the corresponding field.

The reason to use the offset instead of the field index is that an application that shall select an interval of values from a TeaFile, is required to read the Time Section only in order to deliver all values from t1 to t2.