What is HDF5, and why should you use it ?

http://www.hdfgroup.org/why_hdf/

(My summary):
  - very large data sets, very fast access requirements, and complex datasets
  - share data across variety of platforms
  - many open-source and commercial tools that understand HDF
  - self-describing and can specify complex data relationships and dependencies
  - can contain binary data in many representations
  - allow direct access to parts of file without first parsing whole contents
  - hierarchical data objects can be expressed in natural manner (contrast
    experience with realational database tables)
  - n-dimensional datasets and each element in set may be complex object
  - relational databases good for field matching queries but not for
    sequentially processing all records in database or for subsetting data
    based on co-ordinate style lookup
  - custom proprietary binary formats often not portable, not extensible and
    not high-performance.  technical debt to maintain data management part of
    code

I personally find it useful for storing price data for financial instruments, and also economic data.  There are bespoke time series databases, but they come at a price, which is not purely a pecuniary one.

Updated wrappers are here:
https://github.com/Laeeth/d_hdf5

Changes since last time - some fixes to bindings and updates to later version of HDF5 API.  There is more to go to make it accessible idiomatically from D, but it's usable today.  A simple example of mapping D structs to HDF5 types and back again is in the examples/traits directory.

Pull requests and offers to help maintain it are welcome.  It's still at an alpha stage, but already useful.



Laeeth.