D Helpers,

I have a ton of huge financial data to process and analyze statistically.  I am looking for the best way to store/retrieve the data for processing. I am wondering how to write the data to a compressed binary file and then later read from that same file for processing.

I can read/write the binary data just fine, but how to do it with compression?

I appreciate your help and suggestions.

TJB

On Wed, 19 Sep 2012 19:48:49 +0200, TJB wrote:

> D Helpers,
> 
> I have a ton of huge financial data to process and analyze statistically.  I am looking for the best way to store/retrieve the data for processing. I am wondering how to write the data to a compressed binary file and then later read from that same file for processing.
> 
> I can read/write the binary data just fine, but how to do it with compression?
> 
> I appreciate your help and suggestions.
> 
> TJB

Take a look at std.zip and std.zlib.  Personally, I use LZO which is incredibly fast and pretty easy to wrap (it's a C lib).  Using LZO takes a little more work (you need split the data into chunks, manually store compressed/uncompressed lengths, etc.), so the easy route is to use std.zlib.