Hello,

I am trying to save some data to compressed binary files.  I have my data in a struct, which I can write to a binary file just fine:

align(1) struct TradesBin {
  int ttim;
  int prc;
  int siz;
  short g127;
  short corr;
  char[2] cond;
  char ex;
}

auto fout = new BufferedFile(outfile, FileMode.Out);
fout.writeExact(&bin, TradesBin.sizeof);

Where bin is of type TradesBin, filled with data.  The question is how do I now do this with zlib to get compression?  Suggestions and help are mightily appreciated!

TJB

On Mon, 24 Sep 2012 22:10:04 +0200, TJB wrote:

> Hello,
> 
> I am trying to save some data to compressed binary files.  I have my data in a struct, which I can write to a binary file just fine:
> 
> align(1) struct TradesBin {
>    int ttim; int prc;
>    int siz; short g127; short corr; char[2] cond;
>    char ex;
> }
> 
> auto fout = new BufferedFile(outfile, FileMode.Out);
> fout.writeExact(&bin, TradesBin.sizeof);
> 
> Where bin is of type TradesBin, filled with data.  The question is how
> do I now do this with zlib to get compression?
> Suggestions and help are mightily appreciated!
> 
> TJB

Since you're already set up with the Stream interface, try creating a MemoryStream instead of a BufferedFile.  Write to the stream just as you are now, then use the .data() property (on MemoryStream's superclass, TArrayStream) to get an array of raw bytes.  You can feed this array into the compress function in std.zlib to produce a new (hopefully shorter) array of bytes that contains the compressed data.  Simply write this compressed data to a file with `fout.writeExact(compressed.ptr, compressed.length)` and you're done.

Since uncompress works better if you know the exact size of the uncompressed data, you might also consider writing the uncompressed size of the data to your output file before the compressed data.

Justin

> Since you're already set up with the Stream interface, try creating a
> MemoryStream instead of a BufferedFile.  Write to the stream just as you
> are now, then use the .data() property (on MemoryStream's superclass,
> TArrayStream) to get an array of raw bytes.  You can feed this array into
> the compress function in std.zlib to produce a new (hopefully shorter)
> array of bytes that contains the compressed data.  Simply write this
> compressed data to a file with `fout.writeExact(compressed.ptr,
> compressed.length)` and you're done.
>
> Since uncompress works better if you know the exact size of the
> uncompressed data, you might also consider writing the uncompressed size
> of the data to your output file before the compressed data.
>
> Justin

Ok, I think I get it.  Can you help me set up the first part, writing to the MemoryStream?

TJB

On 25-Sep-12 00:10, TJB wrote:
> Hello,
>
> I am trying to save some data to compressed binary files.  I have my
> data in a struct, which I can write to a binary file just fine:
>
> align(1) struct TradesBin {

Just a note: recently compiler changed so this align(1) have no effect on the fields alignment only on a struct as a whole.
So instead
align(1):
inside of struct should probably work.
>    int ttim;
>    int prc;
>    int siz;
>    short g127;
>    short corr;
>    char[2] cond;
>    char ex;
> }
>
> auto fout = new BufferedFile(outfile, FileMode.Out);
> fout.writeExact(&bin, TradesBin.sizeof);
>
> Where bin is of type TradesBin, filled with data.  The question is how
> do I now do this with zlib to get compression? Suggestions and help are
> mightily appreciated!
>
> TJB
>


-- 
Dmitry Olshansky

On Tue, 25 Sep 2012 03:25:37 +0200, TJB wrote:

>> Since you're already set up with the Stream interface, try creating a
>> MemoryStream instead of a BufferedFile.  Write to the stream just as
>> you are now, then use the .data() property (on MemoryStream's
>> superclass,
>> TArrayStream) to get an array of raw bytes.  You can feed this array
>> into the compress function in std.zlib to produce a new (hopefully
>> shorter)
>> array of bytes that contains the compressed data.  Simply write this
>> compressed data to a file with `fout.writeExact(compressed.ptr,
>> compressed.length)` and you're done.
>>
>> Since uncompress works better if you know the exact size of the uncompressed data, you might also consider writing the uncompressed size of the data to your output file before the compressed data.
>>
>> Justin
> 
> Ok, I think I get it.  Can you help me set up the first part, writing to the MemoryStream?
> 
> TJB

I wrote up a quick example program and pasted it here:
  http://dpaste.dzfl.pl/f1699d07

Hope that helps you out.

On Tuesday, 25 September 2012 at 18:08:44 UTC, Justin Whear wrote:

> I wrote up a quick example program and pasted it here:
>   http://dpaste.dzfl.pl/f1699d07
>
> Hope that helps you out.

Justin,

Thanks so much! Very helpful!!!

TJB

On Tue, 25 Sep 2012 18:08:44 +0000, Justin Whear wrote:

> On Tue, 25 Sep 2012 03:25:37 +0200, TJB wrote:
> 
>>> Since you're already set up with the Stream interface, try creating a
>>> MemoryStream instead of a BufferedFile.  Write to the stream just as
>>> you are now, then use the .data() property (on MemoryStream's
>>> superclass,
>>> TArrayStream) to get an array of raw bytes.  You can feed this array
>>> into the compress function in std.zlib to produce a new (hopefully
>>> shorter)
>>> array of bytes that contains the compressed data.  Simply write this
>>> compressed data to a file with `fout.writeExact(compressed.ptr,
>>> compressed.length)` and you're done.
>>>
>>> Since uncompress works better if you know the exact size of the uncompressed data, you might also consider writing the uncompressed size of the data to your output file before the compressed data.
>>>
>>> Justin
>> 
>> Ok, I think I get it.  Can you help me set up the first part, writing to the MemoryStream?
>> 
>> TJB
> 
> I wrote up a quick example program and pasted it here:
>   http://dpaste.dzfl.pl/f1699d07
> 
> Hope that helps you out.

I should point out that in my example, the writing to the MemoryStream is not strictly necessary.  I could have simply cast the foo array to an array of bytes and passed that into the compress function.  But if you're already set up for writing raw data into a stream, the MemoryStream will help you minimize code changes as well as facilitate more complex scenarios.

On Tuesday, 25 September 2012 at 18:08:44 UTC, Justin Whear wrote:

> I wrote up a quick example program and pasted it here:
>   http://dpaste.dzfl.pl/f1699d07
>
> Hope that helps you out.

OK, I think I am working this out.  Let's say I want to write the data to a gzipped file that will be inter-operable with gzip at the command line. I think
I do something like the following:

  auto mem = new MemoryStream();
  mem.writeExact(foo.ptr, foo.length * A.sizeof);

  auto gzdat = new Compress(9, HeaderFormat.gzip);
  gzdat.compress(mem.data);

But then, what do I do with gzdat?  Do I try to write it to a file like this:

  auto fout = new std.stream.File("data.bin.gz", FileMode.OutNew);
  fout.writeExact(gzdat.ptr, gzdat.sizeof);

Something is wrong though.  Hmm ...

TJB