View mode: basic / threaded / horizontal-split · Log in · Help
September 24, 2012
How do I use std.zlib to write a struct of data to a binary file with compression?
Hello,

I am trying to save some data to compressed binary files.  I have 
my data in a struct, which I can write to a binary file just fine:

align(1) struct TradesBin {
  int ttim;
  int prc;
  int siz;
  short g127;
  short corr;
  char[2] cond;
  char ex;
}

auto fout = new BufferedFile(outfile, FileMode.Out);
fout.writeExact(&bin, TradesBin.sizeof);

Where bin is of type TradesBin, filled with data.  The question 
is how do I now do this with zlib to get compression?  
Suggestions and help are mightily appreciated!

TJB
September 24, 2012
Re: How do I use std.zlib to write a struct of data to a binary file with compression?
On Mon, 24 Sep 2012 22:10:04 +0200, TJB wrote:

> Hello,
> 
> I am trying to save some data to compressed binary files.  I have my
> data in a struct, which I can write to a binary file just fine:
> 
> align(1) struct TradesBin {
>    int ttim; int prc;
>    int siz; short g127; short corr; char[2] cond;
>    char ex;
> }
> 
> auto fout = new BufferedFile(outfile, FileMode.Out);
> fout.writeExact(&bin, TradesBin.sizeof);
> 
> Where bin is of type TradesBin, filled with data.  The question is how
> do I now do this with zlib to get compression?
> Suggestions and help are mightily appreciated!
> 
> TJB

Since you're already set up with the Stream interface, try creating a 
MemoryStream instead of a BufferedFile.  Write to the stream just as you 
are now, then use the .data() property (on MemoryStream's superclass, 
TArrayStream) to get an array of raw bytes.  You can feed this array into 
the compress function in std.zlib to produce a new (hopefully shorter) 
array of bytes that contains the compressed data.  Simply write this 
compressed data to a file with `fout.writeExact(compressed.ptr, 
compressed.length)` and you're done.

Since uncompress works better if you know the exact size of the 
uncompressed data, you might also consider writing the uncompressed size 
of the data to your output file before the compressed data.

Justin
September 25, 2012
Re: How do I use std.zlib to write a struct of data to a binary file with compression?
> Since you're already set up with the Stream interface, try 
> creating a
> MemoryStream instead of a BufferedFile.  Write to the stream 
> just as you
> are now, then use the .data() property (on MemoryStream's 
> superclass,
> TArrayStream) to get an array of raw bytes.  You can feed this 
> array into
> the compress function in std.zlib to produce a new (hopefully 
> shorter)
> array of bytes that contains the compressed data.  Simply write 
> this
> compressed data to a file with `fout.writeExact(compressed.ptr,
> compressed.length)` and you're done.
>
> Since uncompress works better if you know the exact size of the
> uncompressed data, you might also consider writing the 
> uncompressed size
> of the data to your output file before the compressed data.
>
> Justin

Ok, I think I get it.  Can you help me set up the first part, 
writing to the MemoryStream?

TJB
September 25, 2012
Re: How do I use std.zlib to write a struct of data to a binary file with compression?
On 25-Sep-12 00:10, TJB wrote:
> Hello,
>
> I am trying to save some data to compressed binary files.  I have my
> data in a struct, which I can write to a binary file just fine:
>
> align(1) struct TradesBin {

Just a note: recently compiler changed so this align(1) have no effect 
on the fields alignment only on a struct as a whole.
So instead
align(1):
inside of struct should probably work.
>    int ttim;
>    int prc;
>    int siz;
>    short g127;
>    short corr;
>    char[2] cond;
>    char ex;
> }
>
> auto fout = new BufferedFile(outfile, FileMode.Out);
> fout.writeExact(&bin, TradesBin.sizeof);
>
> Where bin is of type TradesBin, filled with data.  The question is how
> do I now do this with zlib to get compression? Suggestions and help are
> mightily appreciated!
>
> TJB
>


-- 
Dmitry Olshansky
September 25, 2012
Re: How do I use std.zlib to write a struct of data to a binary file with compression?
On Tue, 25 Sep 2012 03:25:37 +0200, TJB wrote:

>> Since you're already set up with the Stream interface, try creating a
>> MemoryStream instead of a BufferedFile.  Write to the stream just as
>> you are now, then use the .data() property (on MemoryStream's
>> superclass,
>> TArrayStream) to get an array of raw bytes.  You can feed this array
>> into the compress function in std.zlib to produce a new (hopefully
>> shorter)
>> array of bytes that contains the compressed data.  Simply write this
>> compressed data to a file with `fout.writeExact(compressed.ptr,
>> compressed.length)` and you're done.
>>
>> Since uncompress works better if you know the exact size of the
>> uncompressed data, you might also consider writing the uncompressed
>> size of the data to your output file before the compressed data.
>>
>> Justin
> 
> Ok, I think I get it.  Can you help me set up the first part,
> writing to the MemoryStream?
> 
> TJB

I wrote up a quick example program and pasted it here:  
 http://dpaste.dzfl.pl/f1699d07

Hope that helps you out.
September 25, 2012
Re: How do I use std.zlib to write a struct of data to a binary file with compression?
On Tuesday, 25 September 2012 at 18:08:44 UTC, Justin Whear wrote:

> I wrote up a quick example program and pasted it here:
>   http://dpaste.dzfl.pl/f1699d07
>
> Hope that helps you out.

Justin,

Thanks so much! Very helpful!!!

TJB
September 25, 2012
Re: How do I use std.zlib to write a struct of data to a binary file with compression?
On Tue, 25 Sep 2012 18:08:44 +0000, Justin Whear wrote:

> On Tue, 25 Sep 2012 03:25:37 +0200, TJB wrote:
> 
>>> Since you're already set up with the Stream interface, try creating a
>>> MemoryStream instead of a BufferedFile.  Write to the stream just as
>>> you are now, then use the .data() property (on MemoryStream's
>>> superclass,
>>> TArrayStream) to get an array of raw bytes.  You can feed this array
>>> into the compress function in std.zlib to produce a new (hopefully
>>> shorter)
>>> array of bytes that contains the compressed data.  Simply write this
>>> compressed data to a file with `fout.writeExact(compressed.ptr,
>>> compressed.length)` and you're done.
>>>
>>> Since uncompress works better if you know the exact size of the
>>> uncompressed data, you might also consider writing the uncompressed
>>> size of the data to your output file before the compressed data.
>>>
>>> Justin
>> 
>> Ok, I think I get it.  Can you help me set up the first part,
>> writing to the MemoryStream?
>> 
>> TJB
> 
> I wrote up a quick example program and pasted it here:
>   http://dpaste.dzfl.pl/f1699d07
> 
> Hope that helps you out.

I should point out that in my example, the writing to the MemoryStream is 
not strictly necessary.  I could have simply cast the foo array to an 
array of bytes and passed that into the compress function.  But if you're 
already set up for writing raw data into a stream, the MemoryStream will 
help you minimize code changes as well as facilitate more complex 
scenarios.
September 25, 2012
Re: How do I use std.zlib to write a struct of data to a binary file with compression?
On Tuesday, 25 September 2012 at 18:08:44 UTC, Justin Whear wrote:

> I wrote up a quick example program and pasted it here:
>   http://dpaste.dzfl.pl/f1699d07
>
> Hope that helps you out.

OK, I think I am working this out.  Let's say I want to write the 
data to a gzipped file that will be inter-operable with gzip at 
the command line. I think
I do something like the following:

  auto mem = new MemoryStream();
  mem.writeExact(foo.ptr, foo.length * A.sizeof);

  auto gzdat = new Compress(9, HeaderFormat.gzip);
  gzdat.compress(mem.data);

But then, what do I do with gzdat?  Do I try to write it to a 
file like this:

  auto fout = new std.stream.File("data.bin.gz", FileMode.OutNew);
  fout.writeExact(gzdat.ptr, gzdat.sizeof);

Something is wrong though.  Hmm ...

TJB
Top | Discussion index | About this forum | D home