Thread overview | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
September 24, 2012 How do I use std.zlib to write a struct of data to a binary file with compression? | ||||
---|---|---|---|---|
| ||||
Hello, I am trying to save some data to compressed binary files. I have my data in a struct, which I can write to a binary file just fine: align(1) struct TradesBin { int ttim; int prc; int siz; short g127; short corr; char[2] cond; char ex; } auto fout = new BufferedFile(outfile, FileMode.Out); fout.writeExact(&bin, TradesBin.sizeof); Where bin is of type TradesBin, filled with data. The question is how do I now do this with zlib to get compression? Suggestions and help are mightily appreciated! TJB |
September 24, 2012 Re: How do I use std.zlib to write a struct of data to a binary file with compression? | ||||
---|---|---|---|---|
| ||||
Posted in reply to TJB | On Mon, 24 Sep 2012 22:10:04 +0200, TJB wrote: > Hello, > > I am trying to save some data to compressed binary files. I have my data in a struct, which I can write to a binary file just fine: > > align(1) struct TradesBin { > int ttim; int prc; > int siz; short g127; short corr; char[2] cond; > char ex; > } > > auto fout = new BufferedFile(outfile, FileMode.Out); > fout.writeExact(&bin, TradesBin.sizeof); > > Where bin is of type TradesBin, filled with data. The question is how > do I now do this with zlib to get compression? > Suggestions and help are mightily appreciated! > > TJB Since you're already set up with the Stream interface, try creating a MemoryStream instead of a BufferedFile. Write to the stream just as you are now, then use the .data() property (on MemoryStream's superclass, TArrayStream) to get an array of raw bytes. You can feed this array into the compress function in std.zlib to produce a new (hopefully shorter) array of bytes that contains the compressed data. Simply write this compressed data to a file with `fout.writeExact(compressed.ptr, compressed.length)` and you're done. Since uncompress works better if you know the exact size of the uncompressed data, you might also consider writing the uncompressed size of the data to your output file before the compressed data. Justin |
September 25, 2012 Re: How do I use std.zlib to write a struct of data to a binary file with compression? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Justin Whear | > Since you're already set up with the Stream interface, try creating a
> MemoryStream instead of a BufferedFile. Write to the stream just as you
> are now, then use the .data() property (on MemoryStream's superclass,
> TArrayStream) to get an array of raw bytes. You can feed this array into
> the compress function in std.zlib to produce a new (hopefully shorter)
> array of bytes that contains the compressed data. Simply write this
> compressed data to a file with `fout.writeExact(compressed.ptr,
> compressed.length)` and you're done.
>
> Since uncompress works better if you know the exact size of the
> uncompressed data, you might also consider writing the uncompressed size
> of the data to your output file before the compressed data.
>
> Justin
Ok, I think I get it. Can you help me set up the first part, writing to the MemoryStream?
TJB
|
September 25, 2012 Re: How do I use std.zlib to write a struct of data to a binary file with compression? | ||||
---|---|---|---|---|
| ||||
Posted in reply to TJB | On 25-Sep-12 00:10, TJB wrote: > Hello, > > I am trying to save some data to compressed binary files. I have my > data in a struct, which I can write to a binary file just fine: > > align(1) struct TradesBin { Just a note: recently compiler changed so this align(1) have no effect on the fields alignment only on a struct as a whole. So instead align(1): inside of struct should probably work. > int ttim; > int prc; > int siz; > short g127; > short corr; > char[2] cond; > char ex; > } > > auto fout = new BufferedFile(outfile, FileMode.Out); > fout.writeExact(&bin, TradesBin.sizeof); > > Where bin is of type TradesBin, filled with data. The question is how > do I now do this with zlib to get compression? Suggestions and help are > mightily appreciated! > > TJB > -- Dmitry Olshansky |
September 25, 2012 Re: How do I use std.zlib to write a struct of data to a binary file with compression? | ||||
---|---|---|---|---|
| ||||
Posted in reply to TJB | On Tue, 25 Sep 2012 03:25:37 +0200, TJB wrote: >> Since you're already set up with the Stream interface, try creating a >> MemoryStream instead of a BufferedFile. Write to the stream just as >> you are now, then use the .data() property (on MemoryStream's >> superclass, >> TArrayStream) to get an array of raw bytes. You can feed this array >> into the compress function in std.zlib to produce a new (hopefully >> shorter) >> array of bytes that contains the compressed data. Simply write this >> compressed data to a file with `fout.writeExact(compressed.ptr, >> compressed.length)` and you're done. >> >> Since uncompress works better if you know the exact size of the uncompressed data, you might also consider writing the uncompressed size of the data to your output file before the compressed data. >> >> Justin > > Ok, I think I get it. Can you help me set up the first part, writing to the MemoryStream? > > TJB I wrote up a quick example program and pasted it here: http://dpaste.dzfl.pl/f1699d07 Hope that helps you out. |
September 25, 2012 Re: How do I use std.zlib to write a struct of data to a binary file with compression? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Justin Whear | On Tuesday, 25 September 2012 at 18:08:44 UTC, Justin Whear wrote:
> I wrote up a quick example program and pasted it here:
> http://dpaste.dzfl.pl/f1699d07
>
> Hope that helps you out.
Justin,
Thanks so much! Very helpful!!!
TJB
|
September 25, 2012 Re: How do I use std.zlib to write a struct of data to a binary file with compression? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Justin Whear | On Tue, 25 Sep 2012 18:08:44 +0000, Justin Whear wrote:
> On Tue, 25 Sep 2012 03:25:37 +0200, TJB wrote:
>
>>> Since you're already set up with the Stream interface, try creating a
>>> MemoryStream instead of a BufferedFile. Write to the stream just as
>>> you are now, then use the .data() property (on MemoryStream's
>>> superclass,
>>> TArrayStream) to get an array of raw bytes. You can feed this array
>>> into the compress function in std.zlib to produce a new (hopefully
>>> shorter)
>>> array of bytes that contains the compressed data. Simply write this
>>> compressed data to a file with `fout.writeExact(compressed.ptr,
>>> compressed.length)` and you're done.
>>>
>>> Since uncompress works better if you know the exact size of the uncompressed data, you might also consider writing the uncompressed size of the data to your output file before the compressed data.
>>>
>>> Justin
>>
>> Ok, I think I get it. Can you help me set up the first part, writing to the MemoryStream?
>>
>> TJB
>
> I wrote up a quick example program and pasted it here:
> http://dpaste.dzfl.pl/f1699d07
>
> Hope that helps you out.
I should point out that in my example, the writing to the MemoryStream is not strictly necessary. I could have simply cast the foo array to an array of bytes and passed that into the compress function. But if you're already set up for writing raw data into a stream, the MemoryStream will help you minimize code changes as well as facilitate more complex scenarios.
|
September 25, 2012 Re: How do I use std.zlib to write a struct of data to a binary file with compression? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Justin Whear | On Tuesday, 25 September 2012 at 18:08:44 UTC, Justin Whear wrote:
> I wrote up a quick example program and pasted it here:
> http://dpaste.dzfl.pl/f1699d07
>
> Hope that helps you out.
OK, I think I am working this out. Let's say I want to write the data to a gzipped file that will be inter-operable with gzip at the command line. I think
I do something like the following:
auto mem = new MemoryStream();
mem.writeExact(foo.ptr, foo.length * A.sizeof);
auto gzdat = new Compress(9, HeaderFormat.gzip);
gzdat.compress(mem.data);
But then, what do I do with gzdat? Do I try to write it to a file like this:
auto fout = new std.stream.File("data.bin.gz", FileMode.OutNew);
fout.writeExact(gzdat.ptr, gzdat.sizeof);
Something is wrong though. Hmm ...
TJB
|
Copyright © 1999-2021 by the D Language Foundation