Thread overview | |||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
August 02, 2013 Reading a structured binary file? | ||||
---|---|---|---|---|
| ||||
What library commands do i use to read from a structured binary file? I want to read the byte stream 1, 2 maybe 4 bytes at a time and cast these to bytes, shorts and ints respectively. I can't seem to find anything like readByte(). |
August 02, 2013 Re: Reading a structured binary file? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Gary Willoughby | On Friday, 2 August 2013 at 17:49:55 UTC, Gary Willoughby wrote: > What library commands do i use to read from a structured binary file? I want to read the byte stream 1, 2 maybe 4 bytes at a time and cast these to bytes, shorts and ints respectively. I can't seem to find anything like readByte(). http://dlang.org/phobos/std_file.html#.read http://dlang.org/phobos/std_stdio.html#.File.rawRead ? |
August 02, 2013 Re: Reading a structured binary file? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Gary Willoughby | On Fri, 02 Aug 2013 19:49:54 +0200, Gary Willoughby wrote:
> What library commands do i use to read from a structured binary file? I want to read the byte stream 1, 2 maybe 4 bytes at a time and cast these to bytes, shorts and ints respectively. I can't seem to find anything like readByte().
You can use File.rawRead:
ushort[1] myShort;
file.rawRead(myShort);
Or if you have structures in the file:
struct Foo
{
align(1):
int bar;
short k;
char[7] str;
}
Foo[1] foo;
file.rawRead(foo);
|
August 02, 2013 Re: Reading a structured binary file? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Gary Willoughby | On Friday, 2 August 2013 at 17:49:55 UTC, Gary Willoughby wrote:
> What library commands do i use to read from a structured binary file? I want to read the byte stream 1, 2 maybe 4 bytes at a time and cast these to bytes, shorts and ints respectively. I can't seem to find anything like readByte().
How big is the file?
If it's not too huge i'd just read it in with std.file.read and then sort out splitting it up from there.
|
August 02, 2013 Re: Reading a structured binary file? | ||||
---|---|---|---|---|
| ||||
Posted in reply to John Colvin | > How big is the file?
>
> If it's not too huge i'd just read it in with std.file.read and then sort out splitting it up from there.
Quite large so i'll probably stream it. Thanks guys.
|
August 02, 2013 Re: Reading a structured binary file? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Gary Willoughby | On Friday, 2 August 2013 at 17:49:55 UTC, Gary Willoughby wrote: > What library commands do i use to read from a structured binary file? I want to read the byte stream 1, 2 maybe 4 bytes at a time and cast these to bytes, shorts and ints respectively. I can't seem to find anything like readByte(). You've gotten some help already around functions D provides. But I thought I would mention I'd recently tried to do some large file parsing for binary data, and decided to try and blog about it. http://he-the-great.livejournal.com/47550.html I can't say this is the best solution, but it worked. I was parsing a 20 gig OpenStreetMap planet file. |
August 02, 2013 Re: Reading a structured binary file? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Gary Willoughby | On Friday, August 02, 2013 19:49:54 Gary Willoughby wrote:
> What library commands do i use to read from a structured binary file? I want to read the byte stream 1, 2 maybe 4 bytes at a time and cast these to bytes, shorts and ints respectively. I can't seem to find anything like readByte().
I'd probably use std.mmfile and std.bitmanip to do it. MmFile will allow you to efficiently operate on the file as a ubyte[] in memory thanks to mmap, and std.bitmanip's peek and read functions make it easy to convert multiple bytes into integral values.
- Jonathan M Davis
|
August 02, 2013 Re: Reading a structured binary file? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jonathan M Davis | On 2013-08-02 17:13, Jonathan M Davis wrote:
> On Friday, August 02, 2013 19:49:54 Gary Willoughby wrote:
>> What library commands do i use to read from a structured binary
>> file? I want to read the byte stream 1, 2 maybe 4 bytes at a time
>> and cast these to bytes, shorts and ints respectively. I can't
>> seem to find anything like readByte().
>
> I'd probably use std.mmfile and std.bitmanip to do it. MmFile will allow you to
> efficiently operate on the file as a ubyte[] in memory thanks to mmap, and
> std.bitmanip's peek and read functions make it easy to convert multiple bytes
> into integral values.
>
> - Jonathan M Davis
FWIW
i have to deal with big data files that can be a few GB. for some data analysis software i wrote in C a while back i did some testing with caching and such. turns out that for Win7-64 the automatic caching done by the OS is really good and any attempt to speed things up actually slowed it down. no kidding, i have seen more than 2GB of data being automatically cached. of course the system RAM must be larger than the file size (if i remember my tests correctly by a factor of ~2, but this is maybe not a linear relationship, i did not actually change the RAM just the size of the data file) and it will hold it in the cache only as long as there are no concurrent applications requiring RAM or caching. i guess my point is, if your target is Win7 and your files are >5x smaller than the installed RAM i would not bother at all trying to optimize file access. i suppose -nix machine will do a similar good job these days.
/det
|
August 02, 2013 Re: Reading a structured binary file? | ||||
---|---|---|---|---|
| ||||
Posted in reply to captaindet | On Fri, Aug 02, 2013 at 06:38:20PM -0500, captaindet wrote: [...] > FWIW > i have to deal with big data files that can be a few GB. for some data > analysis software i wrote in C a while back i did some testing with > caching and such. turns out that for Win7-64 the automatic caching > done by the OS is really good and any attempt to speed things up > actually slowed it down. no kidding, i have seen more than 2GB of data > being automatically cached. of course the system RAM must be larger > than the file size (if i remember my tests correctly by a factor of > ~2, but this is maybe not a linear relationship, i did not actually > change the RAM just the size of the data file) and it will hold it in > the cache only as long as there are no concurrent applications > requiring RAM or caching. i guess my point is, if your target is Win7 > and your files are >5x smaller than the installed RAM i would not > bother at all trying to optimize file access. i suppose -nix machine > will do a similar good job these days. [...] IIRC, Linux has been caching files (or disk blocks, rather) in memory since the days of Win95. Of course, memory in those days was much scarcer, but file sizes were smaller too. :) There's still a cost to copy the kernel buffers into userspace, though, which should not be disregarded. But if you use mmap, then you're essentially accessing that memory cache directly, which is as good as it gets. I don't know how well mmap works on windows, though, IIRC it doesn't have the same semantics as Posix, so you could accidentally run into performance issues by using it the wrong way on windows. T -- There is no gravity. The earth sucks. |
August 03, 2013 Re: Reading a structured binary file? | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jonathan M Davis | On Friday, 2 August 2013 at 22:13:28 UTC, Jonathan M Davis wrote:
> I'd probably use std.mmfile and std.bitmanip to do it. MmFile will allow you to
> efficiently operate on the file as a ubyte[] in memory thanks to mmap, and
> std.bitmanip's peek and read functions make it easy to convert multiple bytes
> into integral values.
>
> - Jonathan M Davis
This sounds a great idea but once the file has been opened as a MmFile how to i convert this to a ubyte[] so the std.bitmanip functions work with it?
|
Copyright © 1999-2021 by the D Language Foundation