File as string [mmfs as string?] (page 3)

"christopher diggins" <cdiggins@videotron.ca> wrote in message news:d26ker$2gma$1@digitaldaemon.com... > "Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d2214j$rej$1@digitaldaemon.com... >> Chris >> >> Have you tried using string_view + mmf yet? >> >> If you're interested, I'd be happy to sketch out a skeleton example to show how it'd be used. > > Thanks, but I haven't tried yet because Zz mentioned there is a 2(or was it 4) GIG limit on MMF's in windows. Can you either confirm or affirm this? ^^^^^^ refute Sorry. Too early in the morning.

March 27, 2005

Re: File as string [mmfs as string?]

Posted by Zz
in reply to christopher diggins

Permalink

Posted in reply to christopher diggins

Permalink

Here is the info from MATLAB on memory mapped files.

 Maximum Size of a Memory Map.   Due to limits set by the operating system,
the maximum amount of data you can map with a single instance of a memory
map is 2^31 - 1 (or 2 GB). If you need to map more than 2 GB, you can either
create separate maps for different regions of the file, or you can move the
2 GB window of one map to different locations in the file.
The 2 GB limit also applies to 64-bit platforms. However, because 64-bit
platforms have a much larger address space, they can support having many
more map instances in memory at any given time.

Aligned Access on Sol2 and HP-UX.   The Sol2 and HP-UX platforms only support aligned data access. This means that numeric values of type double that are to be read from a memory-mapped file must start at some multiple of 8 bytes from the start of the file. (Note that this is from the start of the file, and not the start of the mapped region.) Furthermore, numeric values of type single and also 32-bit integers must start at multiples of 4 bytes, and 16-bit integers at 2-byte multiples.

If you attempt to map a file on Sol2 or HP-UX that does not take into account these alignment considerations, MATLAB generates an error.


"christopher diggins" <cdiggins@videotron.ca> wrote in message news:d26ku8$2h3q$1@digitaldaemon.com...
> "christopher diggins" <cdiggins@videotron.ca> wrote in message news:d26ker$2gma$1@digitaldaemon.com...
>> "Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d2214j$rej$1@digitaldaemon.com...
>>> Chris
>>>
>>> Have you tried using string_view + mmf yet?
>>>
>>> If you're interested, I'd be happy to sketch out a skeleton example to show how it'd be used.
>>
>> Thanks, but I haven't tried yet because Zz mentioned there is a 2(or was it 4) GIG limit on MMF's in windows. Can you either confirm or affirm this?
>
>  ^^^^^^
>  refute
>
> Sorry. Too early in the morning.
>
>
>
>

"Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d27bcb$6sp$1@digitaldaemon.com... > > I can't offhand, but it does sound familiar. > > I was more meaning that string_view + mmf was an option, rather than the complete solution. Definitely I want to support "string_view + mmf" as an option. If you could post the skeleton example, I will write a yard example using it. >> Do you know if fread() is reliable? > > In what sense? Does it do what it is supposed to in most cases, on most platforms. For instance, Zz pointed out that fseekg(f, 0, SEEK_END) isn't specifically require to do what it is supposed to. I just learned the hard way that ifstream.seekg(0, ios::end) doesn't work on large files. So what I am asking is specifically, whether fread will do what is supposed to on most platforms, even at the end of extremely large files. Thanks a lot, Christopher Diggins http://www.ootl.org

"christopher diggins" <cdiggins@videotron.ca> wrote in message news:d29b60$2a4h$1@digitaldaemon.com... > "Matthew" <admin@stlsoft.dot.dot.dot.dot.org> wrote in message news:d27bcb$6sp$1@digitaldaemon.com... >> >> I can't offhand, but it does sound familiar. >> >> I was more meaning that string_view + mmf was an option, rather than the complete solution. > > Definitely I want to support "string_view + mmf" as an option. If you could post the skeleton example, I will write a yard example using it. Done. Posted. >>> Do you know if fread() is reliable? >> >> In what sense? > > Does it do what it is supposed to in most cases, on most platforms. For instance, Zz pointed out that fseekg(f, 0, SEEK_END) isn't specifically require to do what it is supposed to. I just learned the hard way that ifstream.seekg(0, ios::end) doesn't work on large files. > > So what I am asking is specifically, whether fread will do what is supposed to on most platforms, even at the end of extremely large files. My assumption is that if off_t is 64-bit, then fseek() will work for 64-bit files. If not, then it won't. But that's all supposition. The only times I've worked with 64-bit-sized files is on Win32, whence I've used the Win32 API.

Forums