| |
| Posted by rempas in reply to Ali Çehreli | PermalinkReply |
|
rempas
Posted in reply to Ali Çehreli
| On Sunday, 6 February 2022 at 16:45:59 UTC, Ali Çehreli wrote:
> So big that they can't fit in memory. For example, I benefit from mmap on a 16G system where a file would be 30G.
Oh, this small...
> As others said, it depends on the use case. If the entire file will be read anyway especially in sequential order, then mmap may not have much benefit. In my use case though it is common to just read unknown small amounts of bytes from unknown places of the huge file. (Say, 5G total out of a 30G.)
>
> Instead of my making multiple reads to those interesting parts of the file, mmap handles everything transparently: Just mmap the whole thing as a single array and access parts of that memory as needed.
Thank you! I will have that in mind in case I want to do something like that in the future. In my use-case tho, I will read the whole file.
> One huge improvement is to add madvise(2) system call to the picture to tell the system the exact amount of memory that will be touched so the OS reads in a single shot. Otherwise, the system reads by a default amount, which I think is 4K, which can turn out to be pathetically slow e.g. when the file is accessed over a slow network. (Why read 4K when the need is just 200 bytes and why read in 4K steps when the need is already to be 1M?)
>
> I haven't used mmap on Windows but it's in Phobos, so it should work. After all, mmap uses the virtual memory system of the OS and non-ancient Windows versions do use virtual memory and std.mmfile does include 'version (windows)' sections; so, yes. :)
>
> Ali
"mmap" is a system call that doesn't exist (natively) on Windows. I don't know what D does with Phobos (which I'm not gonna use anyway) but even if it works (how?), I will end up creating my own library so I'm in the same spot. "madvise" seems cool, I'll check it out! Thanks! In the end, I like advising and telling others how to do their work, XD!
|