On Sunday, 6 February 2022 at 09:40:48 UTC, rempas wrote:
> I'm planning to make a change to my program to use "mmap" to the contents of a file rather than "fgetc".  This is because I learned that "mmap" can do it faster.  The thing is, are there any problems that can occur when using "mmap"?

Performance is weird, and depends a lot on your access patterns and constraints.  Mmap is not universally fast and, I would argue, really only makes sense in a few constrained circumstances.  I would not switch to mmap just because you heard it was faster; only consider switching if you know i/o is a bottleneck for your application and know mmap is the solution.

https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf  recent, good read.

On Sunday, 6 February 2022 at 10:08:24 UTC, Elronnd wrote:
> Performance is weird, and depends a lot on your access patterns and constraints.  Mmap is not universally fast and, I would argue, really only makes sense in a few constrained circumstances.  I would not switch to mmap just because you heard it was faster; only consider switching if you know i/o is a bottleneck for your application and know mmap is the solution.
>
> https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf  recent, good read.

Thank you! I will actually make a compiler so it will just open and read the requested files. I don't know if the database example you linked will be similar to my case (I will of course read it tho) so I have to make my research I guess just to be sure.

On Sunday, 6 February 2022 at 10:48:01 UTC, rempas wrote:
> On Sunday, 6 February 2022 at 10:08:24 UTC, Elronnd wrote:
>> Performance is weird, and depends a lot on your access patterns and constraints.  Mmap is not universally fast and, I would argue, really only makes sense in a few constrained circumstances.  I would not switch to mmap just because you heard it was faster; only consider switching if you know i/o is a bottleneck for your application and know mmap is the solution.
>>
>> https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf  recent, good read.
>
> Thank you! I will actually make a compiler so it will just open and read the requested files. I don't know if the database example you linked will be similar to my case (I will of course read it tho) so I have to make my research I guess just to be sure.

Perso i'm almost always use mmap for opening large files for r/w. It IS faster.
Exception are small ones that can be read into the memory using std.file.read for example.

On Sunday, 6 February 2022 at 10:53:49 UTC, Temtaime wrote:
> Perso i'm almost always use mmap for opening large files for r/w. It IS faster.
> Exception are small ones that can be read into the memory using std.file.read for example.

Thank you! For how big files are we talking about? Also like another guy told me in another (C) forum, "mmap" is for Unix systems so do you know if Windows or MacOS can emulate that behavior with their memory allocation system calls?

On 2/6/22 04:21, rempas wrote:
> On Sunday, 6 February 2022 at 10:53:49 UTC, Temtaime wrote:
>> Perso i'm almost always use mmap for opening large files for r/w. It
>> IS faster.

Ditto.

> how big files are we talking about?

So big that they can't fit in memory. For example, I benefit from mmap on a 16G system where a file would be 30G.

As others said, it depends on the use case. If the entire file will be read anyway especially in sequential order, then mmap may not have much benefit. In my use case though it is common to just read unknown small amounts of bytes from unknown places of the huge file. (Say, 5G total out of a 30G.)

Instead of my making multiple reads to those interesting parts of the file, mmap handles everything transparently: Just mmap the whole thing as a single array and access parts of that memory as needed.

One huge improvement is to add madvise(2) system call to the picture to tell the system the exact amount of memory that will be touched so the OS reads in a single shot. Otherwise, the system reads by a default amount, which I think is 4K, which can turn out to be pathetically slow e.g. when the file is accessed over a slow network. (Why read 4K when the need is just 200 bytes and why read in 4K steps when the need is already to be 1M?)

> Also like another guy
> told me in another (C) forum, "mmap" is for Unix systems so do you know
> if Windows or MacOS can emulate that behavior with their memory
> allocation system calls?

I haven't used mmap on Windows but it's in Phobos, so it should work. After all, mmap uses the virtual memory system of the OS and non-ancient Windows versions do use virtual memory and std.mmfile does include 'version (windows)' sections; so, yes. :)

Ali

On Sunday, 6 February 2022 at 16:45:59 UTC, Ali Çehreli wrote:
> So big that they can't fit in memory. For example, I benefit from mmap on a 16G system where a file would be 30G.

Oh, this small...

> As others said, it depends on the use case. If the entire file will be read anyway especially in sequential order, then mmap may not have much benefit. In my use case though it is common to just read unknown small amounts of bytes from unknown places of the huge file. (Say, 5G total out of a 30G.)
>
> Instead of my making multiple reads to those interesting parts of the file, mmap handles everything transparently: Just mmap the whole thing as a single array and access parts of that memory as needed.

Thank you! I will have that in mind in case I want to do something like that in the future. In my use-case tho, I will read the whole file.

> One huge improvement is to add madvise(2) system call to the picture to tell the system the exact amount of memory that will be touched so the OS reads in a single shot. Otherwise, the system reads by a default amount, which I think is 4K, which can turn out to be pathetically slow e.g. when the file is accessed over a slow network. (Why read 4K when the need is just 200 bytes and why read in 4K steps when the need is already to be 1M?)
>
> I haven't used mmap on Windows but it's in Phobos, so it should work. After all, mmap uses the virtual memory system of the OS and non-ancient Windows versions do use virtual memory and std.mmfile does include 'version (windows)' sections; so, yes. :)
>
> Ali

"mmap" is a system call that doesn't exist (natively) on Windows. I don't know what D does with Phobos (which I'm not gonna use anyway) but even if it works (how?), I will end up creating my own library so I'm in the same spot. "madvise" seems cool, I'll check it out! Thanks! In the end, I like advising and telling others how to do their work, XD!

On Sunday, 6 February 2022 at 18:14:51 UTC, rempas wrote:
> On Sunday, 6 February 2022 at 16:45:59 UTC, Ali Çehreli wrote:
>> [...]
>
> Oh, this small...
>
>> [...]
>
> Thank you! I will have that in mind in case I want to do something like that in the future. In my use-case tho, I will read the whole file.
>
>> [...]
>
> "mmap" is a system call that doesn't exist (natively) on Windows. I don't know what D does with Phobos (which I'm not gonna use anyway) but even if it works (how?), I will end up creating my own library so I'm in the same spot. "madvise" seems cool, I'll check it out! Thanks! In the end, I like advising and telling others how to do their work, XD!

Windows has its own API to mmap files. There's no need to reinvent the wheel, phobos MmFile works for me without any problems.
Maybe there's no flush function, but for my use cases it's not so critical.