Phobos v3 idea: File interface

Sep 19

solidstate1991

Sep 19

Paul Backus

Sep 19

Richard (Rikki) Andrew Cattermole

Sep 20

Paul Backus

Sep 20

Richard (Rikki) Andrew Cattermole

Sep 20

Paul Backus

Sep 20

Richard (Rikki) Andrew Cattermole

Sep 22

Adam Wilson

Sep 20

user1234

In a nutshell:

Create either a base class or an interface, which defines basic file accessing capabilities, then let that be used for not just files on the disk, but for things like the network, memory mapped files (would be great for compressed files without writing them to the disk first, etc.).

Current workaround is to read the file at once, and while that has its own uses, also has its own downsides, especially at bigger filesizes.

I kind of did something like that with VFile, trying to copy the API of std.stdio.File as closely as possible, although I didn't end up using it as much as I wanted due to having to work on other libraries of mine instead of my data packaging one (Datapak). At least I'm pretty close to creating a minimalistic D native replacement of SDL, also I'm yet again working on my SDLang/KDL/XDL parser (latter being my own modification of SDLang, with ISO date/time and hexadecimal numbers).

September 20

Re: Phobos v3 idea: File interface

Posted by Richard (Rikki) Andrew Cattermole
in reply to Paul Backus

Permalink

Richard (Rikki) Andrew Cattermole

Posted in reply to Paul Backus

Permalink

On 20/09/2024 9:18 AM, Paul Backus wrote:
> On Thursday, 19 September 2024 at 20:39:33 UTC, solidstate1991 wrote:
>> In a nutshell:
>>
>> Create either a base class or an interface, which defines basic file accessing capabilities, then let that be used for not just files on the disk, but for things like the network, memory mapped files (would be great for compressed files without writing them to the disk first, etc.).
> 
> In Phobos v2, `File` is a wrapper around a C `FILE*`, which means that its feature are limited by what the C library supports.

Worth noting is that ``File`` is a reference counted struct.

Unfortunately this is where we enter hill to die on territory for me.

System handles must be deterministically cleaned up. If you do not do this, you can run out of handles and then crash.

This is very easy especially for the 100k req/s target, think 10 seconds, not 2 minutes to run out of handles.

I am very willing to say, that RC types cannot go into GC memory (the memory itself can be GC, but the cleanup must be done via RC).

Right now RC is blocked by Walter on owner escape analysis. This is why I'm pushing so hard on this design wise currently.

I did ask Steven a couple months ago, and a new GC design will not lift this restriction. This is a hard requirement.

On Thursday, 19 September 2024 at 21:43:42 UTC, Richard (Rikki) Andrew Cattermole wrote: > System handles must be deterministically cleaned up. If you do not do this, you can run out of handles and then crash. > > This is very easy especially for the 100k req/s target, think 10 seconds, not 2 minutes to run out of handles. > > I am very willing to say, that RC types cannot go into GC memory (the memory itself can be GC, but the cleanup must be done via RC). > > Right now RC is blocked by Walter on owner escape analysis. This is why I'm pushing so hard on this design wise currently. I don't see why this has to be a hard blocker. D is already perfectly capable of using RC for the object lifetime (construction and destruction) and GC for the memory lifetime (allocation and deallocation).

On 21/09/2024 5:34 AM, Paul Backus wrote: > On Thursday, 19 September 2024 at 21:43:42 UTC, Richard (Rikki) Andrew Cattermole wrote: >> System handles must be deterministically cleaned up. If you do not do this, you can run out of handles and then crash. >> >> This is very easy especially for the 100k req/s target, think 10 seconds, not 2 minutes to run out of handles. >> >> I am very willing to say, that RC types cannot go into GC memory (the memory itself can be GC, but the cleanup must be done via RC). >> >> Right now RC is blocked by Walter on owner escape analysis. This is why I'm pushing so hard on this design wise currently. > > I don't see why this has to be a hard blocker. D is already perfectly capable of using RC for the object lifetime (construction and destruction) and GC for the memory lifetime (allocation and deallocation). For structs yes. But this thread is all about system handles being owned by classes, which don't offer this capability (currently).

On Friday, 20 September 2024 at 18:03:28 UTC, Richard (Rikki) Andrew Cattermole wrote: > For structs yes. > > But this thread is all about system handles being owned by classes, which don't offer this capability (currently). The original poster in this thread, solidstate1991, is talking about classes. My post said nothing about classes, and in fact I do not think using classes for Phobos v3's file abstraction is either necessary or desirable.

On 21/09/2024 6:57 AM, Paul Backus wrote: > On Friday, 20 September 2024 at 18:03:28 UTC, Richard (Rikki) Andrew Cattermole wrote: >> For structs yes. >> >> But this thread is all about system handles being owned by classes, which don't offer this capability (currently). > > The original poster in this thread, solidstate1991, is talking about classes. My post said nothing about classes, and in fact I do not think using classes for Phobos v3's file abstraction is either necessary or desirable. We seem to be talking a bit past each other. Regardless we both agree for different reasons that classes are out at the API level.

On Thursday, 19 September 2024 at 21:43:42 UTC, Richard (Rikki) Andrew Cattermole wrote: > System handles must be deterministically cleaned up. If you do not do this, you can run out of handles and then crash. I don't see this as a blocker. This can be handled by calling .close() in a scope(exit) block. I'll accept that manual resource management is more difficult to get right, but for now it works just fine. It's also pretty much the state of things right now, so we won't be going backwards.

Forums