I suggests a new module for mapping  files into memory. Instead of viewing the file as array of bytes, it directly maps the object onto file allowing to handle it as regular in-memory object. The data remain persistent after the program exits and may be re-opened and used again or shared between processes.
Dynamic arrays are also supported with some limitations.
The code is at
https://github.com/sdegtiarev/phobos/tree/master/std/experimental/perpetual
Is anybody interested?

On Monday, 17 August 2015 at 18:27:48 UTC, Sergei Degtiarev wrote:
> [...]

There is already this in Phobos: http://dlang.org/phobos/std_mmfile.html

Is there any difference between `std.mmfile` and your solution?

On Monday, 17 August 2015 at 19:00:37 UTC, Liam McSherry wrote:
> Is there any difference between `std.mmfile` and your solution?

Implementation at system level is similar, the approach is different. std.mmfile returns uninitialized chunk of memory, std.perpetual returns an object which can be viewed and manipulated like regular object of the underlying type. This is kind of OOP approach to shared memory.

On 18/08/2015 6:27 a.m., Sergei Degtiarev wrote:
> I suggests a new module for mapping  files into memory. Instead of
> viewing the file as array of bytes, it directly maps the object onto
> file allowing to handle it as regular in-memory object. The data remain
> persistent after the program exits and may be re-opened and used again
> or shared between processes.
> Dynamic arrays are also supported with some limitations.
> The code is at
> https://github.com/sdegtiarev/phobos/tree/master/std/experimental/perpetual
> Is anybody interested?

What I like about this approach is:
- No memory allocation, yes yuppie!
- Uses what ever your type is you want!
- The type you want is initialized, yuppie

What I don't like:
- Does not allow for calling the constructor on the type should it have one
- Does not map to std.mmfile in any way
- No way to say an offset in the file to start working from (also maybe an upper limit for arrays)
- You implement basically everything in std.mmfile all over again

Fix up the things above and maybe with a bit more cleaning up it can go into std.mmfile. It's basically a poor mans (de/)serializer which is quite a nice thing to have.

On 18-Aug-2015 07:45, Rikki Cattermole wrote:
> What I don't like:
> - Does not allow for calling the constructor on the type should it have one

Hm it depends - one thing is that after mmaping a file the type may already be constructed, if it wasn't then emplacement should be the right thing to do. That is there should be a default/string value.

> - Does not map to std.mmfile in any way

Which is a good thing. std.mmfile is crappy 8 year old ported crap.

> - No way to say an offset in the file to

That may be useful.

> - You implement basically everything in std.mmfile all over again

Low-level API might be in order but it's not std.mmfile which is not low-level enough. Separating out a set of portable mmaping primitives would be awesome.

What I don't like is perpetual being a class - too bad as it won't honor scoping and destructor may or may not be called. Just make it a struct.

Also type should be Perputal, then perpetual is helper function that creates it then by means of IFTI type may be deduced. I suggest the following API:

// all parameters after file are either T itself
auto p = perpetual("file-name", 42);// would deduce T=int

// or args to T:
struct RGB{ int r,g,b; }
// if not in file use RGB(1,2,3)
auto p = perpetual!RGB("file-name", 1, 2, 3);

-- 
Dmitry Olshansky

On Tuesday, 18 August 2015 at 04:46:10 UTC, Rikki Cattermole wrote:
> - Does not allow for calling the constructor on the type should it have one
I plan to implement arguments forwarding to constructor, just decided that the concept would be more clear with minimal code. Technically, the object is initialized with .init and is in valid state.

> - Does not map to std.mmfile in any way
> - You implement basically everything in std.mmfile all over again
Does not need to, there are only two system calls involved and it makes no sense to reuse such amount of code.

> - No way to say an offset in the file to start working from (also maybe an upper limit for arrays)
This might be useful in some cases, but in general from my experience (I used this utility for several years in C++), it downgrades the interface, makes you regarding the data as set of bytes rather than an object. It is always possible to wrap up everything in a struct, so that working portion is at right offset.

> it can go into std.mmfile.
The concept is very different from plain shared memory, std.mmfile can be improved with a few lines of code, but it wouldn't change the way it's used.

> It's basically a poor mans (de/)serializer.
Not at all, it allows fast access to the data, so it might be used as effectively as regular values, and insurance from data loss at crash or stop.

On Tuesday, 18 August 2015 at 06:52:47 UTC, Dmitry Olshansky wrote:
> What I don't like is perpetual being a class - too bad as it won't honor scoping and destructor may or may not be called. Just make it a struct.
Why do you think the destructor may not be called? It is, and it syncs memory and unmaps the file. Making perpetual a struct would require implementing reference counting or disabling copying, and all such staff. In my opinion, perpetual should definitely have reference semantics, this is exactly a point where D shines.


> Also type should be Perputal, then perpetual is helper function that creates it then by means of IFTI type may be deduced. I suggest the following API:
>
> // all parameters after file are either T itself
> auto p = perpetual("file-name", 42);// would deduce T=int
>
> // or args to T:
> struct RGB{ int r,g,b; }
> // if not in file use RGB(1,2,3)
> auto p = perpetual!RGB("file-name", 1, 2, 3);
Right, just forgot. Will do, thanks.

On Tuesday, 18 August 2015 at 14:16:30 UTC, Sergei Degtiarev wrote:
> Right, just forgot. Will do, thanks.
I added helper functions with argument forwarding for initialization.

On 18-Aug-2015 17:32, Sergei Degtiarev wrote:
> On Tuesday, 18 August 2015 at 06:52:47 UTC, Dmitry Olshansky wrote:
>> What I don't like is perpetual being a class - too bad as it won't
>> honor scoping and destructor may or may not be called. Just make it a
>> struct.
> Why do you think the destructor may not be called? It is, and it syncs
> memory and unmaps the file. Making perpetual a struct would require
> implementing reference counting or disabling copying, and all such
> staff. In my opinion, perpetual should definitely have reference
> semantics, this is exactly a point where D shines.
>

Class is allocated on GC heap dtor is called on collection in that sense it is a finalizer. Just use struct if you don't need inheritance and/or do resource management.


-- 
Dmitry Olshansky

On Tuesday, 18 August 2015 at 14:32:18 UTC, Sergei Degtiarev wrote:
> On Tuesday, 18 August 2015 at 06:52:47 UTC, Dmitry Olshansky wrote:
>> What I don't like is perpetual being a class - too bad as it won't honor scoping and destructor may or may not be called. Just make it a struct.
> Why do you think the destructor may not be called? It is, and it syncs memory and unmaps the file. Making perpetual a struct would require implementing reference counting or disabling copying, and all such staff. In my opinion, perpetual should definitely have reference semantics, this is exactly a point where D shines.

It might be run, but there is no guarantee a class destructor will ever be called.  A struct destructor will be when it goes out of scope.  Hence Scoped!

http://forum.dlang.org/thread/ikuomwcxhydluojkhjqy@forum.dlang.org