I was going to name this thread "SEX!!" but then I thought "best memory management" would get me more reads ;) Anyway now that I have your attention...
What I want to learn (not debate) is the currently available types, idioms etc. whenever one wants deterministic memory management. Please do not derail it debating how often deterministic should be preferred to GC or not. Just, whenever one should happen to require it, what are the available means? And how do they compare in your daily use, if any? If you want to post your code samples using special types for manual memory management, that would be great.

AFAIK (from here on please correct me wherever I'm wrong) the original D design was, if you don't want to use GC, then malloc() and free() are available from std.c. Pretty solid. I guess the downside is less nice syntax than new/delete, and having to check the returned value instead of exceptions. I guess these were the original reasons why C++ introduced new/delete but I've never been sure.

Then from this nice summary [1] I've learned about the existence of new libraries and Phobos modules: std.typecons, Dlib, and std.experimental.allocator. Sadly in this department D starts to look a bit like C++ in that there are too many possible ways to do one certain thing, and what's worse none of them is the "standard" way, and none of them is deprecated atm either. I've just taken a quick look at them, and I was wondering how many people prefer either, and what are their reasons and their experience.

dlib.core.memory and dlib.memory lack documentation, but according to this wiki page [2] I found, dlib defines New/Delete substitutes without GC a-la-C++, with the nice addition of a "memoryDebug" version (how ironclad is this to debug every memory leak?)

From std.typecons what caught my eye first is scoped() and Unique. std.experimental.allocator sounded quite, well, experimental or advanced, so I stopped reading before trying to wrap my head around all of it. Should I take another look?

scoped() seems to work nicely for auto variables, and if I understood it right, not only it provides deterministic management, but allocates statically/in the stack, so it is like C++ without pointers right? Looking into the implementation, I just hope most of that horribly unsafe casting can be taken care of at compile time. The whole thing looks a bit obscure under the hood and in its usage: auto is mandatory or else allocation doesn't hold, but even reflection cannot tell the different at runtime between T and typeof(scoped!T) //eew. Unfortunately this also makes scoped() extremely unwieldy for member variables: their type has to be explicitly declared as typeof(scoped!T), and they cannot be initialized at the declaration. To me this looks like scoped() could be useful in some cases but it looks hardly recommendable to the same extent as the analogous C++ idiom.

Then Unique seems to be analogous to C++ unique_ptr, fair enough... Or are there significant differences? Your experience?

And am I right in assuming that scoped() and Unique (and dlib.core.memory) prevent the GC from monitoring the memory they manage (just like malloc?), thus also saving those few cycles? This I haven't seen clearly stated in the documentation.


[1] http://forum.dlang.org/post/stohzfatiwjzemqojeol@forum.dlang.org
[2] https://github.com/gecko0307/dlib/wiki/Manual-Memory-Management

On Sunday, 5 March 2017 at 20:54:06 UTC, XavierAP wrote:
> I was going to name this thread "SEX!!" but then I thought "best memory management" would get me more reads ;) Anyway now that I have your attention...
> What I want to learn (not debate) is the currently available types, idioms etc. whenever one wants deterministic memory management. Please do not derail it debating how often deterministic should be preferred to GC or not. Just, whenever one should happen to require it, what are the available means? And how do they compare in your daily use, if any? If you want to post your code samples using special types for manual memory management, that would be great.
>
> AFAIK (from here on please correct me wherever I'm wrong) the original D design was, if you don't want to use GC, then malloc() and free() are available from std.c. Pretty solid. I guess the downside is less nice syntax than new/delete, and having to check the returned value instead of exceptions. I guess these were the original reasons why C++ introduced new/delete but I've never been sure.
>
> Then from this nice summary [1] I've learned about the existence of new libraries and Phobos modules: std.typecons, Dlib, and std.experimental.allocator. Sadly in this department D starts to look a bit like C++ in that there are too many possible ways to do one certain thing, and what's worse none of them is the "standard" way, and none of them is deprecated atm either. I've just taken a quick look at them, and I was wondering how many people prefer either, and what are their reasons and their experience.
>
> dlib.core.memory and dlib.memory lack documentation, but according to this wiki page [2] I found, dlib defines New/Delete substitutes without GC a-la-C++, with the nice addition of a "memoryDebug" version (how ironclad is this to debug every memory leak?)
>
> From std.typecons what caught my eye first is scoped() and Unique. std.experimental.allocator sounded quite, well, experimental or advanced, so I stopped reading before trying to wrap my head around all of it. Should I take another look?
>
> scoped() seems to work nicely for auto variables, and if I understood it right, not only it provides deterministic management, but allocates statically/in the stack, so it is like C++ without pointers right? Looking into the implementation, I just hope most of that horribly unsafe casting can be taken care of at compile time. The whole thing looks a bit obscure under the hood and in its usage: auto is mandatory or else allocation doesn't hold, but even reflection cannot tell the different at runtime between T and typeof(scoped!T) //eew. Unfortunately this also makes scoped() extremely unwieldy for member variables: their type has to be explicitly declared as typeof(scoped!T), and they cannot be initialized at the declaration. To me this looks like scoped() could be useful in some cases but it looks hardly recommendable to the same extent as the analogous C++ idiom.
>
> Then Unique seems to be analogous to C++ unique_ptr, fair enough... Or are there significant differences? Your experience?
>
> And am I right in assuming that scoped() and Unique (and dlib.core.memory) prevent the GC from monitoring the memory they manage (just like malloc?), thus also saving those few cycles? This I haven't seen clearly stated in the documentation.
>
>
> [1] http://forum.dlang.org/post/stohzfatiwjzemqojeol@forum.dlang.org
> [2] https://github.com/gecko0307/dlib/wiki/Manual-Memory-Management

The memory management in D is becoming a mess. Yes, D was developed with the GC in mind and the attempts to make it usable without GC came later. Now std has functions that allocate with GC, there're containers that use malloc/free directly or reference counting for the internal storage, and there is std.experimental.allocator. And it doesn't really get better. There is also some effort to add reference counting directly into the language. I really fear we will have soon signatures like "void myfunc() @safe @nogc @norc..".
Stuff like RefCounted or Unique are similar to C++ analogues, but not the same. They throw exceptions allocated with GC, factory methods (like Unique.create) use GC to create the object.
Also dlib's memory management is a nightmare: some stuff uses "new" and GC, some "New" and "Delete". Some functions allocate memory and returns it and you never know if it will be collected or you should free it, you have to look into the source code each time to see what the function does internally, otherwise you will end up with memory leaks or segmentation faults. dlib has a lot of outdated code that isn't easy to update.

On Monday, 6 March 2017 at 08:26:53 UTC, Eugene Wissner wrote:
> The memory management in D is becoming a mess.

I am aware this is a hot topic, hence the opening joke. But I just want to learn what toolboxes are currently available, not really discuss how adequate they are, or how often GC is adequate enough. Neither how much % GC-haram Phobos or other libraries are internally atm. There are several threads already discussing this, which I've browsed.

My impression so far is that dlib's New/Delete is the most straightforward or efficient tool, and it can kind of cover all the bases (and is GC-halal), as far as I can tell in advance. Plus it has printMemoryLog() as a bonus, which is already better than C++ new/delete. Just an observation that it doesn't provide an option to allocate an uninitialized array, considering that this module is already targeting performance applications.

Not sure if I would use any other tool (besides GC itself). I'm curious about Unique but it's not fully clear to me what happens (regarding lifetime, deterministic destruction, GC monitoring) when you need to call "new" to use it.

On Sunday, 5 March 2017 at 20:54:06 UTC, XavierAP wrote:
> What I want to learn (not debate) is the currently available types, idioms etc. whenever one wants deterministic memory management.

There's nothing like that of C++. Currently you have Unique, RefCounted, scoped and individual people efforts on this. BTW, do you want to manage non-memory resources with these memory management mechanisms too?

On Tuesday, 7 March 2017 at 16:51:23 UTC, Kagamin wrote:
> There's nothing like that of C++.

Don't you think New/Delete from dlib.core.memory fills the bill? for C++ style manual dynamic memory management? It looks quite nice to me, being no more than a simple malloc wrapper with constructor/destructor calling and type safety. Plus printMemoryLog() for debugging, much easier than valgrind.

> do you want to manage non-memory resources with these memory management mechanisms too?

I wasn't thinking about this now, but I'm sure the need will come up.

On Tuesday, 7 March 2017 at 17:37:43 UTC, XavierAP wrote:
> On Tuesday, 7 March 2017 at 16:51:23 UTC, Kagamin wrote:
>> There's nothing like that of C++.
>
> Don't you think New/Delete from dlib.core.memory fills the bill? for C++ style manual dynamic memory management? It looks quite nice to me, being no more than a simple malloc wrapper with constructor/destructor calling and type safety. Plus printMemoryLog() for debugging, much easier than valgrind.
>
>> do you want to manage non-memory resources with these memory management mechanisms too?
>
> I wasn't thinking about this now, but I'm sure the need will come up.

Yes. For simple memory management New/Delete would be enough. But you depend on your libc in this case, that is mostly not a problem. From experience it wasn't enough for some code bases, so the C-world invented some work arounds:

1) Link to an another libc providing a different malloc/free implementations
2) Use macros that default to the libc's malloc/free, but can be set at compile time to an alternative implementation (mbedtls uses for example mbedtls_malloc, mbedtls_calloc and mbedtls_free macros)

To avoid this from the beginning, it may be better to use allocators. You can use "make" and "dispose" from std.experimental.allocator the same way as New/Delete.

I tried to introduce the allocators in dlib but it failed, because dlib is difficult to modify because of other projects based on it (although to be honest it was mostly a communication problem as it often happens), so I started a similar lib from scratch.

On Tuesday, 7 March 2017 at 18:21:43 UTC, Eugene Wissner wrote:
> To avoid this from the beginning, it may be better to use allocators. You can use "make" and "dispose" from std.experimental.allocator the same way as New/Delete.

Thanks! looking into it.

Does std.experimental.allocator have a leak debugging tool like dlib's printMemoryLog()?

On Tuesday, 7 March 2017 at 20:15:37 UTC, XavierAP wrote:
> On Tuesday, 7 March 2017 at 18:21:43 UTC, Eugene Wissner wrote:
>> To avoid this from the beginning, it may be better to use allocators. You can use "make" and "dispose" from std.experimental.allocator the same way as New/Delete.
>
> Thanks! looking into it.
>
> Does std.experimental.allocator have a leak debugging tool like dlib's printMemoryLog()?

Yes, but printMemoryLog is anyway useful only for simple searching for memory leaks. For the advanced debugging it is anyway better to learn some memory debugger or profiler.

On Tuesday, 7 March 2017 at 18:21:43 UTC, Eugene Wissner wrote:
> To avoid this from the beginning, it may be better to use allocators. You can use "make" and "dispose" from std.experimental.allocator the same way as New/Delete.

OK I've been reading on std.experimental.allocator; it looks really powerful and general, more than I need. I see the potential but I don't really have the knowledge to tweak memory management, and the details of the "building blocks" are well beyond me.

But even if I don't go there, I guess it's a good thing that I can change my program's allocator by changing one single line or version assigning theAllocator, and benchmark the results among different possibilities.

I see the default allocator is the same GC heap used by 'new'. Just for my learning curiosity, does this mean that if I theAllocator.make() something and then forget to dispose() it, it will be garbage collected the same once no longer referenced? And so are these objects traversed by the GC?

I've also looked at mallocator, [2] can it be used in some way to provide an allocator instead of the default theAllocator? As far as I can tell mallocator is not enough to implement an IAllocator, is there a reason, or where's the rest, am I missing it?


[1] https://dlang.org/phobos/std_experimental_allocator.html
[2] https://dlang.org/phobos/std_experimental_allocator_mallocator.html

On 03/08/2017 02:15 AM, XavierAP wrote:
> I see the default allocator is the same GC heap used by 'new'. Just for
> my learning curiosity, does this mean that if I theAllocator.make()
> something and then forget to dispose() it, it will be garbage collected
> the same once no longer referenced? And so are these objects traversed
> by the GC?

Yes and yes. GCAllocator.allocate calls core.memory.GC.malloc with does pretty much the same thing as the builtin `new`.

One difference might be with preciseness. `new` can take the type into account and automatically mark the allocation as NO_SCAN. I'm not sure if GCAllocator can do that. Maybe if you use `make`/`makeArray` to make the allocations.

> I've also looked at mallocator, [2] can it be used in some way to
> provide an allocator instead of the default theAllocator? As far as I
> can tell mallocator is not enough to implement an IAllocator, is there a
> reason, or where's the rest, am I missing it?

To make an IAllocator, use std.experimental.allocator.allocatorObject. The example in the documentation shows how it's done.

https://dlang.org/phobos/std_experimental_allocator.html#.allocatorObject