Back when I've first introduced RCString I hinted that we have a larger strategy in mind. Here it is.

The basic tenet of the approach is to reckon and act on the fact that memory allocation (the subject of allocators) is an entirely distinct topic from memory management, and more generally resource management. This clarifies that it would be wrong to approach alternatives to GC in Phobos by means of allocators. GC is not only an approach to memory allocation, but also an approach to memory management. Reducing it to either one is a mistake. In hindsight this looks rather obvious but it has caused me and many people better than myself a lot of headache.

That said allocators are nice to have and use, and I will definitely follow up with std.allocator. However, std.allocator is not the key to a @nogc Phobos.

Nor are ranges. There is an attitude that either output ranges, or input ranges in conjunction with lazy computation, would solve the issue of creating garbage. https://github.com/D-Programming-Language/phobos/pull/2423 is a good illustration of the latter approach: a range would be lazily created by chaining stuff together. A range-based approach would take us further than the allocators, but I see the following issues with it:

(a) the whole approach doesn't stand scrutiny for non-linear outputs, e.g. outputting some sort of associative array or really any composite type quickly becomes tenuous either with an output range (eager) or with exposing an input range (lazy);

(b) makes the style of programming without GC radically different, and much more cumbersome, than programming with GC; as a consequence, programmers who consider changing one approach to another, or implementing an algorithm neutral to it, are looking at a major rewrite;

(c) would make D/@nogc a poor cousin of C++. This is quite out of character; technically, I have long gotten used to seeing most elaborate C++ code like poor emulation of simple D idioms. But C++ has spent years and decades taking to perfection an approach without a tracing garbage collector. A departure from that would need to be superior, and that doesn't seem to be the case with range-based approaches.

===========

Now that we clarified that these existing attempts are not going to work well, the question remains what does. For Phobos I'm thinking of defining and using three policies:

enum MemoryManagementPolicy { gc, rc, mrc }
immutable
    gc = ResourceManagementPolicy.gc,
    rc = ResourceManagementPolicy.rc,
    mrc = ResourceManagementPolicy.mrc;

The three policies are:

(a) gc is the classic garbage-collected style of management;

(b) rc is a reference-counted style still backed by the GC, i.e. the GC will still be able to pick up cycles and other kinds of leaks.

(c) mrc is a reference-counted style backed by malloc.

(It should be possible to collapse rc and mrc together and make the distinction dynamically, at runtime. I'm distinguishing them statically here for expository purposes.)

The policy is a template parameter to functions in Phobos (and elsewhere), and informs the functions e.g. what types to return. Consider:

auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2 ext)
if (...)
{
    static if (mmp == gc) alias S = string;
    else alias S = RCString;
    S result;
    ...
    return result;
}

On the caller side:

auto p1 = setExtension("hello", ".txt"); // fine, use gc
auto p2 = setExtension!gc("hello", ".txt"); // same
auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc

So by default it's going to continue being business as usual, but certain functions will allow passing in a (defaulted) policy for memory management.

Destroy!


Andrei

V Mon, 29 Sep 2014 03:49:52 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com>
napsáno:

> Back when I've first introduced RCString I hinted that we have a larger strategy in mind. Here it is.
> 
> The basic tenet of the approach is to reckon and act on the fact that memory allocation (the subject of allocators) is an entirely distinct topic from memory management, and more generally resource management. This clarifies that it would be wrong to approach alternatives to GC in Phobos by means of allocators. GC is not only an approach to memory allocation, but also an approach to memory management. Reducing it to either one is a mistake. In hindsight this looks rather obvious but it has caused me and many people better than myself a lot of headache.
> 
> That said allocators are nice to have and use, and I will definitely follow up with std.allocator. However, std.allocator is not the key to a @nogc Phobos.
> 
> Nor are ranges. There is an attitude that either output ranges, or input ranges in conjunction with lazy computation, would solve the issue of creating garbage. https://github.com/D-Programming-Language/phobos/pull/2423 is a good illustration of the latter approach: a range would be lazily created by chaining stuff together. A range-based approach would take us further than the allocators, but I see the following issues with it:
> 
> (a) the whole approach doesn't stand scrutiny for non-linear outputs,
> e.g. outputting some sort of associative array or really any
> composite type quickly becomes tenuous either with an output range
> (eager) or with exposing an input range (lazy);
> 
> (b) makes the style of programming without GC radically different, and much more cumbersome, than programming with GC; as a consequence, programmers who consider changing one approach to another, or implementing an algorithm neutral to it, are looking at a major rewrite;
> 
> (c) would make D/@nogc a poor cousin of C++. This is quite out of character; technically, I have long gotten used to seeing most elaborate C++ code like poor emulation of simple D idioms. But C++ has spent years and decades taking to perfection an approach without a tracing garbage collector. A departure from that would need to be superior, and that doesn't seem to be the case with range-based approaches.
> 
> ===========
> 
> Now that we clarified that these existing attempts are not going to work well, the question remains what does. For Phobos I'm thinking of defining and using three policies:
> 
> enum MemoryManagementPolicy { gc, rc, mrc }
> immutable
>      gc = ResourceManagementPolicy.gc,
>      rc = ResourceManagementPolicy.rc,
>      mrc = ResourceManagementPolicy.mrc;
> 
> The three policies are:
> 
> (a) gc is the classic garbage-collected style of management;
> 
> (b) rc is a reference-counted style still backed by the GC, i.e. the GC will still be able to pick up cycles and other kinds of leaks.
> 
> (c) mrc is a reference-counted style backed by malloc.
> 
> (It should be possible to collapse rc and mrc together and make the distinction dynamically, at runtime. I'm distinguishing them statically here for expository purposes.)
> 
> The policy is a template parameter to functions in Phobos (and elsewhere), and informs the functions e.g. what types to return. Consider:
> 
> auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path,
> R2 ext) if (...)
> {
>      static if (mmp == gc) alias S = string;
>      else alias S = RCString;
>      S result;
>      ...
>      return result;
> }
> 
> On the caller side:
> 
> auto p1 = setExtension("hello", ".txt"); // fine, use gc
> auto p2 = setExtension!gc("hello", ".txt"); // same
> auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc
> 
> So by default it's going to continue being business as usual, but certain functions will allow passing in a (defaulted) policy for memory management.
> 
> Destroy!
> 
> 
> Andrei

I would add something like this:

@DefaultMemoryManagementPolicy(rc)
module A;

void main() {
    auto p1 = setExtension("hello", ".txt"); // use rc
}

On 9/29/14, 4:03 AM, Daniel Kozak via Digitalmars-d wrote:
> I would add something like this:
>
> @DefaultMemoryManagementPolicy(rc)
> module A;
>
> void main() {
>      auto p1 = setExtension("hello", ".txt"); // use rc
> }

(please don't overquote!)

Yah, I realized I forgot to mention this: if we play our cards right, a lot of code will build in both approaches to memory management by just flipping a switch. In particular, the switch can be defaulted to something else.

I was thinking of leaving it to the user:

module A;
immutable myMMP = rc;

void main() {
    auto p1 = setExtension!myMMP("hello", ".txt");
}


Andrei

On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu wrote:
> Back when I've first introduced RCString I hinted that we have a larger strategy in mind. Here it is.
>
> The policy is a template parameter to functions in Phobos (and elsewhere), and informs the functions e.g. what types to return. Consider:
>
> auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2 ext)
> if (...)
> {
>     static if (mmp == gc) alias S = string;
>     else alias S = RCString;
>     S result;
>     ...
>     return result;
> }
>

How about having something like ResourceManagementPolicy.infer, which under the hood could work something like below... you could combine it with your original suggestion, with an overridable MemoryManagementPolicy(just removed it to make the example shorter)

auto setExtension(R1, R2)(R1 path, R2 ext)
if (...)
{
    static if(functionAttributes!(__traits(parent, setExtension)) & FunctionAttribute.nogc)
      alias S = RCString;
    else
      alias S = string;
    ...
    return result;
}

Daniel N

On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu wrote:


> entirely distinct topic

Finally!

On Monday, 29 September 2014 at 11:37:00 UTC, eles wrote:
> On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu wrote:
>
>
>> entirely distinct topic
>
> Finally!

Sorry, enthusiasm. I really think this is the key for doing the management of all resources in the right way. For me, the memory should be seen as a resource that simply happens to have the possibility of being manageable in a more flexible way and with specific constraints.

For example, with respect to other kind of resources, you could use a lazy approach to deallocate memory, as unlike many other resources memory is like money: is fungible [1]. Other resources are not. OTOH, the memory comes with some of its own quirks, such as the cycles (these could be, in theory, possible for other kind of resources, but are exceptions).

Memory management is not necessarily deterministic neither. Other resources might require determinism, however.

[1] http://en.wikipedia.org/wiki/Fungibility

On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu wrote:
> auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2 ext)
> if (...)
> {
>     static if (mmp == gc) alias S = string;
>     else alias S = RCString;
>     S result;
>     ...
>     return result;
> }

Is this practically feasible without blowing up Phobos several times in size and complexity?

And I'm not sure adding a template parameter to every function is going to work well, what with all the existing template parameters - especially the optional ones.

Any assumption that library code can go away with some set of pre-defined allocation strategies is crap. This whole discussion was about how important it is to move allocation decisions to user code (ranges are just one tool to achieve that, Don has been presenting examples of how we do that with plain arrays in DConf 2014 talk).

In that regard allocators + ranges are still the way to go in my opinion. Yes, sometimes those result in very hard to use API - providing GC-heavy but friendly alternatives for those shouldn't do any harm. But in general full decoupling of algorithms from allocations is necessary. If that makes D poor cousin of C++ we may have a learn few tricks from C++.

On 9/29/14, 5:06 AM, Vladimir Panteleev wrote:
> On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu wrote:
>> auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2
>> ext)
>> if (...)
>> {
>>     static if (mmp == gc) alias S = string;
>>     else alias S = RCString;
>>     S result;
>>     ...
>>     return result;
>> }
>
> Is this practically feasible without blowing up Phobos several times in
> size and complexity?

I believe so. For the most part implementations will be identical - just look at the RCString primitives, which are virtually the same as string's.

> And I'm not sure adding a template parameter to every function is going
> to work well, what with all the existing template parameters -
> especially the optional ones.

Not all functions, just those that allocate. I agree there will be a few decisions to be made there.


Andrei

On 9/29/14, 5:29 AM, Dicebot wrote:
> Any assumption that library code can go away with some set of
> pre-defined allocation strategies is crap. This whole discussion was
> about how important it is to move allocation decisions to user code
> (ranges are just one tool to achieve that, Don has been presenting
> examples of how we do that with plain arrays in DConf 2014 talk).

That's making exactly the confusion I was - that memory allocation strategy is the same as memory management strategy.

> In that regard allocators + ranges are still the way to go in my
> opinion. Yes, sometimes those result in very hard to use API - providing
> GC-heavy but friendly alternatives for those shouldn't do any harm. But
> in general full decoupling of algorithms from allocations is necessary.
> If that makes D poor cousin of C++ we may have a learn few tricks from C++.

As long as things are trivial they can be done with relative ease, albeit with more pain. But consider e.g. the recent JSON library by Sönke. It needs to create a lookup data structure and return things like strings from it. What primitives do you think could it define?


Andrei