On 9/29/14, 10:25 AM, Jacob Carlborg wrote:
> How does allocators fit in this? Will it be an additional argument to
> the function. Or a separate stack that one can push and pop allocators to?

There would be one allocator per thread (changeable) deferring to a global interlocked allocator. Most algorithms would just use whatever allocator is installed.

I know the notion of a thread-local and then global allocator is liable to cause some an apoplexy attack. But it's time to model things as they are - memory is a global resource and it ought to be treated as such. No need to pass allocators around except for special cases.


Andrei

On Monday, 29 September 2014 at 10:49:53 UTC, Andrei Alexandrescu
wrote:
> Back when I've first introduced RCString I hinted that we have a larger strategy in mind. Here it is.
>
> The basic tenet of the approach is to reckon and act on the fact that memory allocation (the subject of allocators) is an entirely distinct topic from memory management, and more generally resource management. This clarifies that it would be wrong to approach alternatives to GC in Phobos by means of allocators. GC is not only an approach to memory allocation, but also an approach to memory management. Reducing it to either one is a mistake. In hindsight this looks rather obvious but it has caused me and many people better than myself a lot of headache.
>
> That said allocators are nice to have and use, and I will definitely follow up with std.allocator. However, std.allocator is not the key to a @nogc Phobos.
>
> Nor are ranges. There is an attitude that either output ranges, or input ranges in conjunction with lazy computation, would solve the issue of creating garbage. https://github.com/D-Programming-Language/phobos/pull/2423 is a good illustration of the latter approach: a range would be lazily created by chaining stuff together. A range-based approach would take us further than the allocators, but I see the following issues with it:
>
> (a) the whole approach doesn't stand scrutiny for non-linear outputs, e.g. outputting some sort of associative array or really any composite type quickly becomes tenuous either with an output range (eager) or with exposing an input range (lazy);
>
> (b) makes the style of programming without GC radically different, and much more cumbersome, than programming with GC; as a consequence, programmers who consider changing one approach to another, or implementing an algorithm neutral to it, are looking at a major rewrite;
>
> (c) would make D/@nogc a poor cousin of C++. This is quite out of character; technically, I have long gotten used to seeing most elaborate C++ code like poor emulation of simple D idioms. But C++ has spent years and decades taking to perfection an approach without a tracing garbage collector. A departure from that would need to be superior, and that doesn't seem to be the case with range-based approaches.
>
> ===========
>
> Now that we clarified that these existing attempts are not going to work well, the question remains what does. For Phobos I'm thinking of defining and using three policies:
>
> enum MemoryManagementPolicy { gc, rc, mrc }
> immutable
>     gc = ResourceManagementPolicy.gc,
>     rc = ResourceManagementPolicy.rc,
>     mrc = ResourceManagementPolicy.mrc;
>
> The three policies are:
>
> (a) gc is the classic garbage-collected style of management;
>
> (b) rc is a reference-counted style still backed by the GC, i.e. the GC will still be able to pick up cycles and other kinds of leaks.
>
> (c) mrc is a reference-counted style backed by malloc.
>
> (It should be possible to collapse rc and mrc together and make the distinction dynamically, at runtime. I'm distinguishing them statically here for expository purposes.)
>
> The policy is a template parameter to functions in Phobos (and elsewhere), and informs the functions e.g. what types to return. Consider:
>
> auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1 path, R2 ext)
> if (...)
> {
>     static if (mmp == gc) alias S = string;
>     else alias S = RCString;
>     S result;
>     ...
>     return result;
> }
>
> On the caller side:
>
> auto p1 = setExtension("hello", ".txt"); // fine, use gc
> auto p2 = setExtension!gc("hello", ".txt"); // same
> auto p3 = setExtension!rc("hello", ".txt"); // fine, use rc
>
> So by default it's going to continue being business as usual, but certain functions will allow passing in a (defaulted) policy for memory management.
>
> Destroy!
>
>
> Andrei
Internally we should have something like:

---
template String(MemoryManagementPolicy mmp=gc){
     /++ ... +/
}
auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1
path, R2 ext)
if (...)
{
     auto result=String!mmp();
     /++ +/
}
----

or maybe even allowing user types in the template argument(the
original purpose of templates)

---
auto setExtension(String = string, R1, R2)(R1
path, R2){
     /++ +/
}
----

On 9/29/14, 11:44 AM, Shammah Chancellor wrote:
> I don't like the idea of having to pass in template parameters
> everywhere -- even for allocators.

I agree.

> Is there some way we could have
> "allocator contexts"?
>
> E.G.
>
> with( auto allocator = ReferencedCounted() )

Don't confuse memory allocation with memory management. There's no such a thing as a "reference counted allocator".


Andrei

On 9/29/14, 3:11 PM, Freddy wrote:
>>
> Internally we should have something like:
>
> ---
> template String(MemoryManagementPolicy mmp=gc){
>       /++ ... +/
> }
> auto setExtension(MemoryManagementPolicy mmp = gc, R1, R2)(R1
> path, R2 ext)
> if (...)
> {
>       auto result=String!mmp();
>       /++ +/
> }
> ----
>
> or maybe even allowing user types in the template argument(the
> original purpose of templates)
>
> ---
> auto setExtension(String = string, R1, R2)(R1
> path, R2){
>       /++ +/
> }

That's correct. -- Andrei

On 9/29/14, 10:19 AM, Dicebot wrote:
> On Monday, 29 September 2014 at 17:04:54 UTC, Andrei Alexandrescu wrote:
>>> Yes but neither decision belongs to library code except for very rare
>>> cases.
>>
>> You just assert it, so all I can say is "I understand you believe
>> this". I've motivated my argument. You may want to do the same for yours.
>
> I probably have missed the part with arguments :)

No problem, let me paste it again:

> The basic tenet of the approach is to reckon and act on the fact that memory allocation (the subject of allocators) is an entirely distinct topic from memory management, and more generally resource management. This clarifies that it would be wrong to approach alternatives to GC in Phobos by means of allocators. GC is not only an approach to memory allocation, but also an approach to memory management. Reducing it to either one is a mistake. In hindsight this looks rather obvious but it has caused me and many people better than myself a lot of headache.
>
> That said allocators are nice to have and use, and I will definitely follow up with std.allocator. However, std.allocator is not the key to a @nogc Phobos.
>
> Nor are ranges. There is an attitude that either output ranges, or input ranges in conjunction with lazy computation, would solve the issue of creating garbage. https://github.com/D-Programming-Language/phobos/pull/2423 is a good illustration of the latter approach: a range would be lazily created by chaining stuff together. A range-based approach would take us further than the allocators, but I see the following issues with it:
>
> (a) the whole approach doesn't stand scrutiny for non-linear outputs, e.g. outputting some sort of associative array or really any composite type quickly becomes tenuous either with an output range (eager) or with exposing an input range (lazy);
>
> (b) makes the style of programming without GC radically different, and much more cumbersome, than programming with GC; as a consequence, programmers who consider changing one approach to another, or implementing an algorithm neutral to it, are looking at a major rewrite;
>
> (c) would make D/@nogc a poor cousin of C++. This is quite out of character; technically, I have long gotten used to seeing most elaborate C++ code like poor emulation of simple D idioms. But C++ has spent years and decades taking to perfection an approach without a tracing garbage collector. A departure from that would need to be superior, and that doesn't seem to be the case with range-based approaches.

=================

> Your reasoning is not
> fundamentally different from "GC should be enough" but extended to
> several options from single one.

Where's RC in the "GC should be enough"?

> My argument is simple - one can't forsee everything. I remember reading
> book of one guy who has been advocating thing called "policy-based
> design", you may know him ;) Was quite impressed with the simple but
> practical basic idea - decoupling parts of the implementation that are
> not inherently related.

Totally. Then it would be great if you trusted the guy when he makes a judgment call in which reasonable people may disagree.

There are many memory /allocation/ policies but precious few memory /management/ policies. I only know "manual", "scoped", "reference counted", and "tracing" based on... the last 50 years of software development.

>> So you don't have an answer. And again you are confusing memory
>> allocation with memory management.
>
> Yes, sorry, I don't have an answer. Or time do deeply dive into the code
> unless it is really important or my direct responsibility.
>
> Unfortunately, I don't see an answer how your proposal fits our code
> either. Most of Sociomantic code relies on using arrays as ref arguments
> to avoid creating of new GC roots (no, we don't need/want to switch to
> ARC). This was several times called as the reason why Phobos in its
> current shape is largely unusable for out needs even when D2 switch is
> finished. I don't see how proposal in original post changes that.

Passing arrays by reference is plenty adequate with all memory management strategies. You'll need to wait and see how the proposal changes that, but if you naysay, back it up.


Andrei

On Monday, 29 September 2014 at 22:18:38 UTC, Andrei Alexandrescu wrote:
> Passing arrays by reference is plenty adequate with all memory management strategies. You'll need to wait and see how the proposal changes that, but if you naysay, back it up.

Resisting to go on meaningless argument on other points, this pretty much says that focus on things that are important for me is abandoned in favor of something that mostly doesn't matter. Am I supposed to be happy? :) Am I supposed to be twice as happy when you propose to close pull requests that do help because of this proposal?

I am waiting for what comes next but right now "not impressed" is most optimistic way to put this. Sorry :(

On 9/29/14, 3:43 PM, Dicebot wrote:
> On Monday, 29 September 2014 at 22:18:38 UTC, Andrei Alexandrescu wrote:
>> Passing arrays by reference is plenty adequate with all memory
>> management strategies. You'll need to wait and see how the proposal
>> changes that, but if you naysay, back it up.
>
> Resisting to go on meaningless argument on other points, this pretty
> much says that focus on things that are important for me is abandoned in
> favor of something that mostly doesn't matter. Am I supposed to be
> happy? :) Am I supposed to be twice as happy when you propose to close
> pull requests that do help because of this proposal?
>
> I am waiting for what comes next but right now "not impressed" is most
> optimistic way to put this. Sorry :(

I trust you'll be. -- Andrei

On 2014-09-29 22:15:33 +0000, Andrei Alexandrescu said:

> On 9/29/14, 11:44 AM, Shammah Chancellor wrote:
>> I don't like the idea of having to pass in template parameters
>> everywhere -- even for allocators.
> 
> I agree.
> 
>> Is there some way we could have
>> "allocator contexts"?
>> 
>> E.G.
>> 
>> with( auto allocator = ReferencedCounted() )
> 
> Don't confuse memory allocation with memory management. There's no such a thing as a "reference counted allocator".
> 
> Andrei

Sure, but combining the two could be very useful -- as we have noticed with a allocators that work off of a garbage collector.  With regards to reference counting, you could implement one that automatically wraps the type in an RC struct and proxies them.   Being able to redefined aliases during different sections of compilation would be required though.

On Monday, 29 September 2014 at 22:15:32 UTC, Andrei Alexandrescu wrote:
> On 9/29/14, 11:44 AM, Shammah Chancellor wrote:
>> I don't like the idea of having to pass in template parameters
>> everywhere -- even for allocators.
>
> I agree.
>

There was a solution earlier in this thread which avoids that problem. When a function is annotated with @nogc there's sufficient info to chose the correct implementation without any parameters, it's already known whether we are instantiated from a @nogc block or not.

On Monday, 29 September 2014 at 20:07:41 UTC, Uranuz wrote:

> 1. As far as I understand allocation and memory management of
> entities like class (Object), dynamic arrays and associative
> arrays is part of language/ runtime. What is proposed here is
> *fix* to standart library. But that allocation and MM happening
> via GC is not *fault* of standart library but is predefined
> behaviour of D lang itself and it's runtime. The standard library
> becomes a `hostage` of runtime library in this situation. Do you
> really sure that we should "fix" standart library in that way?
> For me it looks like implementing struts for standard lib (which
> is not broken yet ;) ) in order to compensate behaviour of
> runtime lib.

This really hits the nail on the head, and I think your other comments and questions are also quite insightful.

IMO the proposal that started this thread, @nogc, and -vgc are all beating around the bush rather than addressing the fundamental problem.

Mike