On 6 May 2014 16:28, Jacob Carlborg via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On 06/05/14 05:51, HaraldZealot wrote:
>
>> Manu, can you direct me what is ARC? This abbreviation is very misgooglly.
>
>
> Automatic Reference Counting. Like regular RC but the compiler automatically inserts calls to release/free.

And further, more importantly, automatically *eliminates* redundant calls to add/dec ref, which I think has much greater potential in D's type system than in Obj-C.

On Tuesday, 6 May 2014 at 06:39:19 UTC, Ola Fosheim Grøstad wrote:
> a GC friendly set of D constructs where FFI is followed by programmer guaranteed postconditions (specified by library authors).

Btw "postcondition" is the wrong term. In order to be robust need to specify how far into the GC heap a FFI can go while it is live. Consider callbacks from C to D. You don't know the layout of the stack frame, so you have to scan the stack assuming anything unknown is a pointer.

But that is not sufficient, because the C code may be holding references to the GC heap in a malloc'ed struct. So, you have to scan all reachable malloc'ed memory too.

At the end of the day some discipline is required for system level programming because the robust solutions are very expensive.

On Tuesday, 6 May 2014 at 03:40:47 UTC, Manu via Digitalmars-d
wrote:
> On 3 May 2014 18:49, Benjamin Thaut via Digitalmars-d
> <digitalmars-d@puremagic.com> wrote:
>> Am 30.04.2014 22:21, schrieb Andrei Alexandrescu:
>>>
>>> Walter and I have had a long chat in which we figured our current
>>> offering of abstractions could be improved. Here are some thoughts.
>>> There's a lot of work ahead of us on that and I wanted to make sure
>>> we're getting full community buy-in and backup.
>>>
>>> First off, we're considering eliminating destructor calls from within
>>> the GC entirely. It makes for a faster and better GC, but the real
>>> reason here is that destructors are philosophically bankrupt in a GC
>>> environment. I think there's no need to argue that in this community.
>>>
>>> The GC never guarantees calling destructors even today, so this decision
>>> would be just a point in the definition space (albeit an extreme one).
>>>
>>> That means classes that need cleanup (either directly or by having
>>> fields that are structs with destructors) would need to garner that by
>>> other means, such as reference counting or manual. We're considering
>>> deprecating ~this() for classes in the future.
>>>
>>> Also, we're considering a revamp of built-in slices, as follows. Slices
>>> of types without destructors stay as they are.
>>>
>>> Slices T[] of structs with destructors shall be silently lowered into
>>> RCSlice!T, defined inside object.d. That type would occupy THREE words,
>>> one of which being a pointer to a reference count. That type would
>>> redefine all slice primitives to update the reference count accordingly.
>>>
>>> RCSlice!T will not convert implicitly to void[]. Explicit cast(void[])
>>> will be allowed, and will ignore the reference count (so if a void[]
>>> extracted from a T[] via a cast outlives all slices, dangling pointers
>>> will ensue).
>>>
>>> I foresee any number of theoretical and practical issues with this
>>> approach. Let's discuss some of them here.
>>>
>>>
>>> Thanks,
>>>
>>> Andrei
>>
>>
>> Honestly, that sounds like the entierly wrong apporach to me. Your
>> approaching the problem in this way:
>>
>> "We can not implement a propper GC in D because the language design prevents
>> us from doing so. So lets remove destructors to migate the issue of false
>> pointers."
>>
>> While the approach should be.
>>
>> "The language does not allow to implement a propper GC (anything else then
>> dirty mark & sweep), what needs to be changed to allow a implementation of a
>> more sophisticated GC."
>
> Couldn't agree more.
> Abandoning destructors is a disaster.
> Without destructors, you effectively have manual memory management, or
> rather, manual 'resource' management, which is basically the same
> thing, even if you have a GC.
> It totally undermines the point of memory management as a foundational
> element of the language if most things are to require manual
> release/finalisation/destruction or whatever you wanna call it.
>
>
>> Also let me tell you that at work we have a large C# codebase which heavily
>> relies on resource management. So basically every class in there inherits
>> from C#'s IDisposable interface which is used to manually call the finalizer
>> on the class (but the C# GC will also call that finalizer!). Basically the
>> entire codebase feels like manual memory management. You have to think about
>> manually destroying every class and the entire advantage of having a GC,
>> e.g. not having to think about memory management and thus beeing more
>> productive, vanished. It really feels like writing C++ with C# syntax. Do we
>> really want that for D?
>
> This is interesting to hear someone else say this. I have always found
> C# - an alleged GC language - to result in extensive manual memory
> management in practise too.
> I've ranted enough about it already, but I have come to the firm
> conclusion that the entire premise of a mark&sweep GC is practically
> corrupt. Especially in D.
> Given this example that you raise with C#, and my own experience that
> absolutely parallels your example, I realise that GC's failure extends
> into far more cases than just the ones I'm usually representing.
>
> I also maintain that GC isn't future-proof in essence. Computers grow
> exponentially, and GC performance inversely tracks the volume of
> memory in the system. Anything with an exponential growth curve is
> fundamentally not future-proof.
> I predict a 2025 Wikipedia entry: "GC was a cute idea that existed for
> a few years in the early 2000's while memory ranged in the 100's mb -
> few gb's, but quickly became unsustainable as computer technology
> advanced".
>

Java Azul VM GC was already handling 1 TB in 2010.

http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf

GC is not the only way of doing automatic memory management, but
this ongoing discussion steams more from D's current GC status
and respective phobia in C world, and less from what a modern GC
is capable of.

--
Paulo

Paulo Pinto:

> Java Azul VM GC was already handling 1 TB in 2010.
>
> http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf
>
> GC is not the only way of doing automatic memory management, but
> this ongoing discussion steams more from D's current GC status
> and respective phobia in C world, and less from what a modern GC
> is capable of.

On the other hand I don't think you will have available in the D world the amount of work done on the Azul GC. This is the problem with looking at Java.

Bye,
bearophile

On Tuesday, 6 May 2014 at 07:34:13 UTC, bearophile wrote:
> Paulo Pinto:
>
>> Java Azul VM GC was already handling 1 TB in 2010.
>>
>> http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf
>>
>> GC is not the only way of doing automatic memory management, but
>> this ongoing discussion steams more from D's current GC status
>> and respective phobia in C world, and less from what a modern GC
>> is capable of.
>
> On the other hand I don't think you will have available in the D world the amount of work done on the Azul GC. This is the problem with looking at Java.
>
> Bye,
> bearophile

Hence my remark that this whole discussion is more about finding working alternatives in the D world and less about what industrial class GCs are capable of.

I was just doing a heads up, as people tend to misunderstand GCs.

--
Paulo

> That experimental package idea that was discussed months ago
> comes to my mind again. Add that thing as exp.rational and
> have people report bugs or shortcomings to the original
> author. When it seems to be usable by everyone interested it
> can move into Phobos proper after the formal review (that
> includes code style checks, unit tests etc. that mere users
> don't take as seriously).
>
> As long as there is nothing even semi-official, it is tempting
> to write such a module from scratch in a quick&dirty fashion
> and ignore existing work.
> The experimental package makes it clear that this code is
> eventually going to the the official way and home brewed stuff
> wont have a future. Something in the standard library is much
> less likely to be reinvented. On the other hand, once a module
> is in Phobos proper, it is close to impossible to change the
> API to accommodate for a new use case. That's why I think the
> most focused library testing and development can happen in the
> experimental phase of a module. The longer it is, the more
> people will have tried it in their projects before formal
> review, which would greatly improve informed decisions.
> The original std.rationale proposal could have been in active
> use now for months!

+1

The same idea came to my mind yesterday too (that signalize that isn't void idea).

And GNU/Linux community has proof that such model works. I mean Archlinux as example (but as I think other example exists). There are core, extra, community repositories and AUR (arch user repository). Today AUR contains 48489 packages and 11814 of which are orphan. But even orphanity isn't a sign of useless. Possibility to write PKGBUILD and place it in AUR has everybody and it is not such difficult. Only small part of these package goes to official repository, but this part EXISTS. And in my opinion this is the most rapid way for evolution. For example archlinux has the best support of D language through abundance of linux distributives (http://pastebin.com/tPKWS4ga and this is official repositories not in AUR). Oh, I've forgotten to mention all sorts of testing repositories.

And D community have all means for evolute such way. We have dub and http://code.dlang.org/. It rests to do a little: create package (in D sense) which will contain packages that installed through dub; and some UI to vote, orphan and adopt package.

There are example as how that I see:

On http://code.dlang.org/ we can found  package "rationals" (for example) with mark expstd. Than we install package with dub to some environment (it may be fakeroot, peruser import path or even perproject directory (the last likes me the least)) and address this package as "import expstd.rationals;" in our D projects. It's were good when dub or code.dlang.org counts installation. When some package counts enough installation and enough history (includes bug tracking), it become candidate to include in phobos. All trusted users (in therm of Archlinux) receive strong signal about and one of them review such package and become its auxiliary  comaintainer. No one package can exist in official repository without maintainer.

And some words about why "expstd". I think that with such package model comes time to add not only phobos in "namespace". There are many useful libraries which may not become part of phobos (linear algebra for example, or even GUI bindings), but should have some standardization and  highway to distribution. I suggest package "extra" (in D sense) and "bind" (for deimos and perhaps not only deimos packages). Naturally they start their life as "expextra" and "expbind". In the future we will need some other package (in D sense), but for start that three are enough.

As Andrei said "acta non verba". I promise that I make all, considering my life environment, to help and start this project.

P.S. This theme will become independent. And this is yet another one argue, that forum forms not ideal to represent news and current state of community. Important things sink often in other topics.

P.P.S. My apologies for my poor English.

>
> The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available.

So it seems that I have understood more than I've thought.

If really problem rely on this than an idea came to my mind yesterday. What about separate destructors call and garbage collections mechanism? This idea reflects current D state with postblit constructor in structures. Raw memory allocation, default initialization (with T.init field) and after all that calling constructor (especially postblit). In therms of destructors it sound as: after becoming object or structure  needless some automatic mechanism call cascading destructor, but memory still not fried, and GC collects memory in its time. Yes it's sounded monstrous, but perhaps when which mechanism will done only self work, each of them can be lighter. And you always can call some kind of finalizator manually for performance (some kind of flag can exist to protect from double call of destructors).

On Tuesday, 6 May 2014 at 06:07:41 UTC, HaraldZealot wrote:
> I have to say that all this discussion (more precisely the understanding on the side of key developers) make me very upset.
>
> It's good that Andrei agreed with impossibility of the harebrained disallowing of the class destructors. But I was very surprise, that so thought go to such head, because such solution contradicts the D spirit totally. There are many language which are very popular and have many dark moments in their design. I (and I think not only me) go to the D not for its popularity, but for its clarity, power and SANITY (that bases on strong guaranties). The strong solutions found on the strong decision makes D itself. (Examples of such strong solutions: immutabilities, default unshareness, struct and class as distinct being). And way that leads us in state where stucts have dtors and classes haven't but allow struct with dtor as member and people have to call struct dtor manually, isn't D way. Because such way relies on programmers discipline, but how Andrei has written "If there one thing that decades of computing have taught us, it must be that discipline-oriented programming does not scale."[TDPL, p. 399].

Actually, IMO the clarity and sanity you talk about are a strong argument _for_ the deprecation of destructors for GC objects:

Premise 1: Destructors are for things that _must_ be done when an object is destroyed. Destroying (or leaking) an object without calling its destructor can (and often will) lead to wrong program behavior and is therefore an error.

Premise 2: Tracing GCs are by their nature non-deterministic and can not guarantee proper destruction of objects. This is true even if full type information about the managed objects is available (which is currently the case only for classes), as long as they are non-precise (which, realistically, in a systems programming language they will always be at least to some degree).

Conclusion: It's an error to have objects with destructors be managed by the GC. Per Andrei's quote about "discipline-oriented programming", this should therefore be made impossible.

(Apart from that, there are tons of things that you aren't allowed to do in a destructor called from the current GC, but there's currently no way to detect them statically => again "discipline-oriented programming".)

Of course, how exactly this is achieved is up for debate. Also, maybe one of the premises is false...

On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On 06/05/14 08:07, HaraldZealot wrote:
>
>> I notice that I view only part of problem, can anybody link or describe me completely state and problems of current garbage collection and other resource management? It help me in finding of existence solution (at least theoretical).
>
>
> The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available.

I think it's also an important consideration that GC is incompatible with low-memory and real-time environments.

The trouble is, GC runs more frequently as the amount of available
free memory decreases, and trace&collect takes a long time, long
enough that realtime software typically can't tolerate the pause. If
the pause is super rare (minutes/hours apart), maybe it's acceptable,
but if it's closer to every call to alloc (ie, when there is little/no
free memory), then you realise it's fundamentally incompatible.
Then consider that this is typically the default operating state of
embedded systems; to have no free memory at runtime, and therefore D
is effectively incompatible with a subset of computers...
specifically, games consoles; a gigantic industry, and one of the few
that still absolutely relies on native code where D is particularly
compelling.

If people want to call D a native systems language, this seems like a
serious conflict with that idea. A major subset of the language is
incompatible with realtime or embedded use (surely among the most
compelling use cases remaining for native code these days!). Usually
the argument is that I represent a niche, and therefore I should
accept the restrictions of my environment, and revert to full manual
memory management for my work, while letting everyone else have their
fun.
That is a broken argument, not only because it leaves very little
motivation to leave C++, but also because regardless of whether I
adopt such restrictions, library authors won't, and we don't live in
the 1980's where I can write a few lines of self-contained assembly
and call it a program. To say I can't depend on libraries is
practically absurd. Games are gigantic software projects, and depend
on heaps of libraries.

The GC effectively eliminates libraries from embedded/realtime users, which I see as an absolute deal-breaker alone, irrespective of a further bunch more restrictions it places on that subset of users within their own code :/

I'll be happy with any solution that works, really. But as I see it, reference counting is the only garbage collection technology I know which will reliably clean up eagerly as things fall out of scope (also addressing this issue with destructors as bonus), and as such, it seems the logical solution. ARC is a demonstrable success in Obj-C, and works well in realtime and embedded systems. D has type system improvements which would theoretically allow for significant improvements over Obj-C.

On 6 May 2014 17:16, Paulo Pinto via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Tuesday, 6 May 2014 at 03:40:47 UTC, Manu via Digitalmars-d
> wrote:
> Java Azul VM GC was already handling 1 TB in 2010.

Is D compatible with this GC? And how does it go in an environment with 128mb of ram, and no free memory?


> http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf
>
> GC is not the only way of doing automatic memory management, but this ongoing discussion steams more from D's current GC status and respective phobia in C world, and less from what a modern GC is capable of.

It's not a 'phobia', it's an incompatibility. Show me a 'modern' GC
that's compatible with realtime+embedded, and also compatible with D,
and then we can talk... I'm sure there's a very good reason nobody has
raised such a technology for consideration.
Framing the C world in this way is completely unfair. I've spent years
here trying to reconcile my relationship with the GC. I've actually
defended the GC on numerous occasions. I don't want to reject the GC,
it's very convenient... but I think it's reached the point where I
have to face the music. I want to know the plan, and if it remains
incompatible with my industry, then I can't reasonably continue to
advocate D in my field, can I?

If it were my language, I know what I'd do. It may require some radical changes, and I would do it because I simply had _no other choice_. Please, make suggestions! It's been years, and I'm still waiting to hear another realistic proposal.