http://d.puremagic.com/issues/show_bug.cgi?id=8993

           Summary: Implement unique references/isolated memory
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: DMD
        AssignedTo: nobody@puremagic.com
        ReportedBy: sludwig@outerproduct.org


--- Comment #0 from Sönke Ludwig <sludwig@outerproduct.org> 2012-11-10 06:40:09 PST ---
'unique' or 'isolated' is a recursive type qualifier which enforces that there is only a single reference to a given piece of memory, and that the memory itself is only pointing to other data that is either immutable, or only referenced within the same memory cluster.

Any value/reference that is tagged this way can safely be converted to immutable or can safely be passed (moved) to another thread. In conjunction with (weakly) pure functions it is possible to operate almost freely on a unique value, because it can be guaranteed that no visible additional references can be created.

The main advantages of this approach are:

 - Provides a powerful alternative to safely pass mutable data between threads
without requiring locks or synchronization primitives (synchronized/shared)

 - Construction of immutable objects can be done in a natural procedural
fashion and not only within the object constructor or within a pure function

 - Fully compatible with the current type system, builds nicely on top of the
current base

 - The system is proven both in theory and in practice

Microsoft has released a paper describing the system in form of an extension to C#. The paper includes formal proofs for the soundness of the system and mentions a multi-million lines of code project for which it is used:

http://research.microsoft.com/pubs/170528/msr-tr-2012-79.pdf

D specific and in addition to the Microsoft paper, a solution to support partitioning of a uniqe array with unique slices as the result is desirable. But this can be done as a simple library function if needed.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=8993


Alex Rønne Petersen <alex@lycus.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alex@lycus.org


--- Comment #1 from Alex Rønne Petersen <alex@lycus.org> 2012-11-10 15:46:00 CET ---
I think it's a bit too late to add new keywords to the language, other than ones prefixed with "__", and we certainly wouldn't want something this useful to be subject to such annoying-to-type keywords...

This seems like a thing for D3 *if* D3 ever becomes a thing.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=8993


bearophile_hugs@eml.cc changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bearophile_hugs@eml.cc


--- Comment #2 from bearophile_hugs@eml.cc 2012-11-10 08:24:06 PST ---
Writing a detailed DEP requires lot of work, but maybe for a request this complex it's unavoidable.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #3 from Sönke Ludwig <sludwig@outerproduct.org> 2012-11-10 08:38:03 PST ---
(In reply to comment #1)
> I think it's a bit too late to add new keywords to the language, other than ones prefixed with "__", and we certainly wouldn't want something this useful to be subject to such annoying-to-type keywords...
> 
> This seems like a thing for D3 *if* D3 ever becomes a thing.

Even with __ it would be better than nothing. I think it's almost safe to say that current D is almost unusable in conjunction with immutable/shared (objects) and that's ironically one of the areas where it tries to shine. I have tried numerous times to cleanly reimplement certain multithreaded functionality using it and _never_ got it running without just so many casts that the result was not only as unsafe/unchecked as the original, but also unreadable und badly usable.


I apologize in advance for the following (general) rant:

The attitude of implementing ad-hoc solutions, not listening to valid objections and then declaring the language as stable is a very foolish development approach. The way shared was introduced was not the only thing in the past that went this way.

Things like these made me really frustrated with D multiple times, to the point where I was seriously looking out for alternatives - the end result was pure disregard/disinterest for the corresponding language parts. And I think that there is quite a number of people who just gave up on either the language or on certain parts, and the resulting silence creates a dangerous deceived feeling of statisfaction.

The current mode of development is shizophrenic, stabilizing an obviously unfinished language and, from time to time, introducing new, breaking things anyway. There absolutely needs to be a way to still let innovation and refinement happen, be it in a separate branch (called 3.x or something else doesn't matter) or in the form of allowing new features as long as they are backwards-compatible (e.g. using the __ way). Of course the constant breakage of the past is bad, but stagnation at this point could possibly be the start of a slow, starving death.

Sorry again, but I've seen D make so many mistakes in the past and that it has done a lot more things right is no excuse for that.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #4 from Alex Rønne Petersen <alex@lycus.org> 2012-11-10 17:47:59 CET ---
Yeah, no doubt shared is a disaster in general (I have posted several rants about this in particular on the NG). We're effectively doing a lot of false advertising with that type qualifier in particular, and people are writing code as if it results in atomic operations (and then their code will break on !x86).

And immutable/const are only realistically usable with non-class types. I feel you're spot on with the point about perceived satisfaction.

But it's not me you have to convince. It's Walter and Andrei...

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #5 from bearophile_hugs@eml.cc 2012-11-10 09:48:35 PST ---
(In reply to comment #3)

> I apologize in advance for the following (general) rant:

I agree with what you say.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #6 from bearophile_hugs@eml.cc 2012-11-10 09:50:22 PST ---
(In reply to comment #5)

> I agree with what you say.

But in the end this enhancement request is not enough. A detailed DEP is probably necessary. You don't have to write it down all yourself, of course. Once the DEP is written, you will be in a better position to rant :-)

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #7 from Peter Alexander <peter.alexander.au@gmail.com> 2012-11-10 10:04:02 PST ---
Just want to chime in to say that I 100% agree with Sönke and Alex.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #8 from Sönke Ludwig <sludwig@outerproduct.org> 2012-11-10 10:38:42 PST ---
(In reply to comment #6)
> (In reply to comment #5)
> 
> > I agree with what you say.
> 
> But in the end this enhancement request is not enough. A detailed DEP is probably necessary. You don't have to write it down all yourself, of course. Once the DEP is written, you will be in a better position to rant :-)

Agreed, a DIP is probably in order for this change. It would probably be enough to concentrate on the simple parts first (implicit isolated->immutable conversion and free psasing of isolated values between threads) and then implement further relaxations bit by bit (e.g. implicit scoped isolated->writable->isolated conversion). Allocating time for a proper DIP is difficult for me currently, but I guess it could be marked as work-in-progress in the wiki so that it can be completed step by step?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #9 from bearophile_hugs@eml.cc 2012-11-10 11:11:35 PST ---
(In reply to comment #8)

> but I guess it could be marked as work-in-progress
> in the wiki so that it can be completed step by step?

I think this is OK.

In the paper they say the Microsoft programmers have implemented that system one piece at a time. So take a look at those implementation steps (avoiding their eventual mistakes).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------