We need to have a simple recipe on how to define a canonical object. That would include requirements such as:

* should work with mutable, const, immutable, and shared
* the right way to define constructor(s)
* the right way to define copying
* the right way to define destructor(s)

I've asked my student Razvan to document the behavior of this(this). His work:

https://github.com/dlang/dmd/pull/8055
https://github.com/dlang/dlang.org/pull/2281
https://github.com/dlang/dlang.org/pull/2299

... reveals a puzzling array of behaviors. Sometimes the typechecking is wrong, too.

I think it's very important for us to have a simple, correct, and canonical way of defining structs in the D language that work with the language features: qualifiers, pure, safe, and nogc.

Once we have that, we can encapsulate desirable abstractions (such as @nogc safe collections that work in pure code), regardless of how difficult their implementations might be. It seems that currently this(this) does not allow us to do that.

Eduard, another student I work with, has made steps toward a collections library that rant into difficulties:

* @safe is achievable with relative ease

* immutable and const are very difficult, but we have an attack (assuming copy construction gets taken care of)

* @nogc is doable with a couple of conventions for allocators

* pure is difficult

* making them work together is very difficult

We need to offer features, tools, and guidance toward creating simple encapsulated types that work well with D's own abstractions. Once we have that we can build libraries to work satisfactorily for any domain.

I think the way to move forward is to deprecate this(this) entirely and create a DIP that allows people to define truly encapsulated structs. This is important, urgent, and of huge impact.

I am looking for folks to assist me in creating a DIP for that. There will be a _lot_ of work involved, so don't take it lightly.


Thanks,

Andrei

On Saturday, March 31, 2018 19:38:06 Andrei Alexandrescu via Digitalmars-d wrote:
> We need to have a simple recipe on how to define a canonical object. That would include requirements such as:
>
> * should work with mutable, const, immutable, and shared
> * the right way to define constructor(s)
> * the right way to define copying
> * the right way to define destructor(s)
>
> I've asked my student Razvan to document the behavior of this(this). His
> work:
>
> https://github.com/dlang/dmd/pull/8055 https://github.com/dlang/dlang.org/pull/2281 https://github.com/dlang/dlang.org/pull/2299
>
> ... reveals a puzzling array of behaviors. Sometimes the typechecking is wrong, too.
>
> I think it's very important for us to have a simple, correct, and canonical way of defining structs in the D language that work with the language features: qualifiers, pure, safe, and nogc.
>
> Once we have that, we can encapsulate desirable abstractions (such as @nogc safe collections that work in pure code), regardless of how difficult their implementations might be. It seems that currently this(this) does not allow us to do that.
>
> Eduard, another student I work with, has made steps toward a collections library that rant into difficulties:
>
> * @safe is achievable with relative ease
>
> * immutable and const are very difficult, but we have an attack (assuming copy construction gets taken care of)
>
> * @nogc is doable with a couple of conventions for allocators
>
> * pure is difficult
>
> * making them work together is very difficult
>
> We need to offer features, tools, and guidance toward creating simple encapsulated types that work well with D's own abstractions. Once we have that we can build libraries to work satisfactorily for any domain.
>
> I think the way to move forward is to deprecate this(this) entirely and create a DIP that allows people to define truly encapsulated structs. This is important, urgent, and of huge impact.
>
> I am looking for folks to assist me in creating a DIP for that. There will be a _lot_ of work involved, so don't take it lightly.

So, is the idea then that we'd switch to copy constructors?

- Jonathan M Davis

On Sat, Mar 31, 2018 at 07:38:06PM -0400, Andrei Alexandrescu via Digitalmars-d wrote: [...]
> Once we have that, we can encapsulate desirable abstractions (such as @nogc safe collections that work in pure code), regardless of how difficult their implementations might be. It seems that currently this(this) does not allow us to do that.

What exactly is it about this(this) that blocks us from doing that?

Removing this(this) is going to be a huge breaking change far bigger than, say, removing autodecoding ever will be.


> Eduard, another student I work with, has made steps toward a collections library that rant into difficulties:
> 
> * @safe is achievable with relative ease
> 
> * immutable and const are very difficult, but we have an attack (assuming copy construction gets taken care of)

A lot of us here have essentially given up on const except for a few very narrow cases.  The transitive nature of const makes it extremely difficult to work with in the general case, even though the simplest use cases are workable.

One of the biggest stumbling blocks is that whenever ranges are involved, const is practically out of the question, because even though it can be made to work for most cases, there will almost always be that one pathological case where it's impossible / too hard to work around, and that ruins it for everything else, so that it's much simpler to just avoid it altogether.

Also, as far as containers are concerned, the lack of a standard way to construct head-mutable / tail-const types that works analogously with built-in arrays makes it very difficult to write generic containers that work well with const/immutable.  It's not too hard to make it work for specific types, but very difficult to write a truly *generic* container that can be deployed in all situations where const/immutable are involved.


[...]
> * pure is difficult
[...]

The one nagging question I've been having about pure is: how much are we actually taking advantage of the guarantees provided by pure?  We have developed very clever ways of extending the traditional definition of pure and invented creative ways of making more things pure, which is all great.  But AFAIK the only place where it's actually taken advantage of is to elide some redundant function calls inside a single expression. And perhaps infer uniqueness in some cases for implicit casting to immutable.

While these are indisputably useful, they seem so far to be only relatively minor benefits.  If pure is indeed so difficult to support generically, it begs the question, is it worth the effort just to gain these niggling benefits?  Whatever happened to larger-scale benefits conferred by purity?


T

-- 
Computers aren't intelligent; they only think they are.

On 3/31/18 8:25 PM, Jonathan M Davis wrote:
> So, is the idea then that we'd switch to copy constructors?

Something like that. We'll need to define them carefully to obey safety and purity.

On Saturday, March 31, 2018 17:32:10 H. S. Teoh via Digitalmars-d wrote:
> On Sat, Mar 31, 2018 at 07:38:06PM -0400, Andrei Alexandrescu via Digitalmars-d wrote: [...]
>
> > Once we have that, we can encapsulate desirable abstractions (such as @nogc safe collections that work in pure code), regardless of how difficult their implementations might be. It seems that currently this(this) does not allow us to do that.
>
> What exactly is it about this(this) that blocks us from doing that?
>
> Removing this(this) is going to be a huge breaking change far bigger than, say, removing autodecoding ever will be.

Well, as far const goes, you have the inherent problem that the object you have in this(this) has already been initialized, and you can't mutate const. I think that other attribute issues are mostly implementation issues.

And as far as auto-decoding goes, removing postblit constructors would actually be far easier, because it's simply a matter of deprecating a particular function. Some traits that check for postblits would have to be adjusted, but all those really care about is that there's extra code beyond the default copying code, and they could easily be made to work with both postblits and whatever the new solution is at the same time until postblits are actually gone.

Auto-decoding on the other hand, affects far more than just the functions in std.range.primitives, and there is no clear deprecation path. So, while it _might_ be true that deprecating postblits would break more code (and I'm honestly not convinced that it would), postblit constructors would actually have a clean deprecation path. So, while it breaks code, it does so in a way that can easily be managed, whereas the removal of auto-decoding is not straightforward at all. It could be done by just flipping the switch so-to-speak, but AFAIK, no one has yet presented a workable deprecation path. And ultimately, I think that _that_ is what's preventing us from fixing auto-decoding. If a clean deprecation path were found, then we could discuss whether the resulting breakage would be worth it, but without a clean deprecation path, I don't see how we could ever do it.

Also, unless we can fix postblit constructors (and the evidence thus far is that if we can, it's way too complicated - e.g. Kenji's attempt several years ago was rejected because it was way too complicated), if we don't switch to a differnt solution, we're talking about permanently not supporting copying const types that require user-defined copying.

> A lot of us here have essentially given up on const except for a few very narrow cases.  The transitive nature of const makes it extremely difficult to work with in the general case, even though the simplest use cases are workable.
>
> One of the biggest stumbling blocks is that whenever ranges are involved, const is practically out of the question, because even though it can be made to work for most cases, there will almost always be that one pathological case where it's impossible / too hard to work around, and that ruins it for everything else, so that it's much simpler to just avoid it altogether.

I don't think that it's even the case that it can be made to work in most cases - or if it can, it involves a lot of static ifs. The range API does not require const for _anything_ (and really can't due to how restricted const is), so _no_ generic range-based code can assume that even something like length or empty can be called if the range is const or inout. As such, the only ranges that can mark anything with const are ones that either aren't generic or which have a bunch of static ifs presenting const and non-const versions depending on the template arguments, and IMHO, that's just not workable. I've done it before, and it's a mess. As such, ranges and const really don't work together at all. const really only works when you're dealing with a very constrainted set of types where you can actually guarantee that they work with const.

> > * pure is difficult
>
> [...]
>
> The one nagging question I've been having about pure is: how much are we actually taking advantage of the guarantees provided by pure?  We have developed very clever ways of extending the traditional definition of pure and invented creative ways of making more things pure, which is all great.  But AFAIK the only place where it's actually taken advantage of is to elide some redundant function calls inside a single expression. And perhaps infer uniqueness in some cases for implicit casting to immutable.
>
> While these are indisputably useful, they seem so far to be only relatively minor benefits.  If pure is indeed so difficult to support generically, it begs the question, is it worth the effort just to gain these niggling benefits?  Whatever happened to larger-scale benefits conferred by purity?

Honestly, I think that the main benefit of pure is that if you know that a function is pure, you know that it doesn't access any global, mutable state except through its arguments. The secondary benefit is that it can allow for functions which construct immutable objects using mutable state and without casts (though such functions are usually small in scale and don't require that large portions of the code base be pure). I think that the idea that pure is going to result in compiler optimizations is mostly a joke. Not only can it only work with strongly pure functions (which most pure functions aren't and can't be), but the way it's currently implemented, it can only elide calls within a single statement (and it might actually be a single expression - I'm not sure which). Going farther than that requires code-flow analysis, which Walter is almost always against, so it's almost certainly not happening. And even if it did, I don't think that it would help much. How often do you call the same function with the same arguments within a single function body? I expect that that's pretty rare. The place that it would likely be of the most benefit would be math code, and even there, I'm not sure that it happens much.

So, I think that the only large-scale benefit thet exists for pure and really can exist for pure is the fact that you know that the function doesn't access global, mutable state. Everything else it does is just gravy and too limited to be a "large-scale" benefit. Certainly, optimizations are clearly _not_ the main benefit of pure, since they almost don't exist. But over time, we have managed to add more gravy here and there as we've figured out assumptions that can be made based on pure (like the case where we can convert the result of a pure function to immutable).

- Jonathan M Davis

On 3/31/18 8:32 PM, H. S. Teoh wrote:
> On Sat, Mar 31, 2018 at 07:38:06PM -0400, Andrei Alexandrescu via Digitalmars-d wrote:
> [...]
>> Once we have that, we can encapsulate desirable abstractions (such as
>> @nogc safe collections that work in pure code), regardless of how
>> difficult their implementations might be. It seems that currently
>> this(this) does not allow us to do that.
> 
> What exactly is it about this(this) that blocks us from doing that?

See the updated docs. Too many bugs in design and implementation.

> Removing this(this) is going to be a huge breaking change far bigger
> than, say, removing autodecoding ever will be.

We're not removing it as much as evolving it: we define an alternate copying mechanism, and once that is in tip-top shape, we deprecate this(this).

> A lot of us here have essentially given up on const except for a few
> very narrow cases.  The transitive nature of const makes it extremely
> difficult to work with in the general case, even though the simplest use
> cases are workable.

Immutable is where it's at, in terms of usefulness. Const is a mere servant of it (and of mutable).

> One of the biggest stumbling blocks is that whenever ranges are
> involved, const is practically out of the question, because even though
> it can be made to work for most cases, there will almost always be that
> one pathological case where it's impossible / too hard to work around,
> and that ruins it for everything else, so that it's much simpler to just
> avoid it altogether.

Yah, the DIP might address that, too. Consider:

void fun(R)(R arr)
{
    pragma(msg, typeof(arr));
}
void main()
{
    immutable(int[]) arr = [ 1, 2, 3 ];
    pragma(msg, typeof(arr));
    fun(arr);
}

The program prints during compilation:

immutable(int[])
immutable(int)[]

Interesting! So the type of the array changes during template matching, which is an exception to the rule that templates always glom to the exact type passed.

This is a hack introduced in the compiler in response to the issues you mention. But we don't need hacks and special casing - we need a means for types to say "here's what needs to happen when a template parameter is matched against this type".

So the DIP would address manipulating qualified ranges as a perk.

> The one nagging question I've been having about pure is: how much are we
> actually taking advantage of the guarantees provided by pure?

Very little, but that doesn't matter. The problem is it's underspecified. So now it's like a vague threat - whenever we mess with fear somebody comes asking, but what about an aggressive compiler doing some unexpected optimizations based on such and such interpretation?

We need to lock pure down.


Andrei

On 3/31/18 9:01 PM, Jonathan M Davis wrote:
> And as far as auto-decoding goes

Let's keep this about construction and not about auto-decoding. Thanks.

On 01/04/2018 11:38 AM, Andrei Alexandrescu wrote:
> We need to have a simple recipe on how to define a canonical object. That would include requirements such as:
> 
> * should work with mutable, const, immutable, and shared
> * the right way to define constructor(s)
> * the right way to define copying
> * the right way to define destructor(s)

We will also need to make this consistent for classes.

				***

So lets go back to the basics.

From the primordial goo that is My Own Little Language (MOLL heavily WIP):

__userType [Identifier:name] {
	(Declaration|Definition)*
}

Is a value type, acts pretty much like struct.
Why do I not call it a struct? Because it shouldn't be used in user code. This exists only in mental models for D users currently.

So something like:

__userType Foo {
	int x;
	
	void __constructor(int x) {
		this.x = x;
	}

	void __postblit() {
		this.x = x+1;
	}

	void __deconstructor() {
		// free?
	}
}

Is comparable to D code of:

struct Foo {
	int x;

	this(this) {
		this.x = x + 1;
	}

	~this() {
		// free?
	}
}

So what does a class look like?

__userType Foo {
	enum __IS_CLASS = true;
	static TypeInfo_Class TypeInfo = TypeInfo_Class(...);

	static void*[][] __vtables = [
		[&TypeInfo, ...],
		[&TypeInfo, ...]
	];

	void*[] __vtable;
	void* __ptr;

	auto __cast(T)() if(T.__IS_CLASS) {
		// ...
	}
}

Why is this important? Because it shows that our current lifetime management strategies for ref counting with structs can be applied directly to classes.

					***

At this point in time, lets look at how each qualifier is perceived:

mutable: default, good, everything else is a pain
const: library code and the author doesn't want you to go touchy
immutable: in binary only, quite yucky
shared: a message from the library author and not much else

I do not believe we can continue this conversation with the above perceptions.

First and foremost qualifiers are there to tell other developers (and yourself in the future) what you intend on doing with a given bit of memory. Now we do need a way to express head-const. Because it allows us to say that we personally can't modify a bit of memory but we can call a method that does (from it).

Good engineering requires us to consider psychology on these issues. To create a single unified view of this, will be tricky I think, but well worth it.

On Saturday, March 31, 2018 19:38:06 Andrei Alexandrescu via Digitalmars-d wrote:
> We need to have a simple recipe on how to define a canonical object. That would include requirements such as:
>
> * should work with mutable, const, immutable, and shared
> * the right way to define constructor(s)
> * the right way to define copying
> * the right way to define destructor(s)
>
> I've asked my student Razvan to document the behavior of this(this). His
> work:
>
> https://github.com/dlang/dmd/pull/8055 https://github.com/dlang/dlang.org/pull/2281 https://github.com/dlang/dlang.org/pull/2299
>
> ... reveals a puzzling array of behaviors. Sometimes the typechecking is wrong, too.
>
> I think it's very important for us to have a simple, correct, and canonical way of defining structs in the D language that work with the language features: qualifiers, pure, safe, and nogc.
>
> Once we have that, we can encapsulate desirable abstractions (such as @nogc safe collections that work in pure code), regardless of how difficult their implementations might be. It seems that currently this(this) does not allow us to do that.
>
> Eduard, another student I work with, has made steps toward a collections library that rant into difficulties:
>
> * @safe is achievable with relative ease
>
> * immutable and const are very difficult, but we have an attack (assuming copy construction gets taken care of)
>
> * @nogc is doable with a couple of conventions for allocators
>
> * pure is difficult
>
> * making them work together is very difficult
>
> We need to offer features, tools, and guidance toward creating simple encapsulated types that work well with D's own abstractions. Once we have that we can build libraries to work satisfactorily for any domain.
>
> I think the way to move forward is to deprecate this(this) entirely and create a DIP that allows people to define truly encapsulated structs. This is important, urgent, and of huge impact.
>
> I am looking for folks to assist me in creating a DIP for that. There will be a _lot_ of work involved, so don't take it lightly.

Another potential issue is whether any of this does or should relate to

https://github.com/dlang/DIPs/pull/109

and it's solution for hooking into to moves. I'm not at all sure that what happens with that needs to be related to this at all, but it might.

- Jonathan M Davis

Am 01.04.2018 um 01:38 schrieb Andrei Alexandrescu:
> We need to have a simple recipe on how to define a canonical object. That would include requirements such as:
> 
> * should work with mutable, const, immutable, and shared
> * the right way to define constructor(s)
> * the right way to define copying
> * the right way to define destructor(s)
> 
> I've asked my student Razvan to document the behavior of this(this). His
> work:
> 
> https://github.com/dlang/dmd/pull/8055 https://github.com/dlang/dlang.org/pull/2281 https://github.com/dlang/dlang.org/pull/2299
> 
> ... reveals a puzzling array of behaviors. Sometimes the typechecking is wrong, too.
> 
> I think it's very important for us to have a simple, correct, and canonical way of defining structs in the D language that work with the language features: qualifiers, pure, safe, and nogc.
> 
> Once we have that, we can encapsulate desirable abstractions (such as @nogc safe collections that work in pure code), regardless of how difficult their implementations might be. It seems that currently this(this) does not allow us to do that.
> 
> Eduard, another student I work with, has made steps toward a collections library that rant into difficulties:
> 
> * @safe is achievable with relative ease
> 
> * immutable and const are very difficult, but we have an attack (assuming copy construction gets taken care of)
> 
> * @nogc is doable with a couple of conventions for allocators
> 
> * pure is difficult
> 
> * making them work together is very difficult
> 
> We need to offer features, tools, and guidance toward creating simple encapsulated types that work well with D's own abstractions. Once we have that we can build libraries to work satisfactorily for any domain.
> 
> I think the way to move forward is to deprecate this(this) entirely and create a DIP that allows people to define truly encapsulated structs. This is important, urgent, and of huge impact.
> 
> I am looking for folks to assist me in creating a DIP for that. There will be a _lot_ of work involved, so don't take it lightly.
> 
> 
> Thanks,
> 
> Andrei

This seems really sudden, april fool's joke? Not really sure, as there are real problems with this(this)...