On Saturday, 31 March 2018 at 23:38:06 UTC, Andrei Alexandrescu wrote:
> [snip]
>
> * immutable and const are very difficult, but we have an attack (assuming copy construction gets taken care of)
>

Would it be easier if the const/immutable containers were considered separate types? For instance, in the code below, there is InoutFoo and then Foo takes InoutFoo as an alias this (you could do the same thing with immutable, but then you’d have to include two get functions). This would be like inheriting the InoutFoo.

With better syntax, InoutFoo would be something like inout(Foo) and the compiler could recognize that the mutable constructor is also defined and to call that when appropriate.

struct InoutFoo
{
    int a;

    this(int b) inout
    {
        this.a = b;
    }

    int get() inout
    {
        return a;
    }
}

struct Foo
{
    InoutFoo inoutfoo;
    alias inoutfoo this;

    this(int b)
    {
        a = b;
    }

    void set(int b)
    {
        a = b;
    }
}


void main()
{
    auto x = immutable InoutFoo(1);
    auto y = Foo(1);

    assert(is(typeof(y) : typeof(x)));

    //x.a++; //not allowed
    y.a++;

    assert(x.a == 1);
    assert(y.a == 2);

    assert(x.get == 1);
    y.set(3);
    assert(y.get == 3);
}

On 04/03/2018 05:13 PM, Steven Schveighoffer wrote:
> Unfortunately, I found out that it's not just "pre-filled with some values". Member postblits are run before the containing postblit.
> 
> https://run.dlang.io/is/mt6eGa
> 
> So this means, the data that is available to the postblit has already been processed.

There's a similar situation with constructors: A constructor can call another constructor, which can lead to double initialization of fields.

Example:

----
class C
{
    int x;
    this() immutable
    {
        this(42); /* Initializes x. */
        x = 13; /* Breaking immutable, or ok? */
    }
    this(int x) immutable
    {
        this.x = x;
    }
}
----

If there's a problem with running two postblits on the same field, then I think constructors probably have similar issue. I'm having a hard time finding a good example, though. One where we could break immutable in an obvious way or some such.

> It would only make sense to allow const postblits to have the same constructor mechanism if the members all had no postblits.

Less drastically, we could also disallow writing to those fields that have their own postblit.

The analog with constructors would be disallowing writing to fields that have already been initialized by an implicit `super` call. Actually, that seems to be how it works, kinda-sorta:

----
class C
{
    int x;
}

class D : C
{
    int y;
    this() immutable
    {
        y = 1; /* accepted */
        x = 2; /* Error: cannot modify this.x in immutable function */
    }
}
----

Obviously, DMD doesn't check what C's constructor actually does (because it generally can't check that). It just considers initializing x to be C's responsibility.

On 04/03/2018 08:57 PM, Andrei Alexandrescu wrote:
> Well... not really. This is because .init is really an inert state - null indirections, no state allocated etc.

.init can have non-null indirections:

    struct S { int[] a = [1, 2, 3]; }
    static assert(S.init.a.ptr !is null); /* passes */

But maybe it shouldn't. It leads to problems with immutable, because the same .init with the same `a` is used for both mutable and immutable objects. https://issues.dlang.org/show_bug.cgi?id=10376

As far as I see, a const/immutable postblit working like a constructor wouldn't have that particular problem, because the copy is always made from another const/immutable object, so there wouldn't be aliasing with mutable data.

But there might of course be other problems with indirections, which I don't see right now. I'd love to see any examples, though. They probably reveal weaknesses in .init/constructors, too.

> Makes typechecking easy, and calling constructor on top of .init is what happens already. In contrast, the postblit situash is very different - the fields already contain "interesting" data, allocated resources etc. Calling a constructor on top of that is not defined.

Well, the idea would be to define it. But if postblit goes away for other reasons anyway (like the atomic copy thing, or another mechanism being simply superior), then there's no point in pursuing this, of course.

On 4/3/18 4:26 PM, ag0aep6g wrote:
> On 04/03/2018 05:13 PM, Steven Schveighoffer wrote:
>> Unfortunately, I found out that it's not just "pre-filled with some values". Member postblits are run before the containing postblit.
>>
>> https://run.dlang.io/is/mt6eGa
>>
>> So this means, the data that is available to the postblit has already been processed.
> 
> There's a similar situation with constructors: A constructor can call another constructor, which can lead to double initialization of fields.
> 
> Example:
> 
> ----
> class C
> {
>      int x;
>      this() immutable
>      {
>          this(42); /* Initializes x. */
>          x = 13; /* Breaking immutable, or ok? */

IMO, breaking immutable.

>      }
>      this(int x) immutable
>      {
>          this.x = x;
>      }
> }
> ----
> 
> If there's a problem with running two postblits on the same field, then I think constructors probably have similar issue. I'm having a hard time finding a good example, though. One where we could break immutable in an obvious way or some such.

You may NOT want to run a postblit on the member. If all you are going to do, for example, is reassign a variable, then running the postblit, and then the destructor, just so you can overwrite it is pointless.

But more importantly, if the postblit of the member does something crazy like stores a reference to itself as an immutable elsewhere, and then the compiler allows overwriting, we now have problems.

I think the better mechanism for immutable copying would be to have a copy constructor that starts off with T.init, and is passed the object to copy from. That seems to be a direction Andrei is considering.

-Steve

On 04/03/2018 04:26 PM, ag0aep6g wrote:
> On 04/03/2018 05:13 PM, Steven Schveighoffer wrote:
>> Unfortunately, I found out that it's not just "pre-filled with some values". Member postblits are run before the containing postblit.
>>
>> https://run.dlang.io/is/mt6eGa
>>
>> So this means, the data that is available to the postblit has already been processed.
> 
> There's a similar situation with constructors: A constructor can call another constructor, which can lead to double initialization of fields.
> 
> Example:
> 
> ----
> class C
> {
>      int x;
>      this() immutable
>      {
>          this(42); /* Initializes x. */
>          x = 13; /* Breaking immutable, or ok? */
>      }
>      this(int x) immutable
>      {
>          this.x = x;
>      }
> }
> ----

Let's replace "int" with an UDT:

struct S
{
    int x = -1;
    this(int y) immutable { x = y; }
    void opAssign(int) immutable;
}

class C
{
    S x;
    this() immutable
    {
        this(42); /* Initializes x. */
        x = 13; /* Breaking immutable, or ok? */
    }
    this(int x) immutable
    {
        this.x = x;
    }
}

This code compiles, and calls the constructor twice for the same object. Clearly that shouldn't be allowed to pass. I've submitted https://issues.dlang.org/show_bug.cgi?id=18719 - thanks! (The problem seems to occur even without immutable, it's endemic to forwarding constructors.)

Andre

On 04/03/2018 04:29 PM, ag0aep6g wrote:
> But if postblit goes away for other reasons anyway (like the atomic copy thing, or another mechanism being simply superior), then there's no point in pursuing this, of course.

The DIP will definitely need to make a solid case supporting whatever it proposes.

On 04/03/2018 10:51 PM, Steven Schveighoffer wrote:
> On 4/3/18 4:26 PM, ag0aep6g wrote:
[...]
>> If there's a problem with running two postblits on the same field, then I think constructors probably have similar issue. I'm having a hard time finding a good example, though. One where we could break immutable in an obvious way or some such.
> 
> You may NOT want to run a postblit on the member. If all you are going to do, for example, is reassign a variable, then running the postblit, and then the destructor, just so you can overwrite it is pointless.

Same with class constructors: You may not want to run `super` when you're just going to overwrite what it did. But the language doesn't give you a choice. It'll be called one way or another.

I'm not saying that imitating how constructors work will make the best possible copying mechanism. Something else might be superior in every way. It's just that so far the arguments against a constructor-like postblit also seem to apply to constructors as they are implemented.

So I'm thinking that a postblit modeled after constructors could work as well as they do. But maybe the real takeaway is that constructors don't work very well, and shouldn't be imitated.

> But more importantly, if the postblit of the member does something crazy like stores a reference to itself as an immutable elsewhere, and then the compiler allows overwriting, we now have problems.

I'd love to see an example of this in code. The best I can come up with would be something like this (doesn't compile):

----
import std.stdio;

immutable(int)* p;

struct S
{
    int x;
    this(this) immutable
    {
        x = 42; /* First write. */
        .p = &this.x;
        writeln(p, " ", *p); /* Prints some address and 42. */
    }
}

struct T
{
    S s;
    this(this) immutable
    {
        s = S(13); /* Second write. Breaking immutable? */
        writeln(p, " ", *p); /* Same address, but 13. */
    }
}

void main()
{
    immutable T foo;
    immutable bar = foo;
}
----

But that's essentially the same as the class example I posted. `*p` would only change values during the postblit run. Just like a constructor chain can write to the same field multiple times.

That's kinda iffy, but I can't find a way to demonstrate some real, obvious damage.

> I think the better mechanism for immutable copying would be to have a copy constructor that starts off with T.init, and is passed the object to copy from. That seems to be a direction Andrei is considering.

No objection from me. If Andrei et al. can find a better solution, great.

On 01/04/18 04:56, Jonathan M Davis wrote:
> Another potential issue is whether any of this does or should relate to
> 
> https://github.com/dlang/DIPs/pull/109
> 
> and it's solution for hooking into to moves. I'm not at all sure that what
> happens with that needs to be related to this at all, but it might.
> 
> - Jonathan M Davis

I was actually going to start a new thread about it.

On the one hand, nothing in the opMove DIP is directly affected by this. On the other, if we're moving away from "copy and then fix" mentality, then opMove should also reflect this.

The problem is that the alternative solution has a much bigger impact on backward compatibility. I'm really tempted to try and push this DIP as is, only renaming "opMove" to "opPostMove". This way, we can push the simpler version as is, and when (and it will take a while) "this(this)" deprecation finally comes, we can implement "opMove" as an in-process user hook.

Shachar

On 4/3/18 5:44 PM, ag0aep6g wrote:
> On 04/03/2018 10:51 PM, Steven Schveighoffer wrote:
>> On 4/3/18 4:26 PM, ag0aep6g wrote:
> [...]
>>> If there's a problem with running two postblits on the same field, then I think constructors probably have similar issue. I'm having a hard time finding a good example, though. One where we could break immutable in an obvious way or some such.
>>
>> You may NOT want to run a postblit on the member. If all you are going to do, for example, is reassign a variable, then running the postblit, and then the destructor, just so you can overwrite it is pointless.
> 
> Same with class constructors: You may not want to run `super` when you're just going to overwrite what it did. But the language doesn't give you a choice. It'll be called one way or another.

At least you can invoke the one you want, with postblit there is only one "choice".

But this is a red herring -- we already have struct constructors, and that requirement of invoking constructors on members is not present.

With structs, we have the possibility of initialization via different mechanisms: constructor, postblit, .init. All of these are supported by the struct member, but currently you can only invoke postblit if you are in a postblit. And only at the beginning. I would like to see more flexibility for copying.

> I'm not saying that imitating how constructors work will make the best possible copying mechanism. Something else might be superior in every way. It's just that so far the arguments against a constructor-like postblit also seem to apply to constructors as they are implemented.

For structs, using .init is a valid initialization, so it's completely different from classes, where a constructor MUST be invoked. Indeed, there is no mechanism to require calling struct member constructors in the owner's ctor.

Stop thinking class constructors, and think struct constructors instead.

>> But more importantly, if the postblit of the member does something crazy like stores a reference to itself as an immutable elsewhere, and then the compiler allows overwriting, we now have problems.
> 
> I'd love to see an example of this in code. The best I can come up with would be something like this (doesn't compile):
> 
> ----
> import std.stdio;
> 
> immutable(int)* p;
> 
> struct S
> {
>      int x;
>      this(this) immutable
>      {
>          x = 42; /* First write. */
>          .p = &this.x;
>          writeln(p, " ", *p); /* Prints some address and 42. */
>      }
> }
> 
> struct T
> {
>      S s;
>      this(this) immutable
>      {
>          s = S(13); /* Second write. Breaking immutable? */

Of course this doesn't compile, because s is considered immutable by now. What I was saying is that we can't allow postblit to modify data that has already been postblitted, because of the reason this example is showing.

>          writeln(p, " ", *p); /* Same address, but 13. */
>      }
> }
> 
> void main()
> {
>      immutable T foo;
>      immutable bar = foo;
> }
> ----
> 
> But that's essentially the same as the class example I posted. `*p` would only change values during the postblit run. Just like a constructor chain can write to the same field multiple times.

I don't think you should be able to write to the same field multiple times in an immutable/const constructor. If so, that's a bug.

> That's kinda iffy, but I can't find a way to demonstrate some real, obvious damage.

Any place where an immutable can be observed to change between two reads is breaking immutable.

-Steve

On 04/04/2018 12:37 PM, Steven Schveighoffer wrote:
> With structs, we have the possibility of initialization via different mechanisms: constructor, postblit, .init. All of these are supported by the struct member, but currently you can only invoke postblit if you are in a postblit. And only at the beginning. I would like to see more flexibility for copying.
[...]
> For structs, using .init is a valid initialization, so it's completely
> different from classes, where a constructor MUST be invoked. Indeed,
> there is no mechanism to require calling struct member constructors in the owner's ctor.
[...]

To paraphrase your point: We should be able to initialize a member with .init or by calling its constructor (over .init). We should not be forced to initialize it with its postblit.

Makes sense. But I see one tricky case: Can I also choose to use the blitted value of the member without calling its postblit? I'd say that can't be allowed.

So the compiler would have to identify that case and reject it. And now we're in uncharted territory. As far as I see, this is not something that constructors do (struct or class). They happily accept any .init value. But postblits can't accept any blitted value. So we can't say: "Just do what constructors do." We have to come up with new rules.

But we don't want to come up with new rules, we want to say: "Just do what constructors do." So we require the member postblit, and we say that the outer postblit operates on the resulting value like a constructor operates on .init.

And then we see that it breaks the type system, and that any analog behavior of constructors just means that constructors/.init are broken, too.

[...]
>> struct S
>> {
>>      int x;
>>      this(this) immutable
>>      {
>>          x = 42; /* First write. */
[...]
>>      }
>> }
>>
>> struct T
>> {
>>      S s;
>>      this(this) immutable
>>      {
>>          s = S(13); /* Second write. Breaking immutable? */
> 
> Of course this doesn't compile, because s is considered immutable by now.

It doesn't compile, because this(this) is a broken mess currently.

The first write doesn't compile either. Obviously, with a properly implemented immutable this(this), you would at least be allowed to write once.

With mutable(!) `this(this)`s, both writes are accepted.

> What I was saying is that we can't allow postblit to modify data that has already been postblitted, because of the reason this example is showing.

Still, I'd like to have a more explosive example. The breakage here is very localized, and could possibly be defined away somehow. Like: "Unique immutable data is considered 'raw' in constructors and postblit functions. That means, the data can be mutated. Only when the constructor/postblit returns does it become 'cooked' and truly immutable."

[...]
>> ----
[...]
> I don't think you should be able to write to the same field multiple times in an immutable/const constructor. If so, that's a bug.

Are you counting .init as a write, or is a constructor allowed to build on that?

----
import std.stdio;
struct S
{
    int x = 1;
    int y = 3;
    this(int dummy) immutable
    {
        writeln(x); /* 1 */
        ++x; /* Breaking immutable? */
        writeln(x); /* 2 */

        ++y; /* Breaking immutable even though y hasn't been printed yet? */
        writeln(y); /* 4 */
    }
}
void main()
{
    auto s = immutable S(0);
}
----

I think those increments could be considered breaking immutable. Maybe they must be.

If this were to be outlawed, I'd agree a constructors and postblits are fundamentally different.