On Tue, 20 Nov 2018 15:29:50 +0000, Kagamin wrote:
> On Tuesday, 20 November 2018 at 11:11:43 UTC, aliak wrote:
>> This only applies to little scripts and unittests maybe.
>>
>> Not when you're writing any kind of relatively larger application that involves being run for longer or if there's more possible permutations of your state variables.
> 
> Umm... if you write a larger application not knowing what is a reference type, you're into lots and lots of problems.

A pointer to a struct is a reference type.

There are plenty of cases where you need reference semantics for a thing. If you are optimistic about the attentiveness of your future self and potential collaborators, you might add that into a doc comment; people can either manually do escape analysis to check if they can store the thing on the stack, or allocate on the heap and pass a pointer, or embed the thing as a private member of another thing that now must be passed by reference.

If you're pessimistic and don't mind a potential performance decrease, you might defensively use a class to ensure the thing is a reference type.

On Tuesday, 20 November 2018 at 03:38:14 UTC, Jonathan M Davis wrote:
>
> For @safe to function properly, dereferencing null _must_ be guaranteed to be memory safe, and for dmd it is, since it will always segfault. Unfortunately, as understand it, it is currently possible with ldc's optimizer to run into trouble, since it'll do things like see that something must be null and therefore assume that it must never be dereferenced, since it would clearly be wrong to dereference it. And then when the code hits a point where it _does_ try to dereference it, you get undefined behavior. It's something that needs to be fixed in ldc, but based on discussions I had with Johan at dconf this year about the issue, I suspect that the spec is going to have to be updated to be very clear on how dereferencing null has to be handled before the ldc guys do anything about it. As long as the optimizer doesn't get involved everything is fine, but as great as optimizers can be at making code faster, they aren't really written with stuff like @safe in mind.

One big problem is the way people talk and write about this issue. There is a difference between "dereferencing" in the language, and reading from a memory address by the CPU.
Confusing language semantics with what the CPU is doing happens often in the D community and is not helping these debates.

D is proclaiming that dereferencing `null` must segfault but that is not implemented by any of the compilers. It would require inserting null checks upon every dereference. (This may not be as slow as you may think, but it would probably not make code run faster.)

An example:
```
class A {
    int i;
    final void foo() {
     	import std.stdio; writeln(__LINE__);
        // i = 5;
    }
}

void main() {
    A a;
    a.foo();
}
```

In this case, the actual null dereference happens on the last line of main. The program runs fine however since dlang 2.077.
Now when `foo` is modified such that it writes to member field `i`, the program does segfault (writes to address 0).
D does not make dereferencing on class objects explicit, which makes it harder to see where the dereference is happening.

So, I think all compiler implementations are not spec compliant on this point.
I think most people believe that compliance is too costly for the kind of software one wants to write in D; the issue is similar to array bounds checking that people explicitly disable or work around.
For compliance we would need to change the compiler to emit null checks on all @safe dereferences (the opposite direction was chosen in 2.077). It'd be interesting to do the experiment.

-Johan

On 11/20/18 1:04 PM, Johan Engelen wrote:
> On Tuesday, 20 November 2018 at 03:38:14 UTC, Jonathan M Davis wrote:
>>
>> For @safe to function properly, dereferencing null _must_ be guaranteed to be memory safe, and for dmd it is, since it will always segfault. Unfortunately, as understand it, it is currently possible with ldc's optimizer to run into trouble, since it'll do things like see that something must be null and therefore assume that it must never be dereferenced, since it would clearly be wrong to dereference it. And then when the code hits a point where it _does_ try to dereference it, you get undefined behavior. It's something that needs to be fixed in ldc, but based on discussions I had with Johan at dconf this year about the issue, I suspect that the spec is going to have to be updated to be very clear on how dereferencing null has to be handled before the ldc guys do anything about it. As long as the optimizer doesn't get involved everything is fine, but as great as optimizers can be at making code faster, they aren't really written with stuff like @safe in mind.
> 
> One big problem is the way people talk and write about this issue. There is a difference between "dereferencing" in the language, and reading from a memory address by the CPU.

In general, I always consider "dereferencing" the point at which code follows a pointer to read or write its data. The semantics of modifying the type to mean the data vs. the pointer to it, seems less interesting. Types are compiler internal things, the actual reads and writes are what cause the problems.

But really, it's the act of using a pointer to read/write the data it points at which causes the segfault. And in D, we assume that this action is @safe because of the MMU protecting the first page.

> Confusing language semantics with what the CPU is doing happens often in the D community and is not helping these debates.
> 
> D is proclaiming that dereferencing `null` must segfault but that is not implemented by any of the compilers. It would require inserting null checks upon every dereference. (This may not be as slow as you may think, but it would probably not make code run faster.)
> 
> An example:
> ```
> class A {
>      int i;
>      final void foo() {
>           import std.stdio; writeln(__LINE__);
>          // i = 5;
>      }
> }
> 
> void main() {
>      A a;
>      a.foo();
> }
> ```
> 
> In this case, the actual null dereference happens on the last line of main. The program runs fine however since dlang 2.077.

Right, the point is that the segfault happens when null pointers are used to get at the data. If you turn something that is ultimately a pointer into another type of pointer, then you aren't dereferencing it really. This happens when you pass *pointer into a function that takes a reference (or when you pass around a class reference).

In any case, the prior versions to 2.077 didn't segfault, they just had a prelude in front of every function which asserted that this wasn't null (you actually get a nice stack trace).

> Now when `foo` is modified such that it writes to member field `i`, the program does segfault (writes to address 0).
> D does not make dereferencing on class objects explicit, which makes it harder to see where the dereference is happening.

Again, the terms are confusing. You just said the dereference happens at a.foo(), right? I would consider the dereference to happen when the object's data is used. i.e. when you read or write what the pointer points at.

> 
> So, I think all compiler implementations are not spec compliant on this point.

I think if the spec says that dereferencing doesn't mean following a pointer to it's data, and reading/writing that data, and it says null dereferences cause a segfault, then the spec needs to be updated. The @safe segfault is what it should be focused on, not some abstract concept that exists only in the compiler.

If it means changing the terminology, then we should do that.

> I think most people believe that compliance is too costly for the kind of software one wants to write in D; the issue is similar to array bounds checking that people explicitly disable or work around.
> For compliance we would need to change the compiler to emit null checks on all @safe dereferences (the opposite direction was chosen in 2.077). It'd be interesting to do the experiment.

The whole point of using the MMU instead of instrumentation is because we can avoid the performance penalties and still be safe. The only loophole is large structures that may extend beyond the protected data. I would suggest that the compiler inject extra reads of the front of any data type in that case (when @safe is enabled) to cause a segfault properly.

-Steve

On Tuesday, 20 November 2018 at 15:29:50 UTC, Kagamin wrote:
> On Tuesday, 20 November 2018 at 11:11:43 UTC, aliak wrote:
>> This only applies to little scripts and unittests maybe.
>>
>> Not when you're writing any kind of relatively larger application that involves being run for longer or if there's more possible permutations of your state variables.
>
> Umm... if you write a larger application not knowing what is a reference type, you're into lots and lots of problems.

I’m not sure I understood your point? I was saying that you don’t necessarily hit your null dereference within a couple of seconds.

> Try to learn D.
> Put writeln in deconstructor to prove it works as expected
> Make random changes, program never runs again.
> Takes 30+ minutes to realize that writeln("my string") is fine, but writeln("my string " ~ value) is an allocation / garbage collection which crashes the program without a stack.

My favorite D'ism so far:

On Tuesday, 20 November 2018 at 19:11:46 UTC, Steven Schveighoffer wrote:
> On 11/20/18 1:04 PM, Johan Engelen wrote:
>>
>> D does not make dereferencing on class objects explicit, which makes it harder to see where the dereference is happening.
>
> Again, the terms are confusing. You just said the dereference happens at a.foo(), right? I would consider the dereference to happen when the object's data is used. i.e. when you read or write what the pointer points at.

But `a.foo()` is already using the object's data: it is accessing a function of the object and calling it. Whether it is a virtual function, or a final function, that shouldn't matter. There are different ways of implementing class function calls, but here often people seem to pin things down to one specific way. I feel I stand alone in the D community in treating the language in this abstract sense (like C and C++ do, other languages I don't know). It's similar to that people think that local variables and the function return address are put on a stack; even though that is just an implementation detail that is free to be changed (and does often change: local variables are regularly _not_ stored on the stack [*]).

Optimization isn't allowed to change behavior of a program, yet already simple dead-code-elimination would when null dereference is not treated as UB or when it is not guarded by a null check. Here is an example of code that also does what you call a "dereference" (read object data member):
```
class A {
    int i;
    final void foo() {
        int a = i; // no crash with -O
    }
}

void main() {
    A a;
    a.foo();  // dereference happens
}
```

When you don't call `a.foo()` a dereference, you basically say that `this` is allowed to be `null` inside a class member function. (and then it'd have to be normal to do `if (this) ...` inside class member functions...)

These discussions are hard to do on a mailinglist, so I'll stop here. Until next time at DConf, I suppose... ;-)

-Johan

[*] intentionally didn't say where those local variables _are_ stored, so that people can solve that little puzzle for themselves ;-)

On Tuesday, 20 November 2018 at 19:11:46 UTC, Steven Schveighoffer wrote:
>
> But really, it's the act of using a pointer to read/write the data it points at which causes the segfault. And in D, we assume that this action is @safe because of the MMU protecting the first page.
>

This is like me saying I won't bother locking up when I leave the house, cause if the alarm goes off the security company will come around and take care of things anyway.

But by then, it's too late.


>> D is proclaiming that dereferencing `null` must segfault but that is not implemented by any of the compilers. It would require inserting null checks upon every dereference. (This may not be as slow as you may think, but it would probably not make code run faster.)

Aristotle would have immediately solved this dilemma.

Null is a valid value for reference types.
Dereferencing null can lead to bad things happening.
Therefore, check for null before dereferencing a reference type.

Problem solved.

On Tue, 20 Nov 2018 23:14:27 +0000, Johan Engelen wrote:
> When you don't call `a.foo()` a dereference, you basically say that
> `this` is allowed to be `null` inside a class member function. (and then
> it'd have to be normal to do `if (this) ...` inside class member
> functions...)

That's what we have today:

    module scratch;
    import std.stdio;
    class A
    {
        int i;
        final void f()
        {
            writeln(this is null);
            writeln(i);
        }
    }
    void main()
    {
        A a;
        a.f();
    }

This prints `true` and then gets a segfault.

Virtual function calls have to do a dereference to figure out which potentially overrided function to call.

On Tuesday, November 20, 2018 8:38:40 AM MST Kagamin via Digitalmars-d-learn wrote:
> On Monday, 19 November 2018 at 21:23:31 UTC, Jordi Gutiérrez
>
> Hermoso wrote:
> > When I was first playing with D, I managed to create a segfault by doing `SomeClass c;` and then trying do something with the object I thought I had default-created, by analogy with C++ syntax.
>
> D is more similar to Java here and works like languages with reference types - Java, C#, Python, Ruby, JavaScript. Also AFAIK in C++ objects are garbage-created by default, so you would have a similar problem there. To diagnose crashes on linux you can run your program under gdb.

In C++, if the class is put directly on the stack, then you get a similar situation to D's structs, only instead of it being default-initialized, it's default-constructed. So, you don't normally get garbage when you just declare a variable of a class type (though you do with other types, and IIRC, if a class doesn't have a user-defined default constructor, and a member variable's type doesn't have a default constructor, then that member variable does end up being garbage).

However, if you declare a pointer to a class (which is really more analagous to what you're doing when declaring a class reference in D), then it's most definitely garbage, and the behavior is usually _far_ worse than segfaulting. So, while I can see someone getting annoyed about a segfault, because they forgot to initialize a class reference in D, the end result is far, far safer than what C++ does. And in most cases, you catch the bug pretty fast, because pretty much the only way that you don't catch it is if that piece of code is never tested. So, while D's approach is by no means perfect, I don't think that there's really any question that as far as memory safety goes, it's far superior to C++.

- Jonathan M Davis

On Tuesday, November 20, 2018 11:04:08 AM MST Johan Engelen via Digitalmars- d-learn wrote:
> On Tuesday, 20 November 2018 at 03:38:14 UTC, Jonathan M Davis
>
> wrote:
> > For @safe to function properly, dereferencing null _must_ be guaranteed to be memory safe, and for dmd it is, since it will always segfault. Unfortunately, as understand it, it is currently possible with ldc's optimizer to run into trouble, since it'll do things like see that something must be null and therefore assume that it must never be dereferenced, since it would clearly be wrong to dereference it. And then when the code hits a point where it _does_ try to dereference it, you get undefined behavior. It's something that needs to be fixed in ldc, but based on discussions I had with Johan at dconf this year about the issue, I suspect that the spec is going to have to be updated to be very clear on how dereferencing null has to be handled before the ldc guys do anything about it. As long as the optimizer doesn't get involved everything is fine, but as great as optimizers can be at making code faster, they aren't really written with stuff like @safe in mind.
>
> One big problem is the way people talk and write about this issue. There is a difference between "dereferencing" in the language, and reading from a memory address by the CPU. Confusing language semantics with what the CPU is doing happens often in the D community and is not helping these debates.
>
> D is proclaiming that dereferencing `null` must segfault but that is not implemented by any of the compilers. It would require inserting null checks upon every dereference. (This may not be as slow as you may think, but it would probably not make code run faster.)
>
> An example:
> ```
> class A {
>      int i;
>      final void foo() {
>           import std.stdio; writeln(__LINE__);
>          // i = 5;
>      }
> }
>
> void main() {
>      A a;
>      a.foo();
> }
> ```
>
> In this case, the actual null dereference happens on the last
> line of main. The program runs fine however since dlang 2.077.
> Now when `foo` is modified such that it writes to member field
> `i`, the program does segfault (writes to address 0).
> D does not make dereferencing on class objects explicit, which
> makes it harder to see where the dereference is happening.

Yeah. It's one of those areas where the spec will need to be clear. Like C++, D doesn't actually dereference unless it needs to. And IMHO, that's fine. The core issue is that operations that aren't memory safe can't be allowed to happen in @safe code, and the spec needs to be defined in such a way that requires that that be true, though not necessarily by being super specific about every detail about how a compiler is required to do it.

> So, I think all compiler implementations are not spec compliant
> on this point.
> I think most people believe that compliance is too costly for the
> kind of software one wants to write in D; the issue is similar to
> array bounds checking that people explicitly disable or work
> around.
> For compliance we would need to change the compiler to emit null
> checks on all @safe dereferences (the opposite direction was
> chosen in 2.077). It'd be interesting to do the experiment.

Ultimately here, the key thing is that it must be guaranteed that dereferencing null is @safe in @safe code (regardless of whether that involves * or . and regardless of how that is achieved). It must never read from or write to invalid memory. If it can, then dereferencing a null pointer or class reference is not memory safe, and since there's no way to know whether a pointer or class reference is null or not via the type system, dereferencing pointers and references in general would then be @system, and that simply can't be the case, or @safe is completely broken.

Typically, that protection is done right now via segfaults, but we know that that's not always possible. For instance, if the object is large enough (larger than one page size IIRC), then attempting to dereference a null pointer won't necessarily segfault. It can actually end up accessing invalid memory if you try to access a member variable that's deep enough in the object. I know that in that particular case, Walter's answer to the problem is that such objects should be illegal in @safe code, but AFAIK, neither the compiler nor the spec have yet been updated to match that decision, which needs to be fixed. But regardless, in any and all cases where we determine that a segfault won't necessarily protect against accessing invalid memory when a null pointer or reference is dereferenced, then we need to do _something_ to guarantee that that code is @safe - which probably means adding additional null checks in most cases, though in the case of the overly large object, Walter has a different solution.

IMHO, requiring something in the spec like "it must segfault when dereferencing null" as has been suggested before is probably not a good idea is really getting too specific (especially considering that some folks have argued that not all architectures segfault like x86 does), but ultimately, the question needs to be discussed with Walter. I did briefly discuss it with him at this last dconf, but I don't recall exactly what he had to say about the ldc optimization stuff. I _think_ that he was hoping that there was a way to tell the optimizer to just not do that kind of optimization, but I don't remember for sure. Ultimately, the two of you will probably have to discuss it. Either way, I know that he wanted a bugzilla issue on the topic, but I keep forgetting about it. First, I need to at least dig through the spec to figure out what it actually says right now, which probably isn't much.

- Jonathan M Davis