I don't know where the community currently stands on non-nullable types in D, so this idea may be based on a bit of ignorance. Assuming there are some technical issues preventing non-nullable types from being implemented, I had a different idea that may be somewhat of a compromise. As you've gathered by now, it's simply to disallow nullifying references in safe code.

The idea is simply that safe functions can only call other safe functions, so null references should be practically non-existant ... except that's an ideal which can't be reached with this restriction alone. There are two obvious issues:

* There's no way to guarantee input is free of null references
* Trusted functions may return objects with null references; it's currently not convention to avoid null references in trusted code

Albeit that, I think such a restriction could be helpful in preventing bugs/crashes and writing correct code, at least until we can get non-nullable types.

On Saturday, 1 February 2014 at 01:14:07 UTC, Xinok wrote:
> I don't know where the community currently stands on non-nullable types in D, so this idea may be based on a bit of ignorance.

I've written a couple candidates:

http://arsdnet.net/dcode/notnull.d

and

http://arsdnet.net/dcode/notnullsimplified.d

The second one is a much simpler implementation that I think covers the same bases. Gotta try it all in real world code though..

On Saturday, February 01, 2014 01:14:05 Xinok wrote:
> I don't know where the community currently stands on non-nullable types in D, so this idea may be based on a bit of ignorance. Assuming there are some technical issues preventing non-nullable types from being implemented, I had a different idea that may be somewhat of a compromise. As you've gathered by now, it's simply to disallow nullifying references in safe code.
> 
> The idea is simply that safe functions can only call other safe functions, so null references should be practically non-existant ... except that's an ideal which can't be reached with this restriction alone. There are two obvious issues:
> 
> * There's no way to guarantee input is free of null references
> * Trusted functions may return objects with null references; it's
> currently not convention to avoid null references in trusted code
> 
> Albeit that, I think such a restriction could be helpful in preventing bugs/crashes and writing correct code, at least until we can get non-nullable types.

There's nothing unsafe about null pointers/references. @safe is about memory safety, and you can't corrupt memory and otherwise access memory that you're not supposed to with a null pointer or reference.

At some point here, we'll have NonNullable (or NotNull whatever it ends up being called) in Phobos so that folks can have non-nullable references/pointers - e.g. NonNullable!Foo. AFAIK, the only real hold-up is someone completely a fully functional implementation. There's been at least one attempt at it, but as I understand it, there were issues that needed to be worked through before it could be accepted. We'll get there though.

Regardless, we're not adding anything with regards to non-nullable references to the language itself, and there's nothing unsafe about null references. They're just unpleasant to dereference when your code makes that mistake.

- Jonathan M Davis

On Saturday, 1 February 2014 at 01:39:46 UTC, Jonathan M Davis wrote:
> On Saturday, February 01, 2014 01:14:05 Xinok wrote:
>> I don't know where the community currently stands on non-nullable
>> types in D, so this idea may be based on a bit of ignorance.
>> Assuming there are some technical issues preventing non-nullable
>> types from being implemented, I had a different idea that may be
>> somewhat of a compromise. As you've gathered by now, it's simply
>> to disallow nullifying references in safe code.
>> 
>> The idea is simply that safe functions can only call other safe
>> functions, so null references should be practically non-existant
>> ... except that's an ideal which can't be reached with this
>> restriction alone. There are two obvious issues:
>> 
>> * There's no way to guarantee input is free of null references
>> * Trusted functions may return objects with null references; it's
>> currently not convention to avoid null references in trusted code
>> 
>> Albeit that, I think such a restriction could be helpful in
>> preventing bugs/crashes and writing correct code, at least until
>> we can get non-nullable types.
>
> There's nothing unsafe about null pointers/references. @safe is about memory
> safety, and you can't corrupt memory and otherwise access memory that you're
> not supposed to with a null pointer or reference.
>
> At some point here, we'll have NonNullable (or NotNull whatever it ends up
> being called) in Phobos so that folks can have non-nullable
> references/pointers - e.g. NonNullable!Foo. AFAIK, the only real hold-up is
> someone completely a fully functional implementation. There's been at least
> one attempt at it, but as I understand it, there were issues that needed to be
> worked through before it could be accepted. We'll get there though.
>
> Regardless, we're not adding anything with regards to non-nullable references
> to the language itself, and there's nothing unsafe about null references.
> They're just unpleasant to dereference when your code makes that mistake.
>
> - Jonathan M Davis

Dereferencing it is unsafe unless you put runtime check. Which is stupid for something that can be verified at compile time.

On 1/31/14, 5:14 PM, Xinok wrote:
> I don't know where the community currently stands on non-nullable types
> in D, so this idea may be based on a bit of ignorance. Assuming there
> are some technical issues preventing non-nullable types from being
> implemented, I had a different idea that may be somewhat of a
> compromise. As you've gathered by now, it's simply to disallow
> nullifying references in safe code.
>
> The idea is simply that safe functions can only call other safe
> functions, so null references should be practically non-existant ...
> except that's an ideal which can't be reached with this restriction
> alone. There are two obvious issues:
>
> * There's no way to guarantee input is free of null references
> * Trusted functions may return objects with null references; it's
> currently not convention to avoid null references in trusted code
>
> Albeit that, I think such a restriction could be helpful in preventing
> bugs/crashes and writing correct code, at least until we can get
> non-nullable types.

It's an interesting idea, but I don't think it would work well for the reasons others mentioned.

Andrei

On 1/31/14, 5:39 PM, Jonathan M Davis wrote:
> Regardless, we're not adding anything with regards to non-nullable references
> to the language itself [...]

I think the sea is changing here. I've collected hard data that reveals null pointer dereference is a top problem for at least certain important categories of applications. Upon discussing that data with Walter, it became clear that no amount of belief to the contrary, personal anecdote, or rhetoric can stand against the data.

It also became clear that a library solution would improve things but cannot compete with a language solution. The latter can do local flow-sensitive inference and require notations only for function signatures. Consider:

class Widget { ... }

void fun()
{
    // assume fetchWidget() may return null
    Widget w = fetchWidget();
    if (w)
    {
        ... here w is automatically inferred as non-null ...
    }
}

Bottom line: a language change for non-null references is on the table. It will be an important focus of 2014.


Andrei

On Saturday, February 01, 2014 04:01:50 deadalnix wrote:
> > There's nothing unsafe about null pointers/references. @safe is
> > about memory
> > safety, and you can't corrupt memory and otherwise access
> > memory that you're
> > not supposed to with a null pointer or reference.
> > 
> > At some point here, we'll have NonNullable (or NotNull whatever
> > it ends up
> > being called) in Phobos so that folks can have non-nullable
> > references/pointers - e.g. NonNullable!Foo. AFAIK, the only
> > real hold-up is
> > someone completely a fully functional implementation. There's
> > been at least
> > one attempt at it, but as I understand it, there were issues
> > that needed to be
> > worked through before it could be accepted. We'll get there
> > though.
> > 
> > Regardless, we're not adding anything with regards to
> > non-nullable references
> > to the language itself, and there's nothing unsafe about null
> > references.
> > They're just unpleasant to dereference when your code makes
> > that mistake.
> > 
> > - Jonathan M Davis
> 
> Dereferencing it is unsafe unless you put runtime check.

How is it unsafe? It will segfault and kill your program, not corrupt memory. It can't even read any memory. It's a bug to dereference a null pointer or reference, but it's not unsafe, because it can't access _any_ memory, let alone memory that it's not supposed to be accessing, which is precisely what @safe is all about.

>  Which is stupid for something that can be verified at compile time.

In the general case, you can only catch it at compile time if you disallow it completely, which is unnecessarily restrictive. Sure, some basic cases can be caught, but unless the code where the pointer/reference is defined is right next to the code where it's dereferenced, there's no way for the compiler to have any clue whether it's null or not. And yes, there's certainly code where it would make sense to use non-nullable references or pointers, because there's no need for them to be nullable, and having them be non-nullable avoids any risk of forgetting to initialize them, but that doesn't mean that nullable pointers and references aren't useful or that you can catch all instances of a null pointer or reference being dereferenced at compile time.

- Jonathan M Davis

On 2/1/14, 2:14 AM, Jonathan M Davis wrote:
> On Saturday, February 01, 2014 04:01:50 deadalnix wrote:
>> Dereferencing it is unsafe unless you put runtime check.
>
> How is it unsafe? It will segfault and kill your program, not corrupt memory.
> It can't even read any memory. It's a bug to dereference a null pointer or
> reference, but it's not unsafe, because it can't access _any_ memory, let
> alone memory that it's not supposed to be accessing, which is precisely what
> @safe is all about.

This has been discussed to death a number of times. A field access obj.field will use addressing with a constant offset. If that offset is larger than the lowest address allowed to the application, unsafety may occur.

The amount of low-address memory protected is OS-dependent. 4KB can virtually always be counted on. For fields placed beyond than that limit, a runtime test must be inserted. There are few enough 4KB objects out there to make this practically a non-issue. But the checks must be there.

>>   Which is stupid for something that can be verified at compile time.
>
> In the general case, you can only catch it at compile time if you disallow it
> completely, which is unnecessarily restrictive. Sure, some basic cases can be
> caught, but unless the code where the pointer/reference is defined is right
> next to the code where it's dereferenced, there's no way for the compiler to
> have any clue whether it's null or not. And yes, there's certainly code where
> it would make sense to use non-nullable references or pointers, because
> there's no need for them to be nullable, and having them be non-nullable
> avoids any risk of forgetting to initialize them, but that doesn't mean that
> nullable pointers and references aren't useful or that you can catch all
> instances of a null pointer or reference being dereferenced at compile time.

The Java community has a good experience with @Nullable: http://stackoverflow.com/questions/14076296/nullable-annotation-usage


Andrei

On 2/1/2014 12:09 PM, Andrei Alexandrescu wrote:
> On 2/1/14, 2:14 AM, Jonathan M Davis wrote:
>> How is it unsafe? It will segfault and kill your program, not corrupt memory.
>> It can't even read any memory. It's a bug to dereference a null pointer or
>> reference, but it's not unsafe, because it can't access _any_ memory, let
>> alone memory that it's not supposed to be accessing, which is precisely what
>> @safe is all about.
>
> This has been discussed to death a number of times. A field access obj.field
> will use addressing with a constant offset. If that offset is larger than the
> lowest address allowed to the application, unsafety may occur.
>
> The amount of low-address memory protected is OS-dependent. 4KB can virtually
> always be counted on. For fields placed beyond than that limit, a runtime test
> must be inserted. There are few enough 4KB objects out there to make this
> practically a non-issue. But the checks must be there.

Another way to deal with it is to simply disallow @safe objects that are larger than 4K (or whatever the size is on the target system).

On 2/1/14, 1:11 PM, Walter Bright wrote:
> On 2/1/2014 12:09 PM, Andrei Alexandrescu wrote:
>> On 2/1/14, 2:14 AM, Jonathan M Davis wrote:
>>> How is it unsafe? It will segfault and kill your program, not corrupt
>>> memory.
>>> It can't even read any memory. It's a bug to dereference a null
>>> pointer or
>>> reference, but it's not unsafe, because it can't access _any_ memory,
>>> let
>>> alone memory that it's not supposed to be accessing, which is
>>> precisely what
>>> @safe is all about.
>>
>> This has been discussed to death a number of times. A field access
>> obj.field
>> will use addressing with a constant offset. If that offset is larger
>> than the
>> lowest address allowed to the application, unsafety may occur.
>>
>> The amount of low-address memory protected is OS-dependent. 4KB can
>> virtually
>> always be counted on. For fields placed beyond than that limit, a
>> runtime test
>> must be inserted. There are few enough 4KB objects out there to make this
>> practically a non-issue. But the checks must be there.
>
> Another way to deal with it is to simply disallow @safe objects that are
> larger than 4K (or whatever the size is on the target system).

This seems like an arbitrary limitation, and one that's hard to work around without significant code surgery. I think the true solution is runtime checks for fields beyond the 4K barrier. They will be few and far between and performance-conscious coders already know well to lay out all hot data at the beginning of the object.

Andrei