Robert Fraser wrote:
> Walter Bright wrote:
>> My focus is on eliminating bugs that cannot be reliably detected even at run time. This will be a big win for D.
> FWIW, I've _never_ run into a bug const could have prevented.

That isn't really the point of const. The point of const is to be able to write functions that can accommodate both mutable and invariant arguments. The point of invariantness is to be able to prove that code has certain properties. This is much better than relying on your programming team never making a mistake.

For example, you can do functional programming in C++. It's just that the compiler cannot know you're doing that, and so cannot take advantage of it. Furthermore, the compiler cannot detect when code is not functional, and so if someone hands you a million lines of code you have no freakin' way of determining if it adheres to functional principles or not.

This really matters when one starts doing concurrent programming.

Can someone explain what is the plan for when Tango turns 1.0

Will the code be frozen ?

Will the version of D it runs with be frozen ?


cheers
Nick B

On 2008-11-05 03:18:50 -0500, Walter Bright <newshound1@digitalmars.com> said:

> I don't see what you've gained here. The compiler certainly can do flow analysis in some cases to know that a pointer isn't null, but that isn't generalizable. If a function takes a pointer parameter, no flow analysis will tell you if it is null or not.

I'm not sure how you're reading things, but to me having two kinds of pointers (nullable and non-nullable) is exactly what you need to enable nullness flow analysis across function boundaries.

Basically, if you declare some pointer to be restricted to not-null in a function signature, and then try to call the function by passing it a possibly null pointer, the compiler can tell you that you need to check for null at the call site before calling the function.

It then ensue that when given a non-nullable pointer you can call a function requiring a non-nullable pointer directly without any check for null, because you know the pointer you recieved can't be null.

Currently, you can acheive this with proper documentation of functions saying whether arguments accept null and if return values can return null, and write your code with those assumptions in mind. Most often than not however there is no such documentation and you find yourself checking for null a lot more than necessary. If this property about pointers in function parameters and return values were known to the compiler, the compiler could check for you that you're doing things correctly, warn you whenever you're forgetting a null check, and optimise away checks for null on these pointers.

I know the null-dereferencing problem can generally be caught easily at runtime, but sometime your null comes from far away in the program (someone set a global to null for instance) and you're left to wonder who put a null value there in the first place. Non-nullable pointers would help a lot in those cases because you no longer have to test every code path and the error of giving a null value would be caught at the source (with the compiler telling you to check against null), not only where it's being dereferenced.

- - -

That said, I think this could be done using an template. Consider this:

	struct NotNullPtr(Type) {
		private Type* ptr;
		this(Type* ptr) {
			opAssign(ptr);
		}
		void opAssign(Type* ptr) {
			// if this gets inlined and you have already checked for null, then
			// hopefully the optimizer will remove this redundant check.
			if (ptr)
				this.ptr = ptr;
			else
				throw new Exception("Unacceptable null value.");
		}
		void opAssign(NotNullPtr other) {
			this.ptr = other.ptr;
		}
		Type* opCast() {
			return ptr;
		}
		ref Type opDeref() {
			return &ptr;
		}
		alias opDeref opStar;
		// ... implement the rest yourself
	}

(not tested)

You could use this template everywhere you want to be sure a pointer isn't null. It guarenties that its value will never be null, and will throw an exception at the source where you attempt to put a null value in it, not when you attempt to dereference it later, when it's too late and your program has already been put in an incorrect state.

	NotNullPtr!(int) globalThatShouldNotBeNull;

	int* foo();
	globalThatShouldBeNull = foo(); // will throw if you attempt to set it to null.

	void bar(NotNullPtr!(int) arg);
	bar(globalThatShouldNotBeNull); // no need to check for null.

The greatest downside to this template is that since it isn't part of the language, almost no one will use it in their function prototypes and return types. That's not counting that its syntax is verbose and not very appealing (although it's not much worse than boost::shared_ptr or std::auto_ptr).

But still, if you have a global or member variable that must not be null, it can be of use; and if you have a function where you want to put the burden of checking for null on the caller, it can be of use.

-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

Michel Fortin:
> Basically, if you declare some pointer to be restricted to not-null in a function signature, and then try to call the function by passing it a possibly null pointer, the compiler can tell you that you need to check for null at the call site before calling the function.
> 
> It then ensue that when given a non-nullable pointer you can call a function requiring a non-nullable pointer directly without any check for null, because you know the pointer you recieved can't be null.
> 
> Currently, you can acheive this with proper documentation of functions saying whether arguments accept null and if return values can return null, and write your code with those assumptions in mind. Most often than not however there is no such documentation and you find yourself checking for null a lot more than necessary. If this property about pointers in function parameters and return values were known to the compiler, the compiler could check for you that you're doing things correctly, warn you whenever you're forgetting a null check, and optimise away checks for null on these pointers.

The same is true making integral values become range values. If I want to write a function that takes an iterable of results of throwing a dice, I can use an enum, or control every item of the iterable for being in range 1 - 6. If range values are available I can just:

StatsResults stats(Dice[] throwing_results) { ...

Where Dice is:
typedef int:1..7 Dice;

I then don't need to remember to control items for being in 1-6 inside stats(), and the control is pushed up, toward the place where that throwing_results was created (or where it comes from disk, user input, etc). This avoids some bugs and reduces some code.

Bye,
bearophile

On Wed, Nov 5, 2008 at 3:18 AM, Walter Bright <newshound1@digitalmars.com> wrote:
> Jarrett Billingsley wrote:
>>
>> The implication of non-nullable types isn't that nullable types disappear; quite the opposite, in fact.  Nullable types have obvious use for exactly the reason you explain.  The problem arises when nullable types are used in situations where it makes _no sense_ for null to appear.  This is where bugs show up.  In a system that has both nullable and non-null types, nullable types act as a sort of container, preventing you from accessing anything through them as it cannot be statically proven that the access will be legal at runtime. In order to access something from a nullable type, you have to convert it to a non-null type.  Delight uses D's "declare a variable in the condition of an if or while" to great effect here:
>>
>> if(auto f = someFuncThatReturnsNullableFoo()) // f is declared as non-null
>> {
>>    // f is known not to be null.
>> }
>> else
>> {
>>    // something else happened.  Handle it.
>> }
>
> I don't see what you've gained here. The compiler certainly can do flow analysis in some cases to know that a pointer isn't null, but that isn't generalizable. If a function takes a pointer parameter, no flow analysis will tell you if it is null or not.
>

What?  Is your response in response to my post at all?  I am not talking about flow analysis on "normal" pointer types.  I am talking about the typing system actually being modified to allow a programmer to express the idea, with a _type_, and not with static checking, that a reference/pointer value _may not be null_.

In a type system with non-null types, if a function takes a non-null parameter and you pass it a nullable pointer, _you get an error at compile time_.

// foo takes a non-null int*.
void foo(int* x) { writefln("%s", *x); }

// bar returns a nullable int* - an int*?.
int*? bar(int x) { if(x < 10) return new int(x); else return null; }

foo(bar(3)); // compiler error, you can't pass a potentially null type
into a parameter that can't be null, moron

if(auto p = bar(3))
    foo(p); // ok
else
    throw new Exception("Wah wah wah bar returned null");

With nullable types, flow analysis doesn't have to be done.  It is implicit in the types.  It is mangled into function names.  foo _cannot_ take a pointer that may be null.  End of story.

On Wed, Nov 5, 2008 at 10:43 PM, Jarrett Billingsley <jarrett.billingsley@gmail.com> wrote:
> On Wed, Nov 5, 2008 at 3:18 AM, Walter Bright <newshound1@digitalmars.com> wrote:
>> Jarrett Billingsley wrote:
>>> cannot be statically proven that the access will be legal at runtime. In order to access something from a nullable type, you have to convert it to a non-null type.  Delight uses D's "declare a variable in the condition of an if or while" to great effect here:
>>>
>>> if(auto f = someFuncThatReturnsNullableFoo()) // f is declared as non-null
>>> {
>>>    // f is known not to be null.
>>> }
>>> else
>>> {
>>>    // something else happened.  Handle it.
>>> }
>>
>> I don't see what you've gained here. The compiler certainly can do flow analysis in some cases to know that a pointer isn't null, but that isn't generalizable. If a function takes a pointer parameter, no flow analysis will tell you if it is null or not.
>>
>
> What?  Is your response in response to my post at all?  I am not talking about flow analysis on "normal" pointer types.  I am talking about the typing system actually being modified to allow a programmer to express the idea, with a _type_, and not with static checking, that a reference/pointer value _may not be null_.
>
> In a type system with non-null types, if a function takes a non-null parameter and you pass it a nullable pointer, _you get an error at compile time_.
>
> // foo takes a non-null int*.
> void foo(int* x) { writefln("%s", *x); }
>
> // bar returns a nullable int* - an int*?.
> int*? bar(int x) { if(x < 10) return new int(x); else return null; }
>
> foo(bar(3)); // compiler error, you can't pass a potentially null type
> into a parameter that can't be null, moron
>
> if(auto p = bar(3))
>    foo(p); // ok
> else
>    throw new Exception("Wah wah wah bar returned null");
>
> With nullable types, flow analysis doesn't have to be done.  It is implicit in the types.  It is mangled into function names.  foo _cannot_ take a pointer that may be null.  End of story.

I didn't really get what you meant the first time either.  The thing about Delight's use of auto "to great effect" wasn't clear.   I assumed it was basically the same as D's auto inside an if, but I see now that it's not.  Looks like a run-time type deduction, even though its not really.  Kinda neat.

--bb

"Walter Bright" wrote
> Jarrett Billingsley wrote:
>> Don't you think that eliminating something that's
>> always a bug at compile time is a worthwhile investment?
>
> Not always. There's a commensurate increase in complexity that may not make it worth while.
>
> My focus is on eliminating bugs that cannot be reliably detected even at run time. This will be a big win for D.

I was working in C# today, and I realized one very excellent design advantage for D -- the array.  In C#, if null, an array is subject to null dereference errors.  In D, it simply doesn't happen, because the array has a guard that is stored with the reference -- the length.  I think these similar to the kinds of things that Jarrett is referring to.  Something that's like a pointer, but can't ever be null, so you never have to check it for null before using it.  Except Jarrett's idea eliminates it at compile time vs. run time.

Couldn't one design a struct wrapper that implements this behavior? Something like:

NonNullable!(T)
{
   opAssign(T t) {/* make sure t is not null */}
   opAssign(NonNullable!(T) t) {/* no null check */}
   ...
}

-Steve

On Wed, Nov 5, 2008 at 9:33 AM, Bill Baxter <wbaxter@gmail.com> wrote:
> I didn't really get what you meant the first time either.  The thing about Delight's use of auto "to great effect" wasn't clear.   I assumed it was basically the same as D's auto inside an if, but I see now that it's not.  Looks like a run-time type deduction, even though its not really.  Kinda neat.

It's almost the same as D's variable-inside-an-if, with the addition that you can use it to convert a nullable type to a non-null type. Hence, in:

if(auto f = someFunctionThatCanReturnNull())

if someFunctionThatCanReturnNull returns an int*? (nullable pointer to
int), typeof(f) will just be int* (non-null pointer to int), since in
the scope of the if statement, f is provably non-null.

Jarrett Billingsley wrote:
> With nullable types, flow analysis doesn't have to be done.  It is
> implicit in the types.  It is mangled into function names.  foo
> _cannot_ take a pointer that may be null.  End of story.

Sure, which is why I was puzzled at the example given, which is about something else entirely.

What you're talking about is a type constructor to create another kind of pointer. It's a significant increase in complexity. That's why I was talking about complexity being a downside of this - there is a tradeoff.

Steven Schveighoffer wrote:
> Couldn't one design a struct wrapper that implements this behavior? 

If that cannot be done in D, then D needs some design improvements. Essentially, any type should be "wrappable" in a struct which can alter the behavior of the wrapped type.

For example, you should also be able to create a ranged int that can only contain values from n to m:

RangedInt!(N, M) i;

Preserving this property of structs has driven many design choices in D, particularly with regards to how const fits into the type system.