Jon Grant wrote:
> Hi
> Just having a look at the D language.
> Does D still let the programmer allocate memory, cast addresses and read/write
> direct address space as we can from C/C++?
> 
> I'd like to know if it solves this problem, Java and C# don't allow such access.

An important point is that this doesn't solve this problem either. Ok, you get an exception instead of some "undefined behaviour", which is somehow better when debugging, but:

It just doen't solve the problem that you have a bug in your program in the first place. Reread that sentence.

Getting an unexpected exception in a shipped application is a huge bug. In fact it can be:

*) a security risc: the program is taking a path of execution the
programmer haven't thought of.
*) a security risc2: some data might be in a undefined state after the
exception
*) a data risc: the program might be in an undefined state. Saving now,
might save garbage.

[As you might have noticed, these are exactly the problem you have with C, C++ and D]

And, even worse, not only that Java doesn't help you getting the exception handlers right in the code, it doesn't help you avoiding exceptions in the first place (it getting better with the template types (don't know how they call it)).

Sebastian Biallas wrote:
> Steve Horne wrote:
>> In a systems language, pointer arithmetic is a necessity. There are
>> things you simply cannot do without it - data structures that require
>> it, for instance.
> 
> Can you give an example?

http://trac.brainsware.org/ocb/browser/trunk/src/arch/x86/idt.c

This is not D, yet, but it's pointer arithmetic. You need to do this, because you have to split up the address and write lower and higher half to two different places in the structure of the descriptor. (x86 backwards compatibility, yay!)

Best regards,
Alex

Sebastian Biallas wrote:
> Jon Grant wrote:
>> Hi
>> Just having a look at the D language.
>> Does D still let the programmer allocate memory, cast addresses and read/write
>> direct address space as we can from C/C++?
>>
>> I'd like to know if it solves this problem, Java and C# don't allow such access.
> 
> An important point is that this doesn't solve this problem either. Ok,
> you get an exception instead of some "undefined behaviour", which is
> somehow better when debugging, but:
> 
> It just doen't solve the problem that you have a bug in your program in
> the first place. Reread that sentence.

Exceptions can be handled at runtime, still. try (to) catch it and there you go. :)

> 
> Getting an unexpected exception in a shipped application is a huge bug.

That's why you always /test/ everything so good, that you just don't get unexpected exceptions.

> In fact it can be:
> 
> *) a security risc: the program is taking a path of execution the
> programmer haven't thought of.
> *) a security risc2: some data might be in a undefined state after the
> exception
> *) a data risc: the program might be in an undefined state. Saving now,
> might save garbage.
> 
> [As you might have noticed, these are exactly the problem you have with
> C, C++ and D]
> 
> And, even worse, not only that Java doesn't help you getting the
> exception handlers right in the code, it doesn't help you avoiding
> exceptions in the first place (it getting better with the template types
> (don't know how they call it)).

What's the huge problem with exception handlers? They worked for me pretty well so far to avoid a unhandled exceptions that just terminate the program.

Alexander Panek wrote:
> Sebastian Biallas wrote:
>> Jon Grant wrote:
>>> Hi
>>> Just having a look at the D language.
>>> Does D still let the programmer allocate memory, cast addresses and
>>> read/write
>>> direct address space as we can from C/C++?
>>>
>>> I'd like to know if it solves this problem, Java and C# don't allow such access.
>>
>> An important point is that this doesn't solve this problem either. Ok, you get an exception instead of some "undefined behaviour", which is somehow better when debugging, but:
>>
>> It just doen't solve the problem that you have a bug in your program in the first place. Reread that sentence.
> 
> Exceptions can be handled at runtime, still. try (to) catch it and there
> you go. :)

So, how exactly do you /handle/ an exception which "should not be there"?

>> Getting an unexpected exception in a shipped application is a huge bug.
> 
> That's why you always /test/ everything so good, that you just don't get unexpected exceptions.

Well, so you shouldn't need exceptions at all for array handling, right? (Except for debugging, where they really help)

> What's the huge problem with exception handlers?

There is nothing wrong which exception handlers for handling exceptions that the programmer thought of (file-not-found, io-error, etc).

But things like ArrayOutOfBounds, NullPointer and CastErrorException are bugs in the program (usually). Often they get thrown at places where there's no appropriate catch around or transaction logic so they get caught by some high level "catch everything".

So what to do now? The program is now in an unstable state.
And even if you catch the Exception on the right place: The Exception
should never have been thrown. There is a bug, and the bug is not
necessarily at the place where the exception was thrown -- the program
might have been in an inconsistent state for hours.

> They worked for me
> pretty well so far to avoid a unhandled exceptions that just terminate
> the program.

It would be even better if the compiler could prove at compiler-time, that there are no OutOfBound, NullPointer and CastExceptions possible.

While this is hard for ArrayOutOfBounds, there are quite easy ways for avoiding NullPointer and CastExceptions at all. In the language Nice[1] they do this as follows:

NullPointer: They have different types for pointers that can be null
(?Type) and pointer that can't (Type). You can't dereference a ?Type
variable.

So, let's say we have

void foo(?Type t)
{
	// we want to dereference t
	if (t != null) {
		// now the compiler "magically" converts t to type Type
		t.doSomething();
	}
}

TypeCasts: You don't cast with a cast-operator, but with a instance-of-expression:

void bar(A a)
{
	if (a instanceOf B) {
		// now a is of type B, no need to cast
	}
}

I haven't used this language yet, but these are certainly, well, nice features :)

[1] http://nice.sf.net

Sebastian Biallas wrote:
>>> [...]
>>> It just doen't solve the problem that you have a bug in your program in
>>> the first place. Reread that sentence.
>> Exceptions can be handled at runtime, still. try (to) catch it and there
>> you go. :)
> 
> So, how exactly do you /handle/ an exception which "should not be there"?

One has to make sure to catch all "should not be there" exceptions, too (ideally). In the real world, of course, there can happen to be mistakes in the code, hidden bugs and what not else. But the fact is, that in most software, that is really fully tested, those happen very rarely and can be handled with special routines - maybe also automatically sending a bug report to the developers or the IT department of a company. This depends on what kind of software you write/use(libraries). Apart from that, D has this neat scope(fail) thing, that will cover almost all circumstances, except segmentation faults .. ;)

> 
>>> Getting an unexpected exception in a shipped application is a huge bug.
>> That's why you always /test/ everything so good, that you just don't get
>> unexpected exceptions.
> 
> Well, so you shouldn't need exceptions at all for array handling, right?
> (Except for debugging, where they really help)

As soon as the amount of /dynamic/ data you have to handle grows, I don't think there's a way around using at least exceptions to stay safe.

> 
>> What's the huge problem with exception handlers? 
> 
> There is nothing wrong which exception handlers for handling exceptions
> that the programmer thought of (file-not-found, io-error, etc).
> 
> But things like ArrayOutOfBounds, NullPointer and CastErrorException are
> bugs in the program (usually). Often they get thrown at places where
> there's no appropriate catch around or transaction logic so they get
> caught by some high level "catch everything".
> 
> So what to do now? The program is now in an unstable state.
> And even if you catch the Exception on the right place: The Exception
> should never have been thrown. There is a bug, and the bug is not
> necessarily at the place where the exception was thrown -- the program
> might have been in an inconsistent state for hours.

This is, of course, not so good. But then again, you should design the software in a way that avoids such circumstances (yes, this is not possible in any cases, but in very much).

> 
>> They worked for me
>> pretty well so far to avoid a unhandled exceptions that just terminate
>> the program.
> 
> It would be even better if the compiler could prove at compiler-time,
> that there are no OutOfBound, NullPointer and CastExceptions possible.

AFAIK, D handles this as long as it's not dynamic data. As soon as array indexes, pointer arithmetic or other not so exotic things kick in, there's no way to determine what values/states there can be. One way could be, to use a debugger that reflects the whole application execution (there is such a debugger for Java, that records literally everything).

> 
> While this is hard for ArrayOutOfBounds, there are quite easy ways for
> avoiding NullPointer and CastExceptions at all. In the language Nice[1]
> they do this as follows:
> 
> NullPointer: They have different types for pointers that can be null
> (?Type) and pointer that can't (Type). You can't dereference a ?Type
> variable.
> 
> So, let's say we have
> 
> void foo(?Type t)
> {
> 	// we want to dereference t
> 	if (t != null) {
> 		// now the compiler "magically" converts t to type Type
> 		t.doSomething();
> 	}
> }

How about:

void foo ( Type t )
in {
    assert( t != null );
}
body {
    t.doSomething();
}

I find this rather equal.

> 
> TypeCasts: You don't cast with a cast-operator, but with a
> instance-of-expression:
> 
> void bar(A a)
> {
> 	if (a instanceOf B) {
> 		// now a is of type B, no need to cast
> 	}
> }

What exactly does an instance-of-expression? Never seen this, actually.

> 
> I haven't used this language yet, but these are certainly, well, nice
> features :)
> 
> [1] http://nice.sf.net

Best regards,
Alex

Alexander Panek schrieb:
> Sebastian Biallas wrote:
>>>> [...]
>>>> It just doen't solve the problem that you have a bug in your program in
>>>> the first place. Reread that sentence.
>>> Exceptions can be handled at runtime, still. try (to) catch it and there
>>> you go. :)
>>
>> So, how exactly do you /handle/ an exception which "should not be there"?
> 
> One has to make sure to catch all "should not be there" exceptions, too (ideally). In the real world, of course, there can happen to be mistakes in the code, hidden bugs and what not else. But the fact is, that in most software, that is really fully tested, those happen very rarely and can be handled with special routines - 

By what routines?

> maybe also automatically sending a bug report to the developers or the IT department of a company. 

You can do this with C code, too (see talkback from mozilla)

> This depends on what kind of software you write/use(libraries). Apart from that, D has this neat scope(fail) thing, that will cover almost all circumstances, except segmentation faults .. ;)

scope(xx) is nice, that's right.

>>>> Getting an unexpected exception in a shipped application is a huge bug.
>>> That's why you always /test/ everything so good, that you just don't get
>>> unexpected exceptions.
>>
>> Well, so you shouldn't need exceptions at all for array handling, right?
>> (Except for debugging, where they really help)
> 
> As soon as the amount of /dynamic/ data you have to handle grows, I don't think there's a way around using at least exceptions to stay safe.

I'm not sure I understand that sentence.

Maybe I should clearify my point:
Exceptions are a nice way to handle unpredictable situations at runtime, like an IO error. Java helps you even with this, when you use the throw() specifier for your io-functions.

But exceptions are IMHO a wrong way to handle /programming mistakes/. Programming mistakes must be fixed by a programmer -- they can't be fixed at runtime. It's just too late if a buggy program is running.

>> So what to do now? The program is now in an unstable state.
>> And even if you catch the Exception on the right place: The Exception
>> should never have been thrown. There is a bug, and the bug is not
>> necessarily at the place where the exception was thrown -- the program
>> might have been in an inconsistent state for hours.
> 
> This is, of course, not so good. But then again, you should design the software in a way that avoids such circumstances (yes, this is not possible in any cases, but in very much).

This is pretty much the answer you get from someone advocating C or C++ if you ask him why a C or C++ program behaves in such an unpredictable way ;)

>>> They worked for me
>>> pretty well so far to avoid a unhandled exceptions that just terminate
>>> the program.
>>
>> It would be even better if the compiler could prove at compiler-time,
>> that there are no OutOfBound, NullPointer and CastExceptions possible.
> 
> AFAIK, D handles this as long as it's not dynamic data. As soon as array indexes, pointer arithmetic or other not so exotic things kick in, there's no way to determine what values/states there can be. 

You need a careful designed language for this. See e.g. SPARK (I haven't used it yet).

>>
>> While this is hard for ArrayOutOfBounds, there are quite easy ways for
>> avoiding NullPointer and CastExceptions at all. In the language Nice[1]
>> they do this as follows:
>>
>> NullPointer: They have different types for pointers that can be null
>> (?Type) and pointer that can't (Type). You can't dereference a ?Type
>> variable.
>>
>> So, let's say we have
>>
>> void foo(?Type t)
>> {
>>     // we want to dereference t
>>     if (t != null) {
>>         // now the compiler "magically" converts t to type Type
>>         t.doSomething();
>>     }
>> }
> 
> How about:
> 
> void foo ( Type t )
> in {
>     assert( t != null );
> }
> body {
>     t.doSomething();
> }
> 
> I find this rather equal.

It is rather equal. The difference is not in the source code, the difference is how you /can/ write this.

In Nice you just have this one possibility: If you want to dereference a ?pointer, you have to test it against null -- otherwise it won't compile. In other languages this test is optional (and even whether a pointer can be null is not part of the language but of the documentation. And documentation is often not available or wrong).

Of course you can write correct code in D and other languages. The point is: In Nice the code is guaranteed to be correct regarding null-pointer checks once it compiles! There is no need of a "style guide" for this which will be ignored once the project reaches its deadline.

>> TypeCasts: You don't cast with a cast-operator, but with a
>> instance-of-expression:
>>
>> void bar(A a)
>> {
>>     if (a instanceOf B) {
>>         // now a is of type B, no need to cast
>>     }
>> }
> 
> What exactly does an instance-of-expression? Never seen this, actually.

	if (a instanceOf B) {

is the same as

	if (dynamic_cast<B>(a)) {  // in C++

or

	if (cast(B) a) { // in D

Sebastian Biallas wrote:
> Alexander Panek schrieb:
>> Sebastian Biallas wrote:
>>>>> [...]
>>>>> It just doen't solve the problem that you have a bug in your program in
>>>>> the first place. Reread that sentence.
>>>> Exceptions can be handled at runtime, still. try (to) catch it and there
>>>> you go. :)
>>>
>>> So, how exactly do you /handle/ an exception which "should not be there"?
>>
>> One has to make sure to catch all "should not be there" exceptions, too (ideally). In the real world, of course, there can happen to be mistakes in the code, hidden bugs and what not else. But the fact is, that in most software, that is really fully tested, those happen very rarely and can be handled with special routines - 
> 
> By what routines?

The bug report possibility was one example.

> 
>> maybe also automatically sending a bug report to the developers or the IT department of a company. 
> [...]
>>>>> Getting an unexpected exception in a shipped application is a huge bug.
>>>> That's why you always /test/ everything so good, that you just don't get
>>>> unexpected exceptions.
>>>
>>> Well, so you shouldn't need exceptions at all for array handling, right?
>>> (Except for debugging, where they really help)
>>
>> As soon as the amount of /dynamic/ data you have to handle grows, I don't think there's a way around using at least exceptions to stay safe.
> 
> I'm not sure I understand that sentence.

I'm just saying that you have to use proper exception handling, to stay error safe (not bug safe), as soon as (dynamic) data of variable length and kind is to be handled.

> 
> Maybe I should clearify my point:
> Exceptions are a nice way to handle unpredictable situations at runtime, like an IO error. Java helps you even with this, when you use the throw() specifier for your io-functions.

Exactly, that's what I mean.

> 
> But exceptions are IMHO a wrong way to handle /programming mistakes/. Programming mistakes must be fixed by a programmer -- they can't be fixed at runtime. It's just too late if a buggy program is running.

Of course. But a language only may /help/ to avoid programming mistakes. I don't think it's feasable to really restrict pointer usage like in the examples you brought up. But that's just me, I love pointers and what you can mess up with them. :)

>>> So what to do now? The program is now in an unstable state.
>>> And even if you catch the Exception on the right place: The Exception
>>> should never have been thrown. There is a bug, and the bug is not
>>> necessarily at the place where the exception was thrown -- the program
>>> might have been in an inconsistent state for hours.
>>
>> This is, of course, not so good. But then again, you should design the software in a way that avoids such circumstances (yes, this is not possible in any cases, but in very much).
> 
> This is pretty much the answer you get from someone advocating C or C++ if you ask him why a C or C++ program behaves in such an unpredictable way ;)

Aww, got me there. ;)

> 
>>>> They worked for me
>>>> pretty well so far to avoid a unhandled exceptions that just terminate
>>>> the program.
>>>
>>> It would be even better if the compiler could prove at compiler-time,
>>> that there are no OutOfBound, NullPointer and CastExceptions possible.
>>
>> AFAIK, D handles this as long as it's not dynamic data. As soon as array indexes, pointer arithmetic or other not so exotic things kick in, there's no way to determine what values/states there can be. 
> 
> You need a careful designed language for this. See e.g. SPARK (I haven't used it yet).

Where 'careful' is the thing that splits the programmers community in two.. (safe/unsafe and such things). I don't really know SPARK, but I think D does a good job /already/, regarding 'safety'.

>>> While this is hard for ArrayOutOfBounds, there are quite easy ways for
>>> avoiding NullPointer and CastExceptions at all. In the language Nice[1]
>>> they do this as follows:
>>>
>>> NullPointer: They have different types for pointers that can be null
>>> (?Type) and pointer that can't (Type). You can't dereference a ?Type
>>> variable.
>>>
>>> So, let's say we have
>>>
>>> void foo(?Type t)
>>> {
>>>     // we want to dereference t
>>>     if (t != null) {
>>>         // now the compiler "magically" converts t to type Type
>>>         t.doSomething();
>>>     }
>>> }
>>
>> How about:
>>
>> void foo ( Type t )
>> in {
>>     assert( t != null );
>> }
>> body {
>>     t.doSomething();
>> }
>>
>> I find this rather equal.
> 
> It is rather equal. The difference is not in the source code, the difference is how you /can/ write this.
> 
> In Nice you just have this one possibility: If you want to dereference a ?pointer, you have to test it against null -- otherwise it won't compile. In other languages this test is optional (and even whether a pointer can be null is not part of the language but of the documentation. And documentation is often not available or wrong).

Hm. Then again, why not make it automatically: every pointer that is dereferenced gets a comparison to null before being used (imho better than explicitely typing if(t==null) all the time).

> 
> Of course you can write correct code in D and other languages. The point is: In Nice the code is guaranteed to be correct regarding null-pointer checks once it compiles! There is no need of a "style guide" for this which will be ignored once the project reaches its deadline.
> 
>>> TypeCasts: You don't cast with a cast-operator, but with a
>>> instance-of-expression:
>>>
>>> void bar(A a)
>>> {
>>>     if (a instanceOf B) {
>>>         // now a is of type B, no need to cast
>>>     }
>>> }
>>
>> What exactly does an instance-of-expression? Never seen this, actually.
> 
>     if (a instanceOf B) {
> 
> is the same as
> 
>     if (dynamic_cast<B>(a)) {  // in C++
> 
> or
> 
>     if (cast(B) a) { // in D

Mh, well. That's a matter of taste, I'd say. (IIRC, it was even proposed somewhen?)

On Fri, 12 Jan 2007 04:02:50 +0100, Sebastian Biallas <groups.5.sepp@spamgourmet.com> wrote:

>Steve Horne wrote:
>> In a systems language, pointer arithmetic is a necessity. There are things you simply cannot do without it - data structures that require it, for instance.
>
>Can you give an example?

A particular favorite of mine allows doubly-linked list behaviour using a single link per item - very useful if you need extremely large lists of small items (though a custom allocator is essential). Each link is actually the bitwise exclusive-or of the addresses of the previous and next items. This may sound impossible to iterate, but if the iterator contains two pointers pointing to two adjacent items, it is actually quite easy to step in either direction. Stepping always requires a bitwise exclusive-or of a pointer, and inserts and deletes require multiple bitwise exclusive-ors on pointers.

In addition, pointer casting is useful for managing template bloat. The basic approach is to write type-unsafe non-template code, and then use a template to add a typesafe wrapper. But in many cases, more flexibility is needed than can be handled using simple pointer casting.

For example, consider a library to handle flexible tree data structures. The container being implemented may or may not have a key field. It may or may not have a data field. It may or may not have subtree summary data stored within nodes. The non-template part needs a way to access fields such that the position within a node struct (of an unknown datatype) is determined at run-time.

The most efficient way to handle this depends on what kind of operation is being implemented. A single data structure may have two or more ways to access the same field, just to ensure that all operations can be handled efficiently - many accesses will probably be done as part of small functions supplied by the typesafe wrapper, to keep the call overheads down in inner loops. But for many accesses, the best approach is using the offset of a field relative to the start of a node - a relative address. The non-template code gets the relative addresses it needs from a data block provided by the typesafe wrapper, which does a similar job to that done by virtual tables in method calls.

-- 
Remove 'wants' and 'nospam' from e-mail.

On Fri, 12 Jan 2007 09:48:50 +0100, Alexander Panek <a.panek@brainsware.org> wrote:

>> An important point is that this doesn't solve this problem either. Ok, you get an exception instead of some "undefined behaviour", which is somehow better when debugging, but:
>> 
>> It just doen't solve the problem that you have a bug in your program in the first place. Reread that sentence.
>
>Exceptions can be handled at runtime, still. try (to) catch it and there you go. :)

There is a big difference between using exceptions to handle user errors, machine limits and the like compared with handling program bugs.

Put it this way - if your code generates exceptions due to bugs, how exactly can you trust the exception handlers to do any better? If your program is in an unknown illegal state because of an unknown bug, how can your exception handler convert that back to a legal state?

Exceptions are meant to handle exceptional cases cleanly - hence the name. We tend to think of exceptions as errors, but that isn't necessarily the case, and besides, the meaning of error in this context does not include bugs in the code.

Trying to hide bugs behind a catch-all exception handler tends to mean the application stays in an illegal, unstable state that leads to more errors. Instead of exiting at the first sign of trouble with a minimum of data loss, the effects of that error can snowball - files get corrupted with bad data etc etc. Things that seem minor along these lines can actually be the worst cases - an accumulation of minor corruptions can go unnoticed until the point where all existing backups are corrupted, and suddenly the "clever" hiding of bugs has led to bad publicity and paying out for damages.

Exceptions are only better than "undefined behaviour" when debugging since you get clues to how the error arose that allow you to diagnose and fix the bug more easily (providing it is the kind of error that results in an exception). For instance, the debugger will stop on unhandled exceptions, and show you where the exception throw happened, what was on the call stack, etc.

Even then, so much can happen between the initial error and the point where it is detected and an exception thrown that you still have a serious detective job to do. And if you have a bug-hiding catch-all exception handler, there just means that even more will have happened between the initial error and its detection and the detective job will be many times more difficult.

Catch-all exception handlers can be useful for generating diagnostic data, but using them to hide bugs is bad programming.

-- 
Remove 'wants' and 'nospam' from e-mail.

Steve Horne wrote:
> On Fri, 12 Jan 2007 04:02:50 +0100, Sebastian Biallas <groups.5.sepp@spamgourmet.com> wrote:
> 
>> Steve Horne wrote:
>>> In a systems language, pointer arithmetic is a necessity. There are things you simply cannot do without it - data structures that require it, for instance.
>> Can you give an example?
> 
> A particular favorite of mine allows doubly-linked list behaviour using a single link per item - very useful if you need extremely large lists of small items (though a custom allocator is essential).

I know of these lists. But they don't seem to be particulary useful:

a) Where do you need extremely large lists? Lists have a search time in
O(n). Ok, you have a O(1) insertion/deletion here, but only at one
place, since your iterators get void after an insertion/deletion
(contrary to normal double linked lists)
b) They don't work in a GC-enviroment or require special handling.

Are they really used somewhere out of the IOCCC? I'd be curious to see a real world example.

> In addition, pointer casting is useful for managing template bloat. The basic approach is to write type-unsafe non-template code, and then use a template to add a typesafe wrapper.

This works with normal pointer casting (no need for arithmetik). Hey, this was even the normal usage of Java-Containers before they had the template stuff.

> But in many cases, more
> flexibility is needed than can be handled using simple pointer
> casting.
> 
> For example, consider a library to handle flexible tree data structures. The container being implemented may or may not have a key field. It may or may not have a data field. It may or may not have subtree summary data stored within nodes. The non-template part needs a way to access fields such that the position within a node struct (of an unknown datatype) is determined at run-time.

I don't think it's a good idea to trade portability for a few cycles. And this might be even slower, since you confuse the compiler (alias-analysis etc).