On 11/1/2014 4:14 AM, Kagamin wrote:
> On Wednesday, 29 October 2014 at 21:23:00 UTC, Walter Bright wrote:
>> In any case, if the programmer knows than assert error is restricted to a
>> particular domain, and is recoverable, and wants to recover from it, use
>> enforce(), not assert().
>
> But all that does is working around the assert's behavior to ignore cleanups.

It is not "working around" anything unless you're trying to use a screwdriver as a hammer. Cleanups are not appropriate after a program has entered an unknown state.



> Maybe, when it's known, that a failure is not restricted, some different way of
> failure reporting should be used?

assert() and enforce() both work as designed.

On 10/10/2014 2:31 AM, Joseph Rushton Wakeling wrote:
> I still think that was one of the single most important lessons in probability
> that I ever had.

Research shows that humans, even trained statisticians, are spectacularly bad at intuitive probability.

-- "Thinking, Fast and Slow" by Daniel Kahneman

On Saturday, 1 November 2014 at 15:02:53 UTC, H. S. Teoh via Digitalmars-d wrote:
> I never said "component" == "process". All I said was that at the OS
> level, at least with current OSes, processes are the smallest unit
> that is decoupled from each other.

Which is exactly the statement I call wrong. With current OSes processes aren't decoupled units at all - it is all about feature set you stick to. Same with any other units.

> If you go below that level of
> granularity, you have the possibility of shared memory being corrupted
> by one thread (or fibre, or whatever smaller than a process) affecting
> the other threads.

You already have that possibility at process level via shared process memory and kernel mode code. And you still don't have that possibility at thread/fiber level if you don't use mutable shared memory (or any global state in general). It is all about system design.

Pretty much only reliably decoupled units I can imagine are processes running in different restricted virtual machines (or, better, different physical machines). Everything else gives just certain level of expectations.

Walter has experience with certain types of systems where process is indeed most appropriate unit of granularity and calls that a silver bullet by explicitly designing language in a way that makes any other approach inherently complicated and effort-consuming. But there is more than that in software world.

On 11/2/2014 3:48 AM, Dicebot wrote:
> On Saturday, 1 November 2014 at 15:02:53 UTC, H. S. Teoh via Digitalmars-d wrote:
> Which is exactly the statement I call wrong. With current OSes processes aren't
> decoupled units at all - it is all about feature set you stick to. Same with any
> other units.

They have hardware protection against sharing memory between processes. It's a reasonable level of protection.


>> If you go below that level of
>> granularity, you have the possibility of shared memory being corrupted
>> by one thread (or fibre, or whatever smaller than a process) affecting
>> the other threads.
> You already have that possibility at process level via shared process memory

1. very few processes use shared memory
2. those that do should regard it as input/environmental, and not trust it


> and kernel mode code.

Kernel mode code is the responsibility of the OS system, not the app.


> And you still don't have that possibility at thread/fiber
> level if you don't use mutable shared memory (or any global state in general).

A buffer overflow will render all that protection useless.


> It is all about system design.

It's about the probability of coupling and the level of that your system can stand. Process level protection is adequate for most things.


> Pretty much only reliably decoupled units I can imagine are processes running in
> different restricted virtual machines (or, better, different physical machines).
> Everything else gives just certain level of expectations.

Everything is coupled at some level. Again, it's about the level of reliability needed.


> Walter has experience with certain types of systems where process is indeed most
> appropriate unit of granularity and calls that a silver bullet by explicitly
> designing language

I design the language to do what it can. A language cannot compensate for coupling and bugs in the operating system, nor can a language compensate for two machines being plugged into the same power circuit.


> in a way that makes any other approach inherently complicated
> and effort-consuming.

Using enforce is neither complicated nor effort consuming.

The idea that asserts can be recovered from is fundamentally unsound, and makes D unusable for robust critical software. Asserts are for checking for programming bugs. A bug can be tripped because of a buffer overflow, memory corruption, a malicious code injection attack, etc.

NO CODE CAN BE RELIABLY EXECUTED PAST THIS POINT.

Running arbitrary cleanup code at this point is literally undefined behavior. This is not a failure of language design - no language can offer any guarantees about this.

If you want code cleanup to happen, use enforce(). If you are using enforce() to detect programming bugs, well, that's your choice. enforce() isn't any more complicated or effort-consuming than using assert().

On Sunday, 2 November 2014 at 17:53:45 UTC, Walter Bright wrote:
> On 11/2/2014 3:48 AM, Dicebot wrote:
>> On Saturday, 1 November 2014 at 15:02:53 UTC, H. S. Teoh via Digitalmars-d wrote:
>> Which is exactly the statement I call wrong. With current OSes processes aren't
>> decoupled units at all - it is all about feature set you stick to. Same with any
>> other units.
>
> They have hardware protection against sharing memory between processes. It's a reasonable level of protection.

reasonable default - yes
reasoable level of protection in general - no

> 1. very few processes use shared memory
> 2. those that do should regard it as input/environmental, and not trust it

This is no different from:

1. very few threads use shared
2. those that do should regard is as input/environmental

>> and kernel mode code.
>
> Kernel mode code is the responsibility of the OS system, not the app.

In some (many?) large scale server systems OS is the app or at least heavily integrated. Thinking about app as a single independent user-space process is a bit.. outdated.

>> And you still don't have that possibility at thread/fiber
>> level if you don't use mutable shared memory (or any global state in general).
>
> A buffer overflow will render all that protection useless.

Nice we have @safe and default thread-local memory!

>> It is all about system design.
>
> It's about the probability of coupling and the level of that your system can stand. Process level protection is adequate for most things.

Again, I am fine with advocating it as a resonable default. What frustrates me is intentionally making any other design harder than it should be by explicitly allowing normal cleanup to be skipped. This behaviour is easy to achieve by installing custom assert handler (it could be generic Error handler too) but impossible to bail out when it is the default one.

Because of abovementioned avoiding more corruption from cleanup does not sound to me as strong enough benefit to force that on everyone. Ironically in system with decent fault protection and safety redundance it won't even matter (everything it can possibly corrupt is duplicated and proof-checked anyway)

>> Walter has experience with certain types of systems where process is indeed most
>> appropriate unit of granularity and calls that a silver bullet by explicitly
>> designing language
>
> I design the language to do what it can. A language cannot compensate for coupling and bugs in the operating system, nor can a language compensate for two machines being plugged into the same power circuit.

I don't expect you to do magic. My blame is about making decisions that support designs you have great expertise with but hamper something different (but still very real) - decisions that are usually uncharacteristic in D (which I believe is non-opinionated language) and don't really belong to system programming language.

>> in a way that makes any other approach inherently complicated
>> and effort-consuming.
>
> Using enforce is neither complicated nor effort consuming.

> If you want code cleanup to happen, use enforce(). If you are using enforce() to detect programming bugs, well, that's your choice. enforce() isn't any more complicated or effort-consuming than using assert().

I don't have other choice and I don't like it. It is effort consuming because it requires manually maintained exception hierarchy and style rules to keep errors different from exceptions - something that language otherwise provides to you out of the box. And there is always that 3d party library that is hard-coded to throw Error.

It is not something that I realistically expect to change in D and there are specific plans for working with it (thanks for helping with it btw!). Just mentioning it as one of few D design decisions I find rather broken conceptually.

> The idea that asserts can be recovered from is fundamentally unsound, and makes D unusable for robust critical software.

Not "recovered" but "terminate user-defined portion of the system".

> Asserts are for checking for programming bugs. A bug can be tripped because of a buffer overflow, memory corruption, a malicious code injection attack, etc.
>
> NO CODE CAN BE RELIABLY EXECUTED PAST THIS POINT.

As I have already mentioned it almost never can be truly reliable. You simply call one higher reliability chance good enough and other lower one - disastrous. I don't agree this is the language call to make, even if decision is reasonable and fitting 90% cases.

This is really no different than GC usage in Phobos before @nogc push. If language decision may result in fundamental code base fragmentation (even for relatively small portion of users), it is likely to be overly opinionated decision.

> Running arbitrary cleanup code at this point is literally undefined behavior. This is not a failure of language design - no language can offer any guarantees about this.

Some small chance of undefined behaviour vs 100% chance of resource leaks?
Former can be more practical in many cases. And if it isn't for specific application one can always install custom assert handler that kills program right away. Don't see a deal breaker here.

On 11/2/2014 3:44 PM, Dicebot wrote:
>> They have hardware protection against sharing memory between processes. It's a
>> reasonable level of protection.
> reasonable default - yes
> reasoable level of protection in general - no

No language can help when that is the requirement.


>> 1. very few processes use shared memory
>> 2. those that do should regard it as input/environmental, and not trust it
>
> This is no different from:
>
> 1. very few threads use shared
> 2. those that do should regard is as input/environmental

It is absolutely different because of scale; having 1K of shared memory is very different from having 100Mb shared between processes including the stack and program code.


>>> and kernel mode code.
>>
>> Kernel mode code is the responsibility of the OS system, not the app.
>
> In some (many?) large scale server systems OS is the app or at least heavily
> integrated. Thinking about app as a single independent user-space process is a
> bit.. outdated.

Haha, I've used such a system (MSDOS) for many years. Switching to process protection was a huge advance. Sad that we're "modernizing" by reverting to such an awful programming environment.


>>> And you still don't have that possibility at thread/fiber
>>> level if you don't use mutable shared memory (or any global state in general).
>>
>> A buffer overflow will render all that protection useless.
>
> Nice we have @safe and default thread-local memory!

Assert is to catch program bugs that should never happen, not correctly functioning programs. Nor can D possibly guarantee that called C functions are safe.


>>> It is all about system design.
>>
>> It's about the probability of coupling and the level of that your system can
>> stand. Process level protection is adequate for most things.
>
> Again, I am fine with advocating it as a resonable default. What frustrates me
> is intentionally making any other design harder than it should be by explicitly
> allowing normal cleanup to be skipped. This behaviour is easy to achieve by
> installing custom assert handler (it could be generic Error handler too) but
> impossible to bail out when it is the default one.

Running normal cleanup code when the program is in an undefined, possibly corrupted, state can impede proper shutdown.


> Because of abovementioned avoiding more corruption from cleanup does not sound
> to me as strong enough benefit to force that on everyone.

I have considerable experience with what programs can do when continuing to run after a bug. This was on real mode DOS, which infamously does not seg fault on errors.

It's AWFUL. I've had quite enough of having to reboot the operating system after every failure, and even then that often wasn't enough because it might scramble the disk driver code so it won't even boot.

I got into the habit of layering in asserts to stop the program when it went bad. "Do not pass go, do not collect $200" is the only strategy that has a hope of working under such systems.


> I don't expect you to do magic. My blame is about making decisions that support
> designs you have great expertise with but hamper something different (but still
> very real) - decisions that are usually uncharacteristic in D (which I believe
> is non-opinionated language) and don't really belong to system programming
> language.

It is my duty to explain how to use the features of the language correctly, including how and why they work the way they do. The how, why, and best practices are not part of a language specification.


> I don't have other choice and I don't like it. It is effort consuming because it
> requires manually maintained exception hierarchy and style rules to keep errors
> different from exceptions - something that language otherwise provides to you
> out of the box. And there is always that 3d party library that is hard-coded to
> throw Error.
>
> It is not something that I realistically expect to change in D and there are
> specific plans for working with it (thanks for helping with it btw!). Just
> mentioning it as one of few D design decisions I find rather broken conceptually.

I hope to eventually change your mind about it being broken.


>> NO CODE CAN BE RELIABLY EXECUTED PAST THIS POINT.
> As I have already mentioned it almost never can be truly reliable.

That's correct, but not a justification for making it less reliable.


> You simply
> call one higher reliability chance good enough and other lower one - disastrous.
> I don't agree this is the language call to make, even if decision is reasonable
> and fitting 90% cases.

If D changes assert() to do unwinding, then D will become unusable for building reliable systems until I add in yet another form of assert() that does not.


> This is really no different than GC usage in Phobos before @nogc push. If
> language decision may result in fundamental code base fragmentation (even for
> relatively small portion of users), it is likely to be overly opinionated decision.

The reason I initiated this thread is to point out the correct way to use assert() and to get that into the culture of best practices for D. This is because if I don't, then in the vacuum people will tend to fill that vacuum with misunderstandings and misuse.

It is an extremely important topic.


>> Running arbitrary cleanup code at this point is literally undefined behavior.
>> This is not a failure of language design - no language can offer any
>> guarantees about this.
> Some small chance of undefined behaviour vs 100% chance of resource leaks?

If the operating system can't handle resource recovery for a process terminating, it is an unusable operating system.

On Monday, 3 November 2014 at 03:29:05 UTC, Walter Bright wrote:
>
> I have considerable experience with what programs can do when continuing to run after a bug. This was on real mode DOS, which infamously does not seg fault on errors.
>
> It's AWFUL. I've had quite enough of having to reboot the operating system after every failure, and even then that often wasn't enough because it might scramble the disk driver code so it won't even boot.

Yep.  Fun stuff.

http://research.microsoft.com/en-us/people/mickens/thenightwatch.pdf


> I got into the habit of layering in asserts to stop the program when it went bad. "Do not pass go, do not collect $200" is the only strategy that has a hope of working under such systems.

The tough thing is that in D, contracts serve as this early warning system, but the errors this system catches are also sometimes benign and sometimes the programmer knows that they're benign, but if the person writing the test did so with an assert then the decision of how to respond to the error has been made for him.


> It is my duty to explain how to use the features of the language correctly, including how and why they work the way they do. The how, why, and best practices are not part of a language specification.

I don't entirely agree.  The standard library serves as the how and how and best practices for a language, and while a programmer can go against this, it's often like swimming upstream.  For better or worse, we need to establish how parameters are validated and such in Phobos, and this will serve as the template for nearly all code written in D.

The core design goal of Druntime is to make the default behavior as safe as possible, but to allow to discerning user to override this behavior in certain key places.  I kind of see the entire D language like this--it has the power of C/C++ but the ease of use of a much higher level language.  We should strive for all aspects of the language and standard library to have the same basic behavior: a safe, efficient default but the ability to customize in key areas to meet individual needs.  The trick is determining what these key areas are and how much latitude the user should have.


> If D changes assert() to do unwinding, then D will become unusable for building reliable systems until I add in yet another form of assert() that does not.

To be fair, assert currently does unwinding.  It always has.  The proposal is that it should not.


> The reason I initiated this thread is to point out the correct way to use assert() and to get that into the culture of best practices for D. This is because if I don't, then in the vacuum people will tend to fill that vacuum with misunderstandings and misuse.
>
> It is an extremely important topic.

I still feel like there's something really important here that we're all grasping at but it hasn't quite come to the fore yet.  Along the lines of the idea that a @safe program may be able to recover from a logic error.  It seems like a uniquely D thing insofar as systems languages are concerned.


> If the operating system can't handle resource recovery for a process terminating, it is an unusable operating system.

There are all kinds of resources, and not all of them are local to the system.  Everything will eventually recover though, it just won't happen immediately as is the case with resource cleanup within a process.

On 11/2/2014 8:54 PM, Sean Kelly wrote:
> On Monday, 3 November 2014 at 03:29:05 UTC, Walter Bright wrote:
>> I got into the habit of layering in asserts to stop the program when it went
>> bad. "Do not pass go, do not collect $200" is the only strategy that has a
>> hope of working under such systems.
> The tough thing is that in D, contracts serve as this early warning system, but
> the errors this system catches are also sometimes benign

If it is benign is not known until it is debugged by a human.

> and sometimes the
> programmer knows that they're benign, but if the person writing the test did so
> with an assert then the decision of how to respond to the error has been made
> for him.

The person who wrote "assert" decided that it was a non-recoverable programming bug. I deliberately wrote "bug", and not "error".


>> It is my duty to explain how to use the features of the language correctly,
>> including how and why they work the way they do. The how, why, and best
>> practices are not part of a language specification.
> I don't entirely agree.  The standard library serves as the how and how and best
> practices for a language, and while a programmer can go against this, it's often
> like swimming upstream.  For better or worse, we need to establish how
> parameters are validated and such in Phobos, and this will serve as the template
> for nearly all code written in D.

Definitely Phobos should exhibit best practices. Whether bad function argument values are input/environmental errors or bugs is decidable only on a case-by-case basis. There is no overarching rule.

Input/environmental errors must not use assert to detect them.


> To be fair, assert currently does unwinding.  It always has.  The proposal is
> that it should not.

Not entirely - a function with only asserts in it is considered "nothrow" and callers may not have exception handlers for them.


>> The reason I initiated this thread is to point out the correct way to use
>> assert() and to get that into the culture of best practices for D. This is
>> because if I don't, then in the vacuum people will tend to fill that vacuum
>> with misunderstandings and misuse.
>>
>> It is an extremely important topic.
>
> I still feel like there's something really important here that we're all
> grasping at but it hasn't quite come to the fore yet. Along the lines of the
> idea that a @safe program may be able to recover from a logic error.  It seems
> like a uniquely D thing insofar as systems languages are concerned.

It's a false hope. D cannot offer any guarantees of recovery from programming bugs. Asserts, by definition, can never happen. So when they do, something is broken. Broken programs are not recoverable because one cannot know why they broke until they are debugged. As I mentioned to Dicebot, @safe only applies to the function's logic. D programs can call C functions. C functions are not safe. There can be compiler bugs. There can be other threads corrupting memory. There can be hardware failures, operating system bugs, etc., that resulting in tripping the assert.

If a programmer "knows" a bug is benign and wants to recover from it, D has a mechanism for it - enforce(). I do not understand the desire to bash assert() into behaving like enforce(). Just use enforce() in the first place.

The idea was brought up that one may be using a library that uses assert() to detect input/environmental errors. I do not understand using a library in a system that must be made robust, having the source code to the library, and not being allowed to change that source code to fix bugs in it. A robust application cannot be made using such a library - assert() misuse will not be the only problem with it.


>> If the operating system can't handle resource recovery for a process
>> terminating, it is an unusable operating system.
> There are all kinds of resources, and not all of them are local to the system.
> Everything will eventually recover though, it just won't happen immediately as
> is the case with resource cleanup within a process.

I'd say that is a poor design for an operating system. Be that as it may, if you want to recover from assert()s, use enforce() instead.


There are other consequences from trying to make assert() recoverable:

1. functions with assert()s cannot be "nothrow"
2. assert()s cannot provide hints to the optimizer

Those are high prices to pay for a systems performance language.

On 29/10/2014 21:22, Walter Bright wrote:
> On 10/29/2014 5:37 AM, Bruno Medeiros wrote:
>> On 18/10/2014 18:40, Walter Bright wrote:
>>> As I've said before, tripping an assert by definition means the program
>>> has entered an unknown state. I don't believe it is possible for any
>>> language to make guarantees beyond that point.
>>
>> The guarantees (if any), would not be made by the language, but by the
>> programmer. The language cannot know if a program is totally broken and
>> undefined when an assert fails, but a programmer can, for each particular
>> assert, make some assumptions about which fault domains (like Sean put
>> it) can
>> be affected and which are not.
>
> Assumptions are not guarantees.
>

Let me give an example:

double sqrt(double num) {
  assert(num >= 0);
  ...

With just this, then purely from a compiler/language viewpoint, if the assert is triggered the *language* doesn't know if the whole program is corrupted (formatting the hard disk, etc.), or if the fault is localized there, and an error/exception can be thrown cleanly (clean in the sense that other parts of the program are not corrupted).

So the language doesn't know, but the *programmer* can make a reasoning in each particular assert of which domains/components of the program are affected by that assertion failure. In the sqrt() case above, the programmer can easily state that the math library that sqrt is part of is not corrupted, and its state is not totally unknown (as in, it's not deadlocked, nor is it formatting the hard disk!). That being the case, sqrt() can be made to throw an exception, and then that "assertion" failure can be recovered cleanly.

Which leads to what you say next:

> In any case, if the programmer knows than assert error is restricted to
> a particular domain, and is recoverable, and wants to recover from it,
> use enforce(), not assert().
>

Very well then. But then we'll get to the point where enforce() will become much more popular than assert to check for contract conditions. assert() will be relegated to niche and rare situations where the program cant really know how to continue/recover cleanly (memory corruption for example).

That idiom is fine with me actually - but then the documentation for assert should reflect that.

-- 
Bruno Medeiros
https://twitter.com/brunodomedeiros

On Monday, 3 November 2014 at 03:29:05 UTC, Walter Bright wrote:
> On 11/2/2014 3:44 PM, Dicebot wrote:
>>> They have hardware protection against sharing memory between processes. It's a
>>> reasonable level of protection.
>> reasonable default - yes
>> reasoable level of protection in general - no
>
> No language can help when that is the requirement.

Yes, because it is property of system architecture as a whole which is exactly what I am speaking about.

> It is absolutely different because of scale; having 1K of shared memory is very different from having 100Mb shared between processes including the stack and program code.

It is possible to have minimal amount shared mutable memory inside one process. There is nothing inherently blocking one to do so, same as there is nothing inherently preventing one to screw the inter-process shared memory. Being different only because of scale -> not really different.

>>> Kernel mode code is the responsibility of the OS system, not the app.
>>
>> In some (many?) large scale server systems OS is the app or at least heavily
>> integrated. Thinking about app as a single independent user-space process is a
>> bit.. outdated.
>
> Haha, I've used such a system (MSDOS) for many years. Switching to process protection was a huge advance. Sad that we're "modernizing" by reverting to such an awful programming environment.

What is huge advance for user land applciation is a problem for server code. Have you ever heard "OS is the problem, not solution" slogan that is slowly becoming more popular in high load networking world?

>>>> It is all about system design.
>>>
>>> It's about the probability of coupling and the level of that your system can
>>> stand. Process level protection is adequate for most things.
>>
>> Again, I am fine with advocating it as a resonable default. What frustrates me
>> is intentionally making any other design harder than it should be by explicitly
>> allowing normal cleanup to be skipped. This behaviour is easy to achieve by
>> installing custom assert handler (it could be generic Error handler too) but
>> impossible to bail out when it is the default one.
>
> Running normal cleanup code when the program is in an undefined, possibly corrupted, state can impede proper shutdown.

Preventing cleanup can be done with roughly one line of code from user code. Enabling it back is effectively impossible. With this decision you don't trade safer default for more dangerous default - you trade configurable default for unavoidable.

To preserve same safe defaults you could define all thrown Errors to result in plain HLT / abort call with possibility to define user handler that actually throws. That would have addressed all concernc nicely while still not making life of those who want cleanup harder.

>> Because of abovementioned avoiding more corruption from cleanup does not sound
>> to me as strong enough benefit to force that on everyone.
>
> I have considerable experience with what programs can do when continuing to run after a bug. This was on real mode DOS, which infamously does not seg fault on errors.
>
> It's AWFUL. I've had quite enough of having to reboot the operating system after every failure, and even then that often wasn't enough because it might scramble the disk driver code so it won't even boot.

I don't argue necessity to terminate the program. I argue strict relation "program == process" which is impractical and inflexible.

> It is my duty to explain how to use the features of the language correctly, including how and why they work the way they do. The how, why, and best practices are not part of a language specification.

You can't just explain things to make them magically appropriate for user domain. I fully understand how you propose to design applications. Unfortunately, it is completely unacceptable in some cases and quite inconvenient in others. Right now your proposal is effectively "design applications like me or reimplement language / library routines yourself".

>>> NO CODE CAN BE RELIABLY EXECUTED PAST THIS POINT.
>> As I have already mentioned it almost never can be truly reliable.
>
> That's correct, but not a justification for making it less reliable.

It is justification for making it more configurable.

> If D changes assert() to do unwinding, then D will become unusable for building reliable systems until I add in yet another form of assert() that does not.

My personal perfect design would be like this:

- Exceptions work as they do now
- Errors work the same way as exceptions but don't get caught by catch(Exception)
- assert does not throw Error but simply aborts the program (configurable with druntime callback)
- define "die" which is effectively "assert(false)"
- tests don't use assert

That would provide default behaviour similar to one we currently have (with all the good things) but leave much more configurable choices for system designer.

>> Some small chance of undefined behaviour vs 100% chance of resource leaks?
>
> If the operating system can't handle resource recovery for a process terminating, it is an unusable operating system.

There are many unusable operating systems out there then :) And don't forget about remote network resources - while leak will eventually timeout there it will still have a negative impact.