The idea behind removal of the runtime checks is as a performance optimization done on a debugged program. It's like turning on or off array bounds checking. Many leave asserts and array bounds checking on even in released code to ensure memory safety.

At a minimum, turning it off and on will illuminate just what the checks are costing you.

It's at the option of the programmer.

On 05.03.2018 11:25, Walter Bright wrote:
> On 3/4/2018 3:06 PM, Timon Gehr wrote:
>> On 04.03.2018 22:49, Walter Bright wrote:
>>>> Not necessarily. If the code contains an explicit assertion that the index is in bounds, then, according to the language specification, the bounds check may be removed with -release.
>>>
>>> D, as all languages that I know of do implicitly or explicitly, generates code based on the "as if" rule.
>>> ...
>>
>> Impossible. You wrote a Java compiler.
> 
> Even in Java, the compiler generates code that, from the user's point of view, behaves "as if" the code was actually what was specified. For a trivial example, replacing x*2 with x<<1. Not having this means no optimizations can be done.
> ...

I guess I misunderstood what you meant when you said "as if". I thought you meant that for all languages you know, when assertions are disabled, the compiler behaves "as if" the check was actually there and was known to succeed, even though the check is actually not there and may have failed if it was.

> 
>> All languages that use your "as if" rule are memory unsafe.
>> Zero languages that use the "as if" rule have any memory safe subset that includes assertions.
>> In D, assert is @safe, and it should remain @safe.
> 
>>>> I find the reasoning in terms of "on"/"off" confusing anyway.
>>>> Does "off" mean "contract/assertion removed", or does it mean "failure is UB"?
>>>
>>> "Off" means the check is removed. If the check does not hold, the program enters an invalid state, whether or not the check was actually done. An invalid state means subsequent execution is UB.
>>
>> Why is potential memory corruption to be expected when using @safe language features with a flag to disable contract checks?
> 
> Because the checks provide extra information to the compiler that it can use to generate better code. If that extra information is not true, then the better code will be invalid.
> ...

My question is not why it is the case technically, I was asking for a _rationale_ for this apparently silly behavior. I.e., why is this a good idea from the point of view of language design?

Again: assert is @safe. Compiler hints are @system. Why should assert give compiler hints?

> Memory safety is only one class of errors in a program. If the program has entered a state that is not accounted for by the programmer, the rest of the program's execution will be not predictable.
> ...

But the whole point of having memory safety is to not have UB when the programmer screwed up. Behavior not foreseen by the programmer (a bug) is not the same as behavior unconstrained by the language specification (UB).

> 
>> This makes no sense. This is not useful behavior. There are convenient ways to support potentially unsound compilation hints that do not do this. Contracts and compilation hints should be orthogonal. Contracts should be potentially @safe, compilation hints should be @system always.
>>
>> Note that _actual removal_ is the only use case of 'disabling contracts' that I care about, and I think many D programmers who use "off" will also have this behavior in mind. Yet this is not even an option.
> 
> I don't see much use for this behavior, unless you want to continue running the program after an assert failure, which I cannot recommend and the language is not designed to support.

'in'-contracts catch AssertError when being composed. How can the language not be designed to support that?

Except for this case, the assertion is not _supposed_ to fail for my use cases, and I don't really need the language to explicitly "support that use case". The situation is the following:

- I usually don't want UB in programs I am working on. I want the runtime behavior of the programs to be determined by the source code, such that every behavior observed in the wild (intended or unintended) can be traced back to the source code (potentially in a non-deterministic way, e.g. void initialization of an integer constant). This should be the case always, even if me or someone else on my team made a mistake. The @safe D subset is supposed to give this guarantee. What good is @safe if it does not guarantee absence of buffer overrun attacks?

- Checking assertions can be too costly, so it should be possible to disable the check.

- Using existing assertions as compiler hints is not necessary. (Without having checked it, I'm sure that LDC/GDC have a more suitable intrinsic for this already.)

As far as I can discern, forcing disabled asserts to give compiler hints has no upsides.

> But you can always do something like:
> 
>     version (ignore_asserts) { } else { assert(...); }
> ...

I know. Actually version(assert) assert(...); also works. However, this is too verbose, especially in contracts. I'd like a solution that does not require me to change the source code. Ideally, I just want the Java behavior (with reversed defaults).

> which would optionally remove both the runtime check and any compiler use of the assert. Or you could use https://dlang.org/library/std/exception/enforce.html which has no influence on compiler semantics.
> ...

(enforce is _completely unrelated_ to the current discussion.)

Contracts are designed to be used with built-in assert. Built-in assert is convenient.

If the point of 'assert' is to give compiler hints (for me this is not even a use case of assert at all), then it should be @system.

> 
>> At the very least, the DIP should be up-front about this.
>> I'm still not even sure that Mathias Lang intended the UB semantics.
> 
> It being UB was my doing, not Mathias'. DIP1006 is not redefining the semantics of what assert does.

This is not really about assert semantics, this is about the semantics of "disabling the check". There was no "-check=off" flag before.

The DIP uses terminology such as "disable assertions" as opposed to "disable assertion checks (but introduce compiler hints)".

Contracts are supposed to increase safety, not to reduce it (ask Bertrand Meyer).

On Saturday, 3 March 2018 at 16:33:00 UTC, Martin Nowak wrote:
> Doesn't really work that way, we can disable assertions, in contracts, out contracts, and invariants. But not assertions in some contexts while leaving them enabled in other contexts. At least not without modifying all related codegen and introducing context queries (e.g. think mixin templates).

That's a shame, but presumably the fine-grainedness could be extended at some point.

Question: what would -release=assert do to unittests?  Would it not touch them at all?  Or would it disable all asserts including in unittests?

> FWIW -release=assert,in,out,invariant fits out needs well enough.
> Just the use-case that someone wants to disable asserts in functions but still wants to use contracts, required to use a replacement for assert in contracts and invariants.

Yea, there are obviously workarounds.  I think the main concern from my side is to not have hierarchical assumptions about what gets turned on or off, and AFAICS -release=assert,in,out-invariant pretty much fits that.

On 05.03.2018 11:30, Walter Bright wrote:
> The idea behind removal of the runtime checks is as a performance optimization done on a debugged program.

Optimizing performance is fine, but why pessimize safety? The hints will usually not make a significant difference in performance anyway. I guess it is fine to have a compiler option that is all speed no safety, but it should not be the only option.

> It's like turning on or off array bounds checking.

It is not.

void main()@safe{
     int[] x=[];
     writeln(x[0]); // range violation even with -release
                    // defined behavior even with -boundscheck=off (!)
}

If I now add an assertion, I suddenly get UB:

void main()@safe{
    int[] x=[];
    assert(0<x.length); // obviously this should hold, or next line is invalid
    writeln(x[0]); // UB with -release
}

I did not make the code any more wrong by adding the assertion.
Why should I get more UB?


> Many leave asserts and array bounds checking on even in released code to ensure memory safety.
> ...

Maybe the requirements change and it is now too costly to leave contracts on in release mode, or the number of contracts in the code base slowly accumulates until we reach a point where the total cost is too large, or we replace a library, and the new version has costly contracts, etc. Now we have the following options:

- Leave contracts in -- fail performance requirements.

- Remove contracts -- fail safety requirements.

- Track down all 'assert's, even those in external libraries, and replace them by a custom home-cooked solution that is incompatible with everyone else's -- fail maintainability requirements.

To me this situation is ridiculous.

> At a minimum, turning it off and on will illuminate just what the checks are costing you.
> ...

Well, no. If the bug is elusive enough to not have shown up in debug mode, it probably won't be seen early during -release testing, and even if it does, the UB may mask it. (Note that when the program becomes faster, the likelihood of timing-dependent bugs showing up may change.)

I.e., if something goes wrong, it is likely that you won't see the safety costs until it is too late.

> It's at the option of the programmer.

It is not, but I want it to be. That's all I want. [1]

I'm just saying there should be the following option:

- Remove contracts -- sufficient performance and retain memory safety.

FWIW, this is what all contract systems that I'm aware of do, except D, and maybe C asserts in certain implementations (if you want to call that contracts).


[1] Well, maybe add a @system "__assume" intrinsic.

On Monday, 5 March 2018 at 15:48:12 UTC, Timon Gehr wrote:
>
> - Using existing assertions as compiler hints is not necessary. (Without having checked it, I'm sure that LDC/GDC have a more suitable intrinsic for this already.)
>
> As far as I can discern, forcing disabled asserts to give compiler hints has no upsides.
>


In the simple cases, or in anything that looks like a unittest/testsuite, probably not.

There are likely going to be more aggressive optimizations however if CFA can see that a variable will never be outside a given range, i.e:

---
int[5] arr;

if (len < 0 || len >= 5)
{
    unreachable();  // in non-release code, this would throw a RangeError.
}

return arr[len];
---

From this, we aggressively assume that len is a valid index of arr.  Something that happens in optimized non-release builds, but in release builds we must accommodate for the possibility of a range error.

On Monday, 5 March 2018 at 18:44:54 UTC, Joseph Rushton Wakeling wrote:
> On Saturday, 3 March 2018 at 16:33:00 UTC, Martin Nowak wrote:
>> Doesn't really work that way, we can disable assertions, in contracts, out contracts, and invariants. But not assertions in some contexts while leaving them enabled in other contexts. At least not without modifying all related codegen and introducing context queries (e.g. think mixin templates).
>
> That's a shame, but presumably the fine-grainedness could be extended at some point.
>
> Question: what would -release=assert do to unittests?  Would it not touch them at all?  Or would it disable all asserts including in unittests?
>

From memory, it would turn off asserts even in unittests.  You could raise a bug against gdc for that as it's a reasonable suggestion.


>> FWIW -release=assert,in,out,invariant fits out needs well enough.
>> Just the use-case that someone wants to disable asserts in functions but still wants to use contracts, required to use a replacement for assert in contracts and invariants.
>
> Yea, there are obviously workarounds.  I think the main concern from my side is to not have hierarchical assumptions about what gets turned on or off, and AFAICS -release=assert,in,out-invariant pretty much fits that.

N.B: GDC has -f[no]-release, -f[no-]assert, -f[no-]invariant, -f[no-]preconditions, and -f[no-]postconditions  (-f[no-]in and -f[no-]out were removed as they are a little too vague).  And it doesn't matter which order you pass them in, if an option is explicitly set, then they do not get turned on/off by -frelease.

On 05.03.2018 20:41, Iain Buclaw wrote:
> On Monday, 5 March 2018 at 15:48:12 UTC, Timon Gehr wrote:
>>
>> - Using existing assertions as compiler hints is not necessary. (Without having checked it, I'm sure that LDC/GDC have a more suitable intrinsic for this already.)
>>
>> As far as I can discern, forcing disabled asserts to give compiler hints has no upsides.
>>
> 
> 
> In the simple cases, or in anything that looks like a unittest/testsuite, probably not.
> 
> There are likely going to be more aggressive optimizations however if CFA can see that a variable will never be outside a given range, i.e:
> ...

(Note that by "forcing", I mean withholding other options from the user. I'm not saying that using information from asserts can never be useful, just that it can just as well be harmful, and therefore it is unwise to not allow disabling them. I was saying that there are no upsides to not having a flag that actually removes assertions.)

> ---
> int[5] arr;
> 
> if (len < 0 || len >= 5)
> {
>      unreachable();  // in non-release code, this would throw a RangeError.
> }
> 
> return arr[len];
> ---
> 
>  From this, we aggressively assume that len is a valid index of arr.  Something that happens in optimized non-release builds, but in release builds we must accommodate for the possibility of a range error.

I think this particular case is a bit less questionable than doing the same for general assertions (for instance, in @safe code, -release will not actually remove the bounds check unless there is some relevant assertion somewhere). In any case, I don't argue strongly against a flag that turns all assertions into compiler hints, I just think there should also be a flag that disables them safely. Also, maybe -release should commit to either disregarding @safe completely or respecting it completely.

On 3/5/2018 7:48 AM, Timon Gehr wrote:
> Again: assert is @safe. Compiler hints are @system. Why should assert give compiler hints?

Asserts give expressions that must be true. Why not take advantage of them? See Spec# which based an entire language around that notion:

 https://en.wikipedia.org/wiki/Spec_Sharp

Some possible optimizations based on this are:

1. elimination of array bounds checks
2. elimination of null pointer checks
3. by knowing a variable can take on a limited range of values, a cheaper data type can be substituted
4. elimination of checks for 'default' switch values
5. elimination of overflow checks

dmd's optimizer currently does not extract any information from assert's. But why shut the door on that possibility?


> But the whole point of having memory safety is to not have UB when the programmer screwed up. Behavior not foreseen by the programmer (a bug) is not the same as behavior unconstrained by the language specification (UB).

It's the programmer's option to leave those runtime checks in if he wants to.


> 'in'-contracts catch AssertError when being composed. How can the language not be designed to support that?

That is indeed an issue. It's been proposed that in-contracts throw a different exception, say "ContractException" so that it is not UB when they fail. There's a bugzilla ER on this. (It's analogous to asserts in unittests not having UB after they fail.)


> - I usually don't want UB in programs I am working on. I want the runtime behavior of the programs to be determined by the source code, such that every behavior observed in the wild (intended or unintended) can be traced back to the source code (potentially in a non-deterministic way, e.g. void initialization of an integer constant). This should be the case always, even if me or someone else on my team made a mistake. The @safe D subset is supposed to give this guarantee. What good is @safe if it does not guarantee absence of buffer overrun attacks?

It guarantees it at the option of the programmer via a command line switch.


> - Using existing assertions as compiler hints is not necessary. (Without having checked it, I'm sure that LDC/GDC have a more suitable intrinsic for this already.)
> 
> As far as I can discern, forcing disabled asserts to give compiler hints has no upsides.

I suspect that if:

    compiler_hint(i < 10);

were added, there would be nothing but confusion as to its correct usage vs assert vs enforce. There's already enough confusion about the latter two. In fact, I can pretty much guarantee it will be rarely used correctly.


> I know. Actually version(assert) assert(...); also works. However, this is too verbose, especially in contracts.

You can wrap it in a template.


> I'd like a solution that does not require me to change the source code. Ideally, I just want the Java behavior (with reversed defaults).

But you'll have to change the code to compiler_hint().


> (enforce is _completely unrelated_ to the current discussion.)

It does just what you ask (for the regular assert case).


>> It being UB was my doing, not Mathias'. DIP1006 is not redefining the semantics of what assert does.
> This is not really about assert semantics, this is about the semantics of "disabling the check".

It is very much about the semantics of assert.


> There was no "-check=off" flag before.

Yes there was, it's the "release" flag.


> The DIP uses terminology such as "disable assertions" as opposed to "disable assertion checks (but introduce compiler hints)".

Yes, the language could be more precise, but I couldn't blame Mathias for that. I also disagree with the word "hint", because it implies things like "this branch is more likely to be taken" to guide code generation decisions, rather than "assume X is absolutely always incontrovertibly true and you can bet the code on it".

On 3/5/2018 11:34 AM, Timon Gehr wrote:
> On 05.03.2018 11:30, Walter Bright wrote:
>>The hints will usually not make a significant difference in performance anyway.

Reasonable people will disagree on what is significant or not.


>> It's like turning on or off array bounds checking.
> 
> It is not.
> 
> void main()@safe{
>       int[] x=[];
>       writeln(x[0]); // range violation even with -release
>                      // defined behavior even with -boundscheck=off (!)

It is not defined behavior with -boundscheck=off.

> }
> 
> If I now add an assertion, I suddenly get UB:
> 
> void main()@safe{
>      int[] x=[];
>      assert(0<x.length); // obviously this should hold, or next line is invalid
>      writeln(x[0]); // UB with -release
> }
> 
> I did not make the code any more wrong by adding the assertion.
> Why should I get more UB?

Because you put in an assert that did not hold, and disabled the check.


> Now we have the following options:
> 
> - Leave contracts in -- fail performance requirements.
> 
> - Remove contracts -- fail safety requirements.
> 
> - Track down all 'assert's, even those in external libraries, and replace them by a custom home-cooked solution that is incompatible with everyone else's -- fail maintainability requirements.
> 
> To me this situation is ridiculous.

It's completely under the control of the programmer. I know you disagree with that notion. You can even create your own `myassert` to produced your desired semantics.


> FWIW, this is what all contract systems that I'm aware of do, except D, and maybe C asserts in certain implementations (if you want to call that contracts).

D is better (!).

(C's asserts are not part of the language, so impart no semantics to the compiler.)

On 03/05/2018 10:11 PM, Walter Bright wrote:
> On 3/5/2018 11:34 AM, Timon Gehr wrote:
[...]
>>       int[] x=[];
>>       writeln(x[0]); // range violation even with -release
>>                      // defined behavior even with -boundscheck=off (!)
> 
> It is not defined behavior with -boundscheck=off.

Dereferencing null is not defined with -boundscheck=off?