It's been there long enough.

On Tuesday, 23 March 2021 at 21:22:18 UTC, Walter Bright wrote:
> It's been there long enough.


Can we fix all the problems found in this ticket:

https://issues.dlang.org/show_bug.cgi?id=21169

 Issue 21169 - make checkedint as a drop-in replacement of native int/long

On 3/23/2021 2:26 PM, mw wrote:
> Can we fix all the problems found in this ticket:
> 
> https://issues.dlang.org/show_bug.cgi?id=21169
> 
>   Issue 21169 - make checkedint as a drop-in replacement of native int/long

Those are all good enhancement ideas.

On Tuesday, 23 March 2021 at 21:22:18 UTC, Walter Bright wrote:
> It's been there long enough.

Isn't that true meanwhile for everything in std.experimental? I ask, because I've got the feeling, that std.experimental doesn't work as expected. For me it looks more or less like an attic, where stuff is put and then forgotten. Maybe the way we used for sumtype is the better approach...

On Tuesday, 23 March 2021 at 21:26:43 UTC, mw wrote:

> https://issues.dlang.org/show_bug.cgi?id=21169
>
>  Issue 21169 - make checkedint as a drop-in replacement of native int/long

I'm not sure if the first thing can be supported. I would require implicit conversions of custom types, which has always been refused in the past.

I don't think the last one, number 7, can work either. checkedint supports adding arbitrary hooks that are executed during various conditions. I don't see how those could be made atomic.

--
/Jacob Carlborg

On Wednesday, 24 March 2021 at 11:20:52 UTC, Berni44 wrote:
> On Tuesday, 23 March 2021 at 21:22:18 UTC, Walter Bright wrote:
>> It's been there long enough.
>
> Isn't that true meanwhile for everything in std.experimental? I ask, because I've got the feeling, that std.experimental doesn't work as expected. For me it looks more or less like an attic, where stuff is put and then forgotten. Maybe the way we used for sumtype is the better approach...

I have no idea why std.experimental is a thing to begin with. It sounds like a bad idea and it turned out to be one. Moving stuff around in a standard library isn't without some disadvantages: The public import stays as an historic artifact or deprecation is needed, both things that should be avoided. There are cases where it's fine like splitting a module into a package.

A standard library is something expected to be particularly well done and stable. Having experimental stuff in it, is an oxymoron.

DUB packages that are "featured" is a way better approach. If deemed worth it (like sumtype), they can be incorporated into Phobos. We may even introduce a "Phobos candidate" tag. Additionally, that establishes DUB as a core part of the D ecosystem.

Can std.experimental packages be removed without deprecation?

The worst offender is std.experimental.typecons; while I don't really understand the purpose of (un-)wrap, I know enough of Final to be sure it's the kind of thing that must be a language feature or it cannot possibly live up to users' expectations. Final cannot work properly as a library solution. (I can elaborate if needed.) I tried fixing it until I realized it's impossible because it's design goal is unsound. I honestly cannot imagine anyone who uses it. It is cumbersome and has zero advantages.

On Tuesday, 23 March 2021 at 21:26:43 UTC, mw wrote:
> https://issues.dlang.org/show_bug.cgi?id=21169
>
>  Issue 21169 - make checkedint as a drop-in replacement of native int/long

Years ago I submitted a checkedint module of my own for inclusion in Phobos (https://code.dlang.org/packages/checkedint), which was ultimately rejected by Andrei Alexandrescu because my design goals did not align with his well enough, prompting him to write what became std.experimental.checkedint himself.

Maximum convenience and similarity to D's native integer types were high priorities for me, so I spent a lot of time thinking about and experimenting with this problem. My conclusions:

///////////////////////////////////
1) Checked types are different from unchecked types. That's the whole point!

I found that trying too hard to make transitioning between checked and unchecked types seamless created holes in the automated protection against overflow that the checked types are supposed to provide. Implicit conversions from checked to unchecked integers are dangerous for the same reason that implicit conversions from @system to @safe delegates are dangerous.

I think the urge to make that transition seamless comes from the fact that trying to actually use checkedint (whether mine or Andrei's) for defensive programming is extremely tedious and annoying, because no one else is doing so. But, this is the wrong solution: the real answer is that checked operations should have been the default in D from the beginning, with unchecked intrinsics available for those rare cases where wrapping overflow and other strange behaviors of machine integers are actually desired, or where maximum performance is needed.

Unchecked integer operations are mostly just a micro-optimization that is pointless outside of very hot code, like inner loops. (It is very puzzling that people consider memory safety so important, and yet are totally disinterested in integer overflow, which can violate memory safety.)

2) While there are many things that can be done to make the behavior of two types more similar, it is impossible in D to make any custom type an actual drop-in replacement for a different type.

This is because D, by design, has only partial support for implicit conversions, and because template constraints and overload resolution are sensitive to the exact type of the arguments.

Thus, whether to treat two different types as equivalent is ultimately a choice that each and every API that may interact with those types makes for itself, either intentionally or by accident. For example:

V f(V)(V value)
    if(std.traits.isIntegral!V)
{
    // Do something here ...
}

The perfectly reasonable template constraint above rejects checkedint types. Should it? There is no way to answer this question without seeing and understanding the body of the function: while uncommon, it is valid and sometimes desirable to depend upon wrapped integer overflow. So, the API designers must explicitly permit checkedint inputs if they consider that desirable.

Automating good solutions to these ambiguities is possible in many cases, but would require deep, breaking, and controversial changes to the D language.
///////////////////////////////////

TLDR; What you're really asking for is impossible in D2. It would require massive breaking changes to the language to implement without undermining the guarantees that a checked integer type exists to provide.

On 3/24/2021 1:28 PM, tsbockman wrote:
> Unchecked integer operations are mostly just a micro-optimization that is pointless outside of very hot code, like inner loops. (It is very puzzling that people consider memory safety so important, and yet are totally disinterested in integer overflow, which can violate memory safety.)

Integer overflow happening should not result in memory safety errors in a safe language. It can cause other problems, but not that.

The reasons people don't care that much about integer overflow are:

1. they are not the cause of enough problems to be that concerning

2. 2's complement arithmetic fundamentally relies on it

3. it's hard to have signed and unsigned integer types coexist without overflows, and not having unsigned types leads to ugly kludges to get them

4. fast integer arithmetic is fundamental to fast code, not a mere micro-optimization. Who wants an overflow check on every pointer increment?

5. size_t is unsigned, and ptrdiff_t is signed. Yet they have to work together.

On Saturday, 27 March 2021 at 03:25:04 UTC, Walter Bright wrote:
> The reasons people don't care that much about integer overflow are:
>
> 1. they are not the cause of enough problems to be that concerning
>
> 2. 2's complement arithmetic fundamentally relies on it

That's an implementation detail. There is no need at either the software or the hardware level to make it the programmer's problem by default.

Main memory is addressed as one giant byte array, but we interact with it through better abstractions most of the time (the stack and the heap).

> 3. it's hard to have signed and unsigned integer types coexist without overflows, and not having unsigned types leads to ugly kludges to get them

Correctly mixing signed and unsigned integers is hard for programmers to consistently get right, but easy for the computer. That's why the default should be for the computer to do it.

> 4. fast integer arithmetic is fundamental to fast code,

I did benchmarking during the development of checkedint. With good inlining and optimization, even a library solution generally slows integer math code down by less than a factor of two. (I expect a language solution could do even better.)

This is significant, but nowhere near big enough to move the bottleneck in most code away from I/O, memory, floating-point, or integer math for which wrapping is semantically correct (like hashing or encryption). In those cases where integer math code really is the bottleneck, there are often just a few hot spots where the automatic checks in some inner loop need to be replaced with manual checks outside the loop.

> not a mere micro-optimization.

By "micro-optimization" I mean that it does not affect the asymptotic performance of algorithms, does not matter much outside of hot spots, and is unlikely to change where the hot spots are in the average program.

> Who wants an overflow check on every pointer increment?

As with bounds checks, most of the time the compiler should be able to prove the checks can be skipped, or move them outside the inner loop. The required logic is very similar.

On Wednesday, 24 March 2021 at 20:28:39 UTC, tsbockman wrote:
> [snip]
> TLDR; What you're really asking for is impossible in D2. It would require massive breaking changes to the language to implement without undermining the guarantees that a checked integer type exists to provide.

Are you familiar with how Zig handles overflow [1]? They error on overflow by default, but they have additional functions and operators to handle when you want to do wraparound.

Nevertheless, I agree that the ship has sailed for D2 on this.

[1] https://ziglang.org/documentation/master/#Integer-Overflow