Jump to page: 1 28  
Page
Thread overview
@system blocks and safer @trusted (ST) functions
Jul 25, 2021
Bruce Carneal
Jul 25, 2021
jfondren
Jul 25, 2021
Bruce Carneal
Jul 25, 2021
jfondren
Jul 25, 2021
Bruce Carneal
Jul 25, 2021
Paul Backus
Jul 25, 2021
Bruce Carneal
Jul 25, 2021
Paul Backus
Jul 25, 2021
Bruce Carneal
Jul 25, 2021
Walter Bright
Jul 26, 2021
ShadoLight
Jul 26, 2021
Walter Bright
Jul 25, 2021
jfondren
Jul 26, 2021
Bruce Carneal
Jul 26, 2021
Bruce Carneal
Jul 25, 2021
Paul Backus
Jul 25, 2021
Bruce Carneal
Jul 25, 2021
Paul Backus
Jul 25, 2021
Bruce Carneal
Jul 25, 2021
Paul Backus
Jul 25, 2021
Bruce Carneal
Jul 25, 2021
Paul Backus
Jul 25, 2021
claptrap
Jul 25, 2021
jfondren
Jul 25, 2021
Paul Backus
Jul 25, 2021
claptrap
Jul 25, 2021
Paul Backus
Jul 25, 2021
claptrap
Jul 25, 2021
Paul Backus
Jul 26, 2021
Timon Gehr
Jul 26, 2021
Paul Backus
Jul 26, 2021
claptrap
Jul 26, 2021
Paul Backus
Jul 26, 2021
claptrap
Jul 26, 2021
jfondren
Jul 28, 2021
claptrap
Jul 28, 2021
Paul Backus
Jul 28, 2021
claptrap
Jul 28, 2021
Bruce Carneal
Jul 28, 2021
claptrap
Jul 28, 2021
Bruce Carneal
Jul 29, 2021
Bruce Carneal
Jul 29, 2021
claptrap
Jul 29, 2021
Bruce Carneal
Jul 26, 2021
Walter Bright
Jul 26, 2021
claptrap
Jul 25, 2021
claptrap
Jul 25, 2021
Bruce Carneal
Jul 25, 2021
jfondren
Jul 26, 2021
ag0aep6g
Jul 26, 2021
Timon Gehr
Jul 26, 2021
ag0aep6g
Jul 26, 2021
Walter Bright
Jul 26, 2021
jfondren
Jul 26, 2021
ag0aep6g
Jul 26, 2021
jfondren
Jul 26, 2021
ag0aep6g
Jul 26, 2021
Araq
Jul 26, 2021
Paul Backus
Jul 26, 2021
Paul Backus
Jul 26, 2021
Paul Backus
Jul 28, 2021
Paul Backus
Jul 25, 2021
Ali Çehreli
Jul 27, 2021
jmh530
Jul 27, 2021
jfondren
Jul 28, 2021
jmh530
July 25, 2021

At beerconf I committed to putting forward a DIP regarding a new syntactic element to be made available within @trusted functions, the @system block. The presence of one or more @system blocks would enable @safe checking elsewhere in the enclosing @trusted function.

I also committed at beerconf to starting the formal DIP on this in two weeks so I read the DIP docs just now in order to get a jump on things. Boy, that was an eye-opener. From the incessantly cautionary language there I'd say that I dramatically underestimated the effort required to bring a DIP over the finish line (or to see it clearly rejected).

Long story short, I'll still do the DIP unless the ideas are discredited in subsequent discussion but it will have to be quite a bit later in the year as I dont have weeks of time to spend on it in the near future. In the mean time I'd invite others to comment on the ST (safer @trusted) idea as sketched in the first paragraph. For starters, we might come up with a better name...

A few notes:

  1. Since @system blocks would be a new syntactic element I believe ST would be a backward compatible addition.

  2. The problematic @trusted lambda escapes creeping in to "@safe" code could be replaced going forward by a more honestly named form, @trusted code with @system blocks. Best practices could evolve to the point that @safe actually meant @safe again.

  3. I believe quite a bit of @trusted code is @safe but can not currently benefit from automated checking. The proposal would allow a transition of such code to an @safer form while reducing the amount of code that requires intense manual checking.

  4. @safe blocks within @trusted code were considered but left things defaulting in a less useful direction (and they're just plain ugly).

Questions and improvements are very welcome but, in the dlang tradition, I also welcome your help in destroying this proposal.

July 25, 2021

On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:

>

The presence of one or more @system blocks would enable @safe checking elsewhere in the enclosing @trusted function.

This has an unfortunate result: if in maintenance you edit a @trusted function to remove its @system blocks, it'll quietly no longer be checked as @safe.

Of the current behavior, Walter's said that he doesn't want @trusted blocks because they should be discouraged in @safe code. With this change, we'll have exactly what he doesn't want with different names: s/@trusted/@system/, s/@safe/@trusted/, and the exact same behavior: @system blocks are just what @trusted blocks would've been, and @trusted code with @system blocks in it is just @safe code with a different name.

Instead of people accepting that @safe "isn't really @safe" in the presence of @trusted blocks, and that the whole body of the function has to be audited, with this change we'll have s/@safe/@trusted/ blocks that aren't really @safe in the presence of @system blocks, and that the whole body of the function has to be audited. The "you have to audit this" signifier is the same, an internal lower-protection block, and all that's gained is that the function attribute's spelled differently. Is this really worth it?

One way to avoid the "unfortunate result" above is to permit @trusted blocks in @safe code. Which results in Rust equivalent functionality: @safe code is checked as @safe, @safe code with internal @trusted blocks is still checked as @safe but people know to audit it, and @system code isn't checked as @safe.

People like Rust's unsafe system, the current @trusted-lambda abuse is a simulation of unsafe, Phobos already uses @trusted lambdas in exactly the same way as unsafe blocks are used in Rust, and this proposed behavior is 99.9% identical to unsafe except it has this bonus "you can silently, accidentally remove @safe checks from your code now" feature.

I appreciate that there's a vision also to @safe/@trusted/@system, but it doesn't seem to have stuck, with Phobos having more than twice as many @trusted lambdas than @trusted functions:

phobos$ grep -rP '\W\(\) @trusted' --include '*.d'|wc -l
238
phobos$ grep -rP '\w\(\) @trusted' --include '*.d'|wc -l
111

I don't think that Rust has everything right. And, I don't pay attention to the Rust community at all; maybe they've a lot of gripes about how they're using unsafe blocks and unsafe functions. But, just look at all those @trusted lambadas. If you run the first command without the "|wc -l" on the end you'll see they're almost all single statements or expressions.

Adding a @trusted block to @safe code doesn't discard the @safe/@trusted/@system vision, it just lets people follow the unsafe vision that they're already following without so many complaints about how ugly the workaround is, when D's good looks are one of its advantages over Rust.

This proposal also doesn't immediately discard the @safe/@trusted/@system vision, but it introduces a minor footgun because of a subtle conflict with that vision, and as people adopt it they'll also want another --preview switch to deal with the footgun, and that switch will break all current @trusted code that's currently assuming no @safe checks, and so there will a long deprecation cycle...

@trusted blocks win this. Or rather--not to be rude--but if you came out and said that this DIP was just some triangulation to get people to accept @trusted blocks, I would say: "good job! It got me thinking!" If not, I'm sorry.

July 25, 2021

On Sunday, 25 July 2021 at 06:13:41 UTC, jfondren wrote:

>

On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:

>

The presence of one or more @system blocks would enable @safe checking elsewhere in the enclosing @trusted function.

This has an unfortunate result: if in maintenance you edit a @trusted function to remove its @system blocks, it'll quietly no longer be checked as @safe.

As opposed to never being checked for @safe in legacy @trusted? Or always requiring manual vigilance in an "@safe" routine corrupted by an @trusted lambda?

More directly, IIUC, someone explicitly downgrades @safety by removing an @system block encapsulation from within a recently coded (non-legacy) @trusted function and we're supposed to second guess them? They don't actually want to revert to legacy behavior they just happened to remove the @system block in an @trusted function?

I think it would be great if we can get to the point where warnings against legacy trusted reversions are welcome but in the mean time I believe the danger of not protecting against explicit safety degradation maintenance errors is modest when compared to the other issues.

>

Of the current behavior, Walter's said that he doesn't want @trusted blocks because they should be discouraged in @safe code. With this change, we'll have exactly what he doesn't want with different names: s/@trusted/@system/, s/@safe/@trusted/, and the exact same behavior: @system blocks are just what @trusted blocks would've been, and @trusted code with @system blocks in it is just @safe code with a different name.

I think you're misreading Walter on this. He was the one who recommended that I pursue this DIP at beerconf (it was just an idea that I'd thrown out up to that point).

>

Instead of people accepting that @safe "isn't really @safe" in the presence of @trusted blocks, and that the whole body of the function has to be audited, with this change we'll have s/@safe/@trusted/ blocks that aren't really @safe in the presence of @system blocks, and that the whole body of the function has to be audited. The "you have to audit this" signifier is the same, an internal lower-protection block, and all that's gained is that the function attribute's spelled differently. Is this really worth it?

Definitely. We shouldn't "accept" contradictions, cognitive load, manual auditing, ... unless we have no alternative. The proposal opens the door to consistent nesting and naming and additional automated checking. Less "unsprung weight".

>

I appreciate that there's a vision also to @safe/@trusted/@system, but it doesn't seem to have stuck, with Phobos having more than twice as many @trusted lambdas than @trusted functions:

phobos$ grep -rP '\W\(\) @trusted' --include '*.d'|wc -l
238
phobos$ grep -rP '\w\(\) @trusted' --include '*.d'|wc -l
111

If the lambdas are mostly found within @safe code then these stats would support the proposal. Also, the @trusted functions could benefit from @system block upgrades.

>

...

Adding a @trusted block to @safe code doesn't discard the @safe/@trusted/@system vision, it just lets people follow the unsafe vision that they're already following without so many complaints about how ugly the workaround is, when D's good looks are one of its advantages over Rust.

I'd like to localize system code and minimize un-automated checking. An @trusted lambda in an @safe function goes in the other direction. It's not the two or three liner workaround itself that is the main problem, it's the silent expansion of programmer responsibility to the surrounds.

>

This proposal also doesn't immediately discard the @safe/@trusted/@system vision, but it introduces a minor footgun because of a subtle conflict with that vision, and as people adopt it they'll also want another --preview switch to deal with the footgun, and that switch will break all current @trusted code that's currently assuming no @safe checks, and so there will a long deprecation cycle...

I don't see it that way. I see a simple path to safer code. Your "footgun" is, in my opinion and mixing metaphors, small potatoes compared to the additional automated coverage evident in the proposal. As I hope you'll agree, there is a huge gap between "this isn't a big deal, most competent programmers would not mess up here very often..." and "no programmers will ever mess up here because they can't".

>

@trusted blocks win this. Or rather--not to be rude--but if you came out and said that this DIP was just some triangulation to get people to accept @trusted blocks, I would say: "good job! It got me thinking!" If not, I'm sorry.

No need to be sorry at all. If, broadly, people are happy about the current state of affairs and see no significant benefit to truth-in-naming, consistent nesting, and @safe checking within @trusted functions then we'll stick with what we've got.

July 25, 2021

On Sunday, 25 July 2021 at 08:16:41 UTC, Bruce Carneal wrote:

> >

Of the current behavior, Walter's said that he doesn't want @trusted blocks because they should be discouraged in @safe code. With this change, we'll have exactly what he doesn't want with different names: s/@trusted/@system/, s/@safe/@trusted/, and the exact same behavior: @system blocks are just what @trusted blocks would've been, and @trusted code with @system blocks in it is just @safe code with a different name.

I think you're misreading Walter on this. He was the one who recommended that I pursue this DIP at beerconf (it was just an idea that I'd thrown out up to that point).

I think it's a fair reading of https://www.youtube.com/watch?v=nGX75xNeW_Y&t=379s

From "With this change"&c that's all in my voice.

>

I'd like to localize system code and minimize un-automated checking. An @trusted lambda in an @safe function goes in the other direction.

Presently, when you write a function that you want to be mostly @safe but that needs to break those restrictions in part, you call it @safe and you put the violation in a @trusted lambda. With your proposal, when you write a function that you want to be mostly @safe but that needs to break those restrictions in part, you call it @trusted and you put the violation in a @system block.

These seem to me to be so identical that I don't see how they can be moving in opposite directions. You need to audit the same amount of code, and you find that code with the same exertion of effort.

>

No need to be sorry at all. If, broadly, people are happy about the current state of affairs and see no significant benefit to truth-in-naming, consistent nesting, and @safe checking within @trusted functions then we'll stick with what we've got.

I feel like you've started with this problem, "@trusted functions don't benefit from @safe checking at all", and you've found what seems like a good solution to it, but you haven't stepped back from your solution to realize that you've worked your way back to the existing state of affairs.

If someone shows you some code with a long @trusted function, and you would like @safe checks in it, you can change it to a @safe function with @trusted lambdas. You can do that right now, and that completely solves the problem of @trusted functions not getting checked. And this is the 2-to-1 choice of @trusted in Phobos. A new option of "you can leave it as @trusted and add @system blocks" is just the same option that you already have.

(Although if you skip back from the timestamp above, Walter points out that @trusted blocks might not get inlined in some cases. A deliberate language feature wouldn't have that problem.)

July 25, 2021
On 7/25/2021 1:16 AM, Bruce Carneal wrote:
> He was the one who recommended that I pursue this DIP at beerconf (it was just an idea that I'd thrown out up to that point).

That's right. I did not think of the case of removing the @system block from a @trusted function and then silently losing the safety checking in the rest of the @trusted function. Thanks for pointing it out.

One solution would be for the compiler to check such @trusted functions for @safe-ty anyway, and issue an error for @trusted functions with no @system{} blocks and no @safe errors.

The user could silence this error by adding an empty @system block:

    @system { }
July 25, 2021

On Sunday, 25 July 2021 at 06:13:41 UTC, jfondren wrote:

>

On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:

>

The presence of one or more @system blocks would enable @safe checking elsewhere in the enclosing @trusted function.

This has an unfortunate result: if in maintenance you edit a @trusted function to remove its @system blocks, it'll quietly no longer be checked as @safe.

Yes, but there is no point in a @trusted function without @system blocks. The compiler should warn about it so it gets changed (but should not ever be deprecated).

>

Of the current behavior, Walter's said that he doesn't want @trusted blocks because they should be discouraged in @safe code. With this change, we'll have exactly what he doesn't want with different names: s/@trusted/@system/, s/@safe/@trusted/, and the exact same behavior: @system blocks are just what @trusted blocks would've been, and @trusted code with @system blocks in it is just @safe code with a different name.

The name is the important change. A @safe function shouldn't require manual verification. A @trusted function does. The @safe function with @trusted escape gives the impression that the @safe parts don't need checking, but they do. It's also difficult to use tools to check which parts need review when focusing on memory safety.

>

Instead of people accepting that @safe "isn't really @safe" in the presence of @trusted blocks, and that the whole body of the function has to be audited, with this change we'll have s/@safe/@trusted/ blocks that aren't really @safe in the presence of @system blocks, and that the whole body of the function has to be audited. The "you have to audit this" signifier is the same, an internal lower-protection block, and all that's gained is that the function attribute's spelled differently. Is this really worth it?

Yes.

>

One way to avoid the "unfortunate result" above is to permit @trusted blocks in @safe code. Which results in Rust equivalent functionality: @safe code is checked as @safe, @safe code with internal @trusted blocks is still checked as @safe but people know to audit it, and @system code isn't checked as @safe.

We already have that with @trusted lambdas today. This just changes the name to identify which parts are actually trusted and need review.

>

I appreciate that there's a vision also to @safe/@trusted/@system, but it doesn't seem to have stuck, with Phobos having more than twice as many @trusted lambdas than @trusted functions:

phobos$ grep -rP '\W\(\) @trusted' --include '*.d'|wc -l
238
phobos$ grep -rP '\w\(\) @trusted' --include '*.d'|wc -l
111

@trusted lambdas are required inside templates where you want code to be @safe when the type parameters allow it. The resulting function itself needs to be marked @trusted with the lambdas replaced with @system blocks to achieve the same effect, as memory safety review is still needed.

-Steve

July 25, 2021

On Sunday, 25 July 2021 at 09:04:17 UTC, jfondren wrote:

>

On Sunday, 25 July 2021 at 08:16:41 UTC, Bruce Carneal wrote:

> >

Of the current behavior, Walter's said that he doesn't want @trusted blocks because they should be discouraged in @safe code. With this change, we'll have exactly what he doesn't want with different names: s/@trusted/@system/, s/@safe/@trusted/, and the exact same behavior: @system blocks are just what @trusted blocks would've been, and @trusted code with @system blocks in it is just @safe code with a different name.

I think you're misreading Walter on this. He was the one who recommended that I pursue this DIP at beerconf (it was just an idea that I'd thrown out up to that point).

I think it's a fair reading of https://www.youtube.com/watch?v=nGX75xNeW_Y&t=379s

From "With this change"&c that's all in my voice.

It may well have been a fair reading of the past. I reported on the present (yesterday).

> >

I'd like to localize system code and minimize un-automated checking. An @trusted lambda in an @safe function goes in the other direction.

Presently, when you write a function that you want to be mostly @safe but that needs to break those restrictions in part, you call it @safe and you put the violation in a @trusted lambda. With your proposal, when you write a function that you want to be mostly @safe but that needs to break those restrictions in part, you call it @trusted and you put the violation in a @system block.

These seem to me to be so identical that I don't see how they can be moving in opposite directions. You need to audit the same amount of code, and you find that code with the same exertion of effort.

They are similar in some regards, but if trusted lambdas are the only practical option for this type of code going forward, then @safe will require manual checking in perpetuity.

I'm not saying mitigating tooling or procedures to cover this can not be devised/employed. I am saying that there is a qualitative difference between "the compiler asserts" and "my super-duper-xtra-linguistic-wonder-procedure asserts". Moving towards the former is a good idea. Moving towards the latter is not.

> >

No need to be sorry at all. If, broadly, people are happy about the current state of affairs and see no significant benefit to truth-in-naming, consistent nesting, and @safe checking within @trusted functions then we'll stick with what we've got.

I feel like you've started with this problem, "@trusted functions don't benefit from @safe checking at all", and you've found what seems like a good solution to it, but you haven't stepped back from your solution to realize that you've worked your way back to the existing state of affairs.

The initial motivation was concern over @safe practically transitioning from "compiler checkable" to "not checkable (reported) by the compiler".

>

If someone shows you some code with a long @trusted function, and you would like @safe checks in it, you can change it to a @safe function with @trusted lambdas. You can do that right now, and that completely solves the problem of @trusted functions not getting checked. And this is the 2-to-1 choice of @trusted in Phobos. A new option of "you can leave it as @trusted and add @system blocks" is just the same option that you already have.

As hopefully understood from my earlier comments, these are, qualitatively, not the same thing. You will still have to check a conversion to a new style @trusted function manually of course, no work savings there, but you'd gain something pretty important: the compiler's assertions regarding your remaining @safe code might actually mean something.

Also, I'm not sure how choices (2-to-1 or any N-to-1) made prior to the availability of an alternative are to be interpreted. What am I missing?

>

...

Finally, thanks for engaging.

July 25, 2021

On Sunday, 25 July 2021 at 12:05:10 UTC, Steven Schveighoffer wrote:

>

On Sunday, 25 July 2021 at 06:13:41 UTC, jfondren wrote:

>

Instead of people accepting that @safe "isn't really @safe" in the presence of @trusted blocks, and that the whole body of the function has to be audited, with this change we'll have s/@safe/@trusted/ blocks that aren't really @safe in the presence of @system blocks, and that the whole body of the function has to be audited. The "you have to audit this" signifier is the same, an internal lower-protection block, and all that's gained is that the function attribute's spelled differently. Is this really worth it?

Yes.

OK. I'll argue the opposite position for a bit, then.

Here's a @trusted function with a non-@safe component:

ulong getAvailableDiskSpace(scope const(char)[] path) @trusted
{
    ULARGE_INTEGER freeBytesAvailable;
    auto err = GetDiskFreeSpaceExW(path.tempCStringW(), &freeBytesAvailable, null, null);
    cenforce(err != 0, "Cannot get available disk space");
    return freeBytesAvailable.QuadPart;
}

With this proposal, I imagine:

ulong getAvailableDiskSpace(scope const(char)[] path) @trusted
{
    ULARGE_INTEGER freeBytesAvailable;
    auto err = @system GetDiskFreeSpaceExW(path.tempCStringW(), &freeBytesAvailable, null, null);  // expression usage?
    @system{ auto err = GetDiskFreeSpaceExW(path.tempCStringW(), &freeBytesAvailable, null, null); }  // scopeless block?
    cenforce(err != 0, "Cannot get available disk space");
    return freeBytesAvailable.QuadPart;
}

And in current practice:

ulong getAvailableDiskSpace(scope const(char)[] path) @safe
{
    ULARGE_INTEGER freeBytesAvailable;
    auto err = () @trusted {
        return GetDiskFreeSpaceExW(path.tempCStringW(), &freeBytesAvailable, null, null);
    } ();
    cenforce(err != 0, "Cannot get available disk space");
    return freeBytesAvailable.QuadPart;
}

So a naive take is "the last two versions are literally the same", but they become distinct when all three versions are framed by how they came to be, respectively:

  1. a @trusted function as written by someone trying to properly use the language as described by https://dlang.org/spec/memory-safe-d.html

  2. a @trusted function as written after that document is updated to reflect this DIP.

  3. a @trusted function in the current "@trusted functions are bad because they don't check anything, avoid them" fail state, which is ugly because it's a late adaptation to a fault in the language rather than a result of deliberate design.

In the first case, @safe/@trusted/@system functions are dutifully written and then bugs are later found in the @trusted functions. @trusted functions are an attractive nuisance.

In the second case, @safe/@trusted/@system functions are written and that doesn't happen. @trusted functions have been rehabilitated and have their intended role.

In the last case, only @safe/@system functions are written, and some of the @safe functions are secretly, really, @trusted functions that have be written in a weird way to work. @trusted functions are either a mistake or a shorthand for a @safe function whose entire body may as well be in a @trusted block.

Rather than bless the failure state by giving it a better syntax (and continuing to have @trusted as a problem for new programmers), we'd like to fix @trusted so that @trusted functions are worth writing again.

Does that sound about right?

July 25, 2021

On Sunday, 25 July 2021 at 05:05:44 UTC, Bruce Carneal wrote:

>

At beerconf I committed to putting forward a DIP regarding a new syntactic element to be made available within @trusted functions, the @system block. The presence of one or more @system blocks would enable @safe checking elsewhere in the enclosing @trusted function.

Both before and after this proposal, there are 3 kinds of code:

  1. Code that is automatically checked for memory safety.
  2. Code that is assumed by the compiler to be safe, and must be manually checked for memory safety.
  3. Code that is not checked for memory safety, and is assumed to be unsafe.

Currently, (1) is marked @safe, (2) is marked @trusted, and (3) is marked @system.

Under this proposal, (1) would be marked either @safe or @trusted, (2) would be marked either @trusted or @system, and (3) would be marked @system. I do not think this is an improvement relative to the status quo.

>

The problematic @trusted lambda escapes creeping in to "@safe" code could be replaced going forward by a more honestly named form, @trusted code with @system blocks. Best practices could evolve to the point that @safe actually meant @safe again.

What makes @trusted lambdas problematic is that they implicitly depend on everything in their enclosing scope, which makes it easy for a change in the @safe portion of the code to accidentally violate an assumption that the @trusted portion depends on.

If we want to address this issue at the language level, the most direct solution is to require that @trusted lambdas make their dependencies explicit. This can be done by requiring all @trusted nested functions (of which lambdas are a special case) to be static.

Of course, this is a breaking change, so it would require a deprecation process.

July 25, 2021

On Sunday, 25 July 2021 at 12:56:33 UTC, Bruce Carneal wrote:

>

As hopefully understood from my earlier comments, these are, qualitatively, not the same thing. You will still have to check a conversion to a new style @trusted function manually of course, no work savings there, but you'd gain something pretty important: the compiler's assertions regarding your remaining @safe code might actually mean something.

Memory safety is a global property. If even a single line of your new-style @system-block (or old-style @trusted lambda) causes undefined behavior, it does not matter one bit what the compiler asserts about the @safe code in your program: the entire process is corrupted.

« First   ‹ Prev
1 2 3 4 5 6 7 8