5 days ago

On Tuesday, 16 September 2025 at 08:50:00 UTC, Dennis wrote:

>

On Saturday, 13 September 2025 at 16:12:06 UTC, Richard (Rikki) Andrew Cattermole wrote:

>

The fact that D is not architectured to be able to stop these false positives is an intended bug in our design. Not all PL's work like this the ML family certainly don't. They tie very advanced analysis engines including DFA into their type system.

So the answer to my question is: this it not motivated by reported issues from D users, but by other languages doing this. If the only example for D you can give is an "unrealistic" one, can you please give a realistic example from another language then?

I am not trying to improve D based upon what other languages can do. They can only be an inspiration. For instance I take note of how people view other languages and compilers solutions both historically and in real time.

> >

Just because people don't understand that this is possible, doesn't mean it hasn't been an issue. I see it as a case that people don't know what they don't know. So they don't complain.
This is a hole that the C family relies upon having due to historical reasons and so hasn't reached common knowledge.

I understand DFA is an interesting research topic, and if your claim was that it might lead to interesting discoveries or new paradigms I fully agree. But when you say it's necessary to fix DIP1000 specifically, I need some evidence to believe it. There's been over 100 DIP1000-related bug reports with simple non-DFA solutions, and I'm skeptical that people 'don't know' to report DFA related issues. Several users reported issues about Value Range Propagation (VRP) not working across statements. Even when they don't explicitly know DFA, they still experience a problem and can intuit a solution being available.

In my last post, which was a giant wall of text, I explained how this isn't the case. Noting templates exhibit some of the underlying issues that DIP1000 has. https://forum.dlang.org/post/10abme4$2ekj$1@digitalmars.com

Both me and Walter seem to have failed to convey these failures in dmd's architecture.

I will be ignoring the lacking features in DIP1000's attribution capabilities. This is purely engine implementation design stuff.

Here are some code similar to what I've sent Walter over the past year for what I want to work by default for escape analysis in D. These are backed up by all the debugging and helping of other people over the years. This will give the best experience over all I expect.

extern(C) int printf(const char*, ...);

void myFunc1(scope const char* str) {
    printf("Text: %s\n", str); // ok
}

This works by acknowledging that variable state isn't "set" and "not-set" for any given property, its "set", "not-set", and "unknown" at a minimum.

My fast DFA engine does this, you can see it in code that looks like:

void aPrototype(ref bool);

void myFunc2() {
    bool b;
    aPrototype(b);
    assert(b); // ok
}

The truthiness value went from false, to unknown with the function call not being attributed.

Escape into a global variable:

int* global;

void myFunc3(scope int* ptr) {
    global = ptr; // Error: ptr going into global escapes
}

Might not error (depends on how it plays out):

int* global;

struct Foo {
    int* field;

    void method() {
    	global = field;
    }
}

void myFunc4(scope int* ptr) {
    Foo foo;
    foo.field = ptr;
    foo.method;
}

The point of this set of tunings isn't because I want to have opinions.
The point is to get a subset of escape analysis turned on by default that the community will find value in. Without hitting any of the many issues that they have perceptually had with DIP1000.

To implement this, I did review DIP1000 and concluded that it would be a rewrite. It simply isn't designed to handle these kind of properties. At which point you might as well have a DFA for the escape analysis to live in, and up the ceiling of capabilities.

At this months monthly meeting I asked if people would find value in the myFunc3 error, hands did go up. I also raised the double-duty of scope wrt. stack allocation.

Fact is people find value in a subset of errors from escape analysis, but it isn't the same subset that Atila is proposing with his "no inferred scope variable reports" approach. If I'm not convincing for any reason, I suggest doing a survey to see how it plays out. I expect you will get four groups:

  1. No protection (@system only), tiny minority
  2. False positive heavy, as much protection as possible default, tiny minority
  3. False positive heavy, as much protection as possible opt-in, small minority
  4. A subset of protection (@safe and the above examples), the majority

These are based upon my previous survey.

If you can do this without a full rewrite of DIP1000 I will be impressed. However, you will still need to solve borrowing so that I can get reference counting.

5 days ago

On Tuesday, 16 September 2025 at 14:21:07 UTC, Richard (Rikki) Andrew Cattermole wrote:

>

Here are some code similar to what I've sent Walter over the past year for what I want to work by default for escape analysis in D.

You gave 4 code snippets. 1 and 2 already compile as expected, 3 already errors as expected if you add @safe, and 4 has no defined expected behavior. None of these have a source linking them to an existing D user / code base.

To reiterate, the question is:

>

Can you think of any more realistic examples where DFA allows you to infer safety where DIP1000 is currently limited?

As long as you keep dancing around it, I'm going to assume the answer is 'no'.

5 days ago
On 17/09/2025 3:52 AM, Dennis wrote:
> On Tuesday, 16 September 2025 at 14:21:07 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> Here are some code similar to what I've sent Walter over the past year for what I want to work by default for escape analysis in D.
> 
> You gave 4 code snippets. 1 and 2 already compile as expected, 3 already errors as expected if you add `@safe`, and 4 has no defined expected behavior. None of these have a source linking them to an existing D user / code base.
> 
> To reiterate, the question is:

As expected.

These examples show true positives. The problem is eliminating perceived false positives but keeping the intended true positives.

>> Can you think of any more realistic examples where DFA allows you to infer safety where DIP1000 is currently limited?
> 
> As long as you keep dancing around it, I'm going to assume the answer is 'no'.

Okay fine.
Bugzilla is down, so I can't look through the WONTFIX's and I'm not going to go hunting on the N.G. learn forum.

I can spot: https://github.com/dlang/dmd/issues/19679
You tried to fix it and haven't in the past. With a DFA for the infrastructure, that wouldn't be possible to have existed.

Another example from you for the above: https://github.com/dlang/dmd/issues/18113

Here is a ticket, with a comment from you showing why DIP1000 needs to be lint level attribution. https://github.com/dlang/dmd/issues/20302#issuecomment-2542028404
If it isn't you break things like the .di generator since it runs before semantic.

It also results in issues like: https://github.com/dlang/dmd/issues/19853

Another example where DIP1000 loses track of lifetimes: https://github.com/dlang/dmd/issues/18153
Pretty nasty.
To fix that you would need to keep track of a stack of lifetimes, with full knowledge of the variables in a function signature. All things a DFA would be good at.

An example where appropriate CFG processing with a DFA would allow this to not error: https://github.com/dlang/dmd/issues/19659
Sadly my fast DFA isn't able to model it so the statement walker marks the variable as unmodellable which prevents the false positive.
To model this across function boundaries, we need to add an escape set in the CFG of blockexit and on the function declaration.
5 days ago

On Tuesday, 16 September 2025 at 17:09:18 UTC, Richard (Rikki) Andrew Cattermole wrote:

>

Okay fine.
Bugzilla is down, so I can't look through the WONTFIX's and I'm not going to go hunting on the N.G. learn forum.

Thanks a lot! I don't need many, I just need 1 good one, so you delivered already.

>

I can spot: https://github.com/dlang/dmd/issues/19679
You tried to fix it and haven't in the past. With a DFA for the infrastructure, that wouldn't be possible to have existed.

Another example from you for the above: https://github.com/dlang/dmd/issues/18113

The escape checking logic for assigning a parameter with scope inference to a local variable is bad, definitely. Like you say I've been working on a fix which remains unfinished because:

  • dmd/escape.d is full of duplicated spaghetti logic (though it has improved)
  • The test suite and buildkite projects rely on existing quirks
  • dip1000 by default is no longer a priority issue, so I shifted work to more important stuff

Unless you consider my refactoring and WIP solution a rewrite/DFA, I wouldn't say a DFA rewrite is required, but I certainly was tempted to throw out dmd/escape.d and rewrite that hot mess from scratch 🙂.

>

Here is a ticket, with a comment from you showing why DIP1000 needs to be lint level attribution. https://github.com/dlang/dmd/issues/20302#issuecomment-2542028404

You haven't defined what "lint level attribution" means, but that issue would be solved if we add inferred scope parameters to a function type's mangle in a next edition so I don't see how this issue necessitates a rewrite/DFA.

>

If it isn't you break things like the .di generator since it runs before semantic.

The .di generator also needs to run semantic to emit correct inferred function attributes, that change doesn't require rewriting DIP1000 / adding DFA.

>

It also results in issues like:
https://github.com/dlang/dmd/issues/19853

That issue is invalid, see closing comment.

>

Another example where DIP1000 loses track of lifetimes: https://github.com/dlang/dmd/issues/18153
To fix that you would need to keep track of a stack of lifetimes, with full knowledge of the variables in a function signature. All things a DFA would be good at.

I haven't studied that issue, but I suspect it has more to do with the internal opApply rewrite than a lack of DFA.

>

To model this across function boundaries, we need to add an escape set in the CFG of blockexit and on the function declaration.

How would you throw a scope Exception across function boundaries? I can't picture that being valid but I haven't thought very hard.


So far, the most representative issue is definitely https://github.com/dlang/dmd/issues/19679.

I agree that a rewrite using a DFA engine would be sufficient to handle that case, I just don't think it's necessary. And that's because I'm noticing a theme in the code snippets you use to showcase your DFA engine vs. the reduced code snippets from the bug reports you just listed. Notice how unlike DFA examples, the DIP1000 bugs are not about asserts, if-statements or loops. The problematic code almost always exists inside the same basic block. That's why I think considering a function's control-flow graph is currently overkill for DIP1000.

5 days ago
On 17/09/2025 8:00 AM, Dennis wrote:
> On Tuesday, 16 September 2025 at 17:09:18 UTC, Richard (Rikki) Andrew Cattermole wrote:
>> Okay fine.
>> Bugzilla is down, so I can't look through the WONTFIX's and I'm not going to go hunting on the N.G. learn forum.
> 
> Thanks a lot! I don't need many, I just need 1 good one, so you delivered already.

I wish I could have offered more in WONTFIX, it would have been illuminating in false positive/perceived false positive territory.
>> I can spot: https://github.com/dlang/dmd/issues/19679
>> You tried to fix it and haven't in the past. With a DFA for the infrastructure, that wouldn't be possible to have existed.
>>
>> Another example from you for the above: https://github.com/dlang/dmd/ issues/18113
> 
> The escape checking logic for assigning a parameter with scope inference to a local variable is bad, definitely. Like you say I've been working on a fix which remains unfinished because:
> 
> - dmd/escape.d is full of duplicated spaghetti logic (though it has improved)
> - The test suite and buildkite projects rely on existing quirks
> - dip1000 by default is no longer a priority issue, so I shifted work to more important stuff
> 
> Unless you consider my refactoring and WIP solution a rewrite/DFA, I wouldn't say a DFA rewrite is *required*, but I certainly was tempted to throw out dmd/escape.d and rewrite that hot mess from scratch 🙂.Is Atila aware that DIP1000 implementation has these issues requiring a 
rewrite which you determined independently of me?

I suspect he is operating under the assumption that to get it on by default is just a matter of tweaking.
>> It also results in issues like:
>> https://github.com/dlang/dmd/issues/19853
> 
> That issue is invalid, see closing comment.

During this I found a few that needed closing as they were out of date. Sent to Nic to close since I don't have rights to do so.

>> Another example where DIP1000 loses track of lifetimes: https:// github.com/dlang/dmd/issues/18153
>> To fix that you would need to keep track of a stack of lifetimes, with full knowledge of the variables in a function signature. All things a DFA would be good at.
> 
> I haven't studied that issue, but I suspect it has more to do with the internal opApply rewrite than a lack of DFA.
> 
>> To model this across function boundaries, we need to add an escape set in the CFG of blockexit and on the function declaration.
> 
> How would you throw a `scope` Exception across function boundaries? I can't picture that being valid but I haven't thought very hard.

Something akin to:

```d
void myThrow(return MyException e) @throws(MyException) {
	throw e;
}

void caller() {
	scope e = new MyException;
	try {
		myThrow(e);
	} catch(Exception) {
	}
}
```

Why would someone do that? I don't know.

However the main thing is to map function calls to their corresponding catches, allowing the CFG analysis to do its job. Impossible for it to work correctly otherwise in regards to catch statements.

Otherwise any try block with function calls to throwing functions has to be marked the catches unmodellable. Which is rather limiting.

> ---
> 
> So far, the most representative issue is definitely https://github.com/ dlang/dmd/issues/19679.
> 
> I agree that a rewrite using a DFA engine would be sufficient to handle that case, I just don't think it's necessary. And that's because I'm noticing a theme in the code snippets you use to showcase your DFA engine vs. the reduced code snippets from the bug reports you just listed. Notice how unlike DFA examples, the DIP1000 bugs are not about asserts, if-statements or loops. The problematic code almost always exists inside the same basic block. That's why I think considering a function's control-flow graph is currently overkill for DIP1000.

Of course that trend exists, I'm currently working on the truthiness & nullability features!Its all about CFG analysis. But there is a reason for it. Each analysis feeds into each other, giving better results over all.

5 days ago

On Tuesday, 16 September 2025 at 23:02:05 UTC, Richard (Rikki) Andrew Cattermole wrote:

>

I wish I could have offered more in WONTFIX, it would have been illuminating in false positive/perceived false positive territory.

My search gives:

  • 18295 [Scope][dip1000] scope class check too conservative under -dip1000
  • 18637 [scope][DIP1000] "copying & i into allocated memory escapes a reference to local variable i" where it's inappropriate
  • 19301 [DIP1000] missing overload abilities
  • 23941 [DIP1000] Overloading by scope should be allowed

There's nothing interesting there.

>

Is Atila aware that DIP1000 implementation has these issues requiring a
rewrite which you determined independently of me?

I said I was tempted to rewrite, not that it is required. You have to ask Atila whether he is aware, but I did talk about the DIP1000s issues and implementation quality at DConf '23 and '24.

>

I suspect he is operating under the assumption that to get it on by default is just a matter of tweaking.

It is. The thing that's blocking DIP1000 being enabled by default is that existing code relies on slicing local static arrays being freely allowed in @safe functions, but that could be disallowed in a next edition, unless you explicitly opt into scope semantics.

1 2 3 4
Next ›   Last »