Permalink
Reply
Sorry for the attention-grabbing title, but I think it's warranted, because the gist of it is this:
With -preview=dip1000 enabled, the compiler will happily compile valid, @safe D code into memory corrupting machine code.
The root cause is:
Issue 20150 - -dip1000 defeated by pure
The compiler ignores "reference to local variable x assigned to non-scope parameter y" errors when the function is annotated or inferred pure. The idea is, presumably, that pure functions can't escape references because they have no interaction with global variables. This is false of course, since they can still return them or assign them to other parameters.
The deadly part it that using this flawed logic, the compiler sometimes turns GC allocations into stack allocations too eagerly. Here I got memory corruption because the compiler allocated an array literal on the stack instead of the heap:
Issue 21291 - Array literal that escapes scope is allocated on stack
Later I encountered another instance of it where a closure was not heap-allocated, which looked something like this:
import core.thread;
@safe:
void main() {
S s;
s.memberFunc();
}
struct S {
int a;
auto memberFunc() {
auto t = new Thread({
auto pa = &a; // pointer to stack frame of main!
});
}
}
I'm not the only one who encountered memory corruption bugs this way, user Dechcaudron commented on my issue: "This has also happened to me, no idea it could be due to -dip1000".
And most recently:
Issue 21912 - Invalid stack closure when calling delegate inside lambda
Why is this not fixed?
Walter made a PR for fixing the behavior in dmd: (March 2020)
https://github.com/dlang/dmd/pull/10924
Later, aG0aep6G made a better fix: (November 2020)
https://github.com/dlang/dmd/pull/12010
But they're both blocked by the fact that Phobos relies on the bug to compile with -dip1000. This makes sense, because the conversion process was mostly "add scope and return annotations until the compile errors go away". pure functions did not give error messages, so they did not get those annotations.
Regarding this extra work, aG0aep6G commented: (January 2021)
>I had started on it, but it's tedious work tracking down the errors through templates and overloads. If I remember correctly, dup gave me some trouble, too.
So I've put it on ice for the time being. If someone else wants to give it a shot, that would be great.
And that's where we are now.
Future of dip1000
Matthias asked "Is there a plan to enable DIP1000 by default?" during DConf Online 2020 Day One Q & A Livestream, at 4:50:11.
Walter mentioned "we can do it now" and Atila mentioned how the first step would be to change -dip1000 errors into equivalent deprecation warnings.
Clearly, issue 20150 is a blocker for dip1000 by default.
In the meantime, since I absolutely don't want another unfortunate soul debugging memory corruption bugs that dip1000 introduces, this post is meant to raise awareness, and discuss intermediate solutions.
Maybe the compiler can defensively heap-allocate for now, though that would break @nogc code. Or maybe we can add another switch, -preview=dip1000proper, since the fix is a breaking change. What do you think?
