On Wednesday, October 30, 2024 9:25:06 AM MDT Manu via Digitalmars-d wrote:
> I can't think of any time I've ever written a lambda that I actually *want* to allocate a closure to access latent local scope data, but I almost always refer to sibling members of the calling function to implement some sort of callback response.

It's not uncommon - particularly with range-based code - to do something like return a range that accesses the calling stack from inside a lambda, and that range gets returned, e.g. something like

    return range.map!((a)
        {
            if(auto value = a in aa)
                return *value;
            return defaultValue;
        })();

So, it's definitely something that comes up. However, the big problem is that the compiler is utterly terrible at figuring out when it isn't needed, and there is no way to tell the compiler what you actually want to capture or whether you actually want a closure or not (not even an @system mechanism). And of course, because it's silent, the programmer has no clue that it's happening unless they put a bit of effort into verifying whether it's happening (which is particularly difficult in code that can't be @nogc). So, we end up with a ton of closures that we really don't want, which means a ton of stray allocations that the GC has to collect.

The current intention for Phobos v3 is to change how predicates are handled by algorithms in an effort to reduce the number of stray closures that we get, but it's still a fundamental problem with how the language works. It makes certain classes of things "just work", but it comes at a cost that tends to be invisible most of the time - which might be okay if the cost were worth paying most of the time, but while there are lots of lambdas that get written which do need access to their enclosing scope, there are probably more written which don't, and they're usually treated the same, so the cost is often not necessary. It's just that the compiler isn't smart enough to see that.

A related problem which comes up pretty frequently is the compiler deciding that it needs access to the frame pointer when it doesn't have it - the problem being that it can only have access to one such pointer, and there are frequently circumstances where it needs access to multiple (e.g. both the enclosing scope and a this reference). And then you're forced to figure out how to rework your code so that it isn't trying to access multiple scopes at once, which is always more of a pain.

I really don't know what the overall solution here is (and if I understand correctly, part of the problem is simply that the compiler needs more information than it has to determine that a closure isn't needed without the programmer's input), but the overall result is that while lambdas are often extremely useful, they frequently cause issues, and working around those issues is typically far less user-friendly (and far less obvious) than using a lambda.

This is definitely a problem which affects more than @nogc code, but it becomes much more obvious with @nogc code, because then you get errors about using the GC, whereas for most D programmers, they just silently end up with worse performance. And if they knew that that was happening, and the lambda made their life easier for it, then that's not necessarily a problem, but frequently, they don't know that it's happening, _and_ it's only happening because the compiler isn't smart enough to make it not happen. It would be really nice if we could improve the situation so that it didn't happen anywhere near as much as it does.

- Jonathan M Davis