I hear people complain about compiler magic a lot. Yes, being able to do everything in-language is nice, but compiler magic is inevitable and also can be very useful. assert
is my favourite example. Things like Python’s print
are more dubious. You can always make a better print function on your own, right? Whereas one assert is never going to transcend another assert, even if the way it prints its assertion failure is slow, who cares? The program is already functionally dead anyway. Do we really want C’s ‘everyone assert for themself’ problem?
And besides that, isn’t ‘compiler magic’ at its logical conclusion generally applicable to any task performed by the compiler? Exception handling, code optimisation, inlining, template expansion, new
, adding two numbers? Compiler magic.
Thread overview | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
August 02 Fear of Compiler Magic | ||||
---|---|---|---|---|
| ||||
August 02 Re: Fear of Compiler Magic | ||||
---|---|---|---|---|
| ||||
Posted in reply to IchorDev | On 8/2/24 05:22, IchorDev wrote: > I hear people complain about compiler magic a lot. Probably this is partially inspired from here: https://dconf.org/2018/talks/alexandrescu.html > Yes, being able to do everything in-language is nice, but compiler magic is inevitable and also can be very useful. `assert` is my favourite example. int foo(){ enforce(0); } // error int foo(){ assert(0); } // ok "No compiler magic" would e.g. mean: `enforce` can similarly influence definite return analysis. It's not inevitable that this is impossible. > Things like Python’s `print` are more dubious. You can always make a better print function on your own, right? Whereas one assert is never going to transcend another assert, even if the way it prints its assertion failure is slow, who cares? The program is already functionally dead anyway. Do we really want C’s ‘everyone assert for themself’ problem? Well, this is what assert does. The question is how it achieves it, and whether the same tools are accessible to user code that perhaps does _something different than assert_. > And besides that, isn’t ‘compiler magic’ at its logical conclusion generally applicable to *any* task performed by the compiler? Not if the tasks are properly decomposed into orthogonal components that are also available to the user. Anyway, there is quite a bit of existing magic, and it does cause some issues. |
August 02 Re: Fear of Compiler Magic | ||||
---|---|---|---|---|
| ||||
Posted in reply to IchorDev | On Friday, 2 August 2024 at 03:22:28 UTC, IchorDev wrote: >And besides that, isn’t ‘compiler magic’ at its logical conclusion generally applicable to any task performed by the compiler? Exception handling, code optimisation, inlining, template expansion, A good programming language consists of a small set of orthogonal features that combine into a powerful language. Magic features are situational. They add the maintenance burden of an orthogonal feature, but instead of multiplying the language's expressiveness, they only add a constant to it. So what happens in practice when you pile on magic features? To keep technical debt of large code bases under control, refactoring needs to happen. Refactoring relies on performing code transformations that result in equivalent semantics. As a programmer, you want to have actions you can always do safely, based on general facts:
However, proposals for magic features tend to add more and more exceptions. I have seen proposals that would make You would think it's safe to transform this:
Into this:
But with magic printf format string rewrites, that transformation turns correct code into memory corrupting code when x is an int. This doesn't mean magic features are always a bad idea. The In practice of course, p.s. It even used to be the case that |
August 03 Re: Fear of Compiler Magic | ||||
---|---|---|---|---|
| ||||
Posted in reply to Timon Gehr | On 8/2/2024 1:34 AM, Timon Gehr wrote:
> Well, this is what assert does. The question is how it achieves it, and whether the same tools are accessible to user code that perhaps does _something different than assert_.
One reason for some of the builtin stuff is to not tempt people to write their own. Having standardized ways to do common tasks is a big win for making code understandable by others, which advantageous in a team environment.
For example, the `debug` conditionals came about from my discussions with a veteran Microsoft programming manager. He complained that every project invented their own scheme for doing debug conditionals, making it unnecessarily difficult to share code.
Unittests and Ddoc are other successful examples.
Lisp is a language that enables building one's one programming language on top of it. It more or less requires it.
The result is every Lisp user invents their own language, incompatible with any other Lisp user, and so successful Lisp programs don't survive their creators. It's why Lisp has never really caught on, much to the bafflement of Lisp advocates.
|
August 03 Re: Fear of Compiler Magic | ||||
---|---|---|---|---|
| ||||
Posted in reply to Dennis | On 8/2/2024 2:29 AM, Dennis wrote:
> You would think it's safe to transform this:
> ```D
> printf("x = %s\n", x);
> printf("x = %s\n", x);
> ```
>
> Into this:
> ```D
> const(char)* fmt = "x = %s\n";
> printf(fmt, x);
> printf(fmt, x);
> ```
>
> But with magic printf format string rewrites, that transformation turns correct code into memory corrupting code when x is an int.
The transformation won't compile if the call is marked @safe, and won't compile with the various proposals to increase the default safety-ness.
It is in the same box as:
```
int[] array;
x = array[5];
```
and rewriting as:
```
int[] array;
x = *(array.ptr + 5);
```
|
August 03 Re: Fear of Compiler Magic | ||||
---|---|---|---|---|
| ||||
Posted in reply to Walter Bright | On Saturday, 3 August 2024 at 17:02:55 UTC, Walter Bright wrote: >The transformation won't compile if the call is marked @safe, and won't compile with the various proposals to increase the default safety-ness. This is an interesting aspect that I forgot to mention: Whenever I bring up such examples of problematic cases resulting from 'magic', the defense is often akin to: "Sure, it fails in that theoretical / pathological case, but when are you going to find that in REAL code?" Which can be fair. Like I said, a bit of magic doesn't always have to be problematic. And while I can't tell you when or how these problem cases are going to crop up, do beware that there might just be plenty of opportunity, just from a statistical standpoint. It's like Intel saying, in response to the Pentium FDIV bug: "Well, when are you going to divide 4195835 by 3145727 needing all the precision? Give me an example of when you would divide those specific numbers in a REAL application." That's hard to answer upfront, but with enough users doing enough floating point math, eventually you get some great stories: When working on Quake, Michael Abrash spent hours tracking down a graphical glitch, until finding out with the help of a friend from Intel that it was the infamous hardware bug. Going back to printf, it's possible hardly anyone will ever hit this problem. But when eventually there's thousands of calls to magic printf out there (or snprintf which still can't be called in So you could try some sort of risk assessment for magic features, but I prefer to just avoid them as much as possible and look for alternatives. >It is in the same box as: That example is different, because the first program wasn't correct to begin with. Or if it were, then the refactoring would result in the second program also being correct. In my example, only the first program was correct. |
August 04 Re: Fear of Compiler Magic | ||||
---|---|---|---|---|
| ||||
Posted in reply to Walter Bright | On 8/3/24 18:54, Walter Bright wrote: > On 8/2/2024 1:34 AM, Timon Gehr wrote: >> Well, this is what assert does. The question is how it achieves it, and whether the same tools are accessible to user code that perhaps does _something different than assert_. > > One reason for some of the builtin stuff is to not tempt people to write their own. Having standardized ways to do common tasks is a big win for making code understandable by others, which advantageous in a team environment. > ... Sure, I am in favor of built-in assert. This thread is however about magic. > For example, the `debug` conditionals came about from my discussions with a veteran Microsoft programming manager. He complained that every project invented their own scheme for doing debug conditionals, making it unnecessarily difficult to share code. > > Unittests and Ddoc are other successful examples. > ... Off-topic, but yes. > Lisp is a language that enables building one's one programming language on top of it. It more or less requires it. > > The result is every Lisp user invents their own language, incompatible with any other Lisp user, and so successful Lisp programs don't survive their creators. It's why Lisp has never really caught on, much to the bafflement of Lisp advocates. Sure. |
August 04 Re: Fear of Compiler Magic | ||||
---|---|---|---|---|
| ||||
Posted in reply to Walter Bright | On 8/3/24 19:02, Walter Bright wrote: > On 8/2/2024 2:29 AM, Dennis wrote: >> You would think it's safe to transform this: >> ```D >> printf("x = %s\n", x); >> printf("x = %s\n", x); >> ``` >> >> Into this: >> ```D >> const(char)* fmt = "x = %s\n"; >> printf(fmt, x); >> printf(fmt, x); >> ``` >> >> But with magic printf format string rewrites, that transformation turns correct code into memory corrupting code when x is an int. > > The transformation won't compile if the call is marked @safe, and won't compile with the various proposals to increase the default safety-ness. > ... The simple fact is that is that the magic treatment of the string-literal leads to some trouble. I.e., this is a good illustration about how magic instills fear. > It is in the same box as: > > ``` > int[] array; > x = array[5]; > ``` > > and rewriting as: > > ``` > int[] array; > x = *(array.ptr + 5); > ``` Not at all. You just orthogonally removed the range check. This is a completely unrelated case. Nothing surprising happens here. |
August 05 Re: Fear of Compiler Magic | ||||
---|---|---|---|---|
| ||||
Posted in reply to Timon Gehr | On Friday, 2 August 2024 at 08:34:39 UTC, Timon Gehr wrote: >On 8/2/24 05:22, IchorDev wrote: >I hear people complain about compiler magic a lot. Probably this is partially inspired from here: Yes, being able to do everything in-language is nice, but compiler magic is inevitable and also can be very useful. int foo(){ enforce(0); } // error "No compiler magic" would e.g. mean: This specific case could be solved with Enum Parameters: |
August 05 Re: Fear of Compiler Magic | ||||
---|---|---|---|---|
| ||||
Posted in reply to Timon Gehr | On Sunday, 4 August 2024 at 17:41:52 UTC, Timon Gehr wrote: >On 8/3/24 19:02, Walter Bright wrote: >On 8/2/2024 2:29 AM, Dennis wrote: >You would think it's safe to transform this:
Into this:
But with magic printf format string rewrites, that transformation turns correct code into memory corrupting code when x is an int. The transformation won't compile if the call is marked @safe, and won't compile with the various proposals to increase the default safety-ness. The simple fact is that is that the magic treatment of the string-literal leads to some trouble. I.e., this is a good illustration about how magic instills fear. And it’s why I suggested using |