On Sunday, 5 April 2020 at 14:25:27 UTC, Steven Schveighoffer wrote:
> On 4/5/20 8:11 AM, Stefan Koch wrote:
>> What do you guys think?
>
> It seems like a good idea, except what you will see is a linker error if you call it. Which is not very friendly or gives the right impression.
>
> Is there a way we can have the compiler give an error when it sees this during compile-time?
>
> -Steve

Yes. that is possible.

However actually it never happens :)

On 4/5/20 10:26 AM, Stefan Koch wrote:
> On Sunday, 5 April 2020 at 14:25:27 UTC, Steven Schveighoffer wrote:
>> On 4/5/20 8:11 AM, Stefan Koch wrote:
>>> What do you guys think?
>>
>> It seems like a good idea, except what you will see is a linker error if you call it. Which is not very friendly or gives the right impression.
>>
>> Is there a way we can have the compiler give an error when it sees this during compile-time?
>>
> 
> Yes. that is possible.

I'd be fully on board then.

> 
> However actually it never happens :)

What do you mean it never happens?

I get that code that builds and runs correctly today will build with this change. But development is full of writing code that doesn't work, running the compiler, and fixing the errors shown. When the error is a link error, this will be completely confusing.

-Steve

On Sunday, 5 April 2020 at 13:26:22 UTC, Basile B. wrote:
> On Sunday, 5 April 2020 at 12:11:23 UTC, Stefan Koch wrote:
>> Hi Guys,
>>
>> [...]
>>
>> What do you guys think?
>>
>> Cheers,
>>
>> Stefan
>
> I'd prefer a new func attribute `@ctfe`.
> Also I find the current implementation unplesant. Cant you just check the assertion when they are already visited elsewhere e.g in dmd.expressionsem.ExpressionSemanticVisitor.visit(AssertExp) ? Also what if we are in a contract ?

I agree that a function attribute makes more sense for a feature like this, however, this solution doesn't solve the general case (i.e. elide functions that are only called at compile-time but don't have assert(__ctfe) because they are available to be called at runtime).  Given that, I believe solving the general case is only a matter of implementation rather than a language change.  So I would hate to add a new language feature that would become obselete once the tooling is better.

So I say yes, I'd be fine with this "hack" to temporarily provide a path for programs to create CTFE-ONLY functions that won't go to code generation.  It's better than the status-quo and can be removed if we ever solve the general case.

On Sunday, 5 April 2020 at 12:11:23 UTC, Stefan Koch wrote:
> Hi Guys,
> ...
> What do you guys think?

Another, more radical and interesting option is to not perform codegen unless it's actually necessary. Instead of blacklisting `@ctfe` or `assert(__ctfe)` functions, have a whitelist of what functions are allowed to codegen-ed.

Specifically, do not perform codegen for any function unless that function:

A) is marked, or inferred as `export` or called from such function (transitively)
B) is part of an aggregate (module, union, class, struct, interface, template) marked as `export`, or called from a function part of such
(By module marked as `export` I mean `export module foo;`, as opposed to `module foo; export:`)
C) is `main`, or it's called from `main` (in these cases it's inferred as `export`)
D) is a `unittest`, or called from one (under `version(unittest)` `unittest`s (and functions they call) are inferred as `export`)
E) is marked as `extern($lang)` or called from such function (again inferred as `export` in such cases)
F) is virtual, or called from such function (again inferred as `export` in those cases)

`export` must be a function storage class orthogonal to protection modifiers, just like `extern`.

Yeah, it will break the world, so this new compilation strategy should be opt-in via a switch (e.g. `-ix`, `-lazy`, `--as-needed`, etc.). I suspect it will play nicely with the existing `-i` ;)

Implications? From top of my head:
1. No need for a new `@cttfe` function attribute, or to treat `assert(__ctfe)` in special way. Though, on second thoughts, perhaps it should be an error to have non-conditional `assert(__ctfe)` inside `export`-ed functions.
2. Regular functions gain some of the special powers of template ones - specifically, they're codegen-ed only if needed. Before, to make druntime/phobos functions accessible from DasBetterC we had to make them templates. Now this awkward technique (what's the point of having a template with no parameters? Non-template auto-returning functions also have attribute inference) is no longer needed.
3. Compilation speed and code bloat are reduced
4. D libraries become kind of like header-only C++ libraries.
5. Until now object files (or static libraries) were a primary build caching technique. Now a per-function caching technique would be much more helpful.

On Sunday, 5 April 2020 at 18:25:10 UTC, Petar Kirov [ZombineDev] wrote:
> On Sunday, 5 April 2020 at 12:11:23 UTC, Stefan Koch wrote:
>> [...]
>
> Another, more radical and interesting option is to not perform codegen unless it's actually necessary. Instead of blacklisting `@ctfe` or `assert(__ctfe)` functions, have a whitelist of what functions are allowed to codegen-ed.
>
> [...]

Seems like you are reading my mind.
Indeed this is one of next steps ...
assert(__ctfe) did not break a single package, in out CI!
Therefore I do now have enough confidence in crawling the dynamic dependencies outwards from the entry point and exported functions.

That however is for the future and will require me to do some theoretical work, in order to proof to myself that it actually is feasible and won't end up in a nightmare of subtly broken builds.

On Sunday, 5 April 2020 at 18:30:25 UTC, Stefan Koch wrote:
> On Sunday, 5 April 2020 at 18:25:10 UTC, Petar Kirov [ZombineDev] wrote:
>> On Sunday, 5 April 2020 at 12:11:23 UTC, Stefan Koch wrote:
>>> [...]
>>
>> Another, more radical and interesting option is to not perform codegen unless it's actually necessary. Instead of blacklisting `@ctfe` or `assert(__ctfe)` functions, have a whitelist of what functions are allowed to codegen-ed.
>>
>> [...]
>
> Seems like you are reading my mind.
> Indeed this is one of next steps ...
> assert(__ctfe) did not break a single package, in out CI!
> Therefore I do now have enough confidence in crawling the dynamic dependencies outwards from the entry point and exported functions.
>
> That however is for the future and will require me to do some theoretical work, in order to proof to myself that it actually is feasible and won't end up in a nightmare of subtly broken builds.

Great! Perhaps we should write a DIP to specify the semantics of such a lazy compilation model? An even more extreme option would be to not perform semantic analysis on non-`export`-ed functions. This should bring some really high gains, though it would be the trickiest to adopt option for users.

On 05.04.20 14:11, Stefan Koch wrote:
> Hi Guys,
> 
> I am currently working on speeding up dmd. In the presence of templates and ctfe.
> and one thing that stood out is that we do codegen for symbols which are never used.
> (and hopefully eliminated by a smart linker but eh ...)
> 
> I've seen the following pattern in the wild.
> 
> 
> string GenerateMixin(string[] params)
> {
>      assert(__ctfe);
>      .... more code ....
> }
> 
> that means if we see assert(__ctfe) inside the outermost scope of a function body ( not in a branch) then we know that this function would assert if called at runtime.

Make sure it works with `in` contracts:

string generateMixin(string[] params)in{
    assert(__ctfe);
}do{
    .... more code ....
}

(The plain assertion slightly hurts my eyes, because if it fails, that failure would be attributable to the GenerateMixin function, whereas with the `in` contract it is clear that the caller is at fault.)

On Sunday, 5 April 2020 at 20:31:15 UTC, Timon Gehr wrote:
> On 05.04.20 14:11, Stefan Koch wrote:
>> [...]
>
> Make sure it works with `in` contracts:
>
> string generateMixin(string[] params)in{
>     assert(__ctfe);
> }do{
>     .... more code ....
> }
>
> (The plain assertion slightly hurts my eyes, because if it fails, that failure would be attributable to the GenerateMixin function, whereas with the `in` contract it is clear that the caller is at fault.)

Oh good idea.
I don't use contracts myself but that's easy to add.

On 05.04.20 22:39, Stefan Koch wrote:
> On Sunday, 5 April 2020 at 20:31:15 UTC, Timon Gehr wrote:
>> On 05.04.20 14:11, Stefan Koch wrote:
>>> [...]
>>
>> Make sure it works with `in` contracts:
>>
>> string generateMixin(string[] params)in{
>>     assert(__ctfe);
>> }do{
>>     .... more code ....
>> }
>>
>> (The plain assertion slightly hurts my eyes, because if it fails, that failure would be attributable to the GenerateMixin function, whereas with the `in` contract it is clear that the caller is at fault.)
> 
> Oh good idea.
> I don't use contracts myself but that's easy to add.

Great, thanks! :)

On Sunday, 5 April 2020 at 20:31:15 UTC, Timon Gehr wrote:
> On 05.04.20 14:11, Stefan Koch wrote:
>> Hi Guys,
>> 
>> I am currently working on speeding up dmd. In the presence of templates and ctfe.
>> and one thing that stood out is that we do codegen for symbols which are never used.
>> (and hopefully eliminated by a smart linker but eh ...)
>> 
>> I've seen the following pattern in the wild.
>> 
>> 
>> string GenerateMixin(string[] params)
>> {
>>      assert(__ctfe);
>>      .... more code ....
>> }
>> 
>> that means if we see assert(__ctfe) inside the outermost scope of a function body ( not in a branch) then we know that this function would assert if called at runtime.
>
> Make sure it works with `in` contracts:
>
> string generateMixin(string[] params)in{
>     assert(__ctfe);
> }do{
>     .... more code ....
> }
>
> (The plain assertion slightly hurts my eyes, because if it fails, that failure would be attributable to the GenerateMixin function, whereas with the `in` contract it is clear that the caller is at fault.)

We likely get the shorthand version for free as well:
string generateMixin(...)
in(__ctfe)
{
   ...
}