April 07, 2020
Am Mon, 06 Apr 2020 13:49:19 +0000 schrieb Atila Neves:

> On Monday, 6 April 2020 at 13:41:52 UTC, Stefan Koch wrote:
>> On Monday, 6 April 2020 at 10:14:24 UTC, Atila Neves wrote:
>>> On Sunday, 5 April 2020 at 12:11:23 UTC, Stefan Koch wrote:
>>>> [...]
>>>
>>> Wouldn't it be easier to skip codegen for private functions that are never called from non CTFE contexts?
>>
>> Easier from the user-perspective yes.
>> From the compiler perspective,
>> That's another step which may take quite a while to do correctly.
>> The easy thing would be (Essentially an (N*M) loop over all calls and
>> functions),
> 
> Where N and M are all calls and private functions in one module, not all code in a project.
> 

In fact, you can't do that in current D. I thought about it some time ago, consider this valid D code:

---------------------------
module foo;

private void privateFunc() {}

public void publicTemplate(T)()
{
    privateFunc();
}
---------------------------

Let's say I compile module foo into libfoo. libfoo internally never uses publicTemplate. An external user instantiates publicTemplate ==> privateFunc needs to be emitted in libfoo.

Now you can't even check that privateFunc is never called, cause you can't analyze the template without instantiating it. And in general, you can't instantiate it without knowing the template instance type parameters. (In this simple example you might think this should work. But just imagine that I can pass in a type with an enum string member, which I can then mixin in the template, ....)


Generally the only way to solve this seems to finally implement export properly, as it was proposed years ago. Then privateFunc should be flagged export (needed for windows DLL support and linux .so selective exports anyway) and you could probably strip uncalled, non-exported private functions.


TLDR: It's certainly not easy to implement.

-- 
Johannes
April 07, 2020
On Tuesday, 7 April 2020 at 11:32:27 UTC, Johannes Pfau wrote:
>
> And the most important thing: This will vastly improve betterC / nogc code:
>
> --------------------
> string generateMixin(T)(string b)
> {
>     assert(__ctfe);
>     return T.stringof ~ " " ~ b ~ ";";
> }
>
> void main() @nogc
> {
>     mixin(generateMixin!string("b"));
> }
> --------------------


I can confirm with the patch there are no template-instances in the object file

with patch:
0000000000000000 R _D6t_ctfe12__ModuleInfoZ
                 U _d_dso_registry
0000000000000000 W _Dmain
                 U _Dmain
                 U _d_run_main
                 U _GLOBAL_OFFSET_TABLE_
0000000000000000 W main
                 U _main
                 U __start_minfo
                 U __stop_minfo

without patch:
                 U _D12TypeInfo_Aya6__initZ
0000000000000000 R _D6t_ctfe12__ModuleInfoZ
0000000000000000 W _D6t_ctfe__T13generateMixinTAyaZQuFNaNbNfQnZQq
                 U _d_arraycatnTX
                 U _d_assertp
                 U _d_dso_registry
0000000000000000 W _Dmain
                 U _Dmain
                 U _d_run_main
                 U _GLOBAL_OFFSET_TABLE_
0000000000000000 W main
                 U _main
                 U __start_minfo
                 U __stop_minfo
0000000000000000 r _TMP0
0000000000000009 r _TMP1
000000000000000b r _TMP2

April 07, 2020
On Tuesday, 7 April 2020 at 09:16:34 UTC, Atila Neves wrote:
> On Tuesday, 7 April 2020 at 06:53:18 UTC, Walter Bright wrote:
>> There's no reason the CTFE-only function can't be global.
>
> No, there isn't. But in my experience nearly all functions I write that are only ever called from CTFE returning a string to be mixed in are private.

I've done it both ways. If it can be accomplished easily enough might as well support both equally.
April 07, 2020
On Tuesday, 7 April 2020 at 12:09:48 UTC, Stefan Koch wrote:
> I can confirm with the patch there are no template-instances in the object file
>

Very cool! Thanks.
April 07, 2020
Am Tue, 07 Apr 2020 12:09:48 +0000 schrieb Stefan Koch:

> On Tuesday, 7 April 2020 at 11:32:27 UTC, Johannes Pfau wrote:
>>
>> And the most important thing: This will vastly improve betterC / nogc code:
>>
>> --------------------
>> string generateMixin(T)(string b)
>> {
>>     assert(__ctfe);
>>     return T.stringof ~ " " ~ b ~ ";";
>> }
>>
>> void main() @nogc {
>>     mixin(generateMixin!string("b"));
>> }
>> --------------------
> 
> 
> I can confirm with the patch there are no template-instances in the object file

The example above actually already compiles with -betterC, as the check is done in codegen phase! But some betterC checks seem to be done in semantic. This doesn't work yet:

------------------------
string generateMixin(T)(string b)
{
    assert(__ctfe);
    return typeid(T).stringof ~ " " ~ b ~ ";";
}

void main() @nogc
{
    mixin(generateMixin!string("b"));
}
------------------------
dmd -betterC test.d -c
test.d(4): Error: TypeInfo cannot be used with -betterC


-- 
Johannes
April 07, 2020
Am Sun, 05 Apr 2020 16:00:42 +0000 schrieb Jonathan Marler:

> 
> So I say yes, I'd be fine with this "hack" to temporarily provide a path for programs to create CTFE-ONLY functions that won't go to code generation.  It's better than the status-quo and can be removed if we ever solve the general case.

The compiler can really not know if a function is supposed to be callable at runtime, see https://forum.dlang.org/post/ r6hotc$28au$2@digitalmars.com for details. It could probably work if we decide to introduce 'export', but there was no consensus on that for years.

-- 
Johannes
April 07, 2020
Am Sun, 05 Apr 2020 18:40:31 +0000 schrieb Petar Kirov [ZombineDev]:

> On Sunday, 5 April 2020 at 18:30:25 UTC, Stefan Koch wrote:
>> On Sunday, 5 April 2020 at 18:25:10 UTC, Petar Kirov [ZombineDev] wrote:
>>> On Sunday, 5 April 2020 at 12:11:23 UTC, Stefan Koch wrote:
>>>> [...]
>>>
>>> Another, more radical and interesting option is to not perform codegen unless it's actually necessary. Instead of blacklisting `@ctfe` or `assert(__ctfe)` functions, have a whitelist of what functions are allowed to codegen-ed.
>>>
>>> [...]
>>
>> Seems like you are reading my mind.
>> Indeed this is one of next steps ...
>> assert(__ctfe) did not break a single package, in out CI! Therefore I
>> do now have enough confidence in crawling the dynamic dependencies
>> outwards from the entry point and exported functions.
>>
>> That however is for the future and will require me to do some theoretical work, in order to proof to myself that it actually is feasible and won't end up in a nightmare of subtly broken builds.
> 
> Great! Perhaps we should write a DIP to specify the semantics of such a
> lazy compilation model? An even more extreme option would be to not
> perform semantic analysis on non-`export`-ed functions.
> This should bring some really high gains, though it would be the
> trickiest to adopt option for users.

How would such a lazy compilation model work without explicit 'export' anotations? If I wanted to build a proprietary library libfoo.so and provide only .di files with stripped function bodies, all functions have to be compiled into the .so file. But how does the compiler know the result is going to be a shared library in a separate compilation model, where the compiler produces only an object file?

I guess such a lazy compilation model might be quite interesting, but as it is a quite radical idea, it needs lots of thinking, proper specification and testing first. (Testing as you'll probably have to emit everything as weak symbols then, and you probably want to test how linkers deal with that if you make excessive use of it)


-- 
Johannes
April 07, 2020
On 2020-04-07 13:41, Johannes Pfau wrote:
> Am Mon, 06 Apr 2020 13:49:19 +0000 schrieb Atila Neves:
> 
>> On Monday, 6 April 2020 at 13:41:52 UTC, Stefan Koch wrote:
>>> On Monday, 6 April 2020 at 10:14:24 UTC, Atila Neves wrote:
>>>> On Sunday, 5 April 2020 at 12:11:23 UTC, Stefan Koch wrote:
>>>>> [...]
>>>>
>>>> Wouldn't it be easier to skip codegen for private functions that are
>>>> never called from non CTFE contexts?
>>>
>>> Easier from the user-perspective yes.
>>>  From the compiler perspective,
>>> That's another step which may take quite a while to do correctly.
>>> The easy thing would be (Essentially an (N*M) loop over all calls and
>>> functions),
>>
>> Where N and M are all calls and private functions in one module,
>> not all code in a project.
>>
> 
> In fact, you can't do that in current D. I thought about it some time
> ago, consider this valid D code:
> 
> ---------------------------
> module foo;
> 
> private void privateFunc() {}
> 
> public void publicTemplate(T)()
> {
>      privateFunc();
> }
> ---------------------------
> 
> Let's say I compile module foo into libfoo. libfoo internally never uses
> publicTemplate. An external user instantiates publicTemplate ==>
> privateFunc needs to be emitted in libfoo.
> 
> Now you can't even check that privateFunc is never called, cause you
> can't analyze the template without instantiating it. And in general, you
> can't instantiate it without knowing the template instance type
> parameters. (In this simple example you might think this should work. But
> just imagine that I can pass in a type with an enum string member, which
> I can then mixin in the template, ....)
> 
> 
> Generally the only way to solve this seems to finally implement export
> properly, as it was proposed years ago. Then privateFunc should be
> flagged export (needed for windows DLL support and linux .so selective
> exports anyway) and you could probably strip uncalled, non-exported
> private functions.
> 
> 
> TLDR: It's certainly not easy to implement.
> 

You can also bypass "private" using __traits(getMemeber) or using an extern(D) declaration.

-- 
/Jacob Carlborg
April 07, 2020
On 2020-04-05 14:11, Stefan Koch wrote:
> Hi Guys,
> 
> I am currently working on speeding up dmd. In the presence of templates and ctfe.
> and one thing that stood out is that we do codegen for symbols which are never used.
> (and hopefully eliminated by a smart linker but eh ...)
> 
> I've seen the following pattern in the wild.
> 
> 
> string GenerateMixin(string[] params)
> {
>      assert(__ctfe);
>      .... more code ....
> }
> 
> that means if we see assert(__ctfe) inside the outermost scope of a function body ( not in a branch) then we know that this function would assert if called at runtime.
> And therefore no correct code can be calling it.
> which also means we can forgo having code-gen or optimization for the function body.
> and reduce the count of names which has to get mangled.
> 
> They only downside to this, is that giving assert(__ctfe) the special meaning to skip codegen might be confusing to some people .... then again you wouldn't use assert(__ctfe) unless you expect that function to not be available at run-time
> 
> What do you guys think?

For many years I wanted a feature that could indicate a function is only used at compile time.

BTW, this is a breaking change. Consider this code:

import core.exception;

void foo()
{
    assert(__ctfe);
}

void main() @system
{
    try
        foo();
    catch (AssertError)
    {}
}

-- 
/Jacob Carlborg
April 07, 2020
On Tuesday, 7 April 2020 at 17:52:44 UTC, Jacob Carlborg wrote:
> BTW, this is a breaking change. Consider this code:
>
> import core.exception;
>
> void foo()
> {
>     assert(__ctfe);
> }
>
> void main() @system
> {
>     try
>         foo();
>     catch (AssertError)
>     {}
> }

Catching an AssertError thrown by an assert statement is officially undefined behavior:
https://dlang.org/spec/expression.html#assert_expressions
1 2 3 4 5 6 7 8 9