On Friday, 2 October 2020 at 03:11:34 UTC, Stefan Koch wrote:
>  - doesn't work for -betterC

This is trivially easy to fix. Wrap the function in a template and use an eponymous enum to collapse it to a literal:


----
template makeConvMatrix(T...) { // wrapper added
        string helper()
        {
            string result;
            static foreach(t; T)
            {
                result ~= "\t" ~ t.stringof;
            }
            result ~= "\n";
            static foreach(t1; T)
            {
                result ~= t1.stringof;
                static foreach(t2; T)
                {
                    result ~=  "\t" ~ (is(t1:t2) ? "yes" : "no");
                }
                result ~= "\n";
            }
            return result;
        }

        enum makeConvMatrix = helper(); // eponymous call
}

extern(C) // for betterC
void main()
{
    import core.stdc.stdio;
    static immutable convMatrix = makeConvMatrix!(byte, ubyte, short, ushort, int, uint, long, ulong); // no more () there

    printf("%s\n", convMatrix.ptr);
}
------

Generates a reasonably small executable too (this pattern btw is what my change to dmd a couple months ago is able to recognize):

$ ls -lh bc
-rwxr-xr-x 1 me users 19K Oct  1 23:19 bc
$ ls -lh bc.o
-rw-r--r-- 1 me users 2.4K Oct  1 23:19 bc.o
$ nm bc.o
0000000000000000 t
0000000000000000 D _D2bc4mainUZ10convMatrixyAa
                 U _GLOBAL_OFFSET_TABLE_
0000000000000000 r _TMP0
0000000000000000 W main
                 U printf


Which is identical to if I delete the template from the source entirely and just replace it with a string literal. Nothing of it is emitted to the object file, so the linker doesn't even have to strip it.

Worth noting that not all cases work out this well. But this one actually does.

On Thursday, 1 October 2020 at 21:34:54 UTC, Walter Bright wrote:
> On 10/1/2020 1:57 AM, Stefan Koch wrote:
>> However there is a difference in file size.
>> 
>> Type function version:
>> 
>> stat  makeConvMatrix.o
>>    File: 'makeConvMatrix.o'
>>    Size: 2876
>> 
>> VS template version
>> 
>> stat  makeConvMatrix_tmpl.o
>>    File: 'makeConvMatrix_tmpl.o'
>>    Size: 11760
>
> Indeed there is, as the compiler generates code for the function and writes the code to the object file. The code it generates is placed into a COMDAT section which, since it is unreferenced, is supposed to be elided by the linker.

Linkers are not under our control.
The compiler is.

> However, many linkers don't seem to do that, and besides, the compiler is wasting effort generating unused code. This is a more general inefficiency in the design of the compiler internals, and is not necessarily a language design defect.

So you are saying having a mechanism to replace a macro/template with a regular function
does not address a language issue?

> Internally, there is a function called `needsCodegen()` which decides if a template instantiation needs code generated for it (duh!). It clearly errs on the side of being a bit too conservative.

I am intimately familiar with needsCodegen() (- and I still don't fully understand it)
It leads to a stack overflow within the compiler when compiling one our apps with -allinst.
Atila tries to fix it, and it has taken him a couple of weeks already.
Right now the likely solution is to make it much more conservative, because that's the only thing we can think of to avoid linker errors.

> I suggest an improvement to it where it will return false if the only use of a template instantiation is in CTFE. If that is unimplementable (the separate compilation model may make it impractical), a fallback position is to have a pragma or some other mechanism that says "don't generate code for this function".

Been there. Done that.

> I suspect such a compiler improvement would produce considerable benefit to existing template-heavy code.

It requires the programmer to decide if a function needs to be elided which in a template, he may not be able to tell.
Because it may depend on which instance is produced.

On Friday, 2 October 2020 at 02:23:25 UTC, Andrei Alexandrescu wrote:
> Is this a visitation/double dispatch?

ummmm I'm actually not sure. I thought about doing a proper double dispatch but instead the implementation is pretty much single... just there is a dynamic cast in the convert methods so maybe technically it still falls under the definition.

The basic idea though is it doesn't take magic to do this. All the types the Variant needs *are* available to it thanks to normal templates, every type it ever sees are sent in (and ones it never sees it never needs), just the difficulty is they come in two separate calls. Thus it extracts what it needs from them into a runtime class to bridge that gap. And then any compile-time reflection that requires two types simultaneously (like `is(a:b)`) will need to have its logic re-implemented in library code instead of leveraging the compiler's built in magic. That is a legit hassle but not a fatal barrier.

All the runtime data required can be generated by normal templates though normal reflection into normal objects of normal data. Nothing special required.

On Friday, 2 October 2020 at 02:21:03 UTC, Andrei Alexandrescu wrote:
>> template staticMap(alias F, Args...) {
>>      static foreach(ref Arg; Args)
>>          Arg = F!Arg;
>>      alias staticMap = Args;
>
> I don't get the interaction. How does mutating the argument affect order of declaration?

Is typeof(Args[0]) in there the original arg or the mapped arg?

In a normal function, there's a distinct before-and-after that, in theory, doesn't exist outside functions (you can use stuff before they are lexically declared)... but even there, you cannot change the type of an already existing variable.

This would mean you can. Would certainly be strange.... but might work.

On 10/1/20 11:38 PM, Adam D. Ruppe wrote:
> On Friday, 2 October 2020 at 02:21:03 UTC, Andrei Alexandrescu wrote:
>>> template staticMap(alias F, Args...) {
>>>      static foreach(ref Arg; Args)
>>>          Arg = F!Arg;
>>>      alias staticMap = Args;
>>
>> I don't get the interaction. How does mutating the argument affect order of declaration?
> 
> Is typeof(Args[0]) in there the original arg or the mapped arg?
> 
> In a normal function, there's a distinct before-and-after that, in theory, doesn't exist outside functions (you can use stuff before they are lexically declared)... but even there, you cannot change the type of an already existing variable.
> 
> This would mean you can. Would certainly be strange.... but might work.

That's a rather large change - ref to aliases and such. Maybe a more basic facility such as appending to a tuple is simpler?


template staticMap(alias F, Args...) {
    alias Result = AliasSeq!();
    static foreach(Arg; Args)
        Result ~= F!Arg;
    alias staticMap = Result;
}

Now indeed order of evaluation concerns are rather apparent...

This should work with Filter, Reverse etc. as well. Not with DerivedToFront (unless a different algo such as mergesort is used).

On Friday, 2 October 2020 at 02:21:03 UTC, Andrei Alexandrescu wrote:
> On 10/1/20 7:10 PM, Paul Backus wrote:
>> On Thursday, 1 October 2020 at 21:34:50 UTC, Adam D. Ruppe wrote:
>>>
>>> So some facility to turn a ctfe array back into a tuple - the dereifiy step basically - with compiler assistance I think will be practical.
>> 
>> What if we could just mutate tuples locally?
>> 
>> template staticMap(alias F, Args...) {
>>      static foreach(ref Arg; Args)
>>          Arg = F!Arg;
>>      alias staticMap = Args;
>> }
>
> Interesting - this is akin to D's relaxed purity whereby a pure function can mutate its arguments. I'm trying to wrap my head around it, e.g. is Args considered a private copy of the argument much like a static array passed by value?

Kind of. Args is a private copy, not of the arguments themselves, but of a list of *references* to the arguments. (In DMD terms, TemplateInstance.tiargs an array of pointers to RootObjects.) So mutating it just means re-binding the references to point to new things.

> I don't get the interaction. How does mutating the argument affect order of declaration?

You're right; it doesn't matter in this case. It would matter if you wanted to write a `staticFold`, though:

template staticFold(alias F, Args...)
    if (Args.length > 0)
{
    static foreach (Arg; Args[1 .. $])
        Args[0] = F!(Init, Arg);
    alias staticFold = Args[0];
}

On Friday, 2 October 2020 at 04:05:00 UTC, Andrei Alexandrescu wrote:
>
> That's a rather large change - ref to aliases and such. Maybe a more basic facility such as appending to a tuple is simpler?

I doubt it.
Compiler Tuples are somewhat hard to model in a polymorphic context.
They are essentially monads because they can't be modified.
Also you cannot preallocate a tuple.
And they can only work inside polymorphic contexts.

To be able to append to them you quite likely need strict order of declaration rules, as well as changes to D's compilation model as a whole.

Type functions or "static tuple emission functions" represent an addition, rather than a modification.
Which should make it much more tractable, to reason about them than to reason about large scale language changes.

YMMV of course.

On Friday, 2 October 2020 at 03:38:37 UTC, Adam D. Ruppe wrote:
> On Friday, 2 October 2020 at 02:21:03 UTC, Andrei Alexandrescu wrote:
>>> template staticMap(alias F, Args...) {
>>>      static foreach(ref Arg; Args)
>>>          Arg = F!Arg;
>>>      alias staticMap = Args;
>>
>> I don't get the interaction. How does mutating the argument affect order of declaration?
>
> Is typeof(Args[0]) in there the original arg or the mapped arg?
>
> In a normal function, there's a distinct before-and-after that, in theory, doesn't exist outside functions (you can use stuff before they are lexically declared)... but even there, you cannot change the type of an already existing variable.
>
> This would mean you can. Would certainly be strange.... but might work.

What you get from typeof(Args[0]) would depend on when it's evaluated relative to the other declarations in the template body. If it's evaluated before Args[0] is mutated (really, re-bound), you get the type of the original arg; if it's evaluated after, you get the type of the mapped arg.

Obviously the language spec would have to spell out what the order of evaluation is. But that's something it ought to do anyway, regardless of this proposal.

On Friday, 2 October 2020 at 03:26:57 UTC, Adam D. Ruppe wrote:
> On Friday, 2 October 2020 at 03:11:34 UTC, Stefan Koch wrote:
>>  - doesn't work for -betterC
>
> This is trivially easy to fix. Wrap the function in a template and use an eponymous enum to collapse it to a literal:
>
>
> ----
> template makeConvMatrix(T...) { // wrapper added
>         string helper()
>         {
>             string result;
>             static foreach(t; T)
>             {
>                 result ~= "\t" ~ t.stringof;
>             }
>             result ~= "\n";
>             static foreach(t1; T)
>             {
>                 result ~= t1.stringof;
>                 static foreach(t2; T)
>                 {
>                     result ~=  "\t" ~ (is(t1:t2) ? "yes" : "no");
>                 }
>                 result ~= "\n";
>             }
>             return result;
>         }
>
>         enum makeConvMatrix = helper(); // eponymous call
> }
>
> extern(C) // for betterC
> void main()
> {
>     import core.stdc.stdio;
>     static immutable convMatrix = makeConvMatrix!(byte, ubyte, short, ushort, int, uint, long, ulong); // no more () there
>
>     printf("%s\n", convMatrix.ptr);
> }
> ------
>
> Generates a reasonably small executable too (this pattern btw is what my change to dmd a couple months ago is able to recognize):
>
> $ ls -lh bc
> -rwxr-xr-x 1 me users 19K Oct  1 23:19 bc
> $ ls -lh bc.o
> -rw-r--r-- 1 me users 2.4K Oct  1 23:19 bc.o
> $ nm bc.o
> 0000000000000000 t
> 0000000000000000 D _D2bc4mainUZ10convMatrixyAa
>                  U _GLOBAL_OFFSET_TABLE_
> 0000000000000000 r _TMP0
> 0000000000000000 W main
>                  U printf
>
>
> Which is identical to if I delete the template from the source entirely and just replace it with a string literal. Nothing of it is emitted to the object file, so the linker doesn't even have to strip it.
>
> Worth noting that not all cases work out this well. But this one actually does.

You still create unnecessary symbols.
Just use the template twice with different arguments.

--
    static immutable convMatrix = makeConvMatrix!(byte, ubyte, short, ushort, int, uint, long, ulong); // no more () there
    static immutable convMatrix2 = makeConvMatrix!(ubyte, short, ushort, int, uint, long, ulong); // no more () there

    printf("%s\n", convMatrix.ptr);
    printf("%s\n", convMatrix2.ptr);
--

0000000000000000 t
0000000000000008 R _D2t412__ModuleInfoZ
0000000000000000 D _D2t44mainUZ10convMatrixyAa
0000000000000010 D _D2t44mainUZ11convMatrix2yAa
                 U _d_dso_registry
                 U _GLOBAL_OFFSET_TABLE_
0000000000000000 W main
                 U printf
                 U __start_minfo
                 U __stop_minfo
0000000000000000 r _TMP0
0000000000000142 r _TMP1

uplink@uplink-black:~$ ls -lh t4.o
-rw-r--r-- 1 uplink uplink 4,5K Okt  2 06:51 t4.o

On 10/1/2020 7:27 PM, Andrei Alexandrescu wrote:
> auto makeConvMatrix(Ts...)() {
>      bool[T.length[T.length] result;
>      static foreach (i, T : Ts)
>          static foreach (j, U : Ts)
>              result[i][j] = is(T : U);
>      return result;
> }

This piece of code may be useful in re-implementing std.meta.MostDerived.