Thread overview
Getting the names of all the top-level functions in module
Sep 21, 2021
Stefan Koch
Sep 21, 2021
Stefan Koch
Sep 21, 2021
Adam D Ruppe
Sep 21, 2021
Stefan Koch
Sep 21, 2021
Adam D Ruppe
Sep 21, 2021
Stefan Koch
Sep 21, 2021
Stefan Koch
Sep 21, 2021
Stefan Koch
Sep 21, 2021
Stefan Koch
September 21, 2021

Hello there,

I was just working on an example for my upcoming library core.reflect
and I wanted to contrast it with the current template way of doing things.
The issue at hand here is getting all the names of top-level (free)function in a module in a string[].

The way that I found to do this with templates involves a bit of knowledge of is expressions here it is

template functionNames (alias M)
{
    static const functionNames = ()
    {
        string[] names;
        foreach(m;__traits(derivedMembers, M))
        {
            alias sym = __traits(getMember, M, m);
            static if (is(typeof(sym) T) && is(T F == function))
            {
                names ~= __traits(identifier, sym);
            }
        }
        return names;
    } ();
}

Note that on the "callsite" you have to instantiate the helper template as:
static const fNames = functionNames!(mixin(__MODULE__));
the parens around the mixin are not optional it will not parse without them.

now comes the way do that with core.reflect.

@(core.reflect) // annotation prevents code-gen since the `nodeFromName` builtin which is called in there is not a real function and therefore has no body for the codegenerator to look at.
const(FunctionDeclaration)[] getFreeFunctionNamesFromModule(
    string module_ = __MODULE__,
    ReflectFlags flags = ReflectFlags.NoMemberRecursion,
    immutable Scope _scope = currentScope()
)
{
    string[] result;
    auto mod_ = nodeFromName(module_, flags, _scope);
    if (auto mod = cast(Module) mod_)
    foreach(member;mod.members)
    {
        if (auto fd = cast(const FunctionDeclaration) member)
        {
            result ~= fd.name;
        }
    }

    return result;
}

and you use it like this.
static const functionNames = getFreeFunctionNamesFromModule;
I am using a parenthesis less call to make it look like it's a magic builtin but in reality the only bits of magic that are used are the two builtins nodeFromName and currentScope

whereas the template above used 5 magic constructs.

  • 2 different types of pattern matching is expressions.
    and
  • 3 different __traits expressions.
September 21, 2021

On Tuesday, 21 September 2021 at 13:29:44 UTC, Stefan Koch wrote:

>

Hello there,

>

Note that on the "callsite" you have to instantiate the helper template as:
static const fNames = functionNames!(mixin(__MODULE__));

Correction you can't actually use it like that.
Because that introduces a symbol into the module you are trying to reflect over, and the compiler will complain about a circular reference to fNames.

you have to use it directly you cannot assign it to a module level variable.
ah well :-)

September 21, 2021
On Tuesday, 21 September 2021 at 13:29:44 UTC, Stefan Koch wrote:
> Note that on the "callsite" you have to instantiate the helper template as:

Not true. You can pass module names as plain names now, no more need for the special mixin, with the exception of getting the current module (just because __MODULE__ is a string), but if this were a library function, you wouldn't be using the current module anyway.

> whereas the template above used 5 magic constructs.
>  - 2 different types of pattern matching is expressions.
>    and
>  - 3 different __traits expressions.

Not true. There's two extremely basic traits and one is expression actually necessary here. You overcomplicated it again.


This works today:

import arsd.simpledisplay; // or whatever

enum string[] functionNames(alias M) = ()
{
        string[] names;
        foreach(memberName; __traits(derivedMembers, M))
            static if (is(typeof(__traits(getMember, M, memberName)) ==
            function))
                names ~= memberName;
        return names;
}();

pragma(msg, functionNames!(arsd.simpledisplay));



As you can see, there's one trait, and one is expression, to see if it is a function. Not that complicated.


> you have to use it directly you cannot assign it to a module level variable.

Also not true.

---
module qwe.whatever;

enum string[] functionNames(alias M) = ()
{
        string[] names;
        foreach(memberName; __traits(derivedMembers, M))
            static if (is(typeof(__traits(getMember, M, memberName)) == function))
                names ~= memberName;
        return names;
}();

const fNames = functionNames!(qwe.whatever);

void main() {
        import std.stdio;
        writeln(fNames);
}

void foo() {}
---



$ dmd qwe
$ ./qwe
["main", "foo"]
$ nm qwe.o | grep functionNames
// notice this is empty btw
$
September 21, 2021
On Tuesday, 21 September 2021 at 14:20:50 UTC, Adam D Ruppe wrote:
> On Tuesday, 21 September 2021 at 13:29:44 UTC, Stefan Koch

>
>
> This works today:
>
> import arsd.simpledisplay; // or whatever
>
> enum string[] functionNames(alias M) = ()
> {
>         string[] names;
>         foreach(memberName; __traits(derivedMembers, M))
>             static if (is(typeof(__traits(getMember, M, memberName)) ==
>             function))
>                 names ~= memberName;
>         return names;
> }();
>
> pragma(msg, functionNames!(arsd.simpledisplay));
>

This does work. Interesting.

But had I not posted here would I have been able to get to the template you posted above?

I am not trying to over-complicate anything this is what I naturally arrive at.
After spending more than 20 minutes to get it to show me the answer I am looking for.

September 21, 2021
Here's one with param names:

----

module qwe;

struct FunctionInfo {
        string name;
        string[] paramNames;
}

enum FunctionInfo[] functionInfo(alias M) = ()
{
        FunctionInfo[] res;
        foreach(memberName; __traits(derivedMembers, M))
            static if (is(typeof(__traits(getMember, M, memberName)) Params == __parameters)) {
                FunctionInfo fi = FunctionInfo(memberName);
                foreach(idx, param; Params)
                        fi.paramNames ~= __traits(identifier, Params[idx .. idx + 1]);
                res ~= fi;
            }
        return res;
}();

void main() {
        import std.stdio;
        writeln(functionInfo!qwe);
}

void foo(int a, int b){}

----


Of course the weird thing here is the need to slice the params tuple to get the idenfitier. I had to borrow that trick from Phobos but now that I know it it is very useful.

Otherwise the code is still fairly simple. Only hte one is expression since if it doesn't have params it doesn't trigger the condition and if all functions have params (even if it is an empty set)
September 21, 2021
On Tuesday, 21 September 2021 at 15:07:35 UTC, Adam D Ruppe wrote:
> Here's one with param names:
>
> ----
>
> module qwe;
>
> struct FunctionInfo {
>         string name;
>         string[] paramNames;
> }
>
> enum FunctionInfo[] functionInfo(alias M) = ()
> {
>         FunctionInfo[] res;
>         foreach(memberName; __traits(derivedMembers, M))
>             static if (is(typeof(__traits(getMember, M, memberName)) Params == __parameters)) {
>                 FunctionInfo fi = FunctionInfo(memberName);
>                 foreach(idx, param; Params)
>                         fi.paramNames ~= __traits(identifier, Params[idx .. idx + 1]);
>                 res ~= fi;
>             }
>         return res;
> }();
>
> void main() {
>         import std.stdio;
>         writeln(functionInfo!qwe);
> }
>
> void foo(int a, int b){}
>

```d
Here's the core.reflect version for that
struct FunctionInfo
{
    string name;
    string[] params;
}

@(core.reflect)
const(FunctionInfo)[] FunctionsInfoFromModule(string module_ = __MODULE__,
    ReflectFlags flags = ReflectFlags.NoMemberRecursion,
    immutable Scope _scope = currentScope()
)
{
    const(FunctionInfo)[] result;
    auto mod_ = nodeFromName(module_, flags, _scope);
    if (auto mod = cast(Module) mod_)
    foreach(member;mod.members)
    {
        if (auto fd = cast(const FunctionDeclaration) member)
        {
            string[] paramNames;
            foreach(p;fd.type.parameterTypes) //because that's how dmd looks and I didn't smooth that part out yet. it's going to be nicer in the future
            {
                paramNames ~= p.identfier; // TODO maybe rename identifier to name?
            }
            result ~= FunctionInfo(fd.name, paramNames);
        }
    }

    return result;
}
```

Alternatively you could of course just use the `FunctionDeclaration` object ;)
As the information is already bundled.
September 21, 2021

On Tuesday, 21 September 2021 at 15:18:01 UTC, Stefan Koch wrote:

>

On Tuesday, 21 September 2021 at 15:07:35 UTC, Adam D Ruppe wrote:

>

Here's one with param names:


I've done a little benchmark the core.reflect version vs the template version.

Command Mean [ms] Min [ms] Max [ms] Relative
-version=arsd #64 41.3 ± 3.0 34.5 53.8 1.33 ± 0.14
#64 31.1 ± 2.4 23.5 35.9 1.00
-version=arsd #192 72.0 ± 3.4 65.7 79.2 2.32 ± 0.21
#192 36.3 ± 2.7 29.5 45.4 1.17 ± 0.13
-version=arsd #320 96.7 ± 4.5 90.8 107.3 3.11 ± 0.28
#320 41.3 ± 2.7 35.2 47.6 1.33 ± 0.13
-version=arsd #448 131.4 ± 5.0 124.9 140.2 4.23 ± 0.37
#448 46.4 ± 2.9 40.9 53.1 1.49 ± 0.15
-version=arsd #576 163.2 ± 7.0 152.8 174.9 5.25 ± 0.47
#576 52.1 ± 2.7 45.4 58.0 1.68 ± 0.16
-version=arsd #704 199.7 ± 8.9 187.3 215.7 6.43 ± 0.58
#704 56.8 ± 3.5 51.1 66.7 1.83 ± 0.18
-version=arsd #832 240.1 ± 10.1 227.2 252.8 7.73 ± 0.69
#832 62.1 ± 4.3 55.0 77.7 2.00 ± 0.21
-version=arsd #960 271.4 ± 8.5 264.1 292.2 8.74 ± 0.74
#960 68.8 ± 5.6 61.4 89.9 2.22 ± 0.25
-version=arsd #1088 313.0 ± 12.3 299.2 334.8 10.08 ± 0.88
#1088 73.6 ± 3.4 68.7 82.3 2.37 ± 0.21
-version=arsd #1216 365.0 ± 11.6 345.7 380.8 11.75 ± 0.99
#1216 79.6 ± 2.8 74.1 86.3 2.56 ± 0.22
-version=arsd #1344 400.5 ± 14.2 386.7 420.8 12.89 ± 1.11
#1344 84.6 ± 3.2 80.4 92.4 2.72 ± 0.24
-version=arsd #1472 439.6 ± 11.6 426.9 464.9 14.15 ± 1.17
#1472 93.3 ± 17.4 78.6 162.0 3.00 ± 0.61
-version=arsd #1600 521.7 ± 18.1 474.1 542.0 16.80 ± 1.44
#1600 99.1 ± 6.7 87.5 115.3 3.19 ± 0.33
-version=arsd #1728 583.8 ± 20.9 548.6 609.5 18.79 ± 1.62
#1728 111.8 ± 8.4 93.6 127.2 3.60 ± 0.39
-version=arsd #1856 617.6 ± 20.3 576.0 640.7 19.88 ± 1.69
#1856 112.5 ± 4.9 103.0 125.2 3.62 ± 0.32
-version=arsd #1984 673.3 ± 26.3 620.1 697.0 21.68 ± 1.90
#1984 115.8 ± 8.7 105.5 145.1 3.73 ± 0.40

Which corresponds to this graph.

[IMAGE perfgraph]

blue is adams template.
orange is core reflect.
X in correlated to N the number of functions being reflection
and Y is correlated to compile time time

September 21, 2021

On Tuesday, 21 September 2021 at 17:15:23 UTC, Stefan Koch wrote:

>

On Tuesday, 21 September 2021 at 15:18:01 UTC, Stefan Koch wrote:

>

On Tuesday, 21 September 2021 at 15:07:35 UTC, Adam D Ruppe wrote:

>

Here's one with param names:


I've done a little benchmark the core.reflect version vs the template version.

Scratch that.
It turns out I should have tested the output.

nodeFromName will return an Import node in this case, so the dynamic cast to module fails and CTFE it never ran to from the datastructure.

Hence the previous number are garbage.
Here is an updated version.
We can see that the CTFE overhead is clearly dominating here.

Command Mean [ms] Min [ms] Max [ms] Relative
-version=arsd #64 40.9 ± 2.2 35.9 45.9 1.24 ± 0.10
#64 33.0 ± 2.2 28.0 38.0 1.00
-version=arsd #192 71.3 ± 2.7 66.2 78.7 2.16 ± 0.16
#192 44.9 ± 2.1 39.0 48.9 1.36 ± 0.11
-version=arsd #320 94.3 ± 3.1 88.2 101.7 2.86 ± 0.21
#320 57.9 ± 2.3 52.2 62.6 1.75 ± 0.13
-version=arsd #448 128.3 ± 4.6 120.4 138.6 3.88 ± 0.29
#448 72.4 ± 3.6 65.9 82.1 2.19 ± 0.18
-version=arsd #576 162.9 ± 4.8 157.6 174.5 4.93 ± 0.36
#576 87.7 ± 2.9 82.5 94.5 2.66 ± 0.20
-version=arsd #704 198.7 ± 6.5 189.7 209.9 6.02 ± 0.44
#704 103.3 ± 3.9 97.1 112.5 3.13 ± 0.24
-version=arsd #832 235.2 ± 7.1 228.4 246.1 7.12 ± 0.52
#832 121.9 ± 3.6 117.0 132.2 3.69 ± 0.27
-version=arsd #960 275.2 ± 10.7 265.9 294.8 8.33 ± 0.64
#960 139.6 ± 3.0 134.6 147.9 4.23 ± 0.29
-version=arsd #1088 311.1 ± 8.2 305.2 331.6 9.42 ± 0.67
#1088 162.0 ± 3.7 155.8 168.2 4.91 ± 0.34
-version=arsd #1216 354.7 ± 13.6 342.3 381.1 10.74 ± 0.82
#1216 187.3 ± 6.2 180.2 206.4 5.67 ± 0.42
-version=arsd #1344 400.7 ± 16.8 386.1 428.0 12.14 ± 0.95
#1344 210.6 ± 3.5 206.1 216.9 6.38 ± 0.43
-version=arsd #1472 444.4 ± 15.7 430.5 470.7 13.46 ± 1.01
#1472 234.3 ± 3.8 230.2 241.3 7.10 ± 0.48
-version=arsd #1600 504.6 ± 21.4 477.9 529.8 15.28 ± 1.20
#1600 261.9 ± 3.4 258.8 268.3 7.93 ± 0.53
-version=arsd #1728 536.7 ± 18.8 523.3 572.7 16.25 ± 1.21
#1728 284.0 ± 2.2 281.3 288.3 8.60 ± 0.57
-version=arsd #1856 586.4 ± 17.6 570.5 625.1 17.76 ± 1.29
#1856 319.4 ± 13.6 309.4 352.2 9.67 ± 0.76
-version=arsd #1984 630.5 ± 18.9 619.1 683.6 19.09 ± 1.38
#1984 344.8 ± 10.5 329.6 360.8 10.44 ± 0.76

So those are the proper numbers.
And instead of a 10x change we have what almost seems to be a constant difference.
The simple truth of this is that the CTFE concatenation is the weak link here.

plot

blue is the template
red is core reflect.

September 21, 2021

On Tuesday, 21 September 2021 at 22:53:26 UTC, Stefan Koch wrote:

>

On Tuesday, 21 September 2021 at 17:15:23 UTC, Stefan Koch wrote:

>

On Tuesday, 21 September 2021 at 15:18:01 UTC, Stefan Koch wrote:

>

On Tuesday, 21 September 2021 at 15:07:35 UTC, Adam D Ruppe wrote:

>

Here's one with param names:


I've done a little benchmark the core.reflect version vs the template version.

Scratch that.
It turns out I should have tested the output.

The simple truth of this is that the CTFE concatenation is the weak link here.

So if ctfe is the limiting factor how does the graph look when CTFE is taken out of the picture.

That's how.
core.reflect looks linear
while the template shows signs of becoming superlinear.
However I cannot test it on bigger N because there is a restriction to how many synthetic symbol-names the compiler can create ...

                    // Perturb the name mangling so that the symbols can co-exist
                    // instead of colliding
                    s.localNum = cast(ushort)(originalSymbol.localNum + 1);
                    assert(s.localNum);         // 65535 should be enough for anyone

ah well :-)
It should be noted that core.reflect can be used with much higher N because it doesn't create symbols in that way.