I have a library where I was using very many voldemort types a la std.range.

While debugging I had an exception triggered, but I found that the library *pauses* significantly while printing the exception.

What I found is essentially that using voldemort types results in horrible stack traces.

To demonstrate the problem:

struct S(T)
{
    void foo(){ throw new Exception("1");}
}

auto s(T)(T t)
{
    struct Result
    {
        void foo(){ throw new Exception("2");}
    }
    return Result();
}

void main(string[] args)
{
    version(bad)
        auto x = 1.s.s.s.s.s;
    else
        S!(S!(S!(S!(S!(int))))) x;

    x.foo;
}

Building without bad version, and running, I get this as the stack frame for the foo call:

4   testexpansion                       0x0000000103c3fc14 pure @safe void testexpansion.S!(testexpansion.S!(testexpansion.S!(testexpansion.S!(testexpansion.S!(int).S).S).S).S).S.foo() + 144


Now, if I compile with version=bad:

4   testexpansion                       0x000000010fb5dbec pure @safe void testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).s(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).Result).s(testexpansion.s!(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).s(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).Result).Result).s(testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).s(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).Result).s(testexpansion.s!(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).s(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).Result).Result).Result.foo
() + 144

I believe what is happening is both the template parameter and the argument type are being printed, but both are the same! And each level of nesting results in another doubling of the printouts. So you have an exponential effect, and the resulting stack trace is horrendously useless.

what's more, the template bloat factor skyrockets:

dmd -c testexpansion.d
ls -l testexpansion.o
-rw-r--r--+ 1 steves  staff  5664 Feb  7 00:06 testexpansion.o

dmd -c -version=bad testexpansion.d
ls -l testexpansion.o
-rw-r--r--+ 1 steves  staff  15312 Feb  7 00:07 testexpansion.o

as a final test, I tried this:

auto s(T)(T t)
{
    return S!(T)();
}

And the resulting .o file:
-rw-r--r--+ 1 steves  staff  7104 Feb  7 00:11 testexpansion.o

With obviously the exception code printing in the less verbose form. So the cost in template bloat of using a voldemort type over a private type is 8k here, more than double the existing size. With more nesting, I'm sure that factor gets worse.

Is there a better way we should be doing this? I'm wondering if voldemort types are really worth it. They offer a lot of convenience, and are much DRYer than separate private template types. But the bloat cost is not really worth the convenience IMO.

Thoughts?

-Steve

On Sunday, 7 February 2016 at 05:18:39 UTC, Steven Schveighoffer wrote:
> Thoughts?

And no line number. But hey, these are convenience for youngsters. We real program, who type on the keyboard using our balls, don't need such distractions.

On Sunday, 7 February 2016 at 05:18:39 UTC, Steven Schveighoffer wrote:
> 4   testexpansion                       0x000000010fb5dbec pure @safe void testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!

Why "bad" foo is void?

> Is there a better way we should be doing this? I'm wondering if

Yeah would by nice to auto-repacle with testexpansion.S!(...)(...).Result.foo
or even with ...Result.foo

On 2/7/16 5:20 AM, deadalnix wrote:
> On Sunday, 7 February 2016 at 05:18:39 UTC, Steven Schveighoffer wrote:
>> Thoughts?
>
> And no line number. But hey, these are convenience for youngsters. We
> real program, who type on the keyboard using our balls, don't need such
> distractions.
>

Remind me never to borrow your laptop.

But really, if you can't figure out what +144 is, *eyeroll*.

-Steve

On 2/7/16 10:42 AM, Iakh wrote:
> On Sunday, 7 February 2016 at 05:18:39 UTC, Steven Schveighoffer wrote:
>> 4   testexpansion                       0x000000010fb5dbec pure @safe
>> void
>> testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!
>>
>
> Why "bad" foo is void?

Huh? foo returns void in both instances.

> Yeah would by nice to auto-repacle with
> testexpansion.S!(...)(...).Result.foo
> or even with ...Result.foo

A possible fix for the stack printing is to use the template parameter placeholders:

testexpansion.s!(T = testexpansion.s!...)(T).Result.foo

But this doesn't fix the object-file bloat.

-Steve

On Monday, 8 February 2016 at 01:48:32 UTC, Steven Schveighoffer wrote:
> On 2/7/16 10:42 AM, Iakh wrote:
>> On Sunday, 7 February 2016 at 05:18:39 UTC, Steven Schveighoffer wrote:
>>> 4   testexpansion                       0x000000010fb5dbec pure @safe
>>> void
>>> testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!
>>>
>>
>> Why "bad" foo is void?
>
> Huh? foo returns void in both instances.
>
>> Yeah would by nice to auto-repacle with
>> testexpansion.S!(...)(...).Result.foo
>> or even with ...Result.foo
>
> A possible fix for the stack printing is to use the template parameter placeholders:
>
> testexpansion.s!(T = testexpansion.s!...)(T).Result.foo
>
> But this doesn't fix the object-file bloat.
>
> -Steve

I think it was Manu who was complaining about symbol length some time ago and we ended up discussing symbol compression as a possible solution. Did anything ever come of that? If so this seems like an obvious candidate for recursive compression.

Nic

On 2/7/16 12:18 AM, Steven Schveighoffer wrote:
> I have a library where I was using very many voldemort types a la
> std.range.
>

[snip]

> Is there a better way we should be doing this? I'm wondering if
> voldemort types are really worth it. They offer a lot of convenience,
> and are much DRYer than separate private template types. But the bloat
> cost is not really worth the convenience IMO.

I modified all my voldemort-returning functions to return module-level types.

One of my example programs (compiled optimized/inline) went from 10MB to 1MB. The other example program went from 2.1MB to 900k.

-Steve

On Monday, 8 February 2016 at 13:01:44 UTC, Steven Schveighoffer wrote:
> On 2/7/16 12:18 AM, Steven Schveighoffer wrote:
>> I have a library where I was using very many voldemort types a la
>> std.range.
>>
>
> [snip]
>
>> Is there a better way we should be doing this? I'm wondering if
>> voldemort types are really worth it. They offer a lot of convenience,
>> and are much DRYer than separate private template types. But the bloat
>> cost is not really worth the convenience IMO.
>
> I modified all my voldemort-returning functions to return module-level types.
>
> One of my example programs (compiled optimized/inline) went from 10MB to 1MB. The other example program went from 2.1MB to 900k.
>
> -Steve

Just to be sure, you replaced something like this:

auto myFunc(int x){
   struct MyStruct{ int a; }
   return MyStruct(x);
}

with?

private struct MyStruct{ int a; }
auto myFunc(int x){
    return MyStruct(x);
}

On 2/8/16 8:19 AM, wobbles wrote:
> On Monday, 8 February 2016 at 13:01:44 UTC, Steven Schveighoffer wrote:
>> On 2/7/16 12:18 AM, Steven Schveighoffer wrote:
>>> I have a library where I was using very many voldemort types a la
>>> std.range.
>>>
>>
>> [snip]
>>
>>> Is there a better way we should be doing this? I'm wondering if
>>> voldemort types are really worth it. They offer a lot of convenience,
>>> and are much DRYer than separate private template types. But the bloat
>>> cost is not really worth the convenience IMO.
>>
>> I modified all my voldemort-returning functions to return module-level
>> types.
>>
>> One of my example programs (compiled optimized/inline) went from 10MB
>> to 1MB. The other example program went from 2.1MB to 900k.
>>
>
> Just to be sure, you replaced something like this:
>
> auto myFunc(int x){
>     struct MyStruct{ int a; }
>     return MyStruct(x);
> }
>
> with?
>
> private struct MyStruct{ int a; }
> auto myFunc(int x){
>      return MyStruct(x);
> }


Yes, but with template parameters. It's not so much the moving of the struct that reduced the bloat, but the nature of how the template parameters are used. Each function that returns one of these structs wraps another such struct, so the template bloat is exponential because of the repeat of the template parameter for the function argument. By moving the struct into the module, there is only one specification of the template parameter for the name mangling. Basically, I changed a.b.c bloat factor from 2^3 to 1^3.

-Steve