We have a compile time problem, and this is basically the cure. Intuitively, people imagine CTFE is expensive (and it kinda is), but really, the reason our compile times are bad is template instantiation.

This DIP single-handedly fixes compile-time issues in programs I've written by reducing template instantiations by near-100%, in particular, the expensive ones; recursive instantiations, usually implementing some form of static map.

https://github.com/dlang/DIPs/pull/188

This is an RFC on a draft, but I'd like to submit it with a reference implementation soon.

Stefan Koch has helped me with a reference implementation, which has so far
gone surprisingly smoothly, and has shown 50x improvement in compile times
in some artificial tests.
I expect much greater improvements in situations where recursive template
expansion reaches a practical threshold due to quadratic resource
consumption used by recursive expansions (junk template instantiations, and
explosive symbol name lengths).
This should also drastically reduce compiler memory consumption in
meta-programming heavy applications.

In addition to that, it's simple, terse, and reduces program logic indirection via 'utility' template definitions, which I find improves readability substantially.

We should have done this a long time ago.

- Manu

Change it to something like .unpackSequence instead of ... and I would be happy.

On 4/22/20 8:04 AM, Manu wrote:
> We have a compile time problem, and this is basically the cure.
> Intuitively, people imagine CTFE is expensive (and it kinda is), but really, the reason our compile times are bad is template instantiation.
> 
> This DIP single-handedly fixes compile-time issues in programs I've written by reducing template instantiations by near-100%, in particular, the expensive ones; recursive instantiations, usually implementing some form of static map.
> 
> https://github.com/dlang/DIPs/pull/188
> 
> This is an RFC on a draft, but I'd like to submit it with a reference implementation soon.
> 
> Stefan Koch has helped me with a reference implementation, which has so far gone surprisingly smoothly, and has shown 50x improvement in compile times in some artificial tests.

Yes please! Where is the reference implementation? I want to try some things out.

> I expect much greater improvements in situations where recursive template expansion reaches a practical threshold due to quadratic resource consumption used by recursive expansions (junk template instantiations, and explosive symbol name lengths).
> This should also drastically reduce compiler memory consumption in meta-programming heavy applications..

We have a project that I had to completely refactor because the memory consumption was so great during compile time, that I ran out of memory on a 12GB virtual machine on my 16GB macbook. The refactoring made a difference, but now it's back up and I needed to buy a new system with 32GB of RAM just to build. I understand Weka had similar issues, I wonder if anyone on that team can elaborate whether this might help fix those problems.

But I want to see if it actually would fix the problems. It's still a good change, but I'm not sure it will be the solution to all these.

> In addition to that, it's simple, terse, and reduces program logic indirection via 'utility' template definitions, which I find improves readability substantially.

There were some posts on "how do I do this in D" that used C++ parameter pack expansion that just weren't possible or horrible to implement in D.

I wonder if your DIP can solve them? I think it can.

e.g.: https://forum.dlang.org/post/ymlqbjblbjxoitoctevl@forum.dlang.org

I think can be solved just like C++:

int double_int(int val) { return 2 * val; }

T double_int(T val) { return val; }

void double_ints(alias pred, T... args) {
   pred(double_int(args)...);
}

That's awesome.

In my dconf talk last year, I talked about how D's metaprogramming was its crown jewel, and we should prioritize anything that makes this more usable/possible. This is the kind of stuff I was talking about. Well done!

-Steve

On 4/22/20 8:19 AM, rikki cattermole wrote:
> Change it to something like .unpackSequence instead of ... and I would be happy.

unpackSequence is a valid identifier. That would be a breaking change. Plus it would be less obvious. If anything it would have to be unpackSequence!(expr).

Plus there is precedence with C++ for using ...

And it makes intuitive sense -- you define tuples with T...

-Steve

On 23/04/2020 1:22 AM, Steven Schveighoffer wrote:
> On 4/22/20 8:19 AM, rikki cattermole wrote:
>> Change it to something like .unpackSequence instead of ... and I would be happy.
> 
> unpackSequence is a valid identifier. That would be a breaking change. Plus it would be less obvious. If anything it would have to be unpackSequence!(expr).
> 
> Plus there is precedence with C++ for using ...
> 
> And it makes intuitive sense -- you define tuples with T...
> 
> -Steve

Yeah.

But it also has precedence with type properties such as mangleof instead of the approach C took with sizeof and friends.

Right now we use it in parameters, adding it to the argument side with a completely different meaning ugh.

On Wed, Apr 22, 2020 at 11:20 PM Steven Schveighoffer via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On 4/22/20 8:04 AM, Manu wrote:
> > We have a compile time problem, and this is basically the cure. Intuitively, people imagine CTFE is expensive (and it kinda is), but really, the reason our compile times are bad is template instantiation.
> >
> > This DIP single-handedly fixes compile-time issues in programs I've written by reducing template instantiations by near-100%, in particular, the expensive ones; recursive instantiations, usually implementing some form of static map.
> >
> > https://github.com/dlang/DIPs/pull/188
> >
> > This is an RFC on a draft, but I'd like to submit it with a reference implementation soon.
> >
> > Stefan Koch has helped me with a reference implementation, which has so far gone surprisingly smoothly, and has shown 50x improvement in compile times in some artificial tests.
>
> Yes please! Where is the reference implementation? I want to try some things out.
>

The link is in the DIP.
Most tests we've done are working, except template instantiation expansion
is not yet implemented: ie, Thing!(Tuple, x, y)...
That's got a lot of implementation baggage attached to it in DMD.

> I expect much greater improvements in situations where recursive
> > template expansion reaches a practical threshold due to quadratic
> > resource consumption used by recursive expansions (junk template
> > instantiations, and explosive symbol name lengths).
> > This should also drastically reduce compiler memory consumption in
> > meta-programming heavy applications..
>
> We have a project that I had to completely refactor because the memory consumption was so great during compile time, that I ran out of memory on a 12GB virtual machine on my 16GB macbook. The refactoring made a difference, but now it's back up and I needed to buy a new system with 32GB of RAM just to build. I understand Weka had similar issues, I wonder if anyone on that team can elaborate whether this might help fix those problems.
>
> But I want to see if it actually would fix the problems. It's still a good change, but I'm not sure it will be the solution to all these.
>

I expect you won't be able to run practical tests on real code yet without
TemplateInstance expansion.
The problem is that existing code is factored exclusively into
template instantiations, so a trivial experiment will apply to existing
code in that form.

> In addition to that, it's simple, terse, and reduces program logic
> > indirection via 'utility' template definitions, which I find improves readability substantially.
>
> There were some posts on "how do I do this in D" that used C++ parameter pack expansion that just weren't possible or horrible to implement in D.
>

Yup. Now they're possible and awesome!

I wonder if your DIP can solve them? I think it can.
>
> e.g.: https://forum.dlang.org/post/ymlqbjblbjxoitoctevl@forum.dlang.org


Yes, it's basically written specifically to solve that problem! :)

I think can be solved just like C++:
>
> int double_int(int val) { return 2 * val; }
>
> T double_int(T val) { return val; }
>
> void double_ints(alias pred, T... args) {
>     pred(double_int(args)...);
> }
>
> That's awesome.
>

Yes, something like that.

In my dconf talk last year, I talked about how D's metaprogramming was
> its crown jewel, and we should prioritize anything that makes this more usable/possible. This is the kind of stuff I was talking about. Well done!
>

Thanks!

On Wed, Apr 22, 2020 at 11:35 PM rikki cattermole via Digitalmars-d < digitalmars-d@puremagic.com> wrote:

> On 23/04/2020 1:22 AM, Steven Schveighoffer wrote:
> > On 4/22/20 8:19 AM, rikki cattermole wrote:
> >> Change it to something like .unpackSequence instead of ... and I would be happy.
> >
> > unpackSequence is a valid identifier. That would be a breaking change. Plus it would be less obvious. If anything it would have to be unpackSequence!(expr).
> >
> > Plus there is precedence with C++ for using ...
> >
> > And it makes intuitive sense -- you define tuples with T...
> >
> > -Steve
>
> Yeah.
>
> But it also has precedence with type properties such as mangleof instead of the approach C took with sizeof and friends.
>
> Right now we use it in parameters, adding it to the argument side with a completely different meaning ugh.
>

I feel it's a perfectly natural expansion of the existing meaning. I'm 300% happy with the syntax personally, I will have a tantrum if you take it away from me.

On Wednesday, 22 April 2020 at 12:19:25 UTC, rikki cattermole wrote:
> Change it to something like .unpackSequence instead of ... and I would be happy.

I'm for some other thing than ... too, in C++ it's a quite bad syntax and it can become very ugly to maintain with all the different combinations you could use it with from a compiler viewpoint. Additionally if you do miss some combinations or allow more complex operators it will quickly become a mess of "why does this compile but why doesn't this compile" or "why does this do something different"

I'm especially not a fan of allowing all of
(Tup*10)...  -->  ( Tup[0]*10, Tup[1]*10, ... , Tup[$-1]*10 )
and
(Tup1*Tup2)...
and
myArr[Tup + 1]... -> myArr[Tup[0] + 1], myArr[Tup[1] + 1], myArr[Tup[2] + 1]

would this be valid:
f!T...(x)
or rather this?
f!T(x)...

what does (cast(Tup)x)... evaluate to?

is f(Tup) and f(Tup...) now doing the same thing?

if
f(Items.x...)
works then what about
f(Items.MemberTup...) ?

On Wednesday, 22 April 2020 at 13:45:52 UTC, WebFreak001 wrote:
> On Wednesday, 22 April 2020 at 12:19:25 UTC, rikki cattermole wrote:
>> Change it to something like .unpackSequence instead of ... and I would be happy.
>
> I'm for some other thing than ... too, in C++ it's a quite bad syntax and it can become very ugly to maintain with all the different combinations you could use it with from a compiler viewpoint. Additionally if you do miss some combinations or allow more complex operators it will quickly become a mess of "why does this compile but why doesn't this compile" or "why does this do something different"
>
> I'm especially not a fan of allowing all of
> (Tup*10)...  -->  ( Tup[0]*10, Tup[1]*10, ... , Tup[$-1]*10 )
> and
> (Tup1*Tup2)...
> and
> myArr[Tup + 1]... -> myArr[Tup[0] + 1], myArr[Tup[1] + 1], myArr[Tup[2] + 1]
>
> would this be valid:
> f!T...(x)
> or rather this?
> f!T(x)...
>
> what does (cast(Tup)x)... evaluate to?
>
> is f(Tup) and f(Tup...) now doing the same thing?
>
> if
> f(Items.x...)
> works then what about
> f(Items.MemberTup...) ?

Because it is basically just staticMap but with syntax from the compiler, how about something like

Tup->{Tup*4}

where each item in the tuple gets replaces with what is inside the {} ; Might want to instead have some item variable name and maybe even index counter but I can't think of some good syntax here or of any use cases right now.

This might be too powerful though because it's basically AST macros now if you allow this:

StringTup->{StringTup ~ } ":)"    <- like "a" ~ "b" ~ ":)"

or

Tup->{cast(Tup)}c    <- like cast(int)cast(short)c

But if you implemented this syntax in the compiler you would manually specify all the allowed rules inside the {} block, so it would definitely be harder to introduce weird allowed/disallowed syntax rules.

On 4/22/20 9:35 AM, Manu wrote:
> On Wed, Apr 22, 2020 at 11:20 PM Steven Schveighoffer via Digitalmars-d <digitalmars-d@puremagic.com <mailto:digitalmars-d@puremagic.com>> wrote:
> 
>     On 4/22/20 8:04 AM, Manu wrote:
>      > We have a compile time problem, and this is basically the cure.
>      > Intuitively, people imagine CTFE is expensive (and it kinda is), but
>      > really, the reason our compile times are bad is
>     template instantiation.
>      >
>      > This DIP single-handedly fixes compile-time issues in programs I've
>      > written by reducing template instantiations by near-100%, in
>     particular,
>      > the expensive ones; recursive instantiations, usually
>     implementing some
>      > form of static map.
>      >
>      > https://github.com/dlang/DIPs/pull/188
>      >
>      > This is an RFC on a draft, but I'd like to submit it with a
>     reference
>      > implementation soon.
>      >
>      > Stefan Koch has helped me with a reference implementation, which
>     has so
>      > far gone surprisingly smoothly, and has shown 50x improvement in
>     compile
>      > times in some artificial tests.
> 
>     Yes please! Where is the reference implementation? I want to try some
>     things out.
> 
> 
> The link is in the DIP.

Oops, it's at the top, I missed that. Thanks.

> Most tests we've done are working, except template instantiation expansion is not yet implemented: ie, Thing!(Tuple, x, y)...
> That's got a lot of implementation baggage attached to it in DMD.

Ugh, I think I won't be able to test then, most of this stuff works with lists of types, not values.

And come to think of it, staticMap works, but I need Filter as well. I suppose it can help a bit, but I still am going to have tons of "temporary" templates that are going to clog up the memory.

Stefan, where's that implementation of first class types for CTFE-only functions you promised? ;)

> 
> I expect you won't be able to run practical tests on real code yet without TemplateInstance expansion.
> The problem is that existing code is factored exclusively into template instantiations, so a trivial experiment will apply to existing code in that form.

The trivial experiment to test is to take a list of types with possible duplicates and remove all duplicates. It doesn't have to be in any order.

I'm hoping something like this might work:

template NewFilter(pred, T...)
{
   template process(U) {
     static if(pred!U) alias process = U;
     else alias process = AliasSeq!();
   }
   alias NewFilter = AliasSeq!(process!(T)...); // do I need the AliasSeq here?
}

template RemoveDuplicates(T...)
{
   template keepIt(U) {
      enum isSame(V) = __traits(isSame, U, V);
      enum keepIt = NewFilter!(isSame, T).length  == 1;
   }
   alias RemoveDuplicates = NewFilter!(keepIt, T);
}

Will this work better? You will still have a bunch of NewFilter, process, keepIt, and isSame instantiations, all with horribly long symbol names. But of note is there is no recursive instantiation patterns.

One thing I noticed, in order to use a property on a static map expansion (i.e. call a function with the resulting sequence, or access .length), you will need extra parentheses to distinguish the ... token from the . token.

Is there an easier/better way to do this with the new feature?

-Steve