On Sunday, 22 April 2018 at 18:25:29 UTC, Simen Kjærås wrote:
> No lowering occurs here. A lowering is when the compiler takes one piece of syntax and replaces it with a different one, usually one that's more verbose.
>
> In a way, it's kind of like a template being instantiated, in that you write some code and it's being replaced by something else. It's not, however, the same thing.

So, how to distinguish it?

>
> Templates are sort of like compile-time functions - overloads are chosen in a similar way way to how function overloads are chosen, and the result may depend on the parameters given. Two templates, just like two functions, can give the same result without the templates themselves being the same.
>
>
>>> *In this case, there are important differences - in the first case the template itself is marked with a UDA, in the second the enum member is.
>> This is a good point. But you won't be able to get the UDA from an uninstantiated template will you? If you will, then, I'm wrong, I think...
>
> Why not try it?
>
> @("Has foo1_A") template foo1(string s) if (s == "a") {
>     enum foo1 = "foo1_A";
> }
> @("Has foo1_B") template foo1(string s) if (s == "b") {
>     enum foo1 = "foo1_B";
> }
>
> template foo2(string s)
> {
>     static if (s == "a")
>     {
>            @("Has foo2_A") enum foo2 = "foo2_A";
>     }
>     else static if (s == "b")
>     {
>           @("Has foo2_B") enum foo2 = "foo2_B";
>     }
> }
>
> // tuple("Has foo1_A")
> pragma(msg, __traits(getAttributes, foo1));
> // tuple()
> pragma(msg, __traits(getAttributes, foo2));
>
> So yes, we can. Templates are real things and can be manipulated, passed to other templates, etc.
>

The example you give, is not what I mean. Compare it with this:

@("Has foo1_A") template foo1(string s) if (s == "a") {
    enum foo1 = "foo1_A";
}
@("Has foo1_B") template foo1(string s) if (s == "b") {
    enum foo1 = "foo1_B";
}

template foo2(string s)
{
    static if (s == "a")
    {
           @("Has foo2_A") enum foo2 = "foo2_A";
    }
    else static if (s == "b")
    {
          @("Has foo2_B") enum foo2 = "foo2_B";
    }
}

// tuple("Has foo1_A")
pragma(msg, __traits(getAttributes, foo1!"a"));
// tuple("Has foo1_A")
pragma(msg, __traits(getAttributes, foo2!"a"));
// tuple("Has foo1_B")
pragma(msg, __traits(getAttributes, foo1!"b"));
// tuple("Has foo1_B")
pragma(msg, __traits(getAttributes, foo2!"b"));

On Sunday, 22 April 2018 at 19:27:24 UTC, Alex wrote:
> On Sunday, 22 April 2018 at 18:25:29 UTC, Simen Kjærås wrote:
>> No lowering occurs here. A lowering is when the compiler takes one piece of syntax and replaces it with a different one, usually one that's more verbose.
>>
>> In a way, it's kind of like a template being instantiated, in that you write some code and it's being replaced by something else. It's not, however, the same thing.
>
> So, how to distinguish it?

There is a limited set of lowerings, and they are defined in the language, not in user code. They include operator overloading (where `a op b` is translated to `a.opBinary!op(b)`), foreach over ranges (where `foreach(e; range) { }` becomes `for (auto e = range.front; !range.empty(); range.popFront()) { }`), string switching (which is forwarded to a template in druntime), and more that I can't recall right now. This is compiler magic meant to make implementating new features and reasoning about existing features, easier. They are briefly described in the article you linked, but I agree it offers limited insight. I've not found any other great sources of information about it, sadly.


> // tuple("Has foo1_A")
> pragma(msg, __traits(getAttributes, foo1!"a"));
> // tuple("Has foo1_A")
> pragma(msg, __traits(getAttributes, foo2!"a"));
> // tuple("Has foo1_B")
> pragma(msg, __traits(getAttributes, foo1!"b"));
> // tuple("Has foo1_B")
> pragma(msg, __traits(getAttributes, foo2!"b"));

You explicitly stated 'uninstantiated template'. These are instantiated templates.

This seems to indicate that we're talking past each other. I may have misunderstood what you meant when you wrote "But you won't be able to get the UDA from an uninstantiated template will you?", or one of us has misunderstood some other part of the discussion.

--
  Simen

On Monday, 23 April 2018 at 00:26:23 UTC, Simen Kjærås wrote:
>> // tuple("Has foo1_A")
>> pragma(msg, __traits(getAttributes, foo1!"a"));
>> // tuple("Has foo1_A")
>> pragma(msg, __traits(getAttributes, foo2!"a"));
>> // tuple("Has foo1_B")
>> pragma(msg, __traits(getAttributes, foo1!"b"));
>> // tuple("Has foo1_B")
>> pragma(msg, __traits(getAttributes, foo2!"b"));
>
> You explicitly stated 'uninstantiated template'. These are instantiated templates.
>
> This seems to indicate that we're talking past each other. I may have misunderstood what you meant when you wrote "But you won't be able to get the UDA from an uninstantiated template will you?", or one of us has misunderstood some other part of the discussion.

That could be, but we are getting closer :)
So, from the output, we can see, that the both implementation cannot be distinguished (at least by the given pragmas)
However, they would be after the new getOverloads version. And I'm wondering all the time, if this is wanted, as the new getOverloads would imply, that I cannot choose between both versions freely.

> There is a limited set of lowerings, and they are defined in the language, not in user code. They include operator overloading (where `a op b` is translated to `a.opBinary!op(b)`), foreach over ranges (where `foreach(e; range) { }` becomes `for (auto e = range.front; !range.empty(); range.popFront()) { }`), string switching (which is forwarded to a template in druntime), and more that I can't recall right now. This is compiler magic meant to make implementating new features and reasoning about existing features, easier. They are briefly described in the article you linked, but I agree it offers limited insight. I've not found any other great sources of information about it, sadly.
>

This is not true, in my opinion.
As an example, Walter gives the rewrite of a while-loop and a foreach-loop into a for-loop, stating that by the ability to do this, the for-loop is more basic than both of the former.

So, in my mind, every action of rewriting  of something into something else is "lowering". And if you can do this (of course, maintaining the full semantic equivalence), available actions have to remain the same.

Said this, I'm not against the new getOverloads. On the contrary, I find the feature cool. But if it is present, it has to yield the same results for foo1 and foo2, just like the pragmas.
I'm not sure about the procedure... should I post this as a question to the GitHub, so it can be discussed? pull 2351 or 8195?

And yes. I'm lacking information about lowering too. But, if all rewrite processes are "lowering" an explicit definition is not needed, as "everything" is allowed. It has something natural so far...
However, if the PR would be accepted in the form it is now, I would make an enhancement request on the docu.

On Monday, 23 April 2018 at 04:58:38 UTC, Alex wrote:
> On Monday, 23 April 2018 at 00:26:23 UTC, Simen Kjærås wrote:
>> There is a limited set of lowerings, and they are defined in the language, not in user code. They include operator overloading (where `a op b` is translated to `a.opBinary!op(b)`), foreach over ranges (where `foreach(e; range) { }` becomes `for (auto e = range.front; !range.empty(); range.popFront()) { }`), string switching (which is forwarded to a template in druntime), and more that I can't recall right now. This is compiler magic meant to make implementating new features and reasoning about existing features, easier. They are briefly described in the article you linked, but I agree it offers limited insight. I've not found any other great sources of information about it, sadly.
>
> This is not true, in my opinion.
> As an example, Walter gives the rewrite of a while-loop and a foreach-loop into a for-loop, stating that by the ability to do this, the for-loop is more basic than both of the former.
>
> So, in my mind, every action of rewriting  of something into something else is "lowering". And if you can do this (of course, maintaining the full semantic equivalence), available actions have to remain the same.

That's not the definition of lowering used elsewhere, and so will lead to confusion and misunderstanding. I would strongly suggest you rethink your definition of lowering.


> Said this, I'm not against the new getOverloads. On the contrary, I find the feature cool. But if it is present, it has to yield the same results for foo1 and foo2, just like the pragmas.

In the general case, this is impossible. Even just limiting ourselves to simple usages of static if it gets unwieldy. This template is from Phobos (all unnecessary code removed). It has 224 different possible combinations of features:

private struct _Cache(R, bool bidir)
{
    static if (bidir) {}
    else              {}
    static if (isInfinite!R) {}
    else                     {}
    static if (hasLength!R) {}
    version(assert) {}
    static if (isForwardRange!R) {}
    static if (hasSlicing!R) {
        static if (hasEndSlicing) {}
        static if (!isInfinite!R) {}
        else static if (hasEndSlicing) {}
    }
}

And that's before we even instantiate any templates that this template references - any templates used inside _Cache could increase the number of combinations.

But wait, there's more! How many does this have?

struct Bar(string s) {
    mixin(s);
}

If that's just one overload because of the impossibility of generating the options, what if we introduce a single static if inside it? Is that still one, or is it two? Do we count the possibilities introduced by a template mixin, but not by a string mixin? What if the template mixin comes from a template argument?

Sorry if I come off as very dismissive right now - I kind of like the idea, but it seems impossible in practice.


> I'm not sure about the procedure... should I post this as a question to the GitHub, so it can be discussed? pull 2351 or 8195?

What you are suggesting would be a fundamental change in the language, and should be discussed in the digitalmars.D forum. Either PR is not the right place to discuss it. If there's any interest, you will have to write a DIP for how the change will work.

--
  Simen

On Monday, 23 April 2018 at 07:49:39 UTC, Simen Kjærås wrote:
> On Monday, 23 April 2018 at 04:58:38 UTC, Alex wrote:
>> On Monday, 23 April 2018 at 00:26:23 UTC, Simen Kjærås wrote:
>>> There is a limited set of lowerings, and they are defined in the language, not in user code. They include operator overloading (where `a op b` is translated to `a.opBinary!op(b)`), foreach over ranges (where `foreach(e; range) { }` becomes `for (auto e = range.front; !range.empty(); range.popFront()) { }`), string switching (which is forwarded to a template in druntime), and more that I can't recall right now. This is compiler magic meant to make implementating new features and reasoning about existing features, easier. They are briefly described in the article you linked, but I agree it offers limited insight. I've not found any other great sources of information about it, sadly.
>>
>> This is not true, in my opinion.
>> As an example, Walter gives the rewrite of a while-loop and a foreach-loop into a for-loop, stating that by the ability to do this, the for-loop is more basic than both of the former.
>>
>> So, in my mind, every action of rewriting  of something into something else is "lowering". And if you can do this (of course, maintaining the full semantic equivalence), available actions have to remain the same.
>
> That's not the definition of lowering used elsewhere, and so will lead to confusion and misunderstanding. I would strongly suggest you rethink your definition of lowering.
>

There is no official definition. That's because some natural rewrite rules are implied, which are very general, I assume...

>
>> Said this, I'm not against the new getOverloads. On the contrary, I find the feature cool. But if it is present, it has to yield the same results for foo1 and foo2, just like the pragmas.
>
> In the general case, this is impossible. Even just limiting ourselves to simple usages of static if it gets unwieldy. This template is from Phobos (all unnecessary code removed). It has 224 different possible combinations of features:
>
> private struct _Cache(R, bool bidir)
> {
>     static if (bidir) {}
>     else              {}
>     static if (isInfinite!R) {}
>     else                     {}
>     static if (hasLength!R) {}
>     version(assert) {}
>     static if (isForwardRange!R) {}
>     static if (hasSlicing!R) {
>         static if (hasEndSlicing) {}
>         static if (!isInfinite!R) {}
>         else static if (hasEndSlicing) {}
>     }
> }
>
> And that's before we even instantiate any templates that this template references - any templates used inside _Cache could increase the number of combinations.
>
> But wait, there's more! How many does this have?
>
> struct Bar(string s) {
>     mixin(s);
> }
>
> If that's just one overload because of the impossibility of generating the options, what if we introduce a single static if inside it? Is that still one, or is it two? Do we count the possibilities introduced by a template mixin, but not by a string mixin? What if the template mixin comes from a template argument?
>
> Sorry if I come off as very dismissive right now - I kind of like the idea, but it seems impossible in practice.
>
>
>> I'm not sure about the procedure... should I post this as a question to the GitHub, so it can be discussed? pull 2351 or 8195?
>
> What you are suggesting would be a fundamental change in the language, and should be discussed in the digitalmars.D forum. Either PR is not the right place to discuss it. If there's any interest, you will have to write a DIP for how the change will work.
>
> --
>   Simen

My point, is that if it is impossible to catch all cases of template rewriting (which I'm advocating from the beginning) getOverloads should not be extended to templates, as this would be a fundamental change in the language.

On Monday, 23 April 2018 at 08:07:52 UTC, Alex wrote:
> On Monday, 23 April 2018 at 07:49:39 UTC, Simen Kjærås wrote:
>> That's not the definition of lowering used elsewhere, and so will lead to confusion and misunderstanding. I would strongly suggest you rethink your definition of lowering.
>
> There is no official definition. That's because some natural rewrite rules are implied, which are very general, I assume...

How official do you want it to be? That's the only definition in common use by others in the context of compilers.


> My point, is that if it is impossible to catch all cases of template rewriting (which I'm advocating from the beginning) getOverloads should not be extended to templates, as this would be a fundamental change in the language.

There is no template rewriting taking place in the language today - that seems to be a feature you are arguing for. getOverloads should return the overloads of a template as they are defined in the language today. Template overloads are mentioned in the D spec[0], and are clearly a real thing that it's useful to be able to manipulate.

You seem to be arguing against a feature on the basis that if the language were significantly different from what it is, the feature would be confusing. The language isn't that way, so the feature isn't confusing in that way.

Having getOverloads return template overloads solves a real issue right now, and would be useful even if your suggested change were implemented (though the behavior would be slightly different). If your suggested feature is impossible to implement, are you suggesting we simply throw our hands in the air and give up, instead of implementing a useful feature?

--
  Simen

[0]: https://dlang.org/concepts.html

On Monday, 23 April 2018 at 10:57:59 UTC, Simen Kjærås wrote:
>> There is no official definition. That's because some natural rewrite rules are implied, which are very general, I assume...
>
> How official do you want it to be? That's the only definition in common use by others in the context of compilers.

Stating the existence of rewriting rules is a complex thing. They is no place in the docu (which I could find until now) where they are defined. Therefore, I rely on something which is common.
https://en.wikipedia.org/wiki/Natural_transformation

We can discuss, whether a template is a functor, but I think this would go beyond the scope of this thread.

You mentioned a limited set of lowerings, defined by the language. Where can I find them?  I'm very interested in that.

> There is no template rewriting taking place in the language today - that seems to be a feature you are arguing for.

Semantic rewriting is not something which one can take control of. Either you have semantically equal constructs or not.

> getOverloads should return the overloads of a template as they are defined in the language today.

This is the topic of discussion. I argue, that getOverloads can (so far) act only on functions, because only there the term overload is well defined. I see, that it could be defined on templates too, but not in the way you do.

> Template overloads are mentioned in the D spec[0], and are clearly a real thing that it's useful to be able to manipulate.

Yes. But using the term "overload" for templates with the same name does not justify to wrongly apply existing features to more abstract objects. This is what you try to do by using getOverloads with templates.

>
> You seem to be arguing against a feature on the basis that if the language were significantly different from what it is, the feature would be confusing. The language isn't that way, so the feature isn't confusing in that way.

Well, as we both saw with the pragma output, the language is exactly that way I'm describing it. New getOverload functionality would introduce a new constraint in comparison to the current state.

>
> Having getOverloads return template overloads solves a real issue right now, and would be useful even if your suggested change were implemented (though the behavior would be slightly different).

What I'm saying is: it would be useful only then, if my suggested change were implemented. Otherwise, it is a new constraint and not a feature.

> [0]: https://dlang.org/concepts.html

On Monday, 23 April 2018 at 13:32:49 UTC, Alex wrote:
> On Monday, 23 April 2018 at 10:57:59 UTC, Simen Kjærås wrote:
>>> There is no official definition. That's because some natural rewrite rules are implied, which are very general, I assume...
>>
>> How official do you want it to be? That's the only definition in common use by others in the context of compilers.
>
> Stating the existence of rewriting rules is a complex thing. They is no place in the docu (which I could find until now) where they are defined. Therefore, I rely on something which is common.
> https://en.wikipedia.org/wiki/Natural_transformation
>
> We can discuss, whether a template is a functor, but I think this would go beyond the scope of this thread.
>
> You mentioned a limited set of lowerings, defined by the language. Where can I find them?  I'm very interested in that.

As with all things D, the only real spec is the compiler source code. :p :(

Again, lowerings are a compiler technique where one syntax is rewritten into a different syntax. They are hard-coded into the language (and optionally in the compiler - if a compiler writer finds an opportunity to use lowerings to simplify or otherwise improve the compiler without such a lowering being defined in the language, and with the same behavior, all the more power to him/her). Only these things are called lowerings. Template instantiations are not lowerings. Function inlining is not a lowering. Constant folding is not a lowering.

As far as I know, there is no canonical list of lowerings anywhere. :(


>> There is no template rewriting taking place in the language today - that seems to be a feature you are arguing for.
>
> Semantic rewriting is not something which one can take control of. Either you have semantically equal constructs or not.

And we don't, so don't worry about it. :p


>> getOverloads should return the overloads of a template as they are defined in the language today.
>
> This is the topic of discussion. I argue, that getOverloads can (so far) act only on functions, because only there the term overload is well defined. I see, that it could be defined on templates too, but not in the way you do.

But why not? Consider this case:

template foo() {}
template foo(int n) {}

In the compiler, this results in two `TemplateDeclaration`s being instantiated. Since they share the 'foo' name, they are considered an overload set by the compiler. The specific overload is chosen by overload resolution. They are iterated by the function overloadApply. The term template overload is also widely used in C++, from which D has inherited much.

Simply put, template overloads are clearly defined, everyone knows what they are, and many of us want a way to iterate over them, choose a specific overload in a different template, and just plain do sensible work with them.

If not how I define it, then how would the term 'overload' be defined for templates? We've established that creating a new overload for every potential static if breaks down in the presence of mixins, and is questionable in their absence. What I'm reading from your post is simply 'the current way is wrong, and there exists a better way'. If that's true, I'd love to hear what the better way is, but I fail to see that you've described it.


>> Template overloads are mentioned in the D spec[0], and are clearly a real thing that it's useful to be able to manipulate.
>
> Yes. But using the term "overload" for templates with the same name does not justify to wrongly apply existing features to more abstract objects. This is what you try to do by using getOverloads with templates.

It's a term with decades of history in C++, and has been used in D since its inception. Even C# mentions overloads for generic functions or types with different constraints.


>> You seem to be arguing against a feature on the basis that if the language were significantly different from what it is, the feature would be confusing. The language isn't that way, so the feature isn't confusing in that way.
>
> Well, as we both saw with the pragma output, the language is exactly that way I'm describing it.

Then, once more, I must have misunderstood your description. Proving that two templates are equivalent is in general impossible, since any amount of wasted computation could be performed before the end result is returned, and inputs must be tested exhaustively for the proof to be valid. The fact that two templates give the same result in one special case does not mean that they are equivalent in the general case, and the compiler needs to care about the general case.


> New getOverload functionality would introduce a new constraint in comparison to the current state.

The new functionality does not in any way stop you from using getOverloads in exactly the way it's been used before. In addition to that, you may pass an additional argument to it to get template overloads (actually, every symbol by that name in the specified aggregate). How is this in any way a constraint?


>> Having getOverloads return template overloads solves a real issue right now, and would be useful even if your suggested change were implemented (though the behavior would be slightly different).
>
> What I'm saying is: it would be useful only then, if my suggested change were implemented. Otherwise, it is a new constraint and not a feature.

That's just plain false. I have use cases for the feature right now, so it's provably useful. Can you describe in which ways your suggested change is superior, rather than simply asserting that it is?

--
  Simen

On Monday, 23 April 2018 at 14:22:13 UTC, Simen Kjærås wrote:
> As with all things D, the only real spec is the compiler source code. :p :(

:p

> Proving that two templates are equivalent is in general impossible, since any amount of wasted computation could be performed before the end result is returned, and inputs must be tested exhaustively for the proof to be valid. The fact that two templates give the same result in one special case does not mean that they are equivalent in the general case, and the compiler needs to care about the general case.

Ok, thats exactly the point. If you have functions

void foo() {}
void foo(int n) {}

There is no ambiguity which function will be chosen if it will be called.

If you have templates

// form 1
template Foo(int N) if (N & 1)    {} // A
template Foo(int N) if (!(N & 1)) {} // B

OR

// form 2
template foo(int N)
{
	static if(N & 1){} // A
	else{} // B
}

There is also no ambiguity which will be called.

However, getOverloads will behave differently.

This is not bad at all. But you have to admit, that while now, there is no way to distinguish form 1 and form 2, with the new getOverloads there will be.
This seems strange to me, because there is no reason to distinguish form 1 and form 2. (Because the callable code, which will be generated is the same, I hope... ?)

So, in particular, I'm not against the feature. And if the equivalence between form 1 and form 2 is gone, so what. But I don't understand the reasoning why something which is now equal won't be equal any more later?

On Monday, 23 April 2018 at 15:00:38 UTC, Alex wrote:
> Ok, thats exactly the point. If you have functions
>
> void foo() {}
> void foo(int n) {}
>
> There is no ambiguity which function will be chosen if it will be called.
>
> If you have templates
>
> // form 1
> template Foo(int N) if (N & 1)    {} // A
> template Foo(int N) if (!(N & 1)) {} // B
>
> OR
>
> // form 2
> template foo(int N)
> {
> 	static if(N & 1){} // A
> 	else{} // B
> }
>
> There is also no ambiguity which will be called.
>
> However, getOverloads will behave differently.

Now I understand what you are getting at. Thanks a bunch for being so patient with me. :)

And you are right, of course. But I'm just as interested in the templates themselves as the instantiated results.


> This is not bad at all. But you have to admit, that while now, there is no way to distinguish form 1 and form 2, with the new getOverloads there will be.
> This seems strange to me, because there is no reason to distinguish form 1 and form 2. (Because the callable code, which will be generated is the same, I hope... ?)
>
> So, in particular, I'm not against the feature. And if the equivalence between form 1 and form 2 is gone, so what. But I don't understand the reasoning why something which is now equal won't be equal any more later?

Ah, but I'm not looking to instantiate the templates, but to learn about them - how many parameters do they take? Are their UDAs different, so that I should warn the programmer? Must I wrap them in different ways?

I hope we now both understand what the other person means and want to achieve. :)

--
  Simen