On Sunday, 2 August 2020 at 04:08:00 UTC, Bruce Carneal wrote:
>
> I was thinking more about gaining access to a mutable form which could be converted to/from concrete types as represented by the compiler under the hood rather than the current methods of creating types in the source.  Note that "mixin" reduces to the others above and the others, of course, reduce to the compiler's internal form.
>
> ...
>
> Yes, unrestricted type mutation capability is a non goal. So today we have a large and growing zoo of forms that we can utilize to work with types.  A mutable class convertible to/from whatever the compiler is using "under the hood" might be a better way.  It might be used to implement the zoo while limiting further special casing.  Switching metaphors, we'd have less unsprung weight going forward.

Correct me if I'm wrong, but it sounds to me like what you have in mind is something like this:

// type function
alias Tuple(Ts...)
{
    TypeBuilder b;
    b.kind = Kind.struct;
    foreach (T; Ts)
        b.addMember(T, ""); // anonymous member
    return b.toType;
}

That is, we have some mutable representation of a type, which we can manipulate via some compiler-defined API, and once we're done, we convert the result to a "real", immutable type, which can be used in other parts of the program.

I can see how this sort of thing might be useful. Indeed, if you generalize this design from just types to *all* kinds of AST nodes (FunctionBuilder, ExpressionBuilder, etc.), what you end up with is essentially a procedural macro system.

The problem is that D already has features for performing these kinds of AST manipulations: static if, static foreach, and mixins. From a language-design perspective, adding new features that duplicate the functionality of existing ones is generally not a good idea.

Rather that attempt to *replace* D's existing metaprogramming features with something entirely new, I think it would be much better to *extend* them with language features that allow us to overcome their current limitations.


> Factoring out naming from the "anonymous" structural aspects of types seems like a good way to go.  If people want to match on type structure for some reason, great.  If they want to create ginormous names, well, OK.

I don't understand what structural vs. nominal typing has to do with the rest of your post.

On Sunday, 2 August 2020 at 15:04:07 UTC, Paul Backus wrote:
> On Sunday, 2 August 2020 at 12:31:00 UTC, Stefan Koch wrote:
>> Take this for example.
>> int[] iota(int start, int finish)
>> {
>>     int[] result = [];
>>     result.length = finish - start;
>>     foreach(i;start .. finish)
>>     {
>>         result[i - start] = i;
>>     }
>>     return result;
>> }
>>
>> Whereas the recursive function for this,
>> is something I do not even want to put on here.
>> Because it's way to complicated for this simple task.
>
> In fact, the naive recursive version of iota is actually even simpler:
>
> int[] iota(int start, int finish)
> {
>     if (start > finish)
>         return [];
>     else
>         return [start] ~ iota(start + 1, finish);
> }
>
> Again, the problem with this function is not complexity or readability, but performance. It performs many unnecessary memory allocations, and consumes (finish - start) stack frames instead of 1.

How is that easier?

It requires you to have a stack, which also means that you have to keep track of the stack frames in order to execute this in your head.
this particular function allocates N literals array + and does N concatenations.
Which when naively implemented consumes N*N + 2N words of memory.

On Sunday, 2 August 2020 at 17:27:43 UTC, Stefan Koch wrote:
>
> How is that easier?
>
> It requires you to have a stack, which also means that you have to keep track of the stack frames in order to execute this in your head.

To some extent this is a matter of experience and personal taste.

If you are comfortable reading recursive code, it is quite simple to understand a function with a single base case and a single recursive case. There is no need to keep track of individual stack frames in your head.

Perhaps my experience in this regard is different from the rest of the D community? I learned about recursion in my first year of undergraduate education, so I assumed this sort of thing would be considered common knowledge among experienced programmers.

On Sunday, 2 August 2020 at 16:38:30 UTC, Paul Backus wrote:
> On Sunday, 2 August 2020 at 04:08:00 UTC, Bruce Carneal wrote:
>>
>> I was thinking more about gaining access to a mutable form which could be converted to/from concrete types as represented by the compiler under the hood rather than the current methods of creating types in the source.  Note that "mixin" reduces to the others above and the others, of course, reduce to the compiler's internal form.
>>
>> ...
>>
>> Yes, unrestricted type mutation capability is a non goal. So today we have a large and growing zoo of forms that we can utilize to work with types.  A mutable class convertible to/from whatever the compiler is using "under the hood" might be a better way.  It might be used to implement the zoo while limiting further special casing.  Switching metaphors, we'd have less unsprung weight going forward.
>
> Correct me if I'm wrong, but it sounds to me like what you have in mind is something like this:
>
> // type function
> alias Tuple(Ts...)
> {
>     TypeBuilder b;
>     b.kind = Kind.struct;
>     foreach (T; Ts)
>         b.addMember(T, ""); // anonymous member
>     return b.toType;
> }
>
> That is, we have some mutable representation of a type, which we can manipulate via some compiler-defined API, and once we're done, we convert the result to a "real", immutable type, which can be used in other parts of the program.

Yes.  That was the idea.  As sketched the "mutable" form would initially only allow locally verifiable mutations IOW an attempted mutation would either error out or leave you with a valid type.  After thinking about it a little more, and reading about LLVM's woes wrt a more generally mutable type representation, I'd say such mutation constraints would be very useful if not essential long term.

>
> I can see how this sort of thing might be useful. Indeed, if you generalize this design from just types to *all* kinds of AST nodes (FunctionBuilder, ExpressionBuilder, etc.), what you end up with is essentially a procedural macro system.
>
> The problem is that D already has features for performing these kinds of AST manipulations: static if, static foreach, and mixins. From a language-design perspective, adding new features that duplicate the functionality of existing ones is generally not a good idea.
>
> Rather that attempt to *replace* D's existing metaprogramming features with something entirely new, I think it would be much better to *extend* them with language features that allow us to overcome their current limitations.

I think that the metric to use here is capability/complexity going forward.  Legacy "dead weight" is an issue but hopefully the new capabilities can help us there, in a big way.

For me, "complexity" equates almost perfectly to readability.  If performant code written using a new capability isn't more readable, more directly comprehensible, then it's a no-go.  If a new capability doesn't admit a performant readable implementation it's also a no-go.

>
>> Factoring out naming from the "anonymous" structural aspects of types seems like a good way to go.  If people want to match on type structure for some reason, great.  If they want to create ginormous names, well, OK.
>
> I don't understand what structural vs. nominal typing has to do with the rest of your post.

It was just in the context of "if we're introducing this new form, what all should it enable?".  There are some times when you care which names refer to an underlying "anonymous" type and other times when working with the factored form is the focus.

Attributes and linkage directives might also be factored explicitly but I think those would better be handled using filter/set operations.

Obviously I'm winging it here so I appreciate the feedback.  Better/cheaper to weed out bad ideas early.

On Sunday, 2 August 2020 at 18:21:17 UTC, Paul Backus wrote:
> On Sunday, 2 August 2020 at 17:27:43 UTC, Stefan Koch wrote:
>>
>> How is that easier?
>>
>> It requires you to have a stack, which also means that you have to keep track of the stack frames in order to execute this   Iin your head.
>
> To some extent this is a matter of experience and personal taste.
>
> If you are comfortable reading recursive code, it is quite simple to understand a function with a single base case and a single recursive case. There is no need to keep track of individual stack frames in your head.
>
> Perhaps my experience in this regard is different from the rest of the D community? I learned about recursion in my first year of undergraduate education, so I assumed this sort of thing would be considered common knowledge among experienced programmers.

I think it's all about readability given the performance constraints.

Your type function example earlier in this thread employs iteration and is dead simple.  Not much to improve there.

On Sunday, 2 August 2020 at 20:27:53 UTC, Bruce Carneal wrote:
>
> I think it's all about readability given the performance constraints.
>
> Your type function example earlier in this thread employs iteration and is dead simple.  Not much to improve there.

You're conflating two separate issues. The existing alternative to using a TypeBuilder + iteration is `static foreach`--which is also iterative.

The question you should answer, if you want to convince people that TypeBuilder (or something like it) is worth adding, is "how is this better than `static foreach` and `static if`?"

On Sunday, 2 August 2020 at 20:42:35 UTC, Paul Backus wrote:
> On Sunday, 2 August 2020 at 20:27:53 UTC, Bruce Carneal wrote:
>>
>> I think it's all about readability given the performance constraints.
>>
>> Your type function example earlier in this thread employs iteration and is dead simple.  Not much to improve there.
>
> You're conflating two separate issues. The existing alternative to using a TypeBuilder + iteration is `static foreach`--which is also iterative.
>
> The question you should answer, if you want to convince people that TypeBuilder (or something like it) is worth adding, is "how is this better than `static foreach` and `static if`?"

The "conflation" was my attempt to fold in the OT recursion sub-thread that I, unfortunately, engendered early on when I misspoke.


How is this better than static foreach and friends?  WRT types it should be more efficient, more readable, and more general.

The readability and generality claims can be examined via speculative coding,  additional examples to go with your inaugural type function.

The performance claim verification would have to wait for a prototype but in the mean time we have informed opinion from the front-end experts.

My main objective here is to raise the possibility of a broadly applicable meta programming advance that could roll up a bunch of special cases now and expand the reach of "mere mortal" metaprogrammers in the future.

I'm exploring here, not crusading.  If there is interest now, great.  If not, well, that's information too.  Something like what has been sketched in this thread may be a bridge too far.  It may be much better suited to an sdc revival or some other front-end development effort.  It may need to go in to the D3 basket.  It may follow many other ideas in to the dust bin.  We'll see.

On Sunday, 2 August 2020 at 20:42:35 UTC, Paul Backus wrote:
> On Sunday, 2 August 2020 at 20:27:53 UTC, Bruce Carneal wrote:
>>
>> I think it's all about readability given the performance constraints.
>>
>> Your type function example earlier in this thread employs iteration and is dead simple.  Not much to improve there.
>
> You're conflating two separate issues. The existing alternative to using a TypeBuilder + iteration is `static foreach`--which is also iterative.
>
> The question you should answer, if you want to convince people that TypeBuilder (or something like it) is worth adding, is "how is this better than `static foreach` and `static if`?"

static foreach and static if come with
compile time performance prices to pay.

it comes down to having to do semantic processing in a piece-wise fashion multiple times.

The type function approach I am working on does semantic processing of invariant parts only once.

Whereas the static foreach body cannot know about invariant regions, (that's a general statement (in special cases it might be possible to implement such awareness (I would not advise it however)))

On Sunday, 2 August 2020 at 23:04:45 UTC, Stefan Koch wrote:
> On Sunday, 2 August 2020 at 20:42:35 UTC, Paul Backus wrote:
>> On Sunday, 2 August 2020 at 20:27:53 UTC, Bruce Carneal wrote:
>>>
>>> I think it's all about readability given the performance constraints.
>>>
>>> Your type function example earlier in this thread employs iteration and is dead simple.  Not much to improve there.
>>
>> You're conflating two separate issues. The existing alternative to using a TypeBuilder + iteration is `static foreach`--which is also iterative.
>>
>> The question you should answer, if you want to convince people that TypeBuilder (or something like it) is worth adding, is "how is this better than `static foreach` and `static if`?"
>
> static foreach and static if come with
> compile time performance prices to pay.
>
> it comes down to having to do semantic processing in a piece-wise fashion multiple times.
>
> The type function approach I am working on does semantic processing of invariant parts only once.
>
> Whereas the static foreach body cannot know about invariant regions, (that's a general statement (in special cases it might be possible to implement such awareness (I would not advise it however)))

I'd add that working with sets of types should become very readable and efficient.  Doable now but dead simple when types live as normal objects in the compile time environment.  Arrays, slicing, all the good stuff.

I don't see a problem with operating with these objects at runtime either.  In process code gen wouldn't be available, at least not absent something like a LLVM/jit hookup, but you could play with it all in a more debuggable environment.