On Wednesday, 7 October 2020 at 14:01:51 UTC, Daniel K wrote:
> If we boil down the Type Function proposal to its fundamental idea, advantages, and deficiencies. What Do we get?
>
> [...]
>
> Advantage:
> Mutating a set of types in an imperative style. Something that is already quite simple, and requires virtually the same, or even less code in practice when written with existing language features.
>
> Disadvantage: A new feature people will have to learn and understand, document, maintain, bugfix.
> No actual substantial future gain.
>
> Is this wrong?

The big advantage you're missing is that, because type functions are so restricted in what they can do, they can be implemented much more efficiently in the compiler. For large programs that make heavy use of D's metaprogramming features, that can really add up.

On Wednesday, 7 October 2020 at 14:01:51 UTC, Daniel K wrote:
> On Wednesday, 7 October 2020 at 12:56:42 UTC, Adam D. Ruppe wrote:
>> On Wednesday, 7 October 2020 at 12:37:48 UTC, Stefan Koch wrote:
>>> On Wednesday, 7 October 2020 at 12:30:15 UTC, Andrei Alexandrescu wrote:
>>>> Incumbency is a huge matter in programming language design. Of course I would not propose another way of doing the same thing.
>>>
>>> Perhaps you can say this in different words?
>>
>> Basically we just prefer to work with what we have before adding new stuff to the language.
>>
>> So whatever features get added first have an automatic advantage in any comparison just because they are already there.
>
> I agree.
>
> And existing features got added after already having their fundamental ideas, advantages, and deficiencies scrutinized.
>
> The fact that we have meta-programming in the form we have, is not by chance. It is a super strong and capable design.
> I am biased, but I would say "state of the art".
>
> If we boil down the Type Function proposal to its fundamental idea, advantages, and deficiencies. What Do we get?
>
> Fundamental idea:
> Language level reification of a set of types, making a set of types mutable to enable imperative programming style, for what is essentially only changing that set of types.
> But at that, only in the context of a special kind of CTFE-like function.

This is quite close, thanks for your description.
The only thing I would change is "CTFE-like" to CTFE.

The language changes are as follows:
  - Introduce a new basic type "alias" which is the type of types. This type does not have a runtime representation, and can only exist at compile time/CTFE.
  - Variables of this type emulate the interface that Type parameters in templates have.
  - Add an implicit conversion such that any type implicitly converts to alias.
  - Allow functions to be called with Types (TypeExps are a concept which already exists in the compiler but not in the language)
  - Functions which take alias parameters or types drives of alias parameters or (such as alias[], or structs which contain alias fields) or return such values cannot be called at runtime, and will never be generated in the binary (this is because alias does not have an ABI representation but helps out with compile speed as well).
  - Lastly (and this one is actually one I am not quite happy with) but it's needed is the alias[] -> TypeTuple via applying .tupleof to a statically determined (compile-time constant) alias[].

The CTFE functionality is 100% reused.
(which also means when newCTFE hits complex type functions get a speed boost)
The alias type is integrated into the type-system except for the special handling of alias[].tupleof  it behaves as expected.
For example you can do use types as keys for associative arrays if you wish to.
Which then allows more freedom in algorithm design in the domain of introspection/ compile time code-generation in particular.

> Disadvantage: A new feature people will have to learn and understand, document, maintain, bugfix.
> No actual substantial future gain.

As for the disadvantages. Yes it's a new feature but it blends so well with the current language that people on here post correct type function code, (presumably) without having a compiler that is capable of running it. (It's of course possible that claptrap cloned my branch and run the code, but I don't think so ...)

As for the feature gain ... well it's akin to comparing c++ templates with D templates.
They are equivalent in power, just that one easier to write than the other.

@Daniel K: I am thankful for your analysis, would you be opposed if I reused your characterization in the DIP?

On Wed, Oct 07, 2020 at 10:10:56AM -0400, Steven Schveighoffer via Digitalmars-d wrote: [...]
> memchr is also the secret sauce to how iopipe was able to beat Phobos byLine and getline.
> 
> I would like to go back to it at some point and see if it can be improved.
> 
> I know that part of the difference would be due to the opaque function call (this cannot be inlined), though perhaps memchr is an intrinsic?
[...]

I wouldn't be surprised if optimizing compilers like ldc treated it as an intrinsic.  IIRC, ldc already recognizes memcpy as an intrinsic and therefore knows how to inline it / substitute it with suitable instructions in special cases, even though it's supposed to be "opaque".


T

-- 
Жил-был король когда-то, при нём блоха жила.

On Wednesday, 7 October 2020 at 14:10:56 UTC, Steven Schveighoffer wrote:
> I know that part of the difference would be due to the opaque function call (this cannot be inlined), though perhaps memchr is an intrinsic?

You can test individual bytes in a simd register in a single instruction. Will clearly destroy any other implementation...

On Wednesday, 7 October 2020 at 14:40:41 UTC, Stefan Koch wrote:
> On Wednesday, 7 October 2020 at 14:01:51 UTC, Daniel K wrote:
>> On Wednesday, 7 October 2020 at 12:56:42 UTC, Adam D. Ruppe wrote:
>>> On Wednesday, 7 October 2020 at 12:37:48 UTC, Stefan Koch wrote:
>>>> On Wednesday, 7 October 2020 at

> As for the disadvantages. Yes it's a new feature but it blends so well with the current language that people on here post correct type function code, (presumably) without having a compiler that is capable of running it. (It's of course possible that claptrap cloned my branch and run the code, but I don't think so ...)

I just based it off what youve posted in this thread.

Maybe my perspective as a relative newbie to D makes me see that as a much bigger payoff than some of the resident greybeards.

On Wednesday, 7 October 2020 at 14:40:41 UTC, Stefan Koch wrote:
> @Daniel K: I am thankful for your analysis, would you be opposed if I reused your characterization in the DIP?

You are free to use any part of it, if it doesn't reference back to me or "the community".
But if you use it to insinuate endorsement from me or "the community", you will have to include all of the opposing points. No cherry picking.

/Daniel K

On Wednesday, 7 October 2020 at 15:00:16 UTC, Daniel K wrote:
> On Wednesday, 7 October 2020 at 14:40:41 UTC, Stefan Koch wrote:
>> @Daniel K: I am thankful for your analysis, would you be opposed if I reused your characterization in the DIP?
>
> You are free to use any part of it, if it doesn't reference back to me or "the community".
> But if you use it to insinuate endorsement from me or "the community", you will have to include all of the opposing points. No cherry picking.
>
> /Daniel K

That is fair.
Thanks!

On Wednesday, 7 October 2020 at 14:59:04 UTC, claptrap wrote:
> On Wednesday, 7 October 2020 at 14:40:41 UTC, Stefan Koch wrote:
>> On Wednesday, 7 October 2020 at 14:01:51 UTC, Daniel K wrote:
>>> On Wednesday, 7 October 2020 at 12:56:42 UTC, Adam D. Ruppe wrote:
>>>> On Wednesday, 7 October 2020 at 12:37:48 UTC, Stefan Koch wrote:
>>>>> On Wednesday, 7 October 2020 at
>
>> As for the disadvantages. Yes it's a new feature but it blends so well with the current language that people on here post correct type function code, (presumably) without having a compiler that is capable of running it. (It's of course possible that claptrap cloned my branch and run the code, but I don't think so ...)
>
> I just based it off what youve posted in this thread.
>
> Maybe my perspective as a relative newbie to D makes me see that as a much bigger payoff than some of the resident greybeards.

You might be happy to see that this exmaple just works with talias_master:

---
alias type = alias;

struct KV
{
    string key;
    type value;
}

auto makeAA(KV[] kvs ...)
{
    type[string] result;
    foreach(kv;kvs)
    {
        result[kv.key] = kv.value;
    }
    return result;
}

pragma(msg, makeAA(KV("u32", uint), KV("kv", KV)));
// outputs: ["u32":(uint), "kv":(KV)]
--

On Wednesday, 7 October 2020 at 14:41:18 UTC, H. S. Teoh wrote:
> On Wed, Oct 07, 2020 at 10:10:56AM -0400, Steven Schveighoffer via Digitalmars-d wrote: [...]
>> memchr is also the secret sauce to how iopipe was able to beat Phobos byLine and getline.
>> 
>> I would like to go back to it at some point and see if it can be improved.
>> 
>> I know that part of the difference would be due to the opaque function call (this cannot be inlined), though perhaps memchr is an intrinsic?
> [...]
>
> I wouldn't be surprised if optimizing compilers like ldc treated it as an intrinsic.  IIRC, ldc already recognizes memcpy as an intrinsic and therefore knows how to inline it / substitute it with suitable instructions in special cases, even though it's supposed to be "opaque".

It's my understanding that LTO could be used to cross the language boundary as well, allowing otherwise opaque function calls to be inlined. It would require having the function compiled to IR code available at build time.

Related to earlier in this subthread - I regularly work with data file 20 bytes per line and others 5000 bytes per line. It's been my experience that the two can have quite different performance characteristics for the same operation. As pointed out, it is sometimes hard to optimize for both.

--Jon

On Wednesday, 7 October 2020 at 14:43:41 UTC, Ola Fosheim Grøstad wrote:
> You can test individual bytes in a simd register in a single instruction. Will clearly destroy any other implementation...

From 2016:

https://gms.tf/stdfind-and-memchr-optimizations.html