Summary:
Can a typesafe D variadic function, or D variadic template pass its parameters to a C variadic function?

Background:
I maintain a library binding [0] to htslib, a high-performance and very widely used C library for high-throughput sequencing (hts) data files. We use this internally and haven't polished it for a release on the announce forum or biomed twitter, etc. yet.

In the course of upgrading it to support the latest API/ABI (htslib-1.10 / so.3), I need to add support for several new functions.

One of these is a variadic function with the signature:

`int sam_hdr_add_line(sam_hdr_t *h, const char *type, ...);`

Of course, we can call this directly from D with hardcoded parameters. However, one of the focuses of our library is "bindings + wrappers" to make this feel more like native D. Thus, we want a variadic function to which we may pass D strings (it is also a struct member function).

With help from Herringway on IRC, we came up with a solution using mixin:


```
    /// Add a single line to an existing header
    auto addLine(T...)(RecordType type, T kvargs)
    if(kvargs.length > 0 && isSomeString!(T[0]))
    {
        static assert (kvargs.length %2 == 0);   // K-V pairs => even number of variadic args

        string varargMagic(size_t len)
        {
            string args = "sam_hdr_add_line(this.h, type.ptr, ";
            for(int i=0; i<len; i++)
                args ~= "toStringz(kvargs[" ~ i.to!string ~ "]), ";
            args ~= "null)";
            return args;
        }

        return mixin(varargMagic(kvargs.length));
    }
```

Interestingly, compilation fails if the mixin consists only of the comma-separated parameters ("Comma expression" [1])


Question:
If a variadic template, despite presenting to the user a "dynamic array", MUST know its parameter list at compile-time, is there a way (other than with mixins as shown) to pass this parameter list to extern(C) linkage function with variadic parameters?


(bonus question: if yes, can it be done with typesafe variadic function where I believe parameter list is known at either compile time OR runtime, depending on how called)


Thanks all in advance



[0] dhtslib
https://code.dlang.org/packages/dhtslib
https://github.com/blachlylab/dhtslib

[1] https://dlang.org/spec/expression.html#expression

On 9/13/20 12:55 PM, James Blachly wrote:
> Summary:
> Can a typesafe D variadic function, or D variadic template pass its parameters to a C variadic function?
> 
> Background:
> I maintain a library binding [0] to htslib, a high-performance and very widely used C library for high-throughput sequencing (hts) data files. We use this internally and haven't polished it for a release on the announce forum or biomed twitter, etc. yet.
> 
> In the course of upgrading it to support the latest API/ABI (htslib-1.10 / so.3), I need to add support for several new functions.
> 
> One of these is a variadic function with the signature:
> 
> `int sam_hdr_add_line(sam_hdr_t *h, const char *type, ...);`
> 
> Of course, we can call this directly from D with hardcoded parameters. However, one of the focuses of our library is "bindings + wrappers" to make this feel more like native D. Thus, we want a variadic function to which we may pass D strings (it is also a struct member function).
> 
> With help from Herringway on IRC, we came up with a solution using mixin:
> 
> 
> ```
>      /// Add a single line to an existing header
>      auto addLine(T...)(RecordType type, T kvargs)
>      if(kvargs.length > 0 && isSomeString!(T[0]))
>      {
>          static assert (kvargs.length %2 == 0);   // K-V pairs => even number of variadic args
> 
>          string varargMagic(size_t len)
>          {
>              string args = "sam_hdr_add_line(this.h, type.ptr, ";
>              for(int i=0; i<len; i++)
>                  args ~= "toStringz(kvargs[" ~ i.to!string ~ "]), ";
>              args ~= "null)";
>              return args;
>          }
> 
>          return mixin(varargMagic(kvargs.length));
>      }
> ```
> 
> Interestingly, compilation fails if the mixin consists only of the comma-separated parameters ("Comma expression" [1])
> 
> 
> Question:
> If a variadic template, despite presenting to the user a "dynamic array", MUST know its parameter list at compile-time, is there a way (other than with mixins as shown) to pass this parameter list to extern(C) linkage function with variadic parameters?

Was just talking about this exact problem with Adam Ruppe. Unfortunately, because the parameters are an expression tuple, and not a compile-time tuple, you can't use stuff like staticMap.

Manu's ... dip would be perfect for this:

return sam_hdr_add_line(this.h, this.ptr, toStringz(kvargs)..., null);

I think the only way it works today is if you use the mixin technique.

-Steve

On Sunday, 13 September 2020 at 17:23:42 UTC, Steven Schveighoffer wrote:
> On 9/13/20 12:55 PM, James Blachly wrote:
>> 
>> ```
>>      /// Add a single line to an existing header
>>      auto addLine(T...)(RecordType type, T kvargs)
>>      if(kvargs.length > 0 && isSomeString!(T[0]))
>>      {
>>          static assert (kvargs.length %2 == 0);   // K-V pairs => even number of variadic args
>> 
>>          string varargMagic(size_t len)
>>          {
>>              string args = "sam_hdr_add_line(this.h, type.ptr, ";
>>              for(int i=0; i<len; i++)
>>                  args ~= "toStringz(kvargs[" ~ i.to!string ~ "]), ";
>>              args ~= "null)";
>>              return args;
>>          }
>> 
>>          return mixin(varargMagic(kvargs.length));
>>      }
>> ```
>> 
>> Interestingly, compilation fails if the mixin consists only of the comma-separated parameters ("Comma expression" [1])
>> 
>> 
>> Question:
>> If a variadic template, despite presenting to the user a "dynamic array", MUST know its parameter list at compile-time, is there a way (other than with mixins as shown) to pass this parameter list to extern(C) linkage function with variadic parameters?
>
> Was just talking about this exact problem with Adam Ruppe. Unfortunately, because the parameters are an expression tuple, and not a compile-time tuple, you can't use stuff like staticMap.

You actually can, if you define the right kind of helper function:

    /// Add a single line to an existing header
    auto addLine(T...)(RecordType type, T kvargs)
    if(kvargs.length > 0 && isSomeString!(T[0]))
    {
        static assert (kvargs.length %2 == 0);   // K-V pairs => even number of variadic args
        immtuable(char)* argToStringz(alias arg)()
        {
            return toStringz(arg);
        }

        return sam_hdr_add_line(this.h, this.ptr, staticMap!(argToStringz, kvargs), null);
    }

The clever trick here is that, because of optional parentheses [1], `argToStringz!(kvargs[i])` can be interpreted either as the name of a function or a function call, depending on the context it appears in.

[1] https://dlang.org/spec/function.html#optional-parenthesis

> ```
>      /// Add a single line to an existing header
>      auto addLine(T...)(RecordType type, T kvargs)
>      if(kvargs.length > 0 && isSomeString!(T[0]))
>      {
>          static assert (kvargs.length %2 == 0);   // K-V pairs => even number of variadic args
> 
>          string varargMagic(size_t len)
>          {
>              string args = "sam_hdr_add_line(this.h, type.ptr, ";
>              for(int i=0; i<len; i++)
>                  args ~= "toStringz(kvargs[" ~ i.to!string ~ "]), ";
>              args ~= "null)";
>              return args;
>          }
> 
>          return mixin(varargMagic(kvargs.length));
>      }
> ```
[...]
> Question:
> If a variadic template, despite presenting to the user a "dynamic array", MUST know its parameter list at compile-time, is there a way (other than with mixins as shown) to pass this parameter list to extern(C) linkage function with variadic parameters?

Easy peasy:

        import std.meta: Repeat;
        Repeat!(kvargs.length, const(char)*) zs;
        foreach (i, ref z; zs) z = toStringz(kvargs[i]);
        return sam_hdr_add_line(this.h, type.ptr, zs, null);

By the way, `kvargs` is not a dynamic array. Its length is not dynamic, and it's not an array.

Also, you don't just want to pass the parameters forward. That would be trivial: `sam_hdr_add_line(this.h, type.ptr, kvargs, null)`. You want to run them through another function first. That's where the difficulty comes from.

> (bonus question: if yes, can it be done with typesafe variadic function where I believe parameter list is known at either compile time OR runtime, depending on how called)

I don't see when it would matter how the function is called, but you can declare it in two different ways:

1) True typesafe variadic:

    auto addLine(RecordType type, string[] kvargs ...)

2) Template + typesafe variadic:

    auto addLine(size_t n)(RecordType type, string[n] kvargs ...)

In the first one, `kvargs.length` is a dynamic value. You can't use it to generate the arguments for a `sam_hdr_add_line` call.

In the second one, `kvargs.length` is a static value. So you can do the same things as in the `T...` template. And just like the `T...` template, it will generate a new function for every distinct `kvargs.length`.

On 9/13/20 2:35 PM, ag0aep6g wrote:
> Easy peasy:
> 
>          import std.meta: Repeat;
>          Repeat!(kvargs.length, const(char)*) zs;
>          foreach (i, ref z; zs) z = toStringz(kvargs[i]);
>          return sam_hdr_add_line(this.h, type.ptr, zs, null);
> 
Great, thank you!

> By the way, `kvargs` is not a dynamic array. Its length is not dynamic, and it's not an array.

I was incorrectly recalling the error message compiler emitted when using the typesafe variadic FUNCTION (your method 1, below)

> Also, you don't just want to pass the parameters forward. That would be trivial: `sam_hdr_add_line(this.h, type.ptr, kvargs, null)`. You want to run them through another function first. That's where the difficulty comes from.

Right, I indeed left that out of the problem statement; the trivial variadic template params pass right on through which is awesome.

> I don't see when it would matter how the function is called, but you can declare it in two different ways:

My assumption is that if called passing a [runtime] dynamic array, of course the parameter list cannot be known, but the (below) "Type 1 True typesafe variadic" can also be called with a fixed parameter list known at compile-time.

> 1) True typesafe variadic:
> 
>      auto addLine(RecordType type, string[] kvargs ...)
> 
> 2) Template + typesafe variadic:
> 
>      auto addLine(size_t n)(RecordType type, string[n] kvargs ...)
> 
> In the first one, `kvargs.length` is a dynamic value. You can't use it to generate the arguments for a `sam_hdr_add_line` call.

My point of surprise was that -- depending on how invoked ( f(anArray) versus f(1,2,3) the compiler may know at compile-time the parameter list. But from a complexity standpoint it makes sense that it is nonetheless not possible to use.

> In the second one, `kvargs.length` is a static value. So you can do the same things as in the `T...` template. And just like the `T...` template, it will generate a new function for every distinct `kvargs.length`.

Great, I never thought of parameterizing as a static array. This looks the best IMO.

Just a observation, from the questions & answers in this thread and mine[1]: I think meta-programming in D is somehow like C++, it starts becoming a baroque language. The language is complex enough that there may be ways to get things done, but it's just quite difficult for ordinary users to find *THE* way.


[1] https://forum.dlang.org/thread/ujcbioaghvofwowihygq@forum.dlang.org

On 9/13/20 2:35 PM, Paul Backus wrote:
> On Sunday, 13 September 2020 at 17:23:42 UTC, Steven Schveighoffer wrote:
>> On 9/13/20 12:55 PM, James Blachly wrote:
>>>
>>> ```
>>>      /// Add a single line to an existing header
>>>      auto addLine(T...)(RecordType type, T kvargs)
>>>      if(kvargs.length > 0 && isSomeString!(T[0]))
>>>      {
>>>          static assert (kvargs.length %2 == 0);   // K-V pairs => even number of variadic args
>>>
>>>          string varargMagic(size_t len)
>>>          {
>>>              string args = "sam_hdr_add_line(this.h, type.ptr, ";
>>>              for(int i=0; i<len; i++)
>>>                  args ~= "toStringz(kvargs[" ~ i.to!string ~ "]), ";
>>>              args ~= "null)";
>>>              return args;
>>>          }
>>>
>>>          return mixin(varargMagic(kvargs.length));
>>>      }
>>> ```
>>>
>>> Interestingly, compilation fails if the mixin consists only of the comma-separated parameters ("Comma expression" [1])
>>>
>>>
>>> Question:
>>> If a variadic template, despite presenting to the user a "dynamic array", MUST know its parameter list at compile-time, is there a way (other than with mixins as shown) to pass this parameter list to extern(C) linkage function with variadic parameters?
>>
>> Was just talking about this exact problem with Adam Ruppe. Unfortunately, because the parameters are an expression tuple, and not a compile-time tuple, you can't use stuff like staticMap.
> 
> You actually can, if you define the right kind of helper function:
> 
>      /// Add a single line to an existing header
>      auto addLine(T...)(RecordType type, T kvargs)
>      if(kvargs.length > 0 && isSomeString!(T[0]))
>      {
>          static assert (kvargs.length %2 == 0);   // K-V pairs => even number of variadic args
>          immtuable(char)* argToStringz(alias arg)()
>          {
>              return toStringz(arg);
>          }
> 
>          return sam_hdr_add_line(this.h, this.ptr, staticMap!(argToStringz, kvargs), null);
>      }
> 
> The clever trick here is that, because of optional parentheses [1], `argToStringz!(kvargs[i])` can be interpreted either as the name of a function or a function call, depending on the context it appears in.

That's cool. And horrific at the same time :) I mean the templates that you have to instantiate for this...

I would prefer the mixin solution, even though it's uglier. I think something that abstracts that out would be a nice thing to have for std.meta.

-Steve