On Wednesday, 27 January 2021 at 10:33:53 UTC, Mike Parker wrote:
> ...

Reposting this here at Walter's request:

Finally, and I'm sorry if I missed this being described in the DIP, but consider
this case (and similar):

```
void f(T)(T array)
if(isArray!T)
{
    //...
}

f(i"...");
```

What would happen here? My best guess is that it won't compile unless written as `f(i"...".idup)` which then introduces an odd discrepancy where there's certain usages that still require explicitly calling .idup, e.g. `i"...".idup.splitter`

On 1/29/21 6:49 AM, SealabJaster wrote:
> On Wednesday, 27 January 2021 at 10:33:53 UTC, Mike Parker wrote:
>> ...
> 
> Reposting this here at Walter's request:
> 
> Finally, and I'm sorry if I missed this being described in the DIP, but consider
> this case (and similar):
> 
> ```
> void f(T)(T array)
> if(isArray!T)
> {
>      //...
> }
> 
> f(i"...");
> ```
> 
> What would happen here? My best guess is that it won't compile unless written as `f(i"...".idup)` which then introduces an odd discrepancy where there's certain usages that still require explicitly calling .idup, e.g. `i"...".idup.splitter`
> 

Sorry for not responding in the discussion thread.

the above will work fine with the DIP. What will happen is that isArray!(interp!"...") will be false, and there will be no match for f. Therefore, the compiler will rewrite as f(i"...".idup), which will translate correctly to f("...").

If you still have questions, please ask on discussion thread and I can answer them.

-Steve

On 1/29/21 3:28 AM, Walter Bright wrote:
>  > provides a call that is free of sql injection attacks
> 
> This is a strong claim that requires substantiation, especially since sql injection attacks are a critical problem.

It's trivially true. The mysql_query function can know that interp!"SELECT * FROM" type cannot be from sql injection because the string was known at compile time. All runtime parameters are identified because they are NOT interp structs, and therefore can use the correct mechanism (prepared statements) that does not suffer from sql injection attacks.

-Steve

On 1/28/21 3:58 PM, Kagamin wrote:
> auto convoluted = i"${ir"`${"{"}`"}"; // nested string interpolations work.
> assert(convoluted == "`{`");
> 
> +InterpolatedString:
> +    InterpolatedDoubleQuotedString
> +    InterpolatedWysiwygString
> +    InterpolatedAlternateWysiwygString
> +    InterpolatedTokenString
> 
> Interpolated string should obey all escaping rules of the string literal it's derived from, and initial lexing of such string should be done with the same logic, and handling of interpolation sequences should be done on raw content of the lexed string after all due unescaping.
> 
> i`\${.}`
> i"\\${.}"
> These two should have the same meaning of escaped interpolation dollar sign, the escaped backslash becomes just backslash after double quote string unescaping, and this backslash is interpreted as interpolation escape sequence.

DoubleQuotedString is the only string with EscapeSequence processing. Therefore we continued that same expectation. The Wysiwyg string types specifically allow single backslash to represent a backslash, and we did not want to change that behavior.

In TokenString, the sequence ${ tokens } are not valid D tokens, so escaping the sequence isn't fruitful. It may be something that is reasonable inside a string literal inside the token string, but I don't think that's worth the complexity. If you want escapes, use the double quoted form.

Note also, to wait until the entire string is lexed to process the interpolation sequences means the sequences would have to obey the rules of the string. This means something like:

i"hello ${firstname ~ " " ~ lastname}"

if processed as a raw string first, would look to the lexer like 2 string literals, one that is i"hello ${firstname ~ ", and one that is " ~ lastname}". You would have to escape the quotation marks, e.g.:

i"hello ${firstname ~ \" \" ~ lastname}"

making the example more convoluted than just passing the parameters directly. In some string types, this wouldn't be possible (i.e. inside a wysiwyg string, you could not use the end quote type (` or ") in your expression).

The lexer already has the capability of processing token strings, which is essentially what this is. It's just in the middle of another string sequence.

-Steve

On 1/28/21 6:06 PM, Paul Backus wrote:
> On Wednesday, 27 January 2021 at 10:33:53 UTC, Mike Parker wrote:
>> This is the feedback thread for the second round of Community Review of DIP 1036, "String Interpolation Tuple Literals".
> 
> DIP 1036 takes two different approaches to string interpolation and attempts to merge them together into a single proposal. In broad terms, those approaches can be characterized as follows:
> 
> 1. The convenient approach: the language and runtime take care of string conversion and memory allocation for you, and you don't have to worry about any of the details.
> 
> 2. The flexible approach: the language splits the string apart into interpolated and non-interpolated pieces, and it's up to you to decide what to do with them.
> 
> DIP 1036's proposal for #2 is very good, and its proposal for #1, while missing some important details, appears to be fundamentally on the right track. Either one of these proposals would make a fine DIP on its own. The problem with DIP 1036 is in the way it attempts to combine the two.
> 
> Fundamentally, the goal that DIP 1036 is aiming for is to give the programmers who want convenience the convenient version, and to give programmers who want flexibility the flexible version. While this is an admirable goal, fully achieving it requires reading the programmer's mind, which is infeasible given D's current level of compiler technology. So what DIP 1036 does is attempt to *guess* what the programmer wants, using a rather crude heuristic: if the code compiles with the flexible version, the compiler is to assume that's what the programmer wants; otherwise, it assumes they want the convenient version.
> 
> As with any heuristic or approximation, there are edge cases where this breaks down. One of them is called out in the DIP itself--type inference via `auto`--but it is not hard to imagine others. For example, a programmer who writes
> 
>      tuple(i"Good morning ${name}", i"Good evening ${name}")
> 
> ...is probably not going to get what they intended, even though their code compiles.

This is quite the unique edge case though. Any proposal that provides the flexible version is going to have trouble with tuple as it is now. DIP1027, Jonathan Marler's PR, etc would all do something "unintended" (though one could argue it may be intended, depending on the usage).

There are solutions that can be had. For instance, tuple could be instrumented never to accept parameters that contain interpolation literals. Therefore, the idup rewrite happens, and they get what they expect. This is not hard to solve.

But sure, one can definitely find edge cases for any proposal, especially in D where one can just write:

auto foo(T...)(T args) {
   ...
}

> 
> Every D programmer who wants to make effective use of DIP 1036's interpolation literals will have to go through the process of learning when .idup is required, when it's optional, when it's allowed-but-unnecessary, and when it's forbidden--which means that, in practice, they will have to learn how it actually works, under the hood. 

This is not my interpretation at all. I can't think of a reasonable case aside from your tuple example where idup is required (if that's what you want). Can you? And the tuple issue can be solved quite easily.

> This is not a desirable trait for a language feature that's intended to make programming *easier*.

Your logic is not very sound. It's ironic to say the language doing what you expect for 99% of cases is a higher burden than requiring you to write it yourself for 100% of cases.

> 
> Ultimately, I think attempting to guess the programmer's intent is the wrong way to go here. Either force them to spell it out explicitly (with a call to .idup, .text, etc.), or take away the choice and give up on one of the two approaches.

I want to say something about this idea of doing only one or the other.

Let's say you want to create a language feature for string interpolation that always results in a string. Because you need to process the values and convert them to string data at runtime, you need a library function that accepts the data.

How do you write such a library function? If this was D1, the way would be to use variadic parameters, passing the TypeInfo of each item, and a void *, and then you'd have to provide a universal way to convert all types to strings.

But this is D2. So the correct way to do this is to pass a template variadic argument list to a runtime function, and have the runtime function do the conversion.

If you are doing that, you are ALREADY having the compiler generate the expanded form. It's just always preventing anyone from using it directly. I can't imagine a DIP with this kind of "implementation detail" hiding the actual treasure of the feature being acceptable to the community.

If you want to implement just the tuple version, now you run into the unpleasant result of having the simplest expected thing fail. That is, when you assign what looks like a string to a string, it fails. When passing what looks like a string into a string parameter, it fails.

Both of these options are feasible, and can be argued for, but are not what I would expect for a D language feature. The first option is hiding everything useful from the user instead of using the power of D metaprogramming. The second option is right up D's alley, but places a heavy burden on those who want to just use strings. The combination of both is a cohesive whole which leaves very small edge cases to be dealt with by function authors (and provides a straightforward clear way to do so).

If this DIP gets accepted that does not mean the work is done. It means the opportunity is opened for people to take advantage of it. Yes, there are edge cases, but the solutions are apparent and readily available, and the default behavior without such changes is still reasonable. And this does not take away from the fact that the vast vast majority of non-edge-case *just work*.

-Steve

On Wednesday, 27 January 2021 at 10:33:53 UTC, Mike Parker wrote:
> * All posts must be a direct reply to the DIP manager's initial post, with only two exceptions:

A problem I see is that i"..." becomes an interp sequence too easily.
D's templates have a low entrance barrier and passing an interpolated string to a variadic function template is something not too outlandish to do. Then, the by the current form of the proposal, the interpolated becomes an `interp` sequence. This may be completely unexpected as most variadic function templates won't be written with an interp sequence in mind at all.

Imagine someone (especially a newcommer from e.g. C#) naively using
    auto tup = tuple(1, i"I'm ${name}.", 2.3)
expecting Tuple!(int, string, double). First, tup.length != 3 and its content are wild.
Basically, an interpolated string should be a string, except in very specific circumstances; variadic templates aren't even close to being a very specific circumstance.

The typing is better done akin to slices vs static arrays.
As a reminder, to a newcomer,
    auto xs = [ 1, 2, 3 ];
looks like it would infer int[3] as the type of xs. It is obviously the most descriptive type for that literal. Why would it infer int[] forgetting its compile-time known length and even do an allocation? That seems so much worse. At least, for consistency, typeof([1,2,3]) is int[], too, and not int[3]. We know why D does it the way it does, and why it uses int[3] with no allocation only when requested explicitly. You can do that with a template with a flexible length like this:
    void takesStaticArray(T, size_t n)(T[n] staticArray);
Here, `T` and `n` can usually be inferred from the argument:
    takesStaticArray([1, 2, 3]);
won't allocate.

Interpolated strings should behave similarly:
1. typeof(i"...") is `string`.
2. auto str = i"..." infers string (cf. typeof) and gc-allocates if necessary.
3. i"..." has a secondary type akin to [1,2,3] having int[3] as a secondary type. (The secondary type is a sequence, but that's a detail.)

If an interpolated string is bound to a parameter of that secondary type (`interp` in the DIP) it uses its secondary type (cf. calling takesStaticArray with [1,2,3]). In any other case, e.g. `auto` or otherwise generic template parameters will infer string.

My suggestion is that one cannot do this:
    ResultSeq mysql_query(Args...)(Args args) if (Args.length > 0 && isInstanceOf!(interp, Args[0])) { ... }
because Args... is too unspecific to force i"..." to become an interp sequence. Therefore
    auto rseq = mysql_query(i"...");
results in mysql_query!string and then fails the constraint. You have to request an interp sequence explicitly like this
   ResultSeq mysql_query(string first, Args...)(interp!first arg, Args args)
to force the secondary type. This is akin to takesStaticArray(size_t n, T)(T[n]). Since it cannot bind an int[]

However, if you already have an interp!"..." object in hand, of course passing it to a template will use its static type interp!"..." (and not string), like an int[3] variable will, too.
This is fine, because at this point, you know what you're dealing with and that it is not a string.

Akin to `staticArray`, Phobos can (should) supply a template `asInterp` that forces a i"..." literal to be statically typed as a tuple around the respective `interp` sequence.

    auto asInterp(string str, Args...)(interp!str first, Args rest)
    {
        import std.typecons : tuple;
        return tuple(first, rest);
    }

Function templates handling interpolated strings can either provide an overload taking a tuple containing an interp!"..." as its first entry (likely result of asInterp) or require the user to use `expand` on the tuple.
Example: https://run.dlang.io/is/h5LgF4

Getting a string is probably what most users expect most of the time, even in templates. Handling the secondary type must be explicit. It is almost an implementation detail that shouldn't be exposed to the user too easily.

On Wednesday, 3 February 2021 at 17:47:19 UTC, Q. Schroll wrote:
> Akin to `staticArray`, Phobos can (should) supply a template `asInterp` that forces a i"..." literal to be statically typed as a tuple around the respective `interp` sequence.

Something to keep in mind is that tuples cannot actually be wrapped in D. As soon as they are anything other than slices of themselves or directly used as function parameters, they start losing information.

>     auto asInterp(string str, Args...)(interp!str first, Args rest)
>     {
>         import std.typecons : tuple;
>         return tuple(first, rest);
>     }

Like here, if there was any refness in rest, it is gone in the return value. If it was used as a template argument and had alias parameters, that information is lost.

That's the big driver toward the naked tuple - it is just the *only* way to express all the possibilities D has to offer.


Conceptually, dip 1036 is trying to express i"foo" becoming:

struct Interpolated(Pieces...) {
    Pieces... pieces;
    string toString() { return text(pieces); }

    alias toString this;
    alias args this;
}

That is, a new type with two different implicit conversions depending on appropriate context.

But the fact is that's flat-out impossible to today's D, so the dip makes some compromises to get as close as it can with the simple naked tuple on one hand and the implicit conversion rules specified on the other.

> Getting a string is probably what most users expect most of the time, even in templates.

That's why the implicit conversion part is in the dip, but your point about the static arrays is an interesting alternative approach.


BTW this part of the argument barely matters to me. I will probably *never* use the implicit to string thing since it is hard for me to think of a situation where actually passing the tuple to the function isn't a better approach.

On Wednesday, 3 February 2021 at 18:17:03 UTC, Adam D. Ruppe wrote:
> On Wednesday, 3 February 2021 at 17:47:19 UTC, Q. Schroll wrote:
>> Akin to `staticArray`, Phobos can (should) supply a template `asInterp` that forces a i"..." literal to be statically typed as a tuple around the respective `interp` sequence.
>
> Something to keep in mind is that tuples cannot actually be wrapped in D. As soon as they are anything other than slices of themselves or directly used as function parameters, they start losing information.
>
>>     auto asInterp(string str, Args...)(interp!str first, Args rest)
>>     {
>>         import std.typecons : tuple;
>>         return tuple(first, rest);
>>     }
>
> Like here, if there was any refness in rest, it is gone in the return value. If it was used as a template argument and had alias parameters, that information is lost.

I never thought of that `asInterp` as the summit of creation but rather as a proof of concept. I thought about it for like 3 minutes. The much better solution would be something akin to

    alias asInterp(alias interp!str first, string str, rest...) = AliasSeq!(first, rest);

used as

    asInterp!i".."

but unfortunately, this does not compile. (This was my first idea.) D cannot infer `str` from the value passed to it. It can for `interp!str` a run-time parameter type, but not for a template parameter. Since aliases are mere names, `ref`ness and stuff would be preserved (I guess so, since otherwise, forward wouldn't work).
Another, actually very simple, idea would be to give an i"..." a property asInterp that (cf. tuple's expand) returns the interp sequence.

I would have spelled out my critique differently had I read all feedback before; tuple was already mentioned. But the main point is: variadic templates are too common to assume they're all ready to take an interp sequence. As a very simple example different from tuple, consider emplacement functions that forward their arguments to constructors; all of them are variadic templates taking on almost anything. String arguments aren't too uncommon and wherever strings are used, we must assume that sooner or later someone uses an interpolated string. All of those variadic function templates would have to anticipate interpolated strings. When one thinks about handling strings, interpolated ones aren't too far fetched, but when one thinks about arbitrary types, interpolated strings aren't really the first or second thought. Variadic templates are the prime example of thinking of arbitrary types.

The @nogc people will hate it, but immediately making i"..." a string through allocation -- except "very specific circumstances" that leave absolutely no room for doubt -- is the only way this will interact with other language features in an intuitive manner. Having to use idup explicitly in any case is bad design. It makes interpolated strings inaccessible. They'd become a tool of the professionals and gurus, although being first and foremost designed to be an accessible productivity feature.
If code can be @nogc, programmers should have means to do that -- but that's it. It need not be stupidly easy at the expense of making an accessible productivity feature worse than it need be.

On Wednesday, 27 January 2021 at 10:33:53 UTC, Mike Parker wrote:
> This is the feedback thread for the second round of Community Review of DIP 1036, "String Interpolation Tuple Literals".
>

There are many great qualities of DIP1036 and I hope it gets accepted, good luck!
But one area which I think could need improvement is:

1) The lowering (either that or I totally overlooked something obvious again).

writeln(i"I ate ${apples} apples and ${bananas} bananas totalling ${apples + bananas} fruit.");

DIP1036 proposes the following rewrite:
writeln(interp!"I ate "(), apples, interp!" apples and "(), bananas, interp!" bananas totalling "(), apples + bananas, interp!" fruit."());

Wouldn't this lowering be both simpler and more more efficient?
writeln(interp!"I ate "(), apples, " apples and ", bananas, " bananas totalling ", apples + bananas, " fruit.");

Actually couldn't we go even futher?
immutable interp interp_tag; // an instance of some unique type with as low overhead as possible
writeln(interp_tag, "I ate ", apples, " apples and ", bananas, " bananas totalling ", apples + bananas, " fruit.");

This way you can overload with a normal function argument instead of using a template constraint and NO single template instance is created.

2) Also no fan of using abbreviated type-names(interp) in userfacing API:s

On Wednesday, 3 February 2021 at 20:02:57 UTC, Daniel N wrote:
> Wouldn't this lowering be both simpler and more more efficient?

Your change wouldn't expose the string for compile time processing.

That's the real benefit of interp!"string" - it is available for CTFE rewriting by the function. So putting that on the whole thing means a whole format string or translation string or whatever else can be CTFE generated, inspected, etc.

> 2) Also no fan of using abbreviated type-names(interp) in userfacing API:s

yeah, it is ambiguous with "interpretation" too.