The problem
-----------

String literals in D are a little bit magical; they have a trailing \0. This means that is possible to write,

printf("Hello, World!\n");

without including a trailing \0. This is important for compatibility with C. This trailing \0 is mentioned in the spec but only incidentally, and generally in connection with printf.

But the semantics are not well defined.

printf("Hello, W" ~ "orld!\n");

Does this have a trailing \0 ? I think it should, because it improves readability of string literals that are longer than one line. Currently DMD adds a \0, but it is not in the spec.

Now consider array literals.

printf(['H','e', 'l', 'l','o','\n']);

Does this have a trailing \0 ? Currently DMD does not put one in.
How about ['H','e', 'l', 'l','o'] ~ " World!\n"  ?

And "Hello " ~ ['W','o','r','l','d','\n']   ?

And "Hello World!" ~ '\n' ?
And  null ~ "Hello World!\n" ?

Currently DMD puts \0 in some cases but not others, and it's rather random.

The root cause is that this trailing zero is not part of the type, it's part of the literal. There are no rules for how literals are propagated inside expressions, they are just literals. This is a mess.

There is a second difference.
Array literals of char type, have completely different semantics from string literals. In module scope:

char[] x = ['a'];  // OK -- array literals can have an implicit .dup
char[] y = "b";    // illegal

This is a big problem for CTFE, because for CTFE, a string is just a compile-time value, it's neither string literal nor array literal!

See bug 8660 for further details of the problems this causes.


A proposal to clean up this mess
--------------------------------

Any compile-time value of type immutable(char)[] or const(char)[], behaves a string literals currently do, and will have a \0 appended when it is stored in the executable.

ie,

enum hello = ['H', 'e', 'l', 'l', 'o', '\n'];
printf(hello);

will work.

Any value of type char[], which is generated at compile time, will not have the trailing \0, and it will do an implicit dup (as current array literals do).

char [] foo()
{
    return "abc";
}

char [] x = foo();

// x does not have a trailing \0, and it is implicitly duped, even though it was not declared with an array literal.

-------------------
So that the difference between string literals and char array literals would simply be that the latter are polysemous. There would be no semantics associated with the form of the literal itself.


We still have this oddity:


void foo(char qqq = 'b') {

   string x = "abc";            // trailing \0
   string y = ['a', 'b', 'c'];  // trailing \0
   string z = ['a', qqq, 'c'];  // no trailing \0
}

This is because we made the (IMHO mistaken) decision to allow variables inside array literals.
This is the reason why I listed _compile time value_ in the requirement for having a \0, rather than entirely basing it on the type.

We could fix that with a language change: an array literal which contains a variable should not be of immutable type. It should be of mutable type (or const, in the case where it contains other, immutable values).

So char [] w = ['a', qqq, 'c']; should compile (it currently doesn't, even though w is allocated on the heap).

But that's a separate proposal from the one I'm making here. I just need a decision on the main proposal so that I can fix a pile of CTFE bugs.

On Tuesday, 2 October 2012 at 11:10:46 UTC, Don Clugston wrote:
> The problem
> -----------
>
> String literals in D are a little bit magical; they have a trailing \0. This means that is possible to write,
>
> printf("Hello, World!\n");
>
> without including a trailing \0. This is important for compatibility with C. This trailing \0 is mentioned in the spec but only incidentally, and generally in connection with printf.
>
> But the semantics are not well defined.
>
> printf("Hello, W" ~ "orld!\n");
>
If every string literal is \0-terminated, then there should be two \0 in the final string. I guess that's not the case and that's actually my preferred behaviour, but the spec should make it crystal clear in which situations a
string literal gets a terminator and in which not.

Well the whole mess come from the fact that D conflate C string and D string.

The first problem come from the fact that D array are implicitly convertible to pointer. So calling D function that expect a char* is possible with D string even if it is unsafe and will not work in the general case.

The fact that D provide tricks that will make it work in special cases is armful as previous discussion have shown (many D programmer assume that this will always work because of toy tests they have made, where in case it won't and toStringz must be used).

The only sane solution I can think of is to :
 - disallow slice to convert implicitly to pointer. .ptr is made for that.
 - Do not put any trailing 0 in string literal, unless it is specified explicitly ( "foobar\0" ).
 - Except if a const(char)* is expected from the string literal. In case it becomes a Cstring literal, with a trailing 0. This is made to allow uses like printf("foobar");

In other terms, the receiver type is used to decide if the compiler generate a string literal or a Cstring literal.

Other addition of 0 are just confusing, and will make incorrect code work in special cases, which is something you usually don't want. Code that work by accident often backfire in spectacular ways at the least expected moment.

On 10/2/12, Don Clugston <dac@nospam.com> wrote:
> A proposal to clean up this mess
> --------------------------------
>
> Any compile-time value of type immutable(char)[] or const(char)[],
> behaves a string literals currently do, and will have a \0 appended when
> it is stored in the executable.
>
> ie,
>
> enum hello = ['H', 'e', 'l', 'l', 'o', '\n'];
> printf(hello);
>
> will work.

What about these, will these pass?:

enum string x = "foo";
assert(x.length == 3);

void test(string x) { assert(x.length == 3); }
test(x);

If these don't pass the proposal will break code.

On 02/10/12 14:02, Andrej Mitrovic wrote:
> On 10/2/12, Don Clugston <dac@nospam.com> wrote:
>> A proposal to clean up this mess
>> --------------------------------
>>
>> Any compile-time value of type immutable(char)[] or const(char)[],
>> behaves a string literals currently do, and will have a \0 appended when
>> it is stored in the executable.
>>
>> ie,
>>
>> enum hello = ['H', 'e', 'l', 'l', 'o', '\n'];
>> printf(hello);
>>
>> will work.
>
> What about these, will these pass?:
>
> enum string x = "foo";
> assert(x.length == 3);
>
> void test(string x) { assert(x.length == 3); }
> test(x);
>
> If these don't pass the proposal will break code.

Yes, they pass. The \0 is not included in the string length. It's effectively in the data segment, not in the string.

On 02/10/12 13:26, deadalnix wrote:
> Well the whole mess come from the fact that D conflate C string and D
> string.
>
> The first problem come from the fact that D array are implicitly
> convertible to pointer. So calling D function that expect a char* is
> possible with D string even if it is unsafe and will not work in the
> general case.
>
> The fact that D provide tricks that will make it work in special cases
> is armful as previous discussion have shown (many D programmer assume
> that this will always work because of toy tests they have made, where in
> case it won't and toStringz must be used).
>
> The only sane solution I can think of is to :
>   - disallow slice to convert implicitly to pointer. .ptr is made for that.
>   - Do not put any trailing 0 in string literal, unless it is specified
> explicitly ( "foobar\0" ).
>   - Except if a const(char)* is expected from the string literal. In
> case it becomes a Cstring literal, with a trailing 0. This is made to
> allow uses like printf("foobar");
>
> In other terms, the receiver type is used to decide if the compiler
> generate a string literal or a Cstring literal.

This still doesn't solve the problem of the difference between array literals and string literals (the magical implicit .dup), which is the key problem I'm trying to solve.

On 02/10/12 13:18, Tobias Pankrath wrote:
> On Tuesday, 2 October 2012 at 11:10:46 UTC, Don Clugston wrote:
>> The problem
>> -----------
>>
>> String literals in D are a little bit magical; they have a trailing
>> \0. This means that is possible to write,
>>
>> printf("Hello, World!\n");
>>
>> without including a trailing \0. This is important for compatibility
>> with C. This trailing \0 is mentioned in the spec but only
>> incidentally, and generally in connection with printf.
>>
>> But the semantics are not well defined.
>>
>> printf("Hello, W" ~ "orld!\n");
>>
> If every string literal is \0-terminated, then there should be two \0 in
> the final string. I guess that's not the case and that's actually my
> preferred behaviour, but the spec should make it crystal clear in which
> situations a
> string literal gets a terminator and in which not.

The \0 is *not* part of the string, it lies after the string.
It's as if all memory is cleared, then the string literals are copied into it, with a gap of at least one byte between each. The 'trailing 0' is not part of the literal, it's the underlying cleared memory.

At least, that's how I understand it. The spec is very vague.

2012/10/2 Don Clugston <dac@nospam.com>:
> The problem
> -----------
>
> String literals in D are a little bit magical; they have a trailing \0. This means that is possible to write,
>
> printf("Hello, World!\n");
>
> without including a trailing \0. This is important for compatibility with C. This trailing \0 is mentioned in the spec but only incidentally, and generally in connection with printf.
>
> But the semantics are not well defined.
>
> printf("Hello, W" ~ "orld!\n");
>
> Does this have a trailing \0 ? I think it should, because it improves readability of string literals that are longer than one line. Currently DMD adds a \0, but it is not in the spec.
>
> Now consider array literals.
>
> printf(['H','e', 'l', 'l','o','\n']);
>
> Does this have a trailing \0 ? Currently DMD does not put one in. How about ['H','e', 'l', 'l','o'] ~ " World!\n"  ?
>
> And "Hello " ~ ['W','o','r','l','d','\n']   ?
>
> And "Hello World!" ~ '\n' ?
> And  null ~ "Hello World!\n" ?
>
> Currently DMD puts \0 in some cases but not others, and it's rather random.
>
> The root cause is that this trailing zero is not part of the type, it's part of the literal. There are no rules for how literals are propagated inside expressions, they are just literals. This is a mess.
>
> There is a second difference.
> Array literals of char type, have completely different semantics from string
> literals. In module scope:
>
> char[] x = ['a'];  // OK -- array literals can have an implicit .dup char[] y = "b";    // illegal
>
> This is a big problem for CTFE, because for CTFE, a string is just a compile-time value, it's neither string literal nor array literal!
>
> See bug 8660 for further details of the problems this causes.
>
>
> A proposal to clean up this mess
> --------------------------------
>
> Any compile-time value of type immutable(char)[] or const(char)[], behaves a
> string literals currently do, and will have a \0 appended when it is stored
> in the executable.
>
> ie,
>
> enum hello = ['H', 'e', 'l', 'l', 'o', '\n'];
> printf(hello);
>
> will work.
>
> Any value of type char[], which is generated at compile time, will not have the trailing \0, and it will do an implicit dup (as current array literals do).
>
> char [] foo()
> {
>     return "abc";
> }
>
> char [] x = foo();
>
> // x does not have a trailing \0, and it is implicitly duped, even though it was not declared with an array literal.
>
> -------------------
> So that the difference between string literals and char array literals would simply be that the latter are polysemous. There would be no semantics associated with the form of the literal itself.
>
>
> We still have this oddity:
>
>
> void foo(char qqq = 'b') {
>
>    string x = "abc";            // trailing \0
>    string y = ['a', 'b', 'c'];  // trailing \0
>    string z = ['a', qqq, 'c'];  // no trailing \0
> }
>
> This is because we made the (IMHO mistaken) decision to allow variables
> inside array literals.
> This is the reason why I listed _compile time value_ in the requirement for
> having a \0, rather than entirely basing it on the type.
>
> We could fix that with a language change: an array literal which contains a variable should not be of immutable type. It should be of mutable type (or const, in the case where it contains other, immutable values).
>
> So char [] w = ['a', qqq, 'c']; should compile (it currently doesn't, even though w is allocated on the heap).
>
> But that's a separate proposal from the one I'm making here. I just need a decision on the main proposal so that I can fix a pile of CTFE bugs.

Maybe your proposal is correct.
I think the key idea is *polysemous typed string literal*.

When based on the Ideal D Interpreter in my brain, the organized rule will become like follows.

1-1) In semantic level, D should have just one polysemous string
literal, which is "an array of char".
1-2) In token level, D has two represents for the polysemous string
literal, they are "str" and ['s','t','r'].

2) The polysemous string literl is implicitly convertible to
[wd]?char[] and immutable([wd]?char)[] (I think const([wd]?char)[] is
not need, because immutable([wd]?char)[] is implicitly convertible to
them).

3) The concatenation result between polysemous literals is still polysemous, but its representation is different based on the both side of the operator.

   "str" ~ "str";         // "strstr"
   "str" ~ ['s','t','r']; // ['s','t','r','s','t','r']
   "str" ~ 's';           // "strs"
   ['s','t','r'] ~ 's';   // ['s','t','r','s']
   "str" ~ null;          // "str"
   ['s','t','r'] ~ null;  // ['s','t','r']

4) After semantics _and_ optimization, polysemous string literal which
represented as like
 4-1) "str" is typed as immutable([wd]?char)[] (The char type is
depends on the literal suffix).
 4-2) ['s','t','r'] is typed as ([wd]?char)[] (The char type is
depends on the common type of its elements).

5) In object file generating phase, string literal which typed as
  5-1) immutable([wd]?)char[] is stored in the executable and
implicitly terminated with \0.
  5-2) [wd]?char[] are stored in the executable as the original image
and implicitly 'dup'ed in runtime.

----
Additionally, in following case, both concatenation should generate
polysemous string literals in CT and RT.
Because, after concatenation of chars and char arrays, newly allocated
strings are *purely immutable* value and implicitly convertible to
mutable.

immutable char ic = 'a';
pragma(msg, typeof(['s', 't', ic, 'r']));   // prints const(char)[]
immutable(char)[] s = ['s', 't', ic, 'r'];  // BUT, should be allowed

char mc = 'a';
pragma(msg, typeof("st"~mc~"r"));   // prints const(char)[]
char[] s = "st"~mc~"r";             // BUT, should be allowed

Kenji Hara

On Tuesday, 2 October 2012 at 11:10:46 UTC, Don Clugston wrote:
> [SNIP]
> A proposal to clean up this mess
> [SNIP]

While I think it is convenient to be able to write 'printf("world");', as you point out, I think that the fact that it works "inconsistently" (and by that, I mean there are rules and exceptions), is even more dangerous.

If at all possible, I'd rather side with consistency, then the "we got your back... except when we don't" approach: IE: strings are NEVER null terminated.

In theory, how often do you *really* need null terminated strings? And when you do, wouldn't it be safer to just write 'printf("world\0")'? or 'printf(str ~ "world" ~ '\0');' rather than "Am I in a case where it is null terminated? Yeah... 90% confident I am..."

If you want 0 termination, then make it explicit, that's my opinion.

Besides, as you said, the null termination is not documented, so anything relying on it is a bug really. Just an observation of an implementation detail.

Le 02/10/2012 15:12, Don Clugston a écrit :
> On 02/10/12 13:26, deadalnix wrote:
>> Well the whole mess come from the fact that D conflate C string and D
>> string.
>>
>> The first problem come from the fact that D array are implicitly
>> convertible to pointer. So calling D function that expect a char* is
>> possible with D string even if it is unsafe and will not work in the
>> general case.
>>
>> The fact that D provide tricks that will make it work in special cases
>> is armful as previous discussion have shown (many D programmer assume
>> that this will always work because of toy tests they have made, where in
>> case it won't and toStringz must be used).
>>
>> The only sane solution I can think of is to :
>> - disallow slice to convert implicitly to pointer. .ptr is made for that.
>> - Do not put any trailing 0 in string literal, unless it is specified
>> explicitly ( "foobar\0" ).
>> - Except if a const(char)* is expected from the string literal. In
>> case it becomes a Cstring literal, with a trailing 0. This is made to
>> allow uses like printf("foobar");
>>
>> In other terms, the receiver type is used to decide if the compiler
>> generate a string literal or a Cstring literal.
>
> This still doesn't solve the problem of the difference between array
> literals and string literals (the magical implicit .dup), which is the
> key problem I'm trying to solve.
>

OK, infact we have 2 different and unrelated problems here. I have to say I have no idea for the second one.