http://jackstouffer.com/blog/d_auto_decoding_and_you.html

Based on the recent thread in General, I wrote this blog post that's designed to be part beginner tutorial, part objective record of the debate over it, and finally my opinions on the matter.

When I first learned about auto-decoding, I was kinda miffed that there wasn't anything in the docs or on the website that went into more detail. So I wrote this in order to introduce people who are getting into D to the concept, it's benefits, and downsides. When people are confused in Learn why typeof(s.front) == dchar then this can just be linked to them.

If you think there should be any more information included in the article, please let me know so I can add it.

On 5/17/16 10:06 AM, Jack Stouffer wrote:
> http://jackstouffer.com/blog/d_auto_decoding_and_you.html
>
> Based on the recent thread in General, I wrote this blog post that's
> designed to be part beginner tutorial, part objective record of the
> debate over it, and finally my opinions on the matter.
>
> When I first learned about auto-decoding, I was kinda miffed that there
> wasn't anything in the docs or on the website that went into more
> detail. So I wrote this in order to introduce people who are getting
> into D to the concept, it's benefits, and downsides. When people are
> confused in Learn why typeof(s.front) == dchar then this can just be
> linked to them.
>
> If you think there should be any more information included in the
> article, please let me know so I can add it.

Starting to read it, see errors in your examples:

is(s[0] == immutable char) -> is(typeof(s[0]) == immutable(char))
is(s.front == dchar) -> is(typeof(s.front()) == dchar)

I'm not sure if you need the parens after front, but if it's not marked as @property, then this returns a function.

-Steve

On Tuesday, 17 May 2016 at 14:16:48 UTC, Steven Schveighoffer wrote:
> Starting to read it, see errors in your examples:
>
> is(s[0] == immutable char) -> is(typeof(s[0]) == immutable(char))
> is(s.front == dchar) -> is(typeof(s.front()) == dchar)

Thanks, fixed.

Grammar:

"This will tie in later because the string types front has special behavior"

...because for string types, front has...

Content:
"For C style strings, you can use ubyte[] and call std.string.assumeUTF where necessary"

Actually, C style strings are ASCII, no? UTF8 includes ASCII. And D treats C strings as char *, not ubyte[]

Content:

D will look ahead in the string and combine things like e and U+0308 into ë

Nope :) This is a grapheme, and D does not decode these into one dchar.

Grammar:

"about it's inclusion"

it's -> its

Typo:

"Pared with the inability to turn it off,"

Pared -> Paired

Typo:

"Phobos String type would be the best option and a deprecation of the sting front function"

sting -> string

Like the article, pretty much sums up my thoughts too. IMO, the only path forward is something that aliases string to something that auto-decodes, but that is NOT a char array. Then you have to deprecate implicit access to the backing array, and make it explicit. Probably would take 2 years or so to migrate.

-Steve

On Tuesday, 17 May 2016 at 14:44:06 UTC, Steven Schveighoffer wrote:
> ...

Thanks, fixed all issues.

> Like the article, pretty much sums up my thoughts too.

Thanks!

On Tuesday, 17 May 2016 at 14:06:37 UTC, Jack Stouffer wrote:
> http://jackstouffer.com/blog/d_auto_decoding_and_you.html
>
> Based on the recent thread in General, I wrote this blog post that's designed to be part beginner tutorial, part objective record of the debate over it, and finally my opinions on the matter.

I probably would have preferred this split up into two parts with one on the tutorial and one on the record of the debate. It seems to focus more on the debate.

Maybe you could get add a section that was like "for THIS type of string, do X for performance" with clear explanations of why that is the best way and why other ways will be slower. Then, you can have other sub-sections for "for THAT type of string, do Y for performance." You have some of this detail in there, but it's organized more with respect to the context of the debate, I think.

On Tuesday, 17 May 2016 at 14:06:37 UTC, Jack Stouffer wrote:
> http://jackstouffer.com/blog/d_auto_decoding_and_you.html

Thanks for writing this. Great article.

Some remarks:

>    static assert(is(typeof(s.front()) == dchar));

I believe .front is a property (so some ranges can implement it as a field, not a @property function). Hence, no parens.

> So, why is typeof(s.front) == dchar.

Question mark?

> In plain English, this means when iterating over strings in D, D will look ahead in the string and combine any code units that make up a single code point.

Perhaps clarify that this only applies to ranges. `foreach` on a string will iterate over chars, but you can iterate over code points if you specify the dchar type explicitly.

More confusing text on the same issue lower, and in the intro:

> Iterating a char array with C style for loops produces different results than foreach loops due to auto decoding.

> One feature of D that is confusing to a lot of new comers is the behavior of strings in relation to range based features like the foreach statement and range algorithms.

---

> E.g. for ë the code units C3 AB (for UTF-8) would turn into a single code point.

Perhaps choose a character that is not also expressable via composite characters, to avoid potential for confusion.

> string s = "cassé";

Ditto (unless the goal was to complement the example from my .d file below)

>  These glaring inconsistencies are the cause of a lot of confusion for new comers.

(Opinion) I would say that they also cause issues in generic code.

> Every time one wants a generic algorithm to work with both strings and ranges, you wind up special casing via static if-ing narrow strings to defeat the auto decoding, or to decode the ranges. Case in point.

Link to the exact SHA to prevent the link from getting outdated. On Github, just hit 'y' on your keyboard to go to the "permalink" version.

> Auto decoding has two choices when encountering invalid code units: throw, or produce an error dchar like std.utf.byUTF does.

(Aside) This was an interesting discussion on the subject: https://issues.dlang.org/show_bug.cgi?id=14519

> However, in my opinion D is too far along to to suddenly ask people

"to to"

---

Some more info / links on the subject I collected a few years ago:

http://wiki.dlang.org/Language_issues#Unicode_and_ranges

On 5/17/16 1:18 PM, Vladimir Panteleev wrote:
> On Tuesday, 17 May 2016 at 14:06:37 UTC, Jack Stouffer wrote:
>> http://jackstouffer.com/blog/d_auto_decoding_and_you.html
>
> Thanks for writing this. Great article.
>
> Some remarks:
>
>>    static assert(is(typeof(s.front()) == dchar));
>
> I believe .front is a property (so some ranges can implement it as a
> field, not a @property function). Hence, no parens.

Right, but s is a string. So front is a function.

There is an inconsistency in the compiler for this. If s.front is a function is(typeof(s.front)) will not be what front *returns*, but the function type itself. Unless you tag with @property. However, it's perfectly legal for a front function not to be tagged @property.

-Steve

On 17 May 2016 16:21, "Steven Schveighoffer via Digitalmars-d-announce" < digitalmars-d-announce@puremagic.com> wrote:
>
> On 5/17/16 10:06 AM, Jack Stouffer wrote:
>>
>> http://jackstouffer.com/blog/d_auto_decoding_and_you.html
>>
>> Based on the recent thread in General, I wrote this blog post that's designed to be part beginner tutorial, part objective record of the debate over it, and finally my opinions on the matter.
>>
>> When I first learned about auto-decoding, I was kinda miffed that there wasn't anything in the docs or on the website that went into more detail. So I wrote this in order to introduce people who are getting into D to the concept, it's benefits, and downsides. When people are confused in Learn why typeof(s.front) == dchar then this can just be linked to them.
>>
>> If you think there should be any more information included in the article, please let me know so I can add it.
>
>
> Starting to read it, see errors in your examples:
>
> is(s[0] == immutable char) -> is(typeof(s[0]) == immutable(char))
> is(s.front == dchar) -> is(typeof(s.front()) == dchar)
>
> I'm not sure if you need the parens after front, but if it's not marked
as @property, then this returns a function.
>
> -Steve

If I remember correctly adding the brackets then goes against best practices because you can't be sure the underlying implementation of a range is using a function for .front.

On Tuesday, 17 May 2016 at 17:26:59 UTC, Steven Schveighoffer wrote:
> On 5/17/16 1:18 PM, Vladimir Panteleev wrote:
>> On Tuesday, 17 May 2016 at 14:06:37 UTC, Jack Stouffer wrote:
>>> http://jackstouffer.com/blog/d_auto_decoding_and_you.html
>>
>> Thanks for writing this. Great article.
>>
>> Some remarks:
>>
>>>    static assert(is(typeof(s.front()) == dchar));
>>
>> I believe .front is a property (so some ranges can implement it as a
>> field, not a @property function). Hence, no parens.
>
> Right, but s is a string. So front is a function.

Then what happened to writing generic code?

> There is an inconsistency in the compiler for this. If s.front is a function is(typeof(s.front)) will not be what front *returns*, but the function type itself. Unless you tag with @property. However, it's perfectly legal for a front function not to be tagged @property.

There is a simple answer to this, and it is to either use ElementType or do what it does (is(typeof(R.init.front.init) T)).