https://issues.dlang.org/show_bug.cgi?id=22473

I've tried to fix this before, but too many people objected.

Are we fed up with this yet? I sure am.

Who wants to take up this cudgel and fix the durned thing once and for all?

(It's unclear if it would even break existing code.)

On Thursday, 4 November 2021 at 02:26:20 UTC, Walter Bright wrote:
> I've tried to fix this before, but too many people objected.

I proposed a few days ago that Phobos autodecoding, if not completely removed, do this exact same thing too.

I agree it is a good idea. If you want an exception, it is easy enough to just check it in the loop and throw then.

Let's do it.

On Thursday, 4 November 2021 at 02:26:20 UTC, Walter Bright wrote:
> https://issues.dlang.org/show_bug.cgi?id=22473
>
> I've tried to fix this before, but too many people objected.
>
> Are we fed up with this yet? I sure am.
>
> Who wants to take up this cudgel and fix the durned thing once and for all?
>
> (It's unclear if it would even break existing code.)

I still think this is a mistake.

One may disagree about autodecoding; I for one think it's a sensible idea. However, a program should either process data correctly or, if that is impossible, not at all. It should not, ever, silently modify it "for you" while reading! I predict this will lead to cryptic, hair-pulling bugs in user code involving replacement characters appearing far downstream of the error site.

On Thursday, 4 November 2021 at 05:34:29 UTC, FeepingCreature wrote:
> One may disagree about autodecoding; I for one think it's a sensible idea. However, a program should either process data correctly or, if that is impossible, not at all. It should not, ever, silently modify it "for you" while reading! I predict this will lead to cryptic, hair-pulling bugs in user code involving replacement characters appearing far downstream of the error site.

(This is floating point NaN all over again!)

On Thursday, 4 November 2021 at 02:26:20 UTC, Walter Bright wrote:
> https://issues.dlang.org/show_bug.cgi?id=22473
>
> I've tried to fix this before, but too many people objected.
>
> Are we fed up with this yet? I sure am.
>
> Who wants to take up this cudgel and fix the durned thing once and for all?
>
> (It's unclear if it would even break existing code.)

Assuming the comment by Ali on the linked bug is right, I think the current behaviour is correct.

Your complaints:

> It can't be turned off

Sure it can.  You can choose to iterate in another fashion; say, by creating your own iterator which folds invalid utf8 into replacement characters.

> it throws

Is it better to produce an incorrect result?

A high-quality, non-throwing mechanism for error handling exists.  It consists of an _optional_ value which must be explicitly unwrapped.  It is also an out-of-band signal; how will I distinguish invalid utf8 from a correctly-encoded replacement character?

> it may allocate with the gc

So?  If that is the sort of thing you care about, then you will @nogc and find an alternate solution.  Lots of core language features allocate, like arrays and hash tables.

> it's slow

In the hot path it's the same speed.  In the slow path, performance doesn't matter.  In any case, it's useless to give an incorrect result faster.


(Notably, this is not exactly _auto_ decoding; it is explicitly requested decoding.  And your proposed modification doesn't change that fact.)


What is (potentially) questionable imo is that given foreach (c; a), c will be inferred to be dchar; you have to explicitly ask for char.  Perhaps that default should be reversed.  (This will definitely break code, though, and may not be worth it.)

If you want an iterator that generates replacement characters for invalid utf8, just create one.  But the default translation should be faithful, and that means not generating any result if none can be generated.

Part of the problem, as mentioned, is that this throws away information, because text may legitimately contain replacement characters.  (And this makes the 'check if replacement char and throw yourself' approach a non-starter).  But there are lossless encodings.  I think if we are really going to go this route, we should use something like raku's utf8-c8 (https://docs.raku.org/language/unicode#UTF8-C8).

On 04/11/2021 8:51 PM, Elronnd wrote:
> What is (potentially) questionable imo is that given foreach (c; a), c will be inferred to be dchar; you have to explicitly ask for char.  Perhaps that default should be reversed.  (This will definitely break code, though, and may not be worth it.)

I think this is the right answer.

Fix the default. Less surprises, less head aches, everyone is happy.

On Thursday, 4 November 2021 at 07:51:11 UTC, Elronnd wrote:
> What is (potentially) questionable imo is that given foreach (c; a), c will be inferred to be dchar; you have to explicitly ask for char.  Perhaps that default should be reversed.  (This will definitely break code, though, and may not be worth it.)

That's not true. It will always be the type of the thing:

void main() {
        foreach(a; "test")
                pragma(msg, typeof(a)); // immutable(char) NOT dchar
}

On 05/11/2021 12:59 AM, rikki cattermole wrote:
> 
> On 04/11/2021 8:51 PM, Elronnd wrote:
>> What is (potentially) questionable imo is that given foreach (c; a), c will be inferred to be dchar; you have to explicitly ask for char. Perhaps that default should be reversed.  (This will definitely break code, though, and may not be worth it.)
> 
> I think this is the right answer.
> 
> Fix the default. Less surprises, less head aches, everyone is happy.

Correction: the default is correct, I checked.