Thread overview
[Issue 20134] autodecode should use replacementDchar rather than throwing on invalid
Aug 15, 2019
Walter Bright
Aug 16, 2019
Dlang Bot
Aug 16, 2019
Walter Bright
Aug 16, 2019
Vladimir Panteleev
Aug 16, 2019
Vladimir Panteleev
Aug 16, 2019
Jon Degenhardt
Nov 06, 2021
Vladimir Panteleev
Dec 17, 2022
Iain Buclaw
August 15, 2019
https://issues.dlang.org/show_bug.cgi?id=20134

--- Comment #1 from Walter Bright <bugzilla@digitalmars.com> ---
Over time, common practice has evolved from rejecting malformed UTF to replacing it with replacementDchar, which enables the application (like a web browser) to continue processing.

Code should also be faster with this change.

--
August 16, 2019
https://issues.dlang.org/show_bug.cgi?id=20134

Dlang Bot <dlang-bot@dlang.rocks> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |pull

--- Comment #2 from Dlang Bot <dlang-bot@dlang.rocks> ---
@WalterBright updated dlang/phobos pull request #7144 "fix Issue 20134 - autodecode should use replacementDchar rather than throwing on invalid" fixing this issue:

- fix Issue 20134 - autodecode should use replacementDchar rather than throwing on invalid

https://github.com/dlang/phobos/pull/7144

--
August 16, 2019
https://issues.dlang.org/show_bug.cgi?id=20134

Walter Bright <bugzilla@digitalmars.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://issues.dlang.org/sh
                   |                            |ow_bug.cgi?id=14519

--
August 16, 2019
https://issues.dlang.org/show_bug.cgi?id=20134

Vladimir Panteleev <dlang-bugzilla@thecybershadow.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dlang-bugzilla@thecybershad
                   |                            |ow.net

--- Comment #3 from Vladimir Panteleev <dlang-bugzilla@thecybershadow.net> ---
(In reply to Walter Bright from comment #1)
> Over time, common practice has evolved from rejecting malformed UTF to replacing it with replacementDchar, which enables the application (like a web browser) to continue processing.

In applications where not crashing is preferrable to corrupting data, yes, but I don't think we can make that decision in place of the user. Corrupted data spreads and seeps into archives and can be very hard to rectify once it's discovered, but crashes are immediately visible and usually easily fixable.

> Code should also be faster with this change.

So should either assuming that the strings are valid, or throwing Errors instead of Exceptions, right?

--
August 16, 2019
https://issues.dlang.org/show_bug.cgi?id=20134

--- Comment #4 from Vladimir Panteleev <dlang-bugzilla@thecybershadow.net> ---
(In reply to Walter Bright from comment #1)
> Over time, common practice has evolved from rejecting malformed UTF to replacing it with replacementDchar, which enables the application (like a web browser) to continue processing.

BTW, I don't think this is quite correct. Web browsers both raise an error (in the dev console) AND continue processing. By using replacementDchar implicitly, D programs would not know that there was ever a problem.

--
August 16, 2019
https://issues.dlang.org/show_bug.cgi?id=20134

Jon Degenhardt <jrdemail2000-dlang@yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jrdemail2000-dlang@yahoo.co
                   |                            |m

--- Comment #5 from Jon Degenhardt <jrdemail2000-dlang@yahoo.com> ---
Correct handling of invalid UTF sequences is often known only by the application. That is, it is task dependent. And in some applications, the appropriate handling may not be known until runtime, making compile-time decisions problematic.

A related piece of the puzzle is that in many high performance string processing applications, it is useful to switch between modes of processing where strings are handled as bytes for some algorithms, then switch back to modes where strings are character sequences. When operating as bytes, UTF interpretation is not needed or desired (so no detection of invalid UTF sequences). But when algorithms are operating on characters, then invalid UTF detection/handling is desired/required. (Note: Many of these algorithms are possible because ASCII characters in UTF-8 can be used as single byte markers without interpretation of other parts of the byte stream.)

This makes it difficult for libraries to implement a single policy and still nicely support the wide range of application use-cases. Especially when there may be many layers of code between the application layer making a call and the lower level function where opportunity for detection occurs.

As an application developer, what I'd really like to have is a magical context object where the current detection and handling policies are set, and have all code invoked with the scope of that object obey them. I'd gladly take a performance hit to get it. This may too big change, but it's worth considering how well other solutions compare from an application development perspective.

--
November 06, 2021
https://issues.dlang.org/show_bug.cgi?id=20134

Vladimir Panteleev <dlang-bugzilla@thecybershadow.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://issues.dlang.org/sh
                   |                            |ow_bug.cgi?id=22473

--
December 17, 2022
https://issues.dlang.org/show_bug.cgi?id=20134

Iain Buclaw <ibuclaw@gdcproject.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P1                          |P4

--