On 13/10/2022 9:27 PM, bauss wrote:
> This doesn't actually work properly in all languages. It will probably work in most, but it's not entirely correct.
> 
> Ex. Turkish will not work with it properly.
> 
> Very interesting article: http://www.moserware.com/2008/02/does-your-code-pass-turkey-test.html

Yes turkic languages, they require a state machine and quite a bit of LUTs to work correctly.

You also need to provide a language and it has to operate on the whole string, not individual characters.

I didn't think it was relevant since Ascii was in the original post ;)

On Thursday, 13 October 2022 at 08:30:04 UTC, rikki cattermole wrote:
> On 13/10/2022 9:27 PM, bauss wrote:
>> This doesn't actually work properly in all languages. It will probably work in most, but it's not entirely correct.
>> 
>> Ex. Turkish will not work with it properly.
>> 
>> Very interesting article: http://www.moserware.com/2008/02/does-your-code-pass-turkey-test.html
>
> Yes turkic languages, they require a state machine and quite a bit of LUTs to work correctly.
>
> You also need to provide a language and it has to operate on the whole string, not individual characters.
>
> I didn't think it was relevant since Ascii was in the original post ;)

I think it's relevant when it comes to D since D is arguably a unicode language, not ascii.

D should strive to be correct, rather than fast.

On Thursday, 13 October 2022 at 08:35:50 UTC, bauss wrote:
> On Thursday, 13 October 2022 at 08:30:04 UTC, rikki cattermole wrote:
>> On 13/10/2022 9:27 PM, bauss wrote:
>>> This doesn't actually work properly in all languages. It will probably work in most, but it's not entirely correct.
>>> 
>>> Ex. Turkish will not work with it properly.
>>> 
>>> Very interesting article: http://www.moserware.com/2008/02/does-your-code-pass-turkey-test.html
>>
>> Yes turkic languages, they require a state machine and quite a bit of LUTs to work correctly.
>>
>> You also need to provide a language and it has to operate on the whole string, not individual characters.
>>
>> I didn't think it was relevant since Ascii was in the original post ;)
>
> I think it's relevant when it comes to D since D is arguably a unicode language, not ascii.
>
> D should strive to be correct, rather than fast.

Oh and to add onto this, IFF you have to do it the hacky way, then converting to uppercase instead of lowercase should be preferred, because not all lowercase characters can perform round trip, although a small group of characters, then using uppercase fixes it, so that's a relatively easy fix. A round trip is basically converting characters from one culture to another and then back. It's impossible with some characters when converting to lowercase, but should always be possible when converting to uppercase.

On 13/10/2022 9:42 PM, bauss wrote:
> Oh and to add onto this, IFF you have to do it the hacky way, then converting to uppercase instead of lowercase should be preferred, because not all lowercase characters can perform round trip, although a small group of characters, then using uppercase fixes it, so that's a relatively easy fix. A round trip is basically converting characters from one culture to another and then back. It's impossible with some characters when converting to lowercase, but should always be possible when converting to uppercase.

You will want to repeat this process with normalize to NFKC and normalize to NFD before transforming. Otherwise there is a possibility that you will miss some transformations as the simplified mappings are 1:1 for characters and not everything is representable as a single character.

On Thursday, 13 October 2022 at 08:48:49 UTC, rikki cattermole wrote:
> On 13/10/2022 9:42 PM, bauss wrote:
>> Oh and to add onto this, IFF you have to do it the hacky way, then converting to uppercase instead of lowercase should be preferred, because not all lowercase characters can perform round trip, although a small group of characters, then using uppercase fixes it, so that's a relatively easy fix. A round trip is basically converting characters from one culture to another and then back. It's impossible with some characters when converting to lowercase, but should always be possible when converting to uppercase.
>
> You will want to repeat this process with normalize to NFKC and normalize to NFD before transforming. Otherwise there is a possibility that you will miss some transformations as the simplified mappings are 1:1 for characters and not everything is representable as a single character.

Yeah, text isn't easy :D

On 13/10/2022 9:55 PM, bauss wrote:
> Yeah, text isn't easy :D

Indeed!

It has me a bit concerned actually, I'm wondering if my string stuff will even work correctly for UI's due to performance issues.

My string builder for instance allocates like crazy just to do slicing. But hey, at least I can feel confident that my general purpose allocator & infrastructure is working correctly!

On 25/10/2022 5:17 PM, Siarhei Siamashka wrote:
> Wow, I didn't expect anything like this and just thought that the nightmares of handling 8-bit codepages for non-English languages ceased to exist nowadays. Too bad. What are the best practices to deal with Turkish text in D language?

std.uni doesn't support it.

For casing it only supports the simple mappings which are 1:1 and not language dependent.

I haven't got to it yet for my own string handling library, so I can't point you to that (even if it was not ready).

I'm sure somebody has got it but you may end up wanting to use ICU unfortunately.