On Monday, 27 December 2021 at 07:12:24 UTC, rempas wrote:
> I don't understand that. Based on your calculations, the results should have been different. Also how are the numbers fixed? Like you said the amount of bytes of each encoding is not always standard for every character. Even if they were fixed this means 2-bytes for each UTF-16 character and 4-bytes for each UTF-32 character so still the numbers doesn't make sense to me. So still the number of the "length" property should have been the same for every encoding or at least for UTF-16 and UTF-32. So are the sizes of every character fixed or not?
Your string is represented by 8 codepoints. The number of codeunits to represent them in memory depends on the encoding. D supports to work with 3 different encodings (in the Unicode standard there are more than these 3)
string utf8s = "Hello 😂\n";
wstring utf16s = "Hello 😂\n"w;
dstring utf32s = "Hello 😂\n"d;
Here the canonical Unicode representation of your string
H e l l o 😂 \n
U+0048 U+0065 U+006C U+006C U+006F U+0020 U+1F602 U+000a
let's see how these 3 variable are represented in memory:
utf8s : 48 65 6C 6C 6F 20 F0 9F 98 82 0a
11 char in memory using 11 bytes
utf16s: 0048 0065 006C 006C 006F 0020 D83D DE02 000A
9 wchar in memory using 18 bytes
utf16s: 00000048 00000065 0000006C 0000006C 0000006F 00000020 0001F602 0000000A
8 dchar in memory using 32 bytes
As you can see, the most compact form is generally UTF-8, that's why it is the preferred encoding for Unicode.
UTF-16 is supported because of legacy support reason like it is used in the Windows API and also internally in Java.
UTF-32 has one advantage, in that it has a 1 to 1 mapping between codepoint and array index. In practice it is not that much of an advantage as codepoints and characters are disjoint concepts. UTF-32 uses a lot of memory for practically no benefit (when you read in the forum about the big auto-decode error of D it is linked to this).
Permalink
Reply
