I have a part in my code that use remove

buffer.remove(tuple(0, size));

with

char[] buffer

What I discovered is that remove doesn't really remove size number of bytes but also removed entire multibyte characters and consider that one step. The result was of course that I got out of bounds exceptions as it went past the end.

When I changed char[] to ubyte[] my code started to work correctly again.

According to the documentation a char is an "unsigned 8 bit (UTF-8 code unit)" so you really believe you are working on bytes. I presume that under the hood there are range iterators at work and those work multibyte characters. However you can iterate over one byte characters as well as an option and you don't know what happens underneath.

I'm a bit confused, when should I expect that the primitives work with single versus multibyte chars in array operations?

On 10/25/20 12:29 PM, IGotD- wrote:

> What I discovered is that remove doesn't really remove size number of bytes but also removed entire multibyte characters and consider that one step. The result was of course that I got out of bounds exceptions as it went past the end.

This is the infamous "auto decode" at work.

> When I changed char[] to ubyte[] my code started to work correctly again.

Instead of changing the type, you can temporarily treat them as ubyte[] with std.string.representation:

import std.stdio;
import std.string;

void print(R)(R range) {
  writefln!"%-(%s, %)"(range);
}

void main() {
  auto s = "abcçd".dup;
  print(s);
  print(s.representation);
}

Prints:

a, b, c, ç, d
97, 98, 99, 195, 167, 100

Ali