On Wednesday, February 19, 2025 11:13:24 AM MST realhet via Digitalmars-d-learn wrote:
> Hello,
>
> The problematic line is at the very bottom.
>
> I only managed to make it run by precedding the .byChar with .text, but that is unwanted because it would convert the whole InputRange.
>
> How can I do this dchar->char conversion on only the required number of chars? (The kwSearch function will pop only a few chars, it doesn't whant a whole RandomAccessRange)

I don't have time at the moment to decipher your code and figure out what you're doing, but at a glance, it looks like you're expecting strings to be treated as ranges of immutable char, and they're not.

Phobos treats all strings as ranges of dchar. We call it auto-decoding, because it means that the range API is automatically decoding UTF-8 and UTF-16 code units to UTF-32. It was an attempt to make Unicode handling correct by default (by making it harder to accidentally split code points), but it doesn't actually succeed at providing full Unicode-correctness (since graphemes and normalization are a thing), and dealing with auto-decoding can be pretty annoying. So, we'd like to get rid of it in the next major version of Phobos and just treat all arrays as arrays of their actual element types, but for now, we have to deal with the range API treating all arrays of char and wchar as bidirectional ranges of dchar.

So, if you're looking to do anything with ranges of char, you can't use strings directly. byChar is one way to wrap a string to get a range of char. byCodeUnit would be another so long as it's an array of char specifically (rather than an array of wchar or dchar - for those, byCodeUnit would give you a range of wchar and dchar respectively). Neither of them actually converts the underlying range. Rather, they wrap it and lazily convert the elements as you access them.

So you should probably either just make your code operate on ranges of dchar, or you'll need to wrap your ranges using byChar or byCodeUnit in order to get ranges of char. All range-based functions will treat your strings as ranges of dchar. So, if really need to have strings and be treating them as ranges of char without wrapping them and without potentially creating new strings from wrapped ranges, then you can't use any range-based functions to do what you're doing.

If you're using byCodeUnit (or you use byChar on an array of char, which in turn uses byCodeUnit), then you can use the source member on the result to get the underlying string back at whatever point it is in the iteration, but in general, if you pass a string wrapped by byCodeUnit to any range-based function that returns its own range type, then you can't convert back to a string without using something like std.conv.to to allocate a new string.

Range-based functions which are eager rather than lazy (e.g. find) will return the original range, but a large percentage of range-based functions are lazy and will return wrapped ranges. So, depending on what you're doing, it's going to be difficult to do a bunch of range-based operations and then get a string at the end of the result witout allocating a new string - even if strings were treated as ranges of their actual element type.

- Jonathan M Davis

On Wednesday, February 19, 2025 7:48:48 PM MST Jonathan M Davis via Digitalmars-d-learn wrote:
> So you should probably either just make your code operate on ranges of dchar, or you'll need to wrap your ranges using byChar or byCodeUnit in order to get ranges of char. All range-based functions will treat your strings as ranges of dchar. So, if really need to have strings and be treating them as ranges of char without wrapping them and without potentially creating new strings from wrapped ranges, then you can't use any range-based functions to do what you're doing.

To add to this, another option is to use std.string.representation to cast a string to immutable(ubyte)[] - or an array of whatever the corresponding integer type is if you're not dealing with immutable(char)[] - and operate on that instead. But you can't use string as-is and have a range of char.

- Jonathan M Davis

On Thursday, 20 February 2025 at 03:54:28 UTC, Jonathan M Davis wrote:
> On Wednesday, February 19, 2025 7:48:48 PM MST Jonathan M Davis via Digitalmars-d-learn wrote:
>> So you should probably either just make your code operate on ranges of dchar

Thank You for the deep explanation!

This thing checks is a range of characters starts with a small set of keywords and returns the index+1 of the particular keyword.
It does not fetches a whole identifier, it lookus up every character in a tree.  The tree is generated as compile time template functions: every branch is translated to a switch/case statement, and there comes the hardcore optimization by LDC.

Yesterday, at one point I already decided to transition my code from chat to dchar.

My actual use case is also presents a complicated range of dchar-s.

And I just don't constraint the range type with isInputRange(R, immutable dchar). It generates a nasty error message and if popFront and empty is not supported it will show it anyways.


On the performance side of 8bit vs 32bit:
It has now 4x more data to work with with.  (Mostly because "cmp ax, im8" operation was replaced by "cmp eax, im32". So the early I can convert down to 8 bit is the better.)
The keywords I looking for at the beginning of the 'string' are only 8bit ascii characters, "#define", "#ifdef", and so on. Because of the dchar it must ensure that the high 24 bits are zero as well.
But this thing only runs for 1msec (8bit version .5ms) so it will stay in the dchar form, which - as now I learned - was the preferred usage at this moment.

I have better understanding now, Thx!