http://woboq.com/blog/utf-8-processing-using-simd.html

All in the article. As D include Unicode as a language feature, I think it is interesting to mention here.

deadalnix:
> http://woboq.com/blog/utf-8-processing-using-simd.html

So many things to do, so little time to do them :-)

Bye,
bearophile

On 07/30/12 21:13, deadalnix wrote:
> http://woboq.com/blog/utf-8-processing-using-simd.html
> 
> All in the article. As D include Unicode as a language feature, I think it is interesting to mention here.

Very interesting, thx for sharing. This NG definitely is a horn of plenty :)

On 7/30/2012 12:13 PM, deadalnix wrote:
> http://woboq.com/blog/utf-8-processing-using-simd.html
>
> All in the article. As D include Unicode as a language feature, I think it is
> interesting to mention here.

If someone wants to fix std.utf

   http://dlang.org/phobos/std_utf.html

to use SIMD instructions, that would be cool!

Walter Bright:

> to use SIMD instructions, that would be cool!

I think in D the most needed UTF operation is UTF8 -> UTF32.

Bye,
bearophile

On Tuesday, 31 July 2012 at 10:57:23 UTC, bearophile wrote:
> Walter Bright:
>
>> to use SIMD instructions, that would be cool!
>
> I think in D the most needed UTF operation is UTF8 -> UTF32.
>
> Bye,
> bearophile

Where is UTF-32 actually used?

Bernard Helyer:

> Where is UTF-32 actually used?

I think all std.algorithm and std.range yield UTF-32 dchars, when you give them a string in input.

Bye,
bearophile

On Tuesday, 31 July 2012 at 12:11:25 UTC, bearophile wrote:
> Bernard Helyer:
>
>> Where is UTF-32 actually used?
>
> I think all std.algorithm and std.range yield UTF-32 dchars, when you give them a string in input.
>
> Bye,
> bearophile

In addition, foreach over a string with a dchar loop variable does implicit UTF-8 decoding.

On 7/31/2012 5:24 AM, Jakob Ovrum wrote:
> On Tuesday, 31 July 2012 at 12:11:25 UTC, bearophile wrote:
>> Bernard Helyer:
>>
>>> Where is UTF-32 actually used?
>>
>> I think all std.algorithm and std.range yield UTF-32 dchars, when you give
>> them a string in input.
>>
>> Bye,
>> bearophile
>
> In addition, foreach over a string with a dchar loop variable does implicit
> UTF-8 decoding.
>

SIMD isn't going to speed things up at all for decoding one character. It is for transcoding a large array.

On Tuesday, 31 July 2012 at 19:28:03 UTC, Walter Bright wrote:
> On 7/31/2012 5:24 AM, Jakob Ovrum wrote:
>> On Tuesday, 31 July 2012 at 12:11:25 UTC, bearophile wrote:
>>> Bernard Helyer:
>>>
>>>> Where is UTF-32 actually used?
>>>
>>> I think all std.algorithm and std.range yield UTF-32 dchars, when you give
>>> them a string in input.
>>>
>>> Bye,
>>> bearophile
>>
>> In addition, foreach over a string with a dchar loop variable does implicit
>> UTF-8 decoding.
>>
>
> SIMD isn't going to speed things up at all for decoding one character. It is for transcoding a large array.

Duh, good point, I totally forgot the context.