Thread overview
Write UTF-8 bytes directly to stack buffer
Mar 10, 2022
Chris Piker
Mar 10, 2022
H. S. Teoh
Mar 13, 2022
Chris Piker
Mar 13, 2022
Brian Callahan
March 10, 2022

Hi D

There are quite a few string, array and range functions in phobos so I'm getting confused as to the right way to encode string data as UTF-8 directly into a stack buffer while keeping track of the write point.

I have some output packets I'm building up in a tight loop. For speed I'm using the a priori knowledge that output packets will never be larger then 64K. So what's the best way to do this:

ubyte[65536] buf;
ubyte[] usable_buf = buf;

  // part of some tight loop, how to create function writef_utf8 ?
  foreach(input_thing; things){
     usable_buf.writef_utf8!"format str"(input_thing.fieldA, input_thing.fieldB);
  }

size_t used = buf.length - usable_buf.length;
stdout.write(buf[0.. used]);

March 10, 2022
On Thu, Mar 10, 2022 at 05:39:34PM +0000, Chris Piker via Digitalmars-d-learn wrote:
> Hi D
> 
> There are quite a few string, array and range functions in phobos so I'm getting confused as to the right way to encode string data as UTF-8 directly into a stack buffer while keeping track of the write point.
> 
> I have some output packets I'm building up in a tight loop.  For speed I'm using the a priori knowledge that output packets will never be larger then 64K.   So what's the best way to do this:
> 
> ```d
> ubyte[65536] buf;
> ubyte[] usable_buf = buf;
> 
>   // part of some tight loop, how to create function writef_utf8 ?
>   foreach(input_thing; things){
>      usable_buf.writef_utf8!"format str"(input_thing.fieldA, input_thing.fieldB);
>   }
> 
> size_t used = buf.length - usable_buf.length;
> stdout.write(buf[0.. used]);
> 
> ```

Probably what you're looking for is std.format.formattedWrite. For example:

------
import std;
void main() {
	ubyte[65536] buf;
	char[] usable_buf = cast(char[]) buf[];
	usable_buf.formattedWrite!"Blah %d blah %s"(123, "Это UTF-8 строка.");
	auto used = buf.length - usable_buf.length;
	writefln("%(%02X %)", buf[0 .. used]);
}
------

D strings are UTF-8 by default, so for the most part, you don't need to worry about it.


T

-- 
Guns don't kill people. Bullets do.
March 13, 2022

On Thursday, 10 March 2022 at 17:59:33 UTC, H. S. Teoh wrote:

>

Probably what you're looking for is std.format.formattedWrite. For example:

import std;
void main() {
	ubyte[65536] buf;
	char[] usable_buf = cast(char[]) buf[];
	usable_buf.formattedWrite!"Blah %d blah %s"(123, "Это UTF-8 строка.");
	auto used = buf.length - usable_buf.length;
	writefln("%(%02X %)", buf[0 .. used]);
}

Hey thanks! That does work with recent versions of dmd+phobos, but doesn't work in gdc-10. For some reason it produces this error:

error: static assert  "Cannot put a const(char)[] into a char[]."

Is there a work around involving .representation as alluded to in this thread ?

To get around the issue I built gdc-11.2 from source code at the GNU site but the old version of phobos is still included, so no dice.

March 13, 2022

On Sunday, 13 March 2022 at 07:55:01 UTC, Chris Piker wrote:

>

Hey thanks! That does work with recent versions of dmd+phobos, but doesn't work in gdc-10. For some reason it produces this error:

error: static assert  "Cannot put a const(char)[] into a char[]."

Is there a work around involving .representation as alluded to in this thread ?

To get around the issue I built gdc-11.2 from source code at the GNU site but the old version of phobos is still included, so no dice.

Build the latest gdc snapshot: https://mirrors.concertpass.com/gcc/snapshots/12-20220306/

~Brian