On Fri, Jun 12, 2020 at 12:11:44PM +0000, duck_tape via Digitalmars-d-learn wrote:
> On Friday, 12 June 2020 at 12:02:19 UTC, duck_tape wrote:
> > For speedups with getting my hands dirty:
> > - Does writef and company flush on every line? I still haven't found
> > the source of this.

writef, et al, ultimately goes through LockingTextWriter in std.stdio.File:

https://github.com/dlang/phobos/blob/master/std/stdio.d#L2890

Looks like it's doing some Unicode manipulation and writing character by character -- a pretty slow proposition IMO!  It was done this way for Unicode-correctness, AFAICT, but if you already know the final form your output is going to take, directly calling fwrite(), or the D wrapper File.rawWrite(), will probably give you a significant performance boost.


> > - It looks like I could use {f}printf if I really wanted to: https://forum.dlang.org/post/hzcjbanvkxgohkbvjnkv@forum.dlang.org

Be aware that D strings, other than string literals, are generally NOT null-terminated, so you need to call toStringZ before calling fprintf, otherwise you might be in for a nasty surprise. :-P  Other than that, calling C from D is pretty easy:

	extern(C) int printf(char*, ...);

	void myDCode(string data) {
		printf("%s\n", data.toStringZ); // calls C printf
	}


> On Friday, 12 June 2020 at 12:02:19 UTC, duck_tape wrote:
> 
> Switching to using `core.stdc.stdio.printf` shaved off nearly two
> seconds (11->9)!
> 
> Once I wrap this up for submission to biofast I will play with mem memmapping / iopipe / tsvutils buffered writers. Sambamba is also doing some non-standard tweaks to it's outputting as well.
> 
> I'm still convinced that stdout is flushing by line.

It seems likely, if you're outputting to terminal. Otherwise, it's likely the performance slowdown is caused by Unicode manipulation code inside LockingTextWriter.

On that note, somebody should get to the bottom of this, and submit a PR to Phobos with a fast-track path for the (IMO very common) case where the string can just be fwrite'd straight into output. AFAICT, all the extra baggage currently in LockingTextWriter is mainly to deal with the case where the OS expects a (slightly) different encoding for text than is internally represented, e.g., classic 0x0D 0x0A DOS line endings (which I heard are obsolete these days, so even that case may not be as common as it used to be anymore), or outputting UTF-16 to UTF-8 or vice versa.  I'm skeptical whether this is the common case these days, so having a fast path for UTF-8 -> UTF-8 (i.e., just fwrite the whole thing straight to file) will be a good improvement for D.


T

-- 
Nobody is perfect.  I am Nobody. -- pepoluan, GKC forum