View mode: basic / threaded / horizontal-split · Log in · Help
October 24, 2005
Buffered output
I need to create a very simple program module that prints some runtime 
generated strings to the standard output. Ok, that's not a problem. The 
real problem is that I need to output these as fast as possible. The 
output consists of big string blocks (>100.000 chars) that mostly 
contain only two different characters.

I've ported the program from C. The original program doesn't use any 
sort of buffering, just some printf:s like for(i=1;i<n;i++) if 
(foobar()) printf("*.-");

Here are some simple benchmarks using the GNU time utility:


dmd test.c -o test -O3 -s
time ./test_c_printf > /dev/null
real	0m0.117s

dmd test.d -O -release -inline -version=printf
time ./test > /dev/null
real	0m0.140s

dmd test.d -O -release -inline -version=writef
time ./test > /dev/null
real	0m0.225s

How come writef() is so much slower than printf()?
Is there a fast way to do buffered output. I tried to use dynamic & 
static arrays and printf the full arrays, but that wasn't actually any 
faster.
October 24, 2005
Re: Buffered output
You may wish to try Mango.io (Stdout)?  The IO model is designed to take 
advantage of buffering, and I once ran into a problem with the Win32 console 
where it barfed on more than 32K at one time :-)

To illustrate, there's a specific subclass to handle that condition at the 
end of this module: 
http://svn.dsource.org/projects/mango/trunk/mango/io/DeviceConduit.d

Mango.io has formatting options similar to printf(), the full gamut of 
discrete IO methods, and a whole lot more besides. The mango.io equivalent 
to writef() should be faster in that it emits arrays of utf8, rather than 
one dchar at a time; and Stdout employs buffering by default. It's worth 
taking a look here http://mango.dsource.org/ and here 
http://www.dsource.org/forums/viewtopic.php?t=148

Those who prefer OOP may well prefer Mango.io over the phobos IO package(s) 
~ they're designed for different audiences.

- Kris


"Jari-Matti Mäkelä" <jmjmak@invalid_utu.fi> wrote in message 
news:djjlb6$1l22$1@digitaldaemon.com...
>I need to create a very simple program module that prints some runtime 
>generated strings to the standard output. Ok, that's not a problem. The 
>real problem is that I need to output these as fast as possible. The output 
>consists of big string blocks (>100.000 chars) that mostly contain only two 
>different characters.
>
> I've ported the program from C. The original program doesn't use any sort 
> of buffering, just some printf:s like for(i=1;i<n;i++) if (foobar()) 
> printf("*.-");
>
> Here are some simple benchmarks using the GNU time utility:
>
>
> dmd test.c -o test -O3 -s
> time ./test_c_printf > /dev/null
> real 0m0.117s
>
> dmd test.d -O -release -inline -version=printf
> time ./test > /dev/null
> real 0m0.140s
>
> dmd test.d -O -release -inline -version=writef
> time ./test > /dev/null
> real 0m0.225s
>
> How come writef() is so much slower than printf()?
> Is there a fast way to do buffered output. I tried to use dynamic & static 
> arrays and printf the full arrays, but that wasn't actually any faster.
October 24, 2005
Re: Buffered output
In article <djjlb6$1l22$1@digitaldaemon.com>,
=?ISO-8859-1?Q?Jari-Matti_M=E4kel=E4?= says...
>
>I need to create a very simple program module that prints some runtime 
>generated strings to the standard output. Ok, that's not a problem. The 
>real problem is that I need to output these as fast as possible. The 
>output consists of big string blocks (>100.000 chars) that mostly 
>contain only two different characters.
..
>How come writef() is so much slower than printf()?

writef does some fancy things that printf doesn't, though I had expected
performance to be comparable to printf for ASCII character strings.  The
implementation has changed since I last looked at it, but you might want to look
at the source for writef in std.format.d (doFormat) to see if you can find any
hangups.

>Is there a fast way to do buffered output. I tried to use dynamic & 
>static arrays and printf the full arrays, but that wasn't actually any 
>faster.

If you just want to print ASCII strings as fast as possible, printf and writef
are probably not ideal.  You may be better off with puts or fwrite, and possibly
even messing with the buffer settings using setbuf/setvbuf.

Sean
October 24, 2005
Re: Buffered output
printf and writef are already buffered - I believe stdout is line-buffered. 
Posting the actual code might help people help you. I also recommend using 
the -profile compiler option and/or std.perf. If you like you can also wrap 
a std.stream.Stream with a std.stream.BufferedStream but I suspect that you 
can do what you want with regular C buffering. See for example setvbuf.

"Jari-Matti Mäkelä" <jmjmak@invalid_utu.fi> wrote in message 
news:djjlb6$1l22$1@digitaldaemon.com...
>I need to create a very simple program module that prints some runtime 
>generated strings to the standard output. Ok, that's not a problem. The 
>real problem is that I need to output these as fast as possible. The output 
>consists of big string blocks (>100.000 chars) that mostly contain only two 
>different characters.
>
> I've ported the program from C. The original program doesn't use any sort 
> of buffering, just some printf:s like for(i=1;i<n;i++) if (foobar()) 
> printf("*.-");
>
> Here are some simple benchmarks using the GNU time utility:
>
>
> dmd test.c -o test -O3 -s
> time ./test_c_printf > /dev/null
> real 0m0.117s
>
> dmd test.d -O -release -inline -version=printf
> time ./test > /dev/null
> real 0m0.140s
>
> dmd test.d -O -release -inline -version=writef
> time ./test > /dev/null
> real 0m0.225s
>
> How come writef() is so much slower than printf()?
> Is there a fast way to do buffered output. I tried to use dynamic & static 
> arrays and printf the full arrays, but that wasn't actually any faster.
October 25, 2005
Re: Buffered output
Ben Hinkle wrote:
> printf and writef are already buffered - I believe stdout is line-buffered. 
> Posting the actual code might help people help you. I also recommend using 
> the -profile compiler option and/or std.perf. If you like you can also wrap 
> a std.stream.Stream with a std.stream.BufferedStream but I suspect that you 
> can do what you want with regular C buffering. See for example setvbuf.
> 

Yes, you're right. writef & printf are already buffered. One can see 
this with multithreaded CLI applications.

Unfortunately the original code isn't mine. I need to ask for permission 
if I want to post some code here. Ok, I'll give a shot at trying the 
std.stream-functions. Thanks.

The strangest thing here is that according to the time program the 
c-version does much greater amount of work in kernel mode. How is it 
possible at all that the D program seems to do the same I/O in user mode?
October 25, 2005
Re: Buffered output
Sean Kelly wrote:
>>How come writef() is so much slower than printf()?
> 
> 
> writef does some fancy things that printf doesn't, though I had expected
> performance to be comparable to printf for ASCII character strings.  The
> implementation has changed since I last looked at it, but you might want to look
> at the source for writef in std.format.d (doFormat) to see if you can find any
> hangups.
> 
writef is very useful and flexible, but also pretty slow. The 
std.format.d is too complex for me to optimize. I need to take a second 
look at it sometime, but for now another approach may be more useful.
Top | Discussion index | About this forum | D home