Hi,

It seems to me that some parts of the std.stream module are working VERY slow. I've done some tests with file manipulation and some comparissions with C++ and the results are strange.

ccopy.cc: simple file copy application writen in C++ dcopy.d:  simple file copy application writen in D text.txt: a text file with ~4MB

Here are the results:

$ time ./ccopy text.txt test1
real    0m0.985s
user    0m0.854s
sys     0m0.047s

Less than one second to copy 4MB.

$ time ./dcopy text.txt test2
real    1m22.886s
user    0m25.014s
sys     0m57.011s
$

More than one minute to copy 4MB.

Here are the sources (both do the same thing: read and write char by char)

/////////////////////////////////////////
// ccopy.cpp
/////////////////////////////////////////
#include <iostream>
#include <fstream>

void error (const char* p, const char* p2=" ")
{
    std::cerr << p << ' ' << p2 << '\n';
    std::exit(1);
}

int main ( int argc, char** argv )
{
    if (argc != 3)
        error ("usage: copy SOURCE DESTINATION");

    std::ifstream origem (argv[1]);
    if (!origem) error ("error openning file ", argv[1]);

    std::ofstream destino (argv[2]);
    if (!destino) error ("error copying file ", argv[2]);

    char ch;
    while ( origem.get(ch) ) destino.put(ch);

    origem.close();
    destino.close();
}

//////////////////////////////////////////
// dcopy.d
//////////////////////////////////////////
import std.stream;

int main (char[][] args)
{
    if (args.length <= 2) {
        printf ("Usage: dcopy <source> <destiny>\n");
        goto EXIT;
    }

    char c;
    File source = new File (args[1]);
    File dest = new File ();
    dest.create (args[2]);

    while (!source.eof()) {
        dest.write (source.getc());
        //printf ("%c", source.getc());
    }

EXIT:
    return 0;
}


Thanks,

Bruno.

I think the problem is source.eof(). You can open the stream.d (in
source/phobos/std) and see for yourself. This eof() uses size() and size() seeks
to the end of stream to find out the length of it. It is quite slow.
THERE IS A SIMPLE FIX: edit the stream.d a little and change the class Stream.
Make an extra variable called size_at_least. Streams can't shrink they only
grow. (Am I right? Well atleast it should be so with files.) So the function
eof() does not have to call size() every time. It can compare the position first
to the size_at_least and if position is bigger (or equal) then it updates the
size_at_least and compares again, and if position is still bigger (or equal)
then it decides that it really is the end of stream. So 99% of cases it only
uses the variable size_at_least not the function size()

There can be a better fix, but it is the simpliest, I think.
So maybe WALTER BRIGHT can make this little change in Phobos?
But since phobos is open source, you can do it yourself too. (I mean, if you
need the Stream class to work faster.)

Thank you for pointing this problem out! This is very helpful for developers, I think.

Greetengs, Martin.

In article <cb99vc$1uur$1@digitaldaemon.com>, Bruno A. Costa says...
>
>Hi,
>
>It seems to me that some parts of the std.stream module are working VERY slow. I've done some tests with file manipulation and some comparissions with C++ and the results are strange.
>
>ccopy.cc: simple file copy application writen in C++ dcopy.d:  simple file copy application writen in D text.txt: a text file with ~4MB
>
>Here are the results:
>
>$ time ./ccopy text.txt test1
>real    0m0.985s
>user    0m0.854s
>sys     0m0.047s
>
>Less than one second to copy 4MB.
>
>$ time ./dcopy text.txt test2
>real    1m22.886s
>user    0m25.014s
>sys     0m57.011s
>$
>
>More than one minute to copy 4MB.
>
>Here are the sources (both do the same thing: read and write char by char)
>
>/////////////////////////////////////////
>// ccopy.cpp
>/////////////////////////////////////////
>#include <iostream>
>#include <fstream>
> 
>void error (const char* p, const char* p2=" ")
>{
>    std::cerr << p << ' ' << p2 << '\n';
>    std::exit(1);
>}
> 
>int main ( int argc, char** argv )
>{
>    if (argc != 3)
>        error ("usage: copy SOURCE DESTINATION");
> 
>    std::ifstream origem (argv[1]);
>    if (!origem) error ("error openning file ", argv[1]);
> 
>    std::ofstream destino (argv[2]);
>    if (!destino) error ("error copying file ", argv[2]);
> 
>    char ch;
>    while ( origem.get(ch) ) destino.put(ch);
> 
>    origem.close();
>    destino.close();
>}
>
>//////////////////////////////////////////
>// dcopy.d
>//////////////////////////////////////////
>import std.stream;
> 
>int main (char[][] args)
>{
>    if (args.length <= 2) {
>        printf ("Usage: dcopy <source> <destiny>\n");
>        goto EXIT;
>    }
> 
>    char c;
>    File source = new File (args[1]);
>    File dest = new File ();
>    dest.create (args[2]);
> 
>    while (!source.eof()) {
>        dest.write (source.getc());
>        //printf ("%c", source.getc());
>    }
> 
>EXIT:
>    return 0;
>}
>
>
>Thanks,
>
>Bruno.

Martin

std.stream is slow because it's unbuffered. I only discovered this recently, but you can speed it up DRAMATICALLY by constructing a BufferedStream using a File as its construction parameter.

This is not documented. Bah.

Arcane Jill

The standard D streams are non-buffered.  You'll want to use BufferedFileStream to get better performance.  Beyond that, I'm in the process of working on stream.d to add some features to it, and I'll look at performance in the process.  The way buffering is currently done is something I'd like to discuss once I have a better idea of whether there are alternatives I think may work better.  None of my changes are guranteed to make it into Phobos, but I'm hoping that if the community is happy with them then perhaps they will.


Sean

All i can do is laugh at this really, how long has this been a problem ??

You should also checkout Mango's IO system ( http://dsource.org/projects/mango/
) , very clean , and I hear very speedy ( im checking this out myself now ).

Charlie

In article <cb99vc$1uur$1@digitaldaemon.com>, Bruno A. Costa says...
>
>Hi,
>
>It seems to me that some parts of the std.stream module are working VERY slow. I've done some tests with file manipulation and some comparissions with C++ and the results are strange.
>
>ccopy.cc: simple file copy application writen in C++ dcopy.d:  simple file copy application writen in D text.txt: a text file with ~4MB
>
>Here are the results:
>
>$ time ./ccopy text.txt test1
>real    0m0.985s
>user    0m0.854s
>sys     0m0.047s
>
>Less than one second to copy 4MB.
>
>$ time ./dcopy text.txt test2
>real    1m22.886s
>user    0m25.014s
>sys     0m57.011s
>$
>
>More than one minute to copy 4MB.
>
>Here are the sources (both do the same thing: read and write char by char)
>
>/////////////////////////////////////////
>// ccopy.cpp
>/////////////////////////////////////////
>#include <iostream>
>#include <fstream>
> 
>void error (const char* p, const char* p2=" ")
>{
>    std::cerr << p << ' ' << p2 << '\n';
>    std::exit(1);
>}
> 
>int main ( int argc, char** argv )
>{
>    if (argc != 3)
>        error ("usage: copy SOURCE DESTINATION");
> 
>    std::ifstream origem (argv[1]);
>    if (!origem) error ("error openning file ", argv[1]);
> 
>    std::ofstream destino (argv[2]);
>    if (!destino) error ("error copying file ", argv[2]);
> 
>    char ch;
>    while ( origem.get(ch) ) destino.put(ch);
> 
>    origem.close();
>    destino.close();
>}
>
>//////////////////////////////////////////
>// dcopy.d
>//////////////////////////////////////////
>import std.stream;
> 
>int main (char[][] args)
>{
>    if (args.length <= 2) {
>        printf ("Usage: dcopy <source> <destiny>\n");
>        goto EXIT;
>    }
> 
>    char c;
>    File source = new File (args[1]);
>    File dest = new File ();
>    dest.create (args[2]);
> 
>    while (!source.eof()) {
>        dest.write (source.getc());
>        //printf ("%c", source.getc());
>    }
> 
>EXIT:
>    return 0;
>}
>
>
>Thanks,
>
>Bruno.

When I used BufferedFile the D code went twice as fast as the C++ code but
that probably is a function of the default buffer size. I don't know how big
the C++ buffer is, but in D it is 8K.
Also I noticed a bug (well, a bummer if not a bug) in the BufferedStream
class in stream.d. It needs a destructor that closes the stream:
 ~this(){ close(); }
Such a destructor is in File but not in Stream. I wonder if it should be
moved from File to Stream. Otherwise when I ran your code the final buffer
contents wouldn't be written to the output file.

-Ben

"Bruno A. Costa" <bruno@codata.com.br> wrote in message news:cb99vc$1uur$1@digitaldaemon.com...
> Hi,
>
> It seems to me that some parts of the std.stream module are working VERY slow. I've done some tests with file manipulation and some comparissions with C++ and the results are strange.
>
> ccopy.cc: simple file copy application writen in C++ dcopy.d:  simple file copy application writen in D text.txt: a text file with ~4MB
>
> Here are the results:
>
> $ time ./ccopy text.txt test1
> real    0m0.985s
> user    0m0.854s
> sys     0m0.047s
>
> Less than one second to copy 4MB.
>
> $ time ./dcopy text.txt test2
> real    1m22.886s
> user    0m25.014s
> sys     0m57.011s
> $
>
> More than one minute to copy 4MB.
>
> Here are the sources (both do the same thing: read and write char by char)
>
> /////////////////////////////////////////
> // ccopy.cpp
> /////////////////////////////////////////
> #include <iostream>
> #include <fstream>
>
> void error (const char* p, const char* p2=" ")
> {
>     std::cerr << p << ' ' << p2 << '\n';
>     std::exit(1);
> }
>
> int main ( int argc, char** argv )
> {
>     if (argc != 3)
>         error ("usage: copy SOURCE DESTINATION");
>
>     std::ifstream origem (argv[1]);
>     if (!origem) error ("error openning file ", argv[1]);
>
>     std::ofstream destino (argv[2]);
>     if (!destino) error ("error copying file ", argv[2]);
>
>     char ch;
>     while ( origem.get(ch) ) destino.put(ch);
>
>     origem.close();
>     destino.close();
> }
>
> //////////////////////////////////////////
> // dcopy.d
> //////////////////////////////////////////
> import std.stream;
>
> int main (char[][] args)
> {
>     if (args.length <= 2) {
>         printf ("Usage: dcopy <source> <destiny>\n");
>         goto EXIT;
>     }
>
>     char c;
>     File source = new File (args[1]);
>     File dest = new File ();
>     dest.create (args[2]);
>
>     while (!source.eof()) {
>         dest.write (source.getc());
>         //printf ("%c", source.getc());
>     }
>
> EXIT:
>     return 0;
> }
>
>
> Thanks,
>
> Bruno.

In article <cb99vc$1uur$1@digitaldaemon.com>, Bruno A. Costa says...
> 
>    char ch;
>    while ( origem.get(ch) ) destino.put(ch);

By the way, this may be a tad faster than the above:

destino << origem.rdbuf();

I'm planning on putting similar capability in stream.d, though it may not result in much of a speedup.


Sean

Hi,

I switched from File to BufferedFile and it worked pretty well:
$ time ./dcopy text.txt teste9
real    0m0.848s
user    0m0.641s
sys     0m0.052s

Much Better. Thank you.

Bruno.

Ben Hinkle wrote:

> When I used BufferedFile the D code went twice as fast as the C++ code but
> that probably is a function of the default buffer size. I don't know how
> big the C++ buffer is, but in D it is 8K.
> Also I noticed a bug (well, a bummer if not a bug) in the BufferedStream
> class in stream.d. It needs a destructor that closes the stream:
>  ~this(){ close(); }
> Such a destructor is in File but not in Stream. I wonder if it should be
> moved from File to Stream. Otherwise when I ran your code the final buffer
> contents wouldn't be written to the output file.
> 
> -Ben
> 
> "Bruno A. Costa" <bruno@codata.com.br> wrote in message news:cb99vc$1uur$1@digitaldaemon.com...
>> Hi,
>>
>> It seems to me that some parts of the std.stream module are working VERY slow. I've done some tests with file manipulation and some comparissions with C++ and the results are strange.
>>

Or use mango.io to do a

FileConduit in = new FileConduit (inpName, FileStyle.ReadExisting);
FileConduit out = new FileConduit (outName, FileStyle.WriteTruncate);

in.copyTo (out);

<g>




"Sean Kelly" <sean@f4.ca> wrote in message news:cb9vtv$2ma$1@digitaldaemon.com...
> In article <cb99vc$1uur$1@digitaldaemon.com>, Bruno A. Costa says...
> >
> >    char ch;
> >    while ( origem.get(ch) ) destino.put(ch);
>
> By the way, this may be a tad faster than the above:
>
> destino << origem.rdbuf();
>
> I'm planning on putting similar capability in stream.d, though it may not
result
> in much of a speedup.
>
>
> Sean
>
>

In article <cba3cv$8l3$1@digitaldaemon.com>, Kris says...
>
>Or use mango.io to do a
>
>FileConduit in = new FileConduit (inpName, FileStyle.ReadExisting);
>FileConduit out = new FileConduit (outName, FileStyle.WriteTruncate);
>
>in.copyTo (out);
>
><g>

Yes, I'm convinced, mango is brilliant. But much of my work involves writing libraries rather than applications, and there are arguments both for and against any one library being dependent upon any other. If some small portion of Deimos (for example) wanted to write to a file, would it be /fair/ to users of Deimos to require them also to download and link against Mango? Should libraries be independent of each other, or should at least the best of them become intertwined.

Based on what I've seen so far, I reckon Mango ought to become /the/ streams standard for D, in which case, no problem - but at least, I need to ask the question.

Arcane Jill