Thread overview | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
June 22, 2004 std.stream very slow | ||||
---|---|---|---|---|
| ||||
Hi, It seems to me that some parts of the std.stream module are working VERY slow. I've done some tests with file manipulation and some comparissions with C++ and the results are strange. ccopy.cc: simple file copy application writen in C++ dcopy.d: simple file copy application writen in D text.txt: a text file with ~4MB Here are the results: $ time ./ccopy text.txt test1 real 0m0.985s user 0m0.854s sys 0m0.047s Less than one second to copy 4MB. $ time ./dcopy text.txt test2 real 1m22.886s user 0m25.014s sys 0m57.011s $ More than one minute to copy 4MB. Here are the sources (both do the same thing: read and write char by char) ///////////////////////////////////////// // ccopy.cpp ///////////////////////////////////////// #include <iostream> #include <fstream> void error (const char* p, const char* p2=" ") { std::cerr << p << ' ' << p2 << '\n'; std::exit(1); } int main ( int argc, char** argv ) { if (argc != 3) error ("usage: copy SOURCE DESTINATION"); std::ifstream origem (argv[1]); if (!origem) error ("error openning file ", argv[1]); std::ofstream destino (argv[2]); if (!destino) error ("error copying file ", argv[2]); char ch; while ( origem.get(ch) ) destino.put(ch); origem.close(); destino.close(); } ////////////////////////////////////////// // dcopy.d ////////////////////////////////////////// import std.stream; int main (char[][] args) { if (args.length <= 2) { printf ("Usage: dcopy <source> <destiny>\n"); goto EXIT; } char c; File source = new File (args[1]); File dest = new File (); dest.create (args[2]); while (!source.eof()) { dest.write (source.getc()); //printf ("%c", source.getc()); } EXIT: return 0; } Thanks, Bruno. |
June 22, 2004 Re: std.stream very slow | ||||
---|---|---|---|---|
| ||||
Posted in reply to Bruno A. Costa | I think the problem is source.eof(). You can open the stream.d (in source/phobos/std) and see for yourself. This eof() uses size() and size() seeks to the end of stream to find out the length of it. It is quite slow. THERE IS A SIMPLE FIX: edit the stream.d a little and change the class Stream. Make an extra variable called size_at_least. Streams can't shrink they only grow. (Am I right? Well atleast it should be so with files.) So the function eof() does not have to call size() every time. It can compare the position first to the size_at_least and if position is bigger (or equal) then it updates the size_at_least and compares again, and if position is still bigger (or equal) then it decides that it really is the end of stream. So 99% of cases it only uses the variable size_at_least not the function size() There can be a better fix, but it is the simpliest, I think. So maybe WALTER BRIGHT can make this little change in Phobos? But since phobos is open source, you can do it yourself too. (I mean, if you need the Stream class to work faster.) Thank you for pointing this problem out! This is very helpful for developers, I think. Greetengs, Martin. In article <cb99vc$1uur$1@digitaldaemon.com>, Bruno A. Costa says... > >Hi, > >It seems to me that some parts of the std.stream module are working VERY slow. I've done some tests with file manipulation and some comparissions with C++ and the results are strange. > >ccopy.cc: simple file copy application writen in C++ dcopy.d: simple file copy application writen in D text.txt: a text file with ~4MB > >Here are the results: > >$ time ./ccopy text.txt test1 >real 0m0.985s >user 0m0.854s >sys 0m0.047s > >Less than one second to copy 4MB. > >$ time ./dcopy text.txt test2 >real 1m22.886s >user 0m25.014s >sys 0m57.011s >$ > >More than one minute to copy 4MB. > >Here are the sources (both do the same thing: read and write char by char) > >///////////////////////////////////////// >// ccopy.cpp >///////////////////////////////////////// >#include <iostream> >#include <fstream> > >void error (const char* p, const char* p2=" ") >{ > std::cerr << p << ' ' << p2 << '\n'; > std::exit(1); >} > >int main ( int argc, char** argv ) >{ > if (argc != 3) > error ("usage: copy SOURCE DESTINATION"); > > std::ifstream origem (argv[1]); > if (!origem) error ("error openning file ", argv[1]); > > std::ofstream destino (argv[2]); > if (!destino) error ("error copying file ", argv[2]); > > char ch; > while ( origem.get(ch) ) destino.put(ch); > > origem.close(); > destino.close(); >} > >////////////////////////////////////////// >// dcopy.d >////////////////////////////////////////// >import std.stream; > >int main (char[][] args) >{ > if (args.length <= 2) { > printf ("Usage: dcopy <source> <destiny>\n"); > goto EXIT; > } > > char c; > File source = new File (args[1]); > File dest = new File (); > dest.create (args[2]); > > while (!source.eof()) { > dest.write (source.getc()); > //printf ("%c", source.getc()); > } > >EXIT: > return 0; >} > > >Thanks, > >Bruno. Martin |
June 22, 2004 Re: std.stream very slow | ||||
---|---|---|---|---|
| ||||
Posted in reply to Bruno A. Costa | std.stream is slow because it's unbuffered. I only discovered this recently, but you can speed it up DRAMATICALLY by constructing a BufferedStream using a File as its construction parameter. This is not documented. Bah. Arcane Jill |
June 22, 2004 Re: std.stream very slow | ||||
---|---|---|---|---|
| ||||
Posted in reply to Bruno A. Costa | The standard D streams are non-buffered. You'll want to use BufferedFileStream to get better performance. Beyond that, I'm in the process of working on stream.d to add some features to it, and I'll look at performance in the process. The way buffering is currently done is something I'd like to discuss once I have a better idea of whether there are alternatives I think may work better. None of my changes are guranteed to make it into Phobos, but I'm hoping that if the community is happy with them then perhaps they will. Sean |
June 22, 2004 Re: std.stream very slow ( Use Mango's IO ) | ||||
---|---|---|---|---|
| ||||
Posted in reply to Bruno A. Costa | All i can do is laugh at this really, how long has this been a problem ?? You should also checkout Mango's IO system ( http://dsource.org/projects/mango/ ) , very clean , and I hear very speedy ( im checking this out myself now ). Charlie In article <cb99vc$1uur$1@digitaldaemon.com>, Bruno A. Costa says... > >Hi, > >It seems to me that some parts of the std.stream module are working VERY slow. I've done some tests with file manipulation and some comparissions with C++ and the results are strange. > >ccopy.cc: simple file copy application writen in C++ dcopy.d: simple file copy application writen in D text.txt: a text file with ~4MB > >Here are the results: > >$ time ./ccopy text.txt test1 >real 0m0.985s >user 0m0.854s >sys 0m0.047s > >Less than one second to copy 4MB. > >$ time ./dcopy text.txt test2 >real 1m22.886s >user 0m25.014s >sys 0m57.011s >$ > >More than one minute to copy 4MB. > >Here are the sources (both do the same thing: read and write char by char) > >///////////////////////////////////////// >// ccopy.cpp >///////////////////////////////////////// >#include <iostream> >#include <fstream> > >void error (const char* p, const char* p2=" ") >{ > std::cerr << p << ' ' << p2 << '\n'; > std::exit(1); >} > >int main ( int argc, char** argv ) >{ > if (argc != 3) > error ("usage: copy SOURCE DESTINATION"); > > std::ifstream origem (argv[1]); > if (!origem) error ("error openning file ", argv[1]); > > std::ofstream destino (argv[2]); > if (!destino) error ("error copying file ", argv[2]); > > char ch; > while ( origem.get(ch) ) destino.put(ch); > > origem.close(); > destino.close(); >} > >////////////////////////////////////////// >// dcopy.d >////////////////////////////////////////// >import std.stream; > >int main (char[][] args) >{ > if (args.length <= 2) { > printf ("Usage: dcopy <source> <destiny>\n"); > goto EXIT; > } > > char c; > File source = new File (args[1]); > File dest = new File (); > dest.create (args[2]); > > while (!source.eof()) { > dest.write (source.getc()); > //printf ("%c", source.getc()); > } > >EXIT: > return 0; >} > > >Thanks, > >Bruno. |
June 22, 2004 Re: std.stream very slow | ||||
---|---|---|---|---|
| ||||
Posted in reply to Bruno A. Costa | When I used BufferedFile the D code went twice as fast as the C++ code but that probably is a function of the default buffer size. I don't know how big the C++ buffer is, but in D it is 8K. Also I noticed a bug (well, a bummer if not a bug) in the BufferedStream class in stream.d. It needs a destructor that closes the stream: ~this(){ close(); } Such a destructor is in File but not in Stream. I wonder if it should be moved from File to Stream. Otherwise when I ran your code the final buffer contents wouldn't be written to the output file. -Ben "Bruno A. Costa" <bruno@codata.com.br> wrote in message news:cb99vc$1uur$1@digitaldaemon.com... > Hi, > > It seems to me that some parts of the std.stream module are working VERY slow. I've done some tests with file manipulation and some comparissions with C++ and the results are strange. > > ccopy.cc: simple file copy application writen in C++ dcopy.d: simple file copy application writen in D text.txt: a text file with ~4MB > > Here are the results: > > $ time ./ccopy text.txt test1 > real 0m0.985s > user 0m0.854s > sys 0m0.047s > > Less than one second to copy 4MB. > > $ time ./dcopy text.txt test2 > real 1m22.886s > user 0m25.014s > sys 0m57.011s > $ > > More than one minute to copy 4MB. > > Here are the sources (both do the same thing: read and write char by char) > > ///////////////////////////////////////// > // ccopy.cpp > ///////////////////////////////////////// > #include <iostream> > #include <fstream> > > void error (const char* p, const char* p2=" ") > { > std::cerr << p << ' ' << p2 << '\n'; > std::exit(1); > } > > int main ( int argc, char** argv ) > { > if (argc != 3) > error ("usage: copy SOURCE DESTINATION"); > > std::ifstream origem (argv[1]); > if (!origem) error ("error openning file ", argv[1]); > > std::ofstream destino (argv[2]); > if (!destino) error ("error copying file ", argv[2]); > > char ch; > while ( origem.get(ch) ) destino.put(ch); > > origem.close(); > destino.close(); > } > > ////////////////////////////////////////// > // dcopy.d > ////////////////////////////////////////// > import std.stream; > > int main (char[][] args) > { > if (args.length <= 2) { > printf ("Usage: dcopy <source> <destiny>\n"); > goto EXIT; > } > > char c; > File source = new File (args[1]); > File dest = new File (); > dest.create (args[2]); > > while (!source.eof()) { > dest.write (source.getc()); > //printf ("%c", source.getc()); > } > > EXIT: > return 0; > } > > > Thanks, > > Bruno. |
June 22, 2004 Re: std.stream very slow | ||||
---|---|---|---|---|
| ||||
Posted in reply to Bruno A. Costa | In article <cb99vc$1uur$1@digitaldaemon.com>, Bruno A. Costa says... > > char ch; > while ( origem.get(ch) ) destino.put(ch); By the way, this may be a tad faster than the above: destino << origem.rdbuf(); I'm planning on putting similar capability in stream.d, though it may not result in much of a speedup. Sean |
June 22, 2004 Re: std.stream very slow | ||||
---|---|---|---|---|
| ||||
Posted in reply to Ben Hinkle | Hi,
I switched from File to BufferedFile and it worked pretty well:
$ time ./dcopy text.txt teste9
real 0m0.848s
user 0m0.641s
sys 0m0.052s
Much Better. Thank you.
Bruno.
Ben Hinkle wrote:
> When I used BufferedFile the D code went twice as fast as the C++ code but
> that probably is a function of the default buffer size. I don't know how
> big the C++ buffer is, but in D it is 8K.
> Also I noticed a bug (well, a bummer if not a bug) in the BufferedStream
> class in stream.d. It needs a destructor that closes the stream:
> ~this(){ close(); }
> Such a destructor is in File but not in Stream. I wonder if it should be
> moved from File to Stream. Otherwise when I ran your code the final buffer
> contents wouldn't be written to the output file.
>
> -Ben
>
> "Bruno A. Costa" <bruno@codata.com.br> wrote in message news:cb99vc$1uur$1@digitaldaemon.com...
>> Hi,
>>
>> It seems to me that some parts of the std.stream module are working VERY slow. I've done some tests with file manipulation and some comparissions with C++ and the results are strange.
>>
|
June 22, 2004 Re: std.stream very slow | ||||
---|---|---|---|---|
| ||||
Posted in reply to Sean Kelly | Or use mango.io to do a FileConduit in = new FileConduit (inpName, FileStyle.ReadExisting); FileConduit out = new FileConduit (outName, FileStyle.WriteTruncate); in.copyTo (out); <g> "Sean Kelly" <sean@f4.ca> wrote in message news:cb9vtv$2ma$1@digitaldaemon.com... > In article <cb99vc$1uur$1@digitaldaemon.com>, Bruno A. Costa says... > > > > char ch; > > while ( origem.get(ch) ) destino.put(ch); > > By the way, this may be a tad faster than the above: > > destino << origem.rdbuf(); > > I'm planning on putting similar capability in stream.d, though it may not result > in much of a speedup. > > > Sean > > |
June 22, 2004 Re: std.stream very slow | ||||
---|---|---|---|---|
| ||||
Posted in reply to Kris | In article <cba3cv$8l3$1@digitaldaemon.com>, Kris says... > >Or use mango.io to do a > >FileConduit in = new FileConduit (inpName, FileStyle.ReadExisting); >FileConduit out = new FileConduit (outName, FileStyle.WriteTruncate); > >in.copyTo (out); > ><g> Yes, I'm convinced, mango is brilliant. But much of my work involves writing libraries rather than applications, and there are arguments both for and against any one library being dependent upon any other. If some small portion of Deimos (for example) wanted to write to a file, would it be /fair/ to users of Deimos to require them also to download and link against Mango? Should libraries be independent of each other, or should at least the best of them become intertwined. Based on what I've seen so far, I reckon Mango ought to become /the/ streams standard for D, in which case, no problem - but at least, I need to ask the question. Arcane Jill |
Copyright © 1999-2021 by the D Language Foundation