Hello,

i have write a test programm that takes 65 seconds.
File tralala has 213753 lines.

import std.stream;
int main() {
File fd = new File("tralala");
char[][] werte;
while (!fd.eof()) {
fd.readLine();
}
return 0;
}

Give it a better way to read a File line by line? The same perl programm takes 11 seconds.

<manfred@toppoint.de> wrote in message news:bvb54a$1qk5$1@digitaldaemon.com...
> Hello,
>
> i have write a test programm that takes 65 seconds.
> File tralala has 213753 lines.
>
> import std.stream;
> int main() {
> File fd = new File("tralala");
> char[][] werte;
> while (!fd.eof()) {
> fd.readLine();
> }
> return 0;
> }
>
> Give it a better way to read a File line by line?
> The same perl programm takes 11 seconds.

Try std.file.read().

Hmm thats interesting.

std.file.read took about 1 second on a 250K line file , the while loop took about 12 seconds for me , which is a very long time.

is it the readLine call thats making this so slow ?

C

<manfred@toppoint.de> wrote in message news:bvb54a$1qk5$1@digitaldaemon.com...
> Hello,
>
> i have write a test programm that takes 65 seconds.
> File tralala has 213753 lines.
>
> import std.stream;
> int main() {
> File fd = new File("tralala");
> char[][] werte;
> while (!fd.eof()) {
> fd.readLine();
> }
> return 0;
> }
>
> Give it a better way to read a File line by line?
> The same perl programm takes 11 seconds.
>
>
>

This C++ takes 3 seconds to read line by line.

#include <fstream>
#include <string>

using namespace std;

int main () {

 ifstream in("tralala");
 string line;
 while ( !in.eof ()  ) {

  std::getline(in,line);

 }
 return 1;
}

Ill try to use Mattys libs to find the bottleneck , as reading line by line is a common operation.

C

"C" <dont@respond.com> wrote in message news:bvbolu$2qpf$1@digitaldaemon.com...
> Hmm thats interesting.
>
> std.file.read took about 1 second on a 250K line file , the while loop
took
> about 12 seconds for me , which is a very long time.
>
> is it the readLine call thats making this so slow ?
>
> C
>
> <manfred@toppoint.de> wrote in message news:bvb54a$1qk5$1@digitaldaemon.com...
> > Hello,
> >
> > i have write a test programm that takes 65 seconds.
> > File tralala has 213753 lines.
> >
> > import std.stream;
> > int main() {
> > File fd = new File("tralala");
> > char[][] werte;
> > while (!fd.eof()) {
> > fd.readLine();
> > }
> > return 0;
> > }
> >
> > Give it a better way to read a File line by line?
> > The same perl programm takes 11 seconds.
> >
> >
> >
>
>

I believe it is readLine() that is may be making this slow. readLine seems to read a line one byte at a time. See the code for readLine in stream.d.

That is why I've been spending some free time lately creating a BufferedStream class that actually buffers read()s and uses that to get line oriented input. I'll post it when I've had a chance to get away from work enough to finish it. :)

-Damon-


C wrote:
> Hmm thats interesting.
> 
> std.file.read took about 1 second on a 250K line file , the while loop took
> about 12 seconds for me , which is a very long time.
> 
> is it the readLine call thats making this so slow ?
> 
> C
> 
> <manfred@toppoint.de> wrote in message
> news:bvb54a$1qk5$1@digitaldaemon.com...
> 
>>Hello,
>>
>>i have write a test programm that takes 65 seconds.
>>File tralala has 213753 lines.
>>
>>import std.stream;
>>int main() {
>>File fd = new File("tralala");
>>char[][] werte;
>>while (!fd.eof()) {
>>fd.readLine();
>>}
>>return 0;
>>}
>>
>>Give it a better way to read a File line by line?
>>The same perl programm takes 11 seconds.
>>
>>
>>
> 
> 
> 

Ok cool I look forward to it.   I can't believe we don't have a phobos library group yet.  So frustrurating...

C

"Damon Gray" <dontbotherasking@go.away.mr.bad.spammer.net> wrote in message news:40196AB3.5040107@go.away.mr.bad.spammer.net...
> I believe it is readLine() that is may be making this slow. readLine seems to read a line one byte at a time. See the code for readLine in stream.d.
>
> That is why I've been spending some free time lately creating a BufferedStream class that actually buffers read()s and uses that to get line oriented input. I'll post it when I've had a chance to get away from work enough to finish it. :)
>
> -Damon-
>
>
> C wrote:
> > Hmm thats interesting.
> >
> > std.file.read took about 1 second on a 250K line file , the while loop
took
> > about 12 seconds for me , which is a very long time.
> >
> > is it the readLine call thats making this so slow ?
> >
> > C
> >
> > <manfred@toppoint.de> wrote in message news:bvb54a$1qk5$1@digitaldaemon.com...
> >
> >>Hello,
> >>
> >>i have write a test programm that takes 65 seconds.
> >>File tralala has 213753 lines.
> >>
> >>import std.stream;
> >>int main() {
> >>File fd = new File("tralala");
> >>char[][] werte;
> >>while (!fd.eof()) {
> >>fd.readLine();
> >>}
> >>return 0;
> >>}
> >>
> >>Give it a better way to read a File line by line?
> >>The same perl programm takes 11 seconds.
> >>
> >>
> >>
> >
> >
> >
>
>

I don't have the source in front of me right now but I believe it's the eof().  It is implemented in an atrocious fashion if I remember correctly.


In article <bvbolu$2qpf$1@digitaldaemon.com>, C says...
>
>Hmm thats interesting.
>
>std.file.read took about 1 second on a 250K line file , the while loop took about 12 seconds for me , which is a very long time.
>
>is it the readLine call thats making this so slow ?
>
>C
>
><manfred@toppoint.de> wrote in message news:bvb54a$1qk5$1@digitaldaemon.com...
>> Hello,
>>
>> i have write a test programm that takes 65 seconds.
>> File tralala has 213753 lines.
>>
>> import std.stream;
>> int main() {
>> File fd = new File("tralala");
>> char[][] werte;
>> while (!fd.eof()) {
>> fd.readLine();
>> }
>> return 0;
>> }
>>
>> Give it a better way to read a File line by line?
>> The same perl programm takes 11 seconds.
>>
>>
>>
>
>

"C" <dont@respond.com> wrote in message news:bvbpjm$2sjj$1@digitaldaemon.com...
> This C++ takes 3 seconds to read line by line.
>
> #include <fstream>
> #include <string>
>
> using namespace std;
>
> int main () {
>
>  ifstream in("tralala");
>  string line;
>  while ( !in.eof ()  ) {
>
>   std::getline(in,line);
>
>  }
>  return 1;
> }
>
> Ill try to use Mattys libs to find the bottleneck , as reading line by
line
> is a common operation.

Cool. Then you can "suggest" to Walter that we include it in Phobos. I've had no luck in that so far. :)

>
> C
>
> "C" <dont@respond.com> wrote in message news:bvbolu$2qpf$1@digitaldaemon.com...
> > Hmm thats interesting.
> >
> > std.file.read took about 1 second on a 250K line file , the while loop
> took
> > about 12 seconds for me , which is a very long time.
> >
> > is it the readLine call thats making this so slow ?
> >
> > C
> >
> > <manfred@toppoint.de> wrote in message news:bvb54a$1qk5$1@digitaldaemon.com...
> > > Hello,
> > >
> > > i have write a test programm that takes 65 seconds.
> > > File tralala has 213753 lines.
> > >
> > > import std.stream;
> > > int main() {
> > > File fd = new File("tralala");
> > > char[][] werte;
> > > while (!fd.eof()) {
> > > fd.readLine();
> > > }
> > > return 0;
> > > }
> > >
> > > Give it a better way to read a File line by line?
> > > The same perl programm takes 11 seconds.
> > >
> > >
> > >
> >
> >
>
>

> I don't have the source in front of me right now but I believe it's the eof().  It is implemented in an atrocious fashion if I remember correctly.


You weren't kidding:

    bit eof() { return position() == size(); }

It w*nks. Unbelievable stuff!

    ulong position() { return seek(0, SeekPos.Current); }

    // returns size of stream
    ulong size()
    {
        ulong pos = position(), result = seek(0, SeekPos.End);
        position(pos);
        return result;
    }

That's three kernel calls to check whether we've hit EOF!

My confidence in stream.d has just fallen to about 5%.

Walter, did anyone do a review on this before accepting it into Phobos? This needs a serious rewrite, if not a redesign.


> In article <bvbolu$2qpf$1@digitaldaemon.com>, C says...
> >
> >Hmm thats interesting.
> >
> >std.file.read took about 1 second on a 250K line file , the while loop
took
> >about 12 seconds for me , which is a very long time.
> >
> >is it the readLine call thats making this so slow ?
> >
> >C
> >
> ><manfred@toppoint.de> wrote in message news:bvb54a$1qk5$1@digitaldaemon.com...
> >> Hello,
> >>
> >> i have write a test programm that takes 65 seconds.
> >> File tralala has 213753 lines.
> >>
> >> import std.stream;
> >> int main() {
> >> File fd = new File("tralala");
> >> char[][] werte;
> >> while (!fd.eof()) {
> >> fd.readLine();
> >> }
> >> return 0;
> >> }
> >>
> >> Give it a better way to read a File line by line?
> >> The same perl programm takes 11 seconds.
> >>
> >>
> >>
> >
> >
>
>

"Matthew" <matthew.hat@stlsoft.dot.org> wrote in message news:bvc0vu$82o$2@digitaldaemon.com...
> Walter, did anyone do a review on this before accepting it into Phobos?
This
> needs a serious rewrite, if not a redesign.

No. It was very early on, done just to get things off the ground. It could use a rewrite. Anyone up for it?