Hi,

currently I am testing D for log processing.
My perl script is more than ten times faster than my D Prog.
How can I get Lines faster from a File?

Boris

---snip---
private import std.stream;
private import std.stdio;
private import std.string;

void main (char[][] args) {
        int c;
        Stream file = new BufferedFile(args[1]);
        foreach(ulong n, char[] line; file) {
                if(std.regexp.find(line, "horizontal") > -1){
                        c++;
                }
        }

        writefln("%d", c);

}
---snip---

#!/usr/bin/perl

while($line=<>) {
        if ($line=~/horizontal/) {
                $c++;
        }
}

print "$c\n";

---snip---

Boris Bukowski wrote:
> currently I am testing D for log processing.
> My perl script is more than ten times faster than my D Prog.
> How can I get Lines faster from a File?

I don't think the file handling is your problem.

> ---snip---
> private import std.stream;
> private import std.stdio;
> private import std.string;
> 
> void main (char[][] args) {
>         int c;
>         Stream file = new BufferedFile(args[1]);
>         foreach(ulong n, char[] line; file) {
>                 if(std.regexp.find(line, "horizontal") > -1){
>                         c++;
>                 }
>         }
> 
>         writefln("%d", c);
> 
> }

std.regexp.find instantiates a RegExp object, compiles the regex and uses it once, then deletes it. This is fine for one-time searches, but if you're using it for each line of a file, you're allocating and deleting an object for each line and performing unnecessary work to recompile the same regex over and over.

Try something like this instead:
-----
// (untested code)
auto re = new RegExp("horizontal");
foreach (ulong n, char[] line; file) {
    if (re.find(line) > -1) {
// ...
-----
as the start of your foreach loop.
That should be faster.

I don't know how fast it'll be compared to Perl; I don't know anything about the relative performance of D vs. Perl regexes. (In fact, I hardly ever use regexes, and have never used Perl)

> // (untested code)
> auto re = new RegExp("horizontal");
> foreach (ulong n, char[] line; file) {
>      if (re.find(line) > -1) {
> // ...
> -----
> as the start of your foreach loop.
> That should be faster.
> 
> I don't know how fast it'll be compared to Perl; I don't know anything about the relative performance of D vs. Perl regexes. (In fact, I hardly ever use regexes, and have never used Perl)

buko01@dizit:~/d$ time ./lineio.pl access.log
1087

real    0m0.105s
user    0m0.092s
sys     0m0.012s
buko01@dizit:~/d$ time ./lineio2 access.log
1087

real    0m1.547s
user    0m1.528s
sys     0m0.020s

still 15 times slower :-(
Perl strings/IO must be somehow black magic.
Looks like I have to write my own lineReader.

Boris

Boris Bukowski wrote:
> buko01@dizit:~/d$ time ./lineio.pl access.log
> 1087
> 
> real    0m0.105s
> user    0m0.092s
> sys     0m0.012s
> buko01@dizit:~/d$ time ./lineio2 access.log
> 1087
> 
> real    0m1.547s
> user    0m1.528s
> sys     0m0.020s
> 
> still 15 times slower :-(
> Perl strings/IO must be somehow black magic.
> Looks like I have to write my own lineReader.

Some obvious questions:

Did you use -O -inline? If not, try those. I don't think they'll make much difference.

Do you actually search for "horizontal" (or a similar fixed string) ? To search for a non-regex string, std.string.find will likely be faster.


Other than that, I'm out of ideas.

IIRC Perl compiles regexes inline, presumably optimizing them along with the rest of the code, so that might explain why it's faster. This sort of stuff is what Perl was designed for...

Boris Bukowski Wrote:

> 
> > // (untested code)
> > auto re = new RegExp("horizontal");
> > foreach (ulong n, char[] line; file) {
> >      if (re.find(line) > -1) {
> > // ...
> > -----
> > as the start of your foreach loop.
> > That should be faster.
> > 
> > I don't know how fast it'll be compared to Perl; I don't know anything about the relative performance of D vs. Perl regexes. (In fact, I hardly ever use regexes, and have never used Perl)
> 
> buko01@dizit:~/d$ time ./lineio.pl access.log
> 1087
> 
> real    0m0.105s
> user    0m0.092s
> sys     0m0.012s
> buko01@dizit:~/d$ time ./lineio2 access.log
> 1087
> 
> real    0m1.547s
> user    0m1.528s
> sys     0m0.020s
> 
> still 15 times slower :-(
> Perl strings/IO must be somehow black magic.
> Looks like I have to write my own lineReader.
> 
> Boris

Hello,
on my PC the D example is faster than the Perl one, with the data produced by
the Python script

f = open('data','w')
for j in range(1087):
  for i in range(100):
    f.write("%d\n" % i)
  f.write("horizontal\n")
  for i in range(100):
    f.write("%d\n" % i)
f.close()

the Perl example takes on my PC  0.068s,  the D example with auto re takes 0.068s.

Bye
   Mario

Boris Bukowski Wrote:

> 
> > // (untested code)
> > auto re = new RegExp("horizontal");
> > foreach (ulong n, char[] line; file) {
> >      if (re.find(line) > -1) {
> > // ...
> > -----
> > as the start of your foreach loop.
> > That should be faster.
> > 
> > I don't know how fast it'll be compared to Perl; I don't know anything about the relative performance of D vs. Perl regexes. (In fact, I hardly ever use regexes, and have never used Perl)
> 
> buko01@dizit:~/d$ time ./lineio.pl access.log
> 1087
> 
> real    0m0.105s
> user    0m0.092s
> sys     0m0.012s
> buko01@dizit:~/d$ time ./lineio2 access.log
> 1087
> 
> real    0m1.547s
> user    0m1.528s
> sys     0m0.020s
> 
> still 15 times slower :-(
> Perl strings/IO must be somehow black magic.
> Looks like I have to write my own lineReader.
> 
> Boris


CORRECTION:
on my PC the D example is faster than the Perl one, with the data produced by
the Python script

f = open('data','w')
for j in range(1087):
  for i in range(100):
    f.write("%d\n" % i)
  f.write("horizontal\n")
  for i in range(100):
    f.write("%d\n" % i)
f.close()

the Perl example takes on my PC  0.148s,  the D example with auto re takes 0.068s.

Bye
   Mario

I'm a bit tired, but does BufferedFile's opApply use a fixed buffer?  I doubt it does.  In this case, the foreach method is going to be a lot slower than reading lines into a buffer.

Check on the other methods of BufferedFile.

Sorry, I'd give a code example but I'm just doing a drive by.

-[Unknown]


> Hi,
> 
> currently I am testing D for log processing.
> My perl script is more than ten times faster than my D Prog.
> How can I get Lines faster from a File?
> 
> Boris 
> 
> ---snip---
> private import std.stream;
> private import std.stdio;
> private import std.string;
> 
> void main (char[][] args) {
>         int c;
>         Stream file = new BufferedFile(args[1]);
>         foreach(ulong n, char[] line; file) {
>                 if(std.regexp.find(line, "horizontal") > -1){
>                         c++;
>                 }
>         }
> 
>         writefln("%d", c);
> 
> }
> ---snip---
> 
> #!/usr/bin/perl
> 
> while($line=<>) {
>         if ($line=~/horizontal/) {
>                 $c++;
>         }
> }
> 
> print "$c\n";
> 
> ---snip---
> 

On Tue, 30 Jan 2007 13:21:53 +0100, Boris Bukowski wrote:

> currently I am testing D for log processing.
> My perl script is more than ten times faster than my D Prog.
> How can I get Lines faster from a File?

Your example code seemed to be trying to count the number of times a certain string occurred in a file so I didn't bother with working with 'lines' as such. Anyhow, here is one way to do it...

// findtext.d ---------
private import std.file;
private import std.stdio;
private import std.string;

void main (char[][] args) {
        char[] lFileText; // Buffer for file contents.

        int lCnt;   // Number if hits
        int lPos;   // Found at position, or Not Found flag.
        int lFrom;  // Where in the file to look from.

        // Grab the whole file into RAM
        lFileText = cast(char[]) std.file.read(args[1]);

        // Start scanning for the substring.
        lFrom = 0;
        while(lFrom < lFileText.length)
        {
            lPos = std.string.find(lFileText[lFrom..$], args[2]);
            if (lPos != -1)
            {
                // Adjust next starting position.
                lFrom += lPos + args[2].length;
                // And count the hits, of course.
                lCnt++;
            }
            else
            {
                // Force end of scanning.
                lFrom = lFileText.length;
            }
        }

        writefln("Count of '%s' found in '%s': %d",
                        args[2], args[1], lCnt);
}

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"Down with mediocrity!"
31/01/2007 7:18:25 PM

mario pernici wrote:

> Boris Bukowski Wrote:
> 
>> 
>> > // (untested code)
>> > auto re = new RegExp("horizontal");
>> > foreach (ulong n, char[] line; file) {
>> >      if (re.find(line) > -1) {
>> > // ...
>> > -----
>> > as the start of your foreach loop.
>> > That should be faster.
>> > 
>> > I don't know how fast it'll be compared to Perl; I don't know anything about the relative performance of D vs. Perl regexes. (In fact, I hardly ever use regexes, and have never used Perl)
>> 
>> buko01@dizit:~/d$ time ./lineio.pl access.log
>> 1087
>> 
>> real    0m0.105s
>> user    0m0.092s
>> sys     0m0.012s
>> buko01@dizit:~/d$ time ./lineio2 access.log
>> 1087
>> 
>> real    0m1.547s
>> user    0m1.528s
>> sys     0m0.020s
>> 
>> still 15 times slower :-(
>> Perl strings/IO must be somehow black magic.
>> Looks like I have to write my own lineReader.
>> 
>> Boris
> 
> Hello,
> on my PC the D example is faster than the Perl one, with the data produced
> by the Python script
> 
> f = open('data','w')
> for j in range(1087):
>   for i in range(100):
>     f.write("%d\n" % i)
>   f.write("horizontal\n")
>   for i in range(100):
>     f.write("%d\n" % i)
> f.close()
> 
> the Perl example takes on my PC  0.068s,  the D example with auto re takes 0.068s.

Hi,

with that generated data file D is faster, cause perl spends more time in
the loop.
I use a 20MB squid access log for testing.
looks like I have to write my own readline for this.

Boris

Boris Bukowski wrote:
> Hi,
> 
> currently I am testing D for log processing.
> My perl script is more than ten times faster than my D Prog.
> How can I get Lines faster from a File?
> 
> Boris 
> 
> ---snip---
> private import std.stream;
> private import std.stdio;
> private import std.string;
> 
> void main (char[][] args) {
>         int c;
>         Stream file = new BufferedFile(args[1]);
>         foreach(ulong n, char[] line; file) {
>                 if(std.regexp.find(line, "horizontal") > -1){
>                         c++;
>                 }
>         }
> 
>         writefln("%d", c);
> 
> }
> ---snip---
> 
> #!/usr/bin/perl
> 
> while($line=<>) {
>         if ($line=~/horizontal/) {
>                 $c++;
>         }
> }
> 
> print "$c\n";
> 
> ---snip---
> 

I am too lazy to look but does the regexp module cache any regexes passed to it?  Otherwise thats probably the major slowdown.

I am pretty sure all 'fixed' regexen in Perl are  pre-compiled into the AST so they aren't re-evaluated each time they are used.

I may be wrong though, I've only been using Perl about 7 months(and I despise it).

-DavidM