Greetings.

I have a program which handles lots of text files on a per line basis.  I am trying to get some info from the D gurus as to which "if" option is fastest and best for a large amount of text handling lines.  Let us imagine these two instances of code:

Code a:
|char[] GetTheChoice(char[] line)
|{
|  if (std.string.find(line,"A") > -1)
|    return "A";
|  if (std.string.find(line,"B") > -1)
|    return "A";
|  ...
|  ...
|  ...
|  if (std.string.find(line,"Y") > -1)
|    return "A";
|  if (std.string.find(line,"Z") > -1)
|    return "A";
|}


Code b:
|char[] GetTheChoice(char[] line)
|{
|  if (std.string.find(line,"A") > -1 ||
|      std.string.find(line,"B") > -1 ||
|  ...
|  ...
|  ...
|      std.string.find(line,"Y") > -1 ||
|      std.string.find(line,"Z") > -1)
|    return "A";
|}

Also, if there is a faster way than this.

thanks,

josé

On Wed, 03 Aug 2005 12:38:01 -0400, jicman <jicman_member@pathlink.com> wrote:

>
> Greetings.
>
> I have a program which handles lots of text files on a per line basis.  I am
> trying to get some info from the D gurus as to which "if" option is fastest and
> best for a large amount of text handling lines.  Let us imagine these two
> instances of code:
>
> Code a:
> |char[] GetTheChoice(char[] line)
> |{
> |  if (std.string.find(line,"A") > -1)
> |    return "A";
> |  if (std.string.find(line,"B") > -1)
> |    return "A";
> |  ...
> |  ...
> |  ...
> |  if (std.string.find(line,"Y") > -1)
> |    return "A";
> |  if (std.string.find(line,"Z") > -1)
> |    return "A";
> |}
>
>
> Code b:
> |char[] GetTheChoice(char[] line)
> |{
> |  if (std.string.find(line,"A") > -1 ||
> |      std.string.find(line,"B") > -1 ||
> |  ...
> |  ...
> |  ...
> |      std.string.find(line,"Y") > -1 ||
> |      std.string.find(line,"Z") > -1)
> |    return "A";
> |}
>
> Also, if there is a faster way than this.
>
> thanks,
>
> josé
>

Well, std.string.find() needs to start at the beginning of the array for each search, so not using it at all and just looping over the string once would be fastest.

char[] GetTheChoice(char[] line)
{
   foreach(char ch; line)
   {
      switch(ch)
      {
         case 'A', 'B':
            return "A";
         case 'Y', 'Z':
            return "A";
         default: ;
      }
   }
   //return "?";
}

They're the same. In general, though, you need to profile code to see where the real bottlenecks are, as the results can be counterintuitive and surprising. This should get you started:

    www.digitalmars.com/techtips/timing_code.html

jicman <jicman_member@pathlink.com> wrote:

[...]
> Also, if there is a faster way than this.

Have you thaught of

import std.regexP;
//...
  if(( new RegExp( "[A-Z]", "")).test( line)) return( "A");

-manfred

Vathix says...
>
>On Wed, 03 Aug 2005 12:38:01 -0400, jicman <jicman_member@pathlink.com> wrote:
>
>>
>> Greetings.
>>
>> I have a program which handles lots of text files on a per line basis.
>> I am
>> trying to get some info from the D gurus as to which "if" option is
>> fastest and
>> best for a large amount of text handling lines.  Let us imagine these two
>> instances of code:
>>
>> Code a:
>> |char[] GetTheChoice(char[] line)
>> |{
>> |  if (std.string.find(line,"A") > -1)
>> |    return "A";
>> |  if (std.string.find(line,"B") > -1)
>> |    return "A";
>> |  ...
>> |  ...
>> |  ...
>> |  if (std.string.find(line,"Y") > -1)
>> |    return "A";
>> |  if (std.string.find(line,"Z") > -1)
>> |    return "A";
>> |}
>>
>>
>> Code b:
>> |char[] GetTheChoice(char[] line)
>> |{
>> |  if (std.string.find(line,"A") > -1 ||
>> |      std.string.find(line,"B") > -1 ||
>> |  ...
>> |  ...
>> |  ...
>> |      std.string.find(line,"Y") > -1 ||
>> |      std.string.find(line,"Z") > -1)
>> |    return "A";
>> |}
>>
>> Also, if there is a faster way than this.
>>
>> thanks,
>>
>> josé
>>
>
>Well, std.string.find() needs to start at the beginning of the array for each search, so not using it at all and just looping over the string once would be fastest.
>
>char[] GetTheChoice(char[] line)
>{
>    foreach(char ch; line)
>    {
>       switch(ch)
>       {
>          case 'A', 'B':
>             return "A";
>          case 'Y', 'Z':
>             return "A";
>          default: ;
>       }
>    }
>    //return "?";
>}

Well, the problem is that I am not really searching for "one letter".  I should have entered another example. ;-)  My fault.  There is a lot of information missing here, but I need to use std.string.find().

Thanks for the help, though.

jic

Thanks Walter.  Yes, I figured that some if's will be encountered less than others, and those I am placing at the bottom of the list.  This will allow for those most used if's to return faster.  I will try both of these if options and tell you if there is a real time difference between them.  However, don't wait standing up.  I am working on a project which I chosed to do in D, instead of Java or c, and I need to finish it first. ;-)  It's due Friday.  My boss does not know, but when he sees the outcome, he'll be very glad. ;-)

thanks,

jic

Walter says...
>
>They're the same. In general, though, you need to profile code to see where the real bottlenecks are, as the results can be counterintuitive and surprising. This should get you started:
>
>    www.digitalmars.com/techtips/timing_code.html
>
>

Manfred Nowak says...
>
>jicman <jicman_member@pathlink.com> wrote:
>
>[...]
>> Also, if there is a faster way than this.
>
>Have you thaught of
>
>import std.regexP;
>//...
>  if(( new RegExp( "[A-Z]", "")).test( line)) return( "A");

Will this be faster than std.string.find?  I already have some RegExp for something else, but I wouldn't think that opening a RegExp object for every different if would be faster.  Perhaps, I should have explained it better.  I am not just searching for one letter.  It was just an example.  Here is the full test, and perhaps, some of you may think of a better way:

|char[] GetTheChoice(char[] line)
|{
|  //writefln(line);
|  if (std.string.find(line,"JUNKDATA") > -1)
|    return "JUNKDATA";
|  if (std.string.find(line,"Auto stopping job") > -1)
|    return "AutoStopJob";
|  if (std.string.find(line,"Auto starting the next job") > -1)
|    return "AutoStartJob";
|  if (std.string.find(line,"HTTP/NSA job") > -1)
|    return "HTTP/NSA job";
|  if (std.string.find(line,"HTTP/IFax job") > -1)
|    return "HTTP/IFax job";
|  if (std.string.find(line,"IFax processing message") > -1)
|    return "IFax processing message";
|  if (std.string.find(line,"IFax Job received") > -1)
|    return "IFax job";
|  if (std.string.find(line,"IFax Job POP3 job") > -1)
|    return "IFax POP3 job";
|  if (std.string.find(line,"Login failed because password incorrect:") > -1)
|    return "Failed NSA FTP login";
|  if (std.string.find(line,"User logged in:") > -1)
|    return "NSA FTP login";
|  if (std.string.find(line,"NSA job from ") > -1)
|    return "NSA job from ";
|  if (std.string.find(line,"Processing image of Cover Sheet ") > -1)
|    return "Processing image of Cover Sheet ";
|  if (std.string.find(line,"e-mail confirmation of") > -1)
|    return "e-mail confirmation of";
|  if (std.string.find(line,"Printing Confirmation Page") > -1)
|    return "Printing Confirmation Page";
|  if (std.string.find(line,"Web Centre URL ") > -1)
|    return "Web Centre URL ";
|  if (std.string.find(line,"Restarted job: ") > -1)
|    return "Restarted job: ";
|  if (std.string.find(line,"Starting remote job") > -1)
|    return "Starting remote job";
|  if (std.string.find(line,"Error handler reports") > -1)
|    return "Error handler reports";
|  if (std.string.find(line," Attempting to send mail to ") > -1)
|    return "CreatingUser0";
|  if (std.string.find(line," User Created: ") > -1)
|    return "CreatingUser1";
|  if (std.string.find(line,"ERROR ") > -1)
|    return "ERROR ";
|  if (std.string.find(line,"DEBUG ") > -1)
|    return "DEBUG ";
|  if (std.string.find(line,"WEB ") > -1)
|    return "WEB ";
|  if (std.string.find(line,"OCR: begin=") > -1)
|    return "OCR: begin=";
|  if (std.string.find(line,"OCR: fail=") > -1)
|    return "OCR: fail=";
|  if (std.string.find(line,"OCR: end=") > -1)
|    return "OCR: end=";
|  if (std.string.find(line,">>> Rejecting login ") > -1)
|    return "HTTPProgrammaticAccess";
|  if (std.string.find(line,"WARNING ") > -1)
|    return "WARNING ";
|  if (std.string.find(line,"SYSTEM.ERR ") > -1)
|    return "SYSTEM.ERR ";
|  if (std.string.find(line,"SYSTEM.OUT ") > -1)
|    return "SYSTEM.OUT ";
|  if (std.string.find(line,"DocuShare retry loop required ") > -1)
|    return "DSRetryLoop";
|  if (std.string.find(line,"FTP command rejected,") > -1)
|    return "FTP command rejected";
|  if (std.string.find(line,"Login: User ") > -1)
|    return "UserLogin";
|  if (std.string.find(line,"Logout from web") > -1)
|    return "UserLogout";
|  if (std.string.find(line,"Login: Username ") > -1)
|    return "BadPassWord";
|  if (std.string.find(line," Advanced access ") > -1) // this one covers
|    return "AdvancedAccess";  // both the request and the access
|  if (std.string.find(line," The scheduled backup did not occur") > -1)
|    return "MissedBackup";
|  if (std.string.find(line,">>> Last backup time of ") > -1)
|    return "MissedBackup";
|  if (std.string.find(line,"There are ") > -1 &&
|      std.string.find(line," jobs waiting to restart") > -1)
|    return "JobsBackedUp";
|  if (std.string.find(line,"Waiting for ") > -1 &&
|      (std.string.find(line," job to complete") > -1 ||
|      std.string.find(line," jobs to complete") > -1 ))
|    return "JobsBackedUp";
|  if (std.string.find(line," Persistent GC is slow") > -1)
|    return "GCSlow";
|  if (std.string.find(line," Persistent store GC reclaimed ") > -1)
|    return "GarbageCollection";
|  if (std.string.find(line,"Shutdown called:") > -1)
|    return "Shutdown called:";
|  if (std.string.find(line,"About to shutdown due to timeout") > -1)
|    return "TimeOutShutDown";
|  if (std.string.find(line,"Persistent Store has been closed!") > -1 ||
|      std.string.find(line,"Terminating persistent store session") > -1)
|    return "ClosedPersistentStore";
|  if (
|      std.string.find(line," >>> About to do GC: ") > -1 ||
|      std.string.find(line," >>> Did GC: ") > -1 ||
|      std.string.find(line," Initial class load") > -1 ||
|      std.string.find(line," Cleaning Database prior ") > -1 ||
|      std.string.find(line," Creating unpack.ini") > -1 ||
|      std.string.find(line," Fixing internal state ") > -1 ||
|      std.string.find(line," Recording Unpack ") > -1 ||
|      std.string.find(line," Deleting info inserted ") > -1 ||
|      std.string.find(line," Packing Properties and ") > -1 ||
|      std.string.find(line," Packing Independent Entities") > -1 ||
|      std.string.find(line," Noting Replaceable objects") > -1 ||
|      std.string.find(line," Starting to unpack objects") > -1 ||
|      std.string.find(line," >>> Credentials support provided ") > -1 ||
|      std.string.find(line," >>> Gathered ") > -1 ||
|      std.string.find(line," >>> Finished unpacking") > -1 ||
|      std.string.find(line," >>> Finished Unpacking") > -1 ||
|      std.string.find(line," >>> SystemParameter ") > -1 ||
|      std.string.find(line," Writing objects to ") > -1 ||
|      std.string.find(line," Exporting database objects...") > -1 ||
|      std.string.find(line," Importing database objects...") > -1 ||
|      std.string.find(line,"Administrative Notice Logging Facility") > -1 ||
|      std.string.find(line,"Starting NSA FTP with server") > -1 ||
|      std.string.find(line," >>> Unable to read ") > -1 ||
|      std.string.find(line," >>> Deleting ") > -1 ||
|      std.string.find(line," >>> Installing Patch ") > -1 ||
|      std.string.find(line," Installing services and ") > -1 ||
|      std.string.find(line," >>> Finished Packing") > -1 ||
|      std.string.find(line,"Starting NSA FTP server") > -1 ||
|      std.string.find(line,"Using Hostname class from ") > -1 ||
|      std.string.find(line,"Started POP3 poller for ") > -1 ||
|      std.string.find(line," >>> POP3 polling enabled") > -1 ||
|      std.string.find(line," >>> POP3 enabled polling") > -1 ||
|      std.string.find(line,"Starting POP3 poller for ") > -1 ||
|      std.string.find(line,"Loading ApplicationJarEntities") > -1 ||
|      std.string.find(line,">>> Loading application ") > -1 ||
|      std.string.find(line,">>> Loading service ") > -1 ||
|      std.string.find(line," Debugging and Error Logging Faci") > -1 ||
|      std.string.find(line,"FlowPort Version FlowPort") > -1 ||
|      std.string.find(line,"TextBridge SDK 5.0-PP") > -1 ||
|      std.string.find(line," Deflate compressed in pass ") > -1 ||
|      std.string.find(line," Terminating persistent store GC ") > -1 ||
|      std.string.find(line,"Attempting to start IFax SMTP") > -1 ||
|      std.string.find(line,"Enabled SMTP IFax server COM.") > -1 ||
|      std.string.find(line,"Started SMTP IFax server COM.") > -1
|     )
|    return "IgnoreEntry"; // ignore these STATUS entries... not used
|  return "NotUsableData";
|}

thanks,

josé

"jicman" <jicman_member@pathlink.com> wrote in message news:dcr6dh$1kv5$1@digitaldaemon.com...
>
> Manfred Nowak says...
>>
>>jicman <jicman_member@pathlink.com> wrote:
>>
>>[...]
>>> Also, if there is a faster way than this.
>>
>>Have you thaught of
>>
>>import std.regexP;
>>//...
>>  if(( new RegExp( "[A-Z]", "")).test( line)) return( "A");
>
> Will this be faster than std.string.find?  I already have some RegExp for
> something else, but I wouldn't think that opening a RegExp object for
> every
> different if would be faster.  Perhaps, I should have explained it better.
> I am
> not just searching for one letter.  It was just an example.  Here is the
> full
> test, and perhaps, some of you may think of a better way:

If you could find a way to use string comparison instead of find then you might be able to use an associative array with the keys the strings like "JUNKDATA", "Auto stopping job" etc and the values strings like "JUNKDATA","AutoStopJob", etc. Without any assumptions about where in the line they keys can appear it would be tough to do any better than what you are currently doing.

Ben Hinkle says...
>
>
>"jicman" <jicman_member@pathlink.com> wrote in message news:dcr6dh$1kv5$1@digitaldaemon.com...
>>
>> Manfred Nowak says...
>>>
>>>jicman <jicman_member@pathlink.com> wrote:
>>>
>>>[...]
>>>> Also, if there is a faster way than this.
>>>
>>>Have you thaught of
>>>
>>>import std.regexP;
>>>//...
>>>  if(( new RegExp( "[A-Z]", "")).test( line)) return( "A");
>>
>> Will this be faster than std.string.find?  I already have some RegExp for
>> something else, but I wouldn't think that opening a RegExp object for
>> every
>> different if would be faster.  Perhaps, I should have explained it better.
>> I am
>> not just searching for one letter.  It was just an example.  Here is the
>> full
>> test, and perhaps, some of you may think of a better way:
>
>If you could find a way to use string comparison instead of find then you might be able to use an associative array with the keys the strings like "JUNKDATA", "Auto stopping job" etc and the values strings like "JUNKDATA","AutoStopJob", etc. Without any assumptions about where in the line they keys can appear it would be tough to do any better than what you are currently doing.
>
>

Ok, thanks.  These values are returned to a case which then, depending on the return string will execute more string splits, checks, etc.  I just wanted to get an idea, since the amount of data is getting bigger and bigger.

thanks again folks.

jic

jicman <jicman_member@pathlink.com> wrote:

[...]
>  Here is the full test, and
> perhaps, some of you may think of a better way:

There is no predefined solution for your problem in D.

If you want a fast running solution for a big data pool you have to implement a lexer.

If you want a prototype to show how fast developing in D can be, you are already done.

-manfred