Brad Roberts wrote:
> My plan at this point is to replace the makefile+do_test.sh with a c++ or d executable.  I'm tempted to use D, but that introduces a bootstrapping concern as well as a reliability concern.
>
> 
I'd also like to see the tests being run from a D program (hey, rdmd is supposed to be used like a script language, so it should be up to the task). The ddmd-project has started building the test suite with such a tool recently, but I think it does not have make functionality yet.

> I've enabled the dmd tests in the win32 auto-tester, so those results will start showing up hourly now too.
>
> 
That's already pretty cool to have. A few thoughts:

- I think the svn-revisions of the checkouts would be helpful
- to avoid too many identical builds, it should not rebuild if there are
no changes to the repositories.
- I'd like the most recent tests per platform to be separated a little
from the rest, so you can immediately see the current status.

> Later,
> Brad
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>
> 

On Tue, 7 Sep 2010, Rainer Schuetze wrote:

> Brad Roberts wrote:
> > My plan at this point is to replace the makefile+do_test.sh with a c++ or d
> > executable.  I'm tempted to use D, but that introduces a bootstrapping
> > concern
> > as well as a reliability concern.
> > 
> > 
> I'd also like to see the tests being run from a D program (hey, rdmd is supposed to be used like a script language, so it should be up to the task). The ddmd-project has started building the test suite with such a tool recently, but I think it does not have make functionality yet.

That's several votes for having a D based driver.  I'm still concerned about the same two issues I already raised:

1) bootstrapping D for new platforms -- the more that has to work before the tests can be easily executed the more pain involved.

2) if the D compiler is broken then there's a chance the test suite could show false success.

Both aren't terribly problematic, just give me pause.

I'm really shocked at how bad msys and cygwin are compared to linux.  Are windows process startup times really that bad?

> > I've enabled the dmd tests in the win32 auto-tester, so those results will
> > start
> > showing up hourly now too.
> > 
> > 
> That's already pretty cool to have. A few thoughts:
> 
> - I think the svn-revisions of the checkouts would be helpful
> - to avoid too many identical builds, it should not rebuild if there are no
> changes to the repositories.
> - I'd like the most recent tests per platform to be separated a little from
> the rest, so you can immediately see the current status.

If you've looked under the covers at the script.. that support is sorta there.  It was one of the first things I wanted to do, but decided that getting it just iterating first was more important.

I like that last one.. I was looking at the table this morning before heading to work and did a little pondering on what it needed.  Some basic filtering would be useful too.. like show just platform X.

It'd be nice to add ldc and gdc into the mix too, though I'm focused pretty much just on v2 at the moment.  I'd also be happy to let someone submit results for osx, fbsd, etc if there's a volunteer to host the builder client.

Later,
Brad

Brad Roberts wrote:
> On Tue, 7 Sep 2010, Rainer Schuetze wrote:
>
> 
>> Brad Roberts wrote:
>> 
>>> My plan at this point is to replace the makefile+do_test.sh with a c++ or d
>>> executable.  I'm tempted to use D, but that introduces a bootstrapping
>>> concern
>>> as well as a reliability concern.
>>>
>>> 
>>> 
>> I'd also like to see the tests being run from a D program (hey, rdmd is
>> supposed to be used like a script language, so it should be up to the task).
>> The ddmd-project has started building the test suite with such a tool
>> recently, but I think it does not have make functionality yet.
>> 
>
> That's several votes for having a D based driver.  I'm still concerned about the same two issues I already raised:
>
> 1) bootstrapping D for new platforms -- the more that has to work before
> the tests can be easily executed the more pain involved.
> 2) if the D compiler is broken then there's a chance the test suite could show false success.
>
> Both aren't terribly problematic, just give me pause.
> 
The test suite might not help a lot in bootstrapping: the "compilable" tests mostly check semantic analysis (which is independent of the platform), not if the object file contains sensible code. The "runnable" tests require a working druntime/phobos library, so there must already be some more or less working compiler available. But having the point where it becomes usable as early as possible is still desirable.

> I'm really shocked at how bad msys and cygwin are compared to linux.  Are
> windows process startup times really that bad?
> 
I guess the bash-startup-time is bad on windows. The runnable tests take about 1min25 when compiled and run from a batch file (no output verifications).

Rainer Schuetze wrote:
>
>
> (*) running these tests caused other problems due to cr/lf pairs: the shell would not execute scripts with CR in them, so I had to convert the *.sh files. Maybe these should not have svn:eol-style native set, but LF.
>

Fer cryin' out loud, I thought these kinds of issues would have been fixed 10 years ago.

An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/dmd-internals/attachments/20100907/95d4c046/attachment.html>

On 9/7/10 17:08 CDT, Walter Bright wrote:
>
>
> Don Clugston wrote:
>> On 7 September 2010 09:03, Brad Roberts<braddr at puremagic.com>  wrote:
>>
>>>
>>> My plan at this point is to replace the makefile+do_test.sh with a c++ or d executable.  I'm tempted to use D, but that introduces a bootstrapping concern as well as a reliability concern.
>>>
>>
>> I think we should be dogfooding as much as we can. Also, we could do some more interesting things with a D test program.
>>
>>
>
> I already wrote a test script runner, it's called shell.exe. (It's
> written in (boo, hiss) C.)

I boo and hiss not at the language, but at the existence of shell.exe. It's an extremely peculiar program with a weird spec and with an extremely generic name. Define testrunner or something, not shell.exe.

Andrei

Andrei Alexandrescu wrote:
>
> I boo and hiss not at the language, but at the existence of shell.exe. It's an extremely peculiar program with a weird spec and with an extremely generic name. Define testrunner or something, not shell.exe.
>

Renaming shell is fine, and is certainly easier than reimplementing it.

Whether it is peculiar or not is irrelevant, what is relevant is is it good for running test suites?

On 9/7/10 18:27 CDT, Walter Bright wrote:
>
>
> Andrei Alexandrescu wrote:
>>
>> I boo and hiss not at the language, but at the existence of shell.exe. It's an extremely peculiar program with a weird spec and with an extremely generic name. Define testrunner or something, not shell.exe.
>>
>
> Renaming shell is fine, and is certainly easier than reimplementing it.
>
> Whether it is peculiar or not is irrelevant, what is relevant is is it good for running test suites?

I'd very much rather use standard available tools with a minimum of custom scripting, but I don't know how far this could take us under Windows without cygwin. The many small binaries defined to support the test suite reek really foul to a Unix user.

Andrei

On Tue, 7 Sep 2010, Walter Bright wrote:

> Andrei Alexandrescu wrote:
> > 
> > I boo and hiss not at the language, but at the existence of shell.exe. It's an extremely peculiar program with a weird spec and with an extremely generic name. Define testrunner or something, not shell.exe.
> > 
> 
> Renaming shell is fine, and is certainly easier than reimplementing it.
> 
> Whether it is peculiar or not is irrelevant, what is relevant is is it good for running test suites?

As far as I'm concerned, one gigantic script of all the commands for all the tests per platform isn't an acceptable solution.  It 'works', but it's well below my 'good enough' bar.  I do agree that the code, as it stands, is unacceptable on windows.

Regardless.. with what I got done this weekend, I think we're several important steps ahead of where we've been for the last couple years.

Andrei Alexandrescu wrote:
> On 9/7/10 18:27 CDT, Walter Bright wrote:
>>
>>
>> Andrei Alexandrescu wrote:
>>>
>>> I boo and hiss not at the language, but at the existence of shell.exe. It's an extremely peculiar program with a weird spec and with an extremely generic name. Define testrunner or something, not shell.exe.
>>>
>>
>> Renaming shell is fine, and is certainly easier than reimplementing it.
>>
>> Whether it is peculiar or not is irrelevant, what is relevant is is it good for running test suites?
>
> I'd very much rather use standard available tools with a minimum of custom scripting, but I don't know how far this could take us under Windows without cygwin. The many small binaries defined to support the test suite reek really foul to a Unix user.
>

There are only two:

shell.exe
diff.exe

And, of course, about any diff will work in place of the latter, and on the Unix side the Unix diff is used. On Unix there is only one binary.

I've also seen a python script to run a test suite for C++. It compiles & runs all the .cpp files in a directory, and keys off things like "fail" in the filename to see which ones must fail, and if there's a "main" in it the test must link & run. I converted it to D so I could use it (at the end). It isn't powerful enough for what we need, but it's still an interesting method.

============= runtest.py =============
#!/usr/bin/env python2.2
import unittest, fnmatch, os, sys, cpptests

def doTests(fullpath, directory):
   dirlist = os.listdir(fullpath)
   runner = unittest.TextTestRunner()
   suite = unittest.TestSuite()
   for fname in dirlist:
      if os.path.isfile(fullpath+'/'+fname)\
         and fnmatch.fnmatch(fname, "*.cpp"):
         gen = cpptests.CppTestCase("testExecute",\
            fname[:-4])
         suite.addTest( gen )
   runner.run(suite)

def cleanUp(fullpath):
   dirlist = os.listdir(fullpath)
   for fname in dirlist:
      if os.path.isfile(fullpath+'/'+fname) and\
         (fnmatch.fnmatch(fname, "*.o")
         or fnmatch.fnmatch(fname, "*.obj")
         or fnmatch.fnmatch(fname, "*.tds")
         or fnmatch.fnmatch(fname, "*.ti")
         or fnmatch.fnmatch(fname, "*.exe")):
         os.remove(fname)

if __name__ == "__main__":
   if len(sys.argv) < 2:
      print "usage: ", sys.argv[0], " <clause directory>"
      print "usage: ", sys.argv[0], " <clause directory>/<filename>"
   else:
      directory = sys.argv[1]
      fullpath = os.getcwd()+'/'+directory
      if os.path.isdir(fullpath):
         os.chdir(fullpath)
         cpptests.failedTestList = []
         doTests(fullpath, directory)
         cleanUp(fullpath)
         print "\nFailures:\n"
         print cpptests.failures
         print len(cpptests.failures), " tests failed"
      else:
         print directory, " is not in this directory"
         print "Current directory is: ", os.getcwd()
============== cpptests.py ==================
import unittest, os, os.path, shutil, re, string

COMPILER_TYPE = 6 ############################################################################ # User must specify the compiler call and link options:                    # # 0 = keystone, 1 = gcc2.9x, 2 = gcc3.0.x, 3 = VC++, 4 = Borland, 5 = MIPS # # 6 = edg, 7 = Portland, 8 = comeau, 9 = Intel, 10 - Watcom                # ############################################################################

class ListFailures:
   def __init__(self):
      self.failedTestList = []
   def add(self, fname):
      self.failedTestList += [ fname ]
   def __len__(self):
      return len(self.failedTestList)
   def __str__(self):
      return "\n".join(self.failedTestList)

failures = ListFailures()
class CppTestCase(unittest.TestCase):
   def __init__(self, testfun, fname):
      unittest.TestCase.__init__(self, testfun)
      #Compiler and file extensions
      self.compile = [ \
                     "/home/malloy/projects/parser/keystone/run %s.cpp",
                     "g++ -c -Wall -DGCC29x %s.cpp",
                     #"g++ -Wall -w -c %s.cpp",
                     "g++ -c -Wall -w -pedantic-errors -O0 -w %s.cpp",
                     "cl /Za /W4 /c -DMSVC6x %s.cpp",
                     "bcc32 -w- -q -c -DBORLAND55 %s.cpp",
                     "CC -c -DMIPS %s.cpp",
                     #"/usr2/edg/release_3.2/bin/eccp --g++ -c %s.cpp",
                     "/usr2/edg/release_3.2/bin/eccp --strict -c %s.cpp",
                     "pgCC -w -O0 -Xa -c %s.cpp",
                     "como -c %s.cpp",
                     "icc -ansi -w -O0 -c %s.cpp",
                            "cl /Za /w /c %s.cpp"
                     ]
      self.link = [ \
                     "keystone does not link", \
                     "g++ -o %s.exe %s.o", \
                     "g++ -o %s.exe %s.o", \
                     "cl /nologo /w /Fe%s.exe %s.obj", \
                     "bcc32 -q -e%s.exe %s.obj", \
                     "CC -o %s.exe %s.o", \
                   "/usr2/edg/release_3.1/bin/eccp --strict -o %s.exe
%s.cpp",\
                     "pgCC -o %s.exe %s.o",\
                     "como -o %s.exe %s.o",\
                     "icc -o %s.exe %s.o", \
                            "cl /w /Fe%s.exe %s.obj"
                     ]

      #The rest are set automatically:
      self.fileName = fname
      self.toPass = not (fname[:4] == "fail")
      self.hasMain = 0
      self.directory = os.getcwd()

   def setUp(self):
      print "\n********************************************"
      print "\nTesting: %s.cpp" % self.fileName
      oldFile = open(self.fileName+".cpp", "r")
      currentline = oldFile.readline()
      while currentline:
         if re.search("main", currentline):
            self.hasMain = 1
            break;
         currentline = oldFile.readline()
      oldFile.close()

   def tearDown(self): pass

   # This function determines whether a test case conforms
   # to the ISO standard:
   def testExecute(self):
      executed = 0
      # different compilers require different combinations of calls
      compiled = (os.system(self.compile[COMPILER_TYPE] % self.fileName)
== 0)
      if compiled and self.hasMain:
         linked = (os.system(self.link[COMPILER_TYPE] % \
            (self.fileName, self.fileName)) == 0)
         if linked:
            executed = (os.system("%s.exe" % self.fileName) == 0)

      if self.toPass and self.hasMain and compiled and executed:
         print "PASS: Semantics properly supported"

      elif self.toPass and self.hasMain and compiled and not executed:
         print "FAIL: did not execute properly"
         failures.add(self.fileName)

      elif self.toPass and self.hasMain and not compiled and not executed:
         print "FAIL: should have compiled"
         failures.add(self.fileName)

      elif self.toPass and not self.hasMain and compiled and not executed:
         print "PASS: compiled as expected"

      elif self.toPass and not self.hasMain and not compiled and not
executed:
         print "FAIL: should have compiled"
         failures.add(self.fileName)

      elif not self.toPass and self.hasMain and compiled and executed:
         print "FAIL: executed but should't have"
         failures.add(self.fileName)

      elif not self.toPass and self.hasMain and compiled and not executed:
         print "PASS: program executed, semantics failed as expected"

      elif not self.toPass and self.hasMain and not compiled and not
executed:
         print "PASS: didn't compile as expected"

      elif not self.toPass and not self.hasMain and compiled and not
executed:
         print "FAIL: should not have compiled"
         failures.add(self.fileName)

      elif not self.toPass and not self.hasMain and \
         not compiled and not executed:
         print "PASS: didn't compile as expected"

      else:
         print "logic errors"
================= pyclean.py ================
#!/usr/bin/env python2.2

import fnmatch, os
import sys

def filelister(dummy, dirname, filesindir):
   for fname in filesindir:
      if os.path.isfile( os.path.join(dirname, fname) ):
        if  fnmatch.fnmatch(fname, "*.pyc"):
           print "deleting: ", fname
           os.remove(  os.path.join(dirname, fname) );
        if  fname[-3:] == "exe":
           print "deleting: ", fname
           os.remove(  os.path.join(dirname, fname) );
        if  fname[-2:] == "ti":
           print "deleting: ", fname
           os.remove(  os.path.join(dirname, fname) );
        if  fname[-1:] == "o":
           print "deleting: ", fname
           os.remove(  os.path.join(dirname, fname) );
        if  fname == "core":
           print "deleting: ", fname
           os.remove(  os.path.join(dirname, fname) );
        if  fname == "*.swp":
           print "deleting: ", fname
           os.remove(  os.path.join(dirname, fname) );

def runClean():
  #os.path.walk(os.getcwd(),  filelister, None)
  os.path.walk('.',  filelister, None)

if __name__ == "__main__":
   runClean()
=============== runtest.d ==========================
//import unittest, fnmatch, os, sys, cpptests

import std.file;
import std.path;
import std.string;
import std.process;


int main(char[][] args)
{
   if (args.length < 2)
   {
      printf("usage: %.*s <clause directory>\n", args[0]);
      printf("usage: %.*s <clause directory>/<filename>\n", args[0]);
   }
   else
   {
      char[] directory = args[1];
      char[] fullpath = std.path.join(std.file.getcwd(), directory);

      if (std.file.isdir(fullpath))
      {
         std.file.chdir(fullpath);
     failures = new ListFailures();
         doTests(fullpath, directory);
         cleanUp(fullpath);
         printf("\nFailures:\n");
         printf("%.*s\n", failures.toString());
         printf("%d tests failed\n", failures.length());
     if (failures.length() == 0)
        printf("Success\n");
     return failures.length() != 0;
      }
      else
      {
         printf("%.*s is not in this directory\n", directory);
         printf("Current directory is: %.*s\n", std.file.getcwd());
      }
   }
   return 1;
}

void doTests(char[] fullpath, char[] directory)
{
   char[][] dirlist = std.file.listdir(fullpath);

   CppTestCase[] suite;
   foreach (char[] fname; dirlist)
   {
      if (std.file.isfile(std.path.join(fullpath, fname))
         && std.path.fnmatch(fname, "*.cpp"))
      {  CppTestCase gen;

         gen = new CppTestCase(fname[0 .. fname.length - 4]);
         suite ~= gen;
      }
   }
   foreach (CppTestCase testcase; suite)
   {
    testcase.setUp();
    testcase.testExecute();
    testcase.tearDown();
   }
}

void cleanUp(char[] fullpath)
{
   char[][] dirlist = std.file.listdir(fullpath);
   foreach (char[] fname; dirlist)
   {
      if (std.file.isfile(std.path.join(fullpath, fname)) &&
         (std.path.fnmatch(fname, "*.o")
         || std.path.fnmatch(fname, "*.obj")
         || std.path.fnmatch(fname, "*.tds")
         || std.path.fnmatch(fname, "*.ti")
         || std.path.fnmatch(fname, "*.exe")))

         std.file.remove(fname);
   }
}


const int COMPILER_TYPE = 11;

/****************
 * User must specify the compiler call and link options:
 * 0 = keystone, 1 = gcc2.9x, 2 = gcc3.0.x, 3 = VC++, 4 = Borland, 5 = MIPS
 * 6 = edg, 7 = Portland, 8 = comeau, 9 = Intel, 10 - Watcom, 11 =
Digital Mars
 */

class ListFailures
{
   char[][] failedTestList;

   void add(char[] fname)
   {
    failedTestList ~= fname;
   }

   int length()
   {
    return failedTestList.length;
   }

   char[] toString()
   {
      return "\n" ~ std.string.join(failedTestList, "\n");
   }
}

ListFailures failures;

class CppTestCase
{
    static char[][] compile =
    [
    "/home/malloy/projects/parser/keystone/run %s.cpp",
    "g++ -c -Wall -DGCC29x %s.cpp",
    "g++ -c -Wall -w -pedantic-errors -O0 -w %s.cpp",
    "cl /Za /W4 /c -DMSVC6x %s.cpp",
    "bcc32 -w- -q -c -DBORLAND55 %s.cpp",
    "CC -c -DMIPS %s.cpp",
    "/usr2/edg/release_3.2/bin/eccp --strict -c %s.cpp",
    "pgCC -w -O0 -Xa -c %s.cpp",
    "como -c %s.cpp",
    "icc -ansi -w -O0 -c %s.cpp",
    "cl /Za /w /c %s.cpp",
    "dmc -c -A %s.cpp",
    ];

    static char[][] link =
    [
    "keystone does not link",
    "g++ -o %s.exe %s.o",
    "g++ -o %s.exe %s.o",
    "cl /nologo /w /Fe%s.exe %s.obj",
    "bcc32 -q -e%s.exe %s.obj",
    "CC -o %s.exe %s.o",
    "/usr2/edg/release_3.1/bin/eccp --strict -o %s.exe %s.cpp",
    "pgCC -o %s.exe %s.o",
    "como -o %s.exe %s.o",
    "icc -o %s.exe %s.o",
    "cl /w /Fe%s.exe %s.obj",
    "dmc %s.obj",
    ];

    char[] fileName;
    char[] directory;
    bit toPass;
    bit hasMain;

    this(char[] fname)
    {
    fileName = fname;
    toPass = (fname[0 .. 4] != "fail");
    hasMain = false;
    directory = std.file.getcwd();
    }

    void setUp()
    {
    printf("\n********************************************\n");
    printf("\nTesting: %.*s.cpp\n", fileName);

    char[] buffer = cast(char[]) std.file.read(fileName ~ ".cpp");
    hasMain = std.string.find(buffer, "main") != -1;
    }

   void tearDown() { }

   // This function determines whether a test case conforms
   // to the ISO standard:
   void testExecute()
   {
      int executed;
      int compiled;
      int linked;

      // different compilers require different combinations of calls
      compiled =
(std.process.system(std.string.format(compile[COMPILER_TYPE], fileName))
== 0);

      if (compiled && hasMain)
      {
         linked =
(std.process.system(std.string.format(link[COMPILER_TYPE], fileName,
fileName)) == 0);
         if (linked)
            executed = (std.process.system(std.string.format("%s.exe",
fileName)) == 0);
      }

      if (toPass && hasMain && compiled && executed)
         printf("PASS: Semantics properly supported\n");

      else if (toPass && hasMain && compiled && !executed)
      {  printf("FAIL: did not execute properly\n");
         failures.add(fileName);
      }

      else if (toPass && hasMain && !compiled && !executed)
      {  printf("FAIL: should have compiled\n");
         failures.add(fileName);
      }

      else if (toPass && !hasMain && compiled && !executed)
         printf("PASS: compiled as expected\n");

      else if (toPass && !hasMain && !compiled && !executed)
      {  printf("FAIL: should have compiled\n");
         failures.add(fileName);
      }

      else if (!toPass && hasMain && compiled && executed)
      {  printf("FAIL: executed but should't have\n");
         failures.add(fileName);
      }

      else if (!toPass && hasMain && compiled && !executed)
         printf("PASS: program executed, semantics failed as expected\n");

      else if (!toPass && hasMain && !compiled && !executed)
         printf("PASS: didn't compile as expected\n");

      else if (!toPass && !hasMain && compiled && !executed)
      {  printf("FAIL: should not have compiled\n");
         failures.add(fileName);
      }

      else if (!toPass && !hasMain && !compiled && !executed)
         printf("PASS: didn't compile as expected\n");

      else
         printf("logic errors\n");
    }
}