On Thursday, 1 May 2014 at 04:50:30 UTC, Jonathan M Davis via Digitalmars-d wrote:
> std.file's unit tests would break immediately. It wouldn't surprise me
> if std.socket's unit tests broke. std.datetime's unit tests would
> probably break on Posix systems, because some of them temporarily set
> the local time zone - which sets it for the whole program, not just the
> current thread (those tests aren't done on Windows, because Windows only
> lets you set it for the whole OS, not just the program). Any tests which
> aren't pure risk breakage due to changes in whatever global, mutable
> state they're accessing.

We really should think about separating Phobos tests into unit tests and higher level ones (in separate top-level source folder).

The fact that importing std.file in my code with `rdmd -unittest` may trigger file I/O makes me _extremely_ disgusted. How did we even get here? >_<

Walter Bright:

> On 4/30/2014 8:54 AM, bearophile wrote:
>> I'd also like some built-in way (or partially built-in) to use a module only as
>> "main module" (to run its demos) or as module to be imported. This problem is
>> solved in Python with the "if __name__ == "__main__":" idiom.
>
>     dmd foo.d -unittest -main

I think you are greatly missing the point. The unittests are mainly for the developer, while the demo is for the user of the demo. The unittests validate the code, while the demo shows how to use the module (and surely the demo is not meant to run in parallel with the other unittests). Sometimes the demo part is more than a demo, it contains usable and useful code, with a command-line interface, to allow stand alone usage of the module functionality. Currently this is how I do it:


module foobar;

// Here functions and
// classes with their unittests

version (foobar_main) {
    void main() {
        // foobar module demo code here.
    }
}


I can't use just "version(main)", because the module could import other modules with their own demo sections. So I need something to tell them apart from each other.

This could be simplified with a solution similar to the Python one:

version (__is_main) {
    void main() {
        // foobar module demo code here.
    }
}


Now if I have module A and B, where B imports A, and both have the demo main, I can use this to compile A stand-alone with its demo:

dmd A.d

And I can use this to not compile the demo of A and compile the demo section of B:

dmd A.d B.d

This is just the basic idea, and perhaps people suggested something better than this.

Bye,
bearophile

On Thu, 01 May 2014 07:26:59 +0000
Dicebot via Digitalmars-d <digitalmars-d@puremagic.com> wrote:

> On Thursday, 1 May 2014 at 04:50:30 UTC, Jonathan M Davis via Digitalmars-d wrote:
> > std.file's unit tests would break immediately. It wouldn't
> > surprise me
> > if std.socket's unit tests broke. std.datetime's unit tests
> > would
> > probably break on Posix systems, because some of them
> > temporarily set
> > the local time zone - which sets it for the whole program, not
> > just the
> > current thread (those tests aren't done on Windows, because
> > Windows only
> > lets you set it for the whole OS, not just the program). Any
> > tests which
> > aren't pure risk breakage due to changes in whatever global,
> > mutable
> > state they're accessing.
> 
> We really should think about separating Phobos tests into unit tests and higher level ones (in separate top-level source folder).
> 
> The fact that importing std.file in my code with `rdmd -unittest` may trigger file I/O makes me _extremely_ disgusted. How did we even get here? >_<

Honestly, I see no problem with std.file's unit tests triggering I/O. That's what the module _does_. And it specifically uses the system's temp directory so that it doesn't screw with anything else on the system. Separating the tests out into some other set of tests wouldn't buy us anything IMHO. The tests need to be run regardless, and they need to be run with the same frequency regardless. Splitting those tests out would just make them harder for developers to run, because now they'd have to worry about running two sets of tests instead of just one. As far as I can see, splitting out tests that do I/O would be purely for ideological reasons and would be of no practical benefit. In fact, it would be _less_ practical if we were to do so.

- Jonathan M Davis

On Thursday, 1 May 2014 at 07:47:27 UTC, Jonathan M Davis via Digitalmars-d wrote:
> Honestly, I see no problem with std.file's unit tests triggering I/O.
> That's what the module _does_. And it specifically uses the system's
> temp directory so that it doesn't screw with anything else on the
> system. Separating the tests out into some other set of tests wouldn't
> buy us anything IMHO. The tests need to be run regardless, and they
> need to be run with the same frequency regardless. Splitting those
> tests out would just make them harder for developers to run, because
> now they'd have to worry about running two sets of tests instead of
> just one. As far as I can see, splitting out tests that do I/O would be
> purely for ideological reasons and would be of no practical benefit. In
> fact, it would be _less_ practical if we were to do so.
>
> - Jonathan M Davis

I have just recently went through some of out internal projects removing all accidental I/O tests for the very reason that /tmp was full and those tests were randomly failing when testing my own program that have used the library. This _sucks_. You can't do any test with I/O without verifying the environment (free space, concurrent access from other processes, file system access etc). And once you do it properly such beast has no longer fits into the same module because of sheer size.

There is a very practical reason to separate tests - to become sure that you always can run -unittest build to verify basic sanity of your program and it will never spuriously fail or take eternity to complete.

On Thursday, 1 May 2014 at 07:32:43 UTC, bearophile wrote:
> This is just the basic idea, and perhaps people suggested something better than this.
>
> Bye,
> bearophile

Yeah I sometimes have commented out main() for that purpose.
Sounds like a useful generic addition.

Am Wed, 30 Apr 2014 13:31:30 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> I'd argue for regular identifiers instead of strings - they can be seen in stack traces, accessed with __FUNCTION__ etc. -- Andrei

If we actually want to make unittests work just like functions
(__FUNCTION__, identifier which are visible in stacktraces) then we
could also simply declare functions and mark them as (@)unittest:

unittest void myUnittest()
{

}

This then allows for further improvements in the future, for example a unit test can then return a result value (skipped, error, ..) or it could optionally receive parameters (like some kind of state from a unittest framework)

Am Wed, 30 Apr 2014 22:19:24 +0200
schrieb Jacob Carlborg <doob@me.com>:

> __traits(getUnitTests)

I've always wondered, but never asked: Doesn't __traits(getUnitTests) usage suffer from the same problem std.benchmark had?

That in order to make it work for all modules you
a) have to explicitly mention every module
b) use module constructors, leading to module constructor dependency
   hell (afaik the main reason we don't have a std.benchmark now)
?

On Thursday, 1 May 2014 at 08:51:47 UTC, Johannes Pfau wrote:
> Am Wed, 30 Apr 2014 22:19:24 +0200
> schrieb Jacob Carlborg <doob@me.com>:
>
>> __traits(getUnitTests)
>
> I've always wondered, but never asked: Doesn't __traits(getUnitTests)
> usage suffer from the same problem std.benchmark had?
>
> That in order to make it work for all modules you
> a) have to explicitly mention every module
> b) use module constructors, leading to module constructor dependency
>    hell (afaik the main reason we don't have a std.benchmark now)
> ?

You only need to make sure all modules are transitively imported from initial one. Everything else can be done via recursive reflection with __traits(allMembers), module constructors are not needed either.

On Thursday, 1 May 2014 at 08:51:47 UTC, Johannes Pfau wrote:
> Am Wed, 30 Apr 2014 22:19:24 +0200
> schrieb Jacob Carlborg <doob@me.com>:
>
>> __traits(getUnitTests)
>
> I've always wondered, but never asked: Doesn't __traits(getUnitTests)
> usage suffer from the same problem std.benchmark had?
>
> That in order to make it work for all modules you
> a) have to explicitly mention every module
> b) use module constructors, leading to module constructor dependency
>    hell (afaik the main reason we don't have a std.benchmark now)
> ?

You only need to make sure all modules are transitively imported
from initial one. Everything else can be done via recursive
reflection with __traits(allMembers), module constructors are not
needed either.

One can also generate special test entry module that imports all
project modules using build system help.

On 2014-04-30 23:25, Jonathan M Davis via Digitalmars-d wrote:

> Sure, that helps, but it's trivial to write a unittest block which
> depends on a previous unittest block

There for the tests should be run in random order.

-- 
/Jacob Carlborg