On 08/24/2016 01:44 AM, Edwin van Leeuwen wrote:
> On Tuesday, 23 August 2016 at 17:55:41 UTC, Andrei Alexandrescu wrote:
>> Why does "dub test" rebuild the library? It should just re-run
>> unittests. -- Andrei
>
> To be honest I prefer it to rebuild it. Why would you want to rerun the
> tests without a rebuild? It should give the same failure/success as last
> time.

Randomized testing. -- Andrei

On 08/24/2016 04:14 AM, Edwin van Leeuwen wrote:
> I might be dense, but the only other thing than integration tests that I
> can think of is if you use random data for testing, but that would be
> more correctly solved by using more random data during the unittests.
> Nothing is worse than tests that only sometimes fail.

Randomized unit testing is a respected approach to testing: https://hackage.haskell.org/package/QuickCheck, https://github.com/rickynils/scalacheck, https://blogs.msdn.microsoft.com/dsyme/2008/08/08/fscheck-0-2/, etc. Some of the Phobos tests I wrote use it (and virtually all of my current code e.g. on median computation), and when they fail I just print the seed of the RNG and then hardcode it to reproduce the test. -- Andrei

On Wednesday, 24 August 2016 at 12:00:32 UTC, Andrei Alexandrescu wrote:
> On 08/24/2016 04:14 AM, Edwin van Leeuwen wrote:
>> I might be dense, but the only other thing than integration tests that I
>> can think of is if you use random data for testing, but that would be
>> more correctly solved by using more random data during the unittests.
>> Nothing is worse than tests that only sometimes fail.
>
> Randomized unit testing is a respected approach to testing: https://hackage.haskell.org/package/QuickCheck, https://github.com/rickynils/scalacheck, https://blogs.msdn.microsoft.com/dsyme/2008/08/08/fscheck-0-2/, etc. Some of the Phobos tests I wrote use it (and virtually all of my current code e.g. on median computation), and when they fail I just print the seed of the RNG and then hardcode it to reproduce the test. -- Andrei

Do you remember that there is a pending PR for this at Phobos that is blocked because of missing feedback / acceptance?

https://github.com/dlang/phobos/pull/2995

On 08/24/2016 09:25 AM, Seb wrote:
> On Wednesday, 24 August 2016 at 12:00:32 UTC, Andrei Alexandrescu wrote:
>> On 08/24/2016 04:14 AM, Edwin van Leeuwen wrote:
>>> I might be dense, but the only other thing than integration tests that I
>>> can think of is if you use random data for testing, but that would be
>>> more correctly solved by using more random data during the unittests.
>>> Nothing is worse than tests that only sometimes fail.
>>
>> Randomized unit testing is a respected approach to testing:
>> https://hackage.haskell.org/package/QuickCheck,
>> https://github.com/rickynils/scalacheck,
>> https://blogs.msdn.microsoft.com/dsyme/2008/08/08/fscheck-0-2/, etc.
>> Some of the Phobos tests I wrote use it (and virtually all of my
>> current code e.g. on median computation), and when they fail I just
>> print the seed of the RNG and then hardcode it to reproduce the test.
>> -- Andrei
>
> Do you remember that there is a pending PR for this at Phobos that is
> blocked because of missing feedback / acceptance?
>
> https://github.com/dlang/phobos/pull/2995

I know. What I need to write:

unittest
{
    void theFunctionToTest(Gen!(int, 1, 5) a, Gen!(float, 0.0, 10.0) b)
    {
        // This will always be true
        assert(a >= 1 && a <= 5);
        assert(a >= 0.0 && a <= 10.0);

        // super expensive operation
        auto rslt = (a + b);
        doNotOptimizeAway(rslt);

        debug
        {
            assert(rslt > 1.0);
        }
    }

    benchmark!theFunctionToTest();
}

What I want to write:

@Benchmark(ParamDomain!1(1, 5), ParamDomain!2(0, 10), Aggregate.median)
void theFunctionToTest(int a, float b)
{
    // This will always be true
    assert(a >= 1 && a <= 5);
    assert(a >= 0.0 && a <= 10.0);

    // super expensive operation
    auto rslt = (a + b);
    doNotOptimizeAway(rslt);
    debug
    {
        assert(rslt > 1.0);
    }
}

Attributes are a very attractive (if not the best) way to set up benchmarks. There are a variety of Java benchmarking frameworks we could draw inspiration from, see list at http://stackoverflow.com/questions/7146207/what-is-the-best-macro-benchmarking-tool-framework-to-measure-a-single-threade. Take a look e.g. at https://github.com/google/caliper/blob/master/examples/src/main/java/examples/BitSetBenchmark.java, which is so clear and simple.


Andrei

Lets move that back to https://github.com/dlang/phobos/pull/2995

On Wed, Aug 24, 2016 at 08:00:32AM -0400, Andrei Alexandrescu via Digitalmars-d wrote: [...]
> Some of the Phobos tests I wrote use it (and virtually all of my current code e.g. on median computation), and when they fail I just print the seed of the RNG and then hardcode it to reproduce the test. -- Andrei

Just a side note: this depends on the assumption that the PRNG is explicitly chosen, or that the default PRNG will never change; and that the implementation of the RNG is independent of environmental factors (such as different CPU's resulting in different answers).

Otherwise, if you find a bug with a specific seed and then hardcode that seed in a unittest to prevent regression, a regression may go undetected once somebody changes the default RNG, or tweaks the implementation (e.g. to fix a RNG weakness due to security concerns) yielding different results for the same seed, or is run in an environment other than the one you tested it in.


T

-- 
An elephant: A mouse built to government specifications. -- Robert Heinlein

On 08/24/2016 10:41 AM, H. S. Teoh via Digitalmars-d wrote:
> Just a side note: this depends on the assumption that the PRNG is
> explicitly chosen, or that the default PRNG will never change;

Correctamundo, it should be explicitly chosen. -- Andrei

On 08/24/2016 10:40 AM, Robert burner Schadek wrote:
> Lets move that back to https://github.com/dlang/phobos/pull/2995

D'accordo. -- Andrei

On Wednesday, 24 August 2016 at 12:00:32 UTC, Andrei Alexandrescu wrote:
> On 08/24/2016 04:14 AM, Edwin van Leeuwen wrote:
>> I might be dense, but the only other thing than integration tests that I
>> can think of is if you use random data for testing, but that would be
>> more correctly solved by using more random data during the unittests.
>> Nothing is worse than tests that only sometimes fail.
>
> Randomized unit testing is a respected approach to testing: https://hackage.haskell.org/package/QuickCheck, https://github.com/rickynils/scalacheck, https://blogs.msdn.microsoft.com/dsyme/2008/08/08/fscheck-0-2/, etc. Some of the Phobos tests I wrote use it (and virtually all of my current code e.g. on median computation), and when they fail I just print the seed of the RNG and then hardcode it to reproduce the test. -- Andrei

unit-threaded has had QuickCheck-like property-based testing features for a while now. As mentioned previously in the forum, I wrote this test in cerealed:


@Types!(bool, byte, ubyte, short, ushort, int, uint, long, ulong,
        float, double,
        char, wchar, dchar,
        ubyte[], ushort[], int[], long[], float[], double[])
void testEncodeDecodeProperty(T)() {
    check!((T val) {
        auto enc = Cerealiser();
        enc ~= val;
        auto dec = Decerealiser(enc.bytes);
        return dec.value!T == val;
    });
}


Checks that for every type in the list that given a random value of that type, serialising then deserialising should yield the same value back.


Atila

On Wednesday, 24 August 2016 at 13:52:43 UTC, Andrei Alexandrescu wrote:
>     // This will always be true
>     assert(a >= 1 && a <= 5);
>     assert(a >= 0.0 && a <= 10.0);

Pretty sure you have a bug here :)