Hi,

Thinking of trying to do the next project in D rather than Rust, but…

Rust has built in unit testing on a module basis. D has this so no problem.

Rust allows for integration tests in the tests directory of a project. These are automatically build and run along with all unit tests as part of "cargo test".

Does D have any integrated support for integration tests in the way
Rust does?

-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

On Friday, 17 April 2020 at 16:56:57 UTC, Russel Winder wrote:
> Hi,
>
> Thinking of trying to do the next project in D rather than Rust, but…
>
> Rust has built in unit testing on a module basis. D has this so no problem.
>
> Rust allows for integration tests in the tests directory of a project. These are automatically build and run along with all unit tests as part of "cargo test".
>
> Does D have any integrated support for integration tests in the way
> Rust does?

Automated testing is important, perhaps you describe further what's needed? I haven't worked with Rust test frameworks, but I took a look at the description of the integration tests and unit tests. It wasn't immediately obvious what can be done with the Rust integration test framework that cannot be done with D's unittest framework.

An important concept described was testing a module as an external caller. That would seem very be doable using D's unittest framework. For example, one could create a set of tests against Phobos, put them in a separate location (e.g. a separate file), and arrange to have the unittests run as part of a CI process run along with a build.

My look was very superficial, perhaps you could explain more.

On Fri, 2020-04-17 at 17:51 +0000, Jon Degenhardt via Digitalmars-d- learn wrote:
> On Friday, 17 April 2020 at 16:56:57 UTC, Russel Winder wrote:
> > Hi,
> > 
> > Thinking of trying to do the next project in D rather than Rust, but…
> > 
> > Rust has built in unit testing on a module basis. D has this so no problem.
> > 
> > Rust allows for integration tests in the tests directory of a project. These are automatically build and run along with all unit tests as part of "cargo test".
> > 
> > Does D have any integrated support for integration tests in the
> > way
> > Rust does?
> 
> Automated testing is important, perhaps you describe further what's needed? I haven't worked with Rust test frameworks, but I took a look at the description of the integration tests and unit tests. It wasn't immediately obvious what can be done with the Rust integration test framework that cannot be done with D's unittest framework.

I should point out that Rust is far behind Python's PyTest in terms of testing sophistication, and I fear D lags behind Rust.

I think the following is just saying what everyone knows, but it seems making it explicit for this thread.

Let us distinguish unit testing, integration testing, and system testing using the definitions: unit testing is self standing testing the code with no extra dependencies; integration testing is testing some functionality of the system with all external resources mocked out (possibly by using a process to simulate the external resource, which is what I want for this project); system testing is testing some functionality with real external resources.

D, like Rust, supports unit testing very nicely, well unit_threading seems essential, on a per module basis. Python unit testing with PyTest is very straightforward.

Rust supports integration testing explicitly, D does not. Python PyTest does but only because with Python you need no extra infrastructure. Integration testing must not have access to the internals of a module or group of modules, but should appear as a using application. Rust/Cargo integrates this using the tests directory as a peer to the src directory: each file in the tests directory represents a crate that uses the code under test as a crate.

Integration and automated system testing in Rust is really a question of whether external resources are mocked or used for real. Python PyTest integration and system testing is very much the same.

As far as I know, D/Dub, D/Meson, D/SCons, D/CMake, and D/Make do not provide an out of the box way of building integration tests.

Hidden agenda item: system testing really needs to be in a sandbox of some sort.

> An important concept described was testing a module as an external caller. That would seem very be doable using D's unittest framework. For example, one could create a set of tests against Phobos, put them in a separate location (e.g. a separate file), and arrange to have the unittests run as part of a CI process run along with a build.
> 
> My look was very superficial, perhaps you could explain more.

The important difference between using a unit test infrastructure and an integration/system test infrastructure is that:

– unit tests are inside the system testing functions and other code
features;
– integration tests are outside the system testing functionality with
mocked external resources.

So the build system needs to build all the codes that realise the mocks of the external resources. So for a vestigial Rust project:

.
├── Cargo.lock
├── Cargo.toml
├── README.md
├── src
│   ├── arcam_protocol.rs
│   ├── lib.rs
│   └── main.rs
└── tests
    ├── integration_tests.rs
    └── mock_avr850.rs

src/arcam_protocol/rs has all the unit tests in it. D can do the equivalent.

tests/mock_avr850.rs is an application that mocks a real amplifier; tests/integration_tests is an application that uses the project as an external crate and starts the mock amplifier for each of the tests. As far as I can tell there is no Dub, Meson, Scons, CMake, Make infrastructure for building D code that supports this sort of thing.

Python using PyTest can do all the above really very trivially, D/Cargo has yet to catch up!


-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

On Friday, 17 April 2020 at 16:56:57 UTC, Russel Winder wrote:
> Hi,
>
> Thinking of trying to do the next project in D rather than Rust, but…
>
> Rust has built in unit testing on a module basis. D has this so no problem.
>
> Rust allows for integration tests in the tests directory of a project. These are automatically build and run along with all unit tests as part of "cargo test".
>
> Does D have any integrated support for integration tests in the way
> Rust does?

D does not recognise any special testing directories afaik. But you'll get the same affect as in rust if you have your integration tests in a file that is not the modules you are testing and run it after building the main library - or during even.

With Dub it's quite easy to set up, you can see the optional package as an example [0]. There's a tests directory that does not have access to the optional package's internals. There's also a unittest-compat configuration in dub.json that tests the package alongside vibed (another d package)

For auto mocking, the only thing I've see that provides some functionality is the unit-threaded package. But I've never tried it.

[0]: https://github.com/aliak00/optional

On Mon, 2020-04-20 at 20:19 +0000, aliak via Digitalmars-d-learn wrote:
> 
> […]

> [0]: https://github.com/aliak00/optional

Rust has Option and Result, and most languages are rapidly introducing at least Option if not Result – and yes it is almost certain all this comes from Haskell.

Is Option intended for adding to Phobos?

-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

On Tuesday, 21 April 2020 at 16:30:15 UTC, Russel Winder wrote:
> On Mon, 2020-04-20 at 20:19 +0000, aliak via Digitalmars-d-learn wrote:
>> 
>> […]
>
>> [0]: https://github.com/aliak00/optional
>
> Rust has Option and Result, and most languages are rapidly introducing at least Option if not Result – and yes it is almost certain all this comes from Haskell.
>
> Is Option intended for adding to Phobos?

There is Nullable in Phobos. This is not really an Option, I know. But just in case.

But back to integration tests, there are indeed only few possibilities to mock something. However, the basics are there.
I found a section of code generation a while ago in std.typecons.
https://dlang.org/library/std/typecons.html

There are Black and White holes there, for example, which could help to implement a basic mocking framework.

On Tuesday, 21 April 2020 at 16:30:15 UTC, Russel Winder wrote:
> On Mon, 2020-04-20 at 20:19 +0000, aliak via Digitalmars-d-learn wrote:
>> 
>> […]
>
>> [0]: https://github.com/aliak00/optional
>
> Rust has Option and Result, and most languages are rapidly introducing at least Option if not Result – and yes it is almost certain all this comes from Haskell.

Yeah it's a great abstraction that's part of modern languages now. Some static languages are even building syntax for it right in (e.g. swift, kotlin, v, zig - to name a few).

There've been a few attempts at building a Result type:

https://code.dlang.org/search?q=expect
And here: https://github.com/aliak00/ddash/blob/master/utils/source/ddash/utils/expect.d

>
> Is Option intended for adding to Phobos?

Not that I am aware of. There was an attempt to PR an Option type way back when which never made it: https://github.com/dlang/phobos/pull/3915

There was a post here: https://forum.dlang.org/thread/hqtdekjtdgbhhbjgyvvo@forum.dlang.org

I ended up creating the following project structure:

.
├── dub.sdl
├── dub.selections.json
├── source
│   ├── arcam_protocol.d
│   └── main.d
├── tests
│   └── integration_tests.d
└── test_support
    └── mock_avr850
        └── main.d

with the following Dub control file:

name "arcamclient"
description "arcamclient is a Rust/gtk-rs/GTK+ desktop application to control an Arcam amplifier over the Ethernet connection."
authors "Russel Winder"
copyright "Copyright © 2020 Russel Winder."
license "GPL-3.0"
targetType "executable"
targetPath "bin"

configuration "application" {
}

configuration "unittest" {
    targetName "arcamclient_test"
    dependency "unit-threaded" version="*"
    mainSourceFile "bin/ut.d"
    excludedSourceFiles "source/main.d"
    preBuildCommands "$DUB run --compiler=$$DC unit-threaded -c gen_ut_main -- -f bin/ut.d -d $DUB"
    preBuildCommands "$DUB build arcamclient:mock_avr850"
    importPaths "tests"
    sourcePaths "tests"
}

subPackage {
    name "mock_avr850"
    targetName "mock_avr850"
    targetType "executable"
    targetPath "bin"
    sourcePaths "source" "test_support/mock_avr850"
    importPaths "source" "test_support/mock_avr850"
    excludedSourceFiles "source/main.d"
}

This seems a bit more sensible that what I have been able to achieve with Rust, but is still second rate compared to how easy things are using Python.

Now I discover Python, Rust, and Go have far nicer abstractions for writing Internet code than D does. Does D really not have a TcpListener abstraction?

To date all I can get is:

std.socket.SocketOSException@std/socket.d(2792): Unable to bind socket: Bad file descriptor

when trying to open a TCP server on 127.0.0.1:50000, with Python, Rust, or Go it all worked first time. This is really sad given D has so many advantages over Rust. :-(

-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

On Wednesday, 22 April 2020 at 10:32:48 UTC, Russel Winder wrote:
>
> Now I discover Python, Rust, and Go have far nicer abstractions for writing Internet code than D does. Does D really not have a TcpListener abstraction?

It really doesn't :(

And D has so much potential as server tech with the potential combination of fibers + TLS + shared + static introspection.

The package Vibe-d is quite nice though. I don't know if you've tried it but it's very simple to get a listener up with it.

>
> To date all I can get is:
>
> std.socket.SocketOSException@std/socket.d(2792): Unable to bind socket: Bad file descriptor
>
> when trying to open a TCP server on 127.0.0.1:50000, with Python, Rust, or Go it all worked first time. This is really sad given D has so many advantages over Rust. :-(

On Tue, 2020-04-21 at 21:29 +0000, aliak via Digitalmars-d-learn wrote:
> On Tuesday, 21 April 2020 at 16:30:15 UTC, Russel Winder wrote:
[…]
> 
> There've been a few attempts at building a Result type:
> 
> https://code.dlang.org/search?q=expect
> And here:
> https://github.com/aliak00/ddash/blob/master/utils/source/ddash/utils/expect.d
> 
> > Is Option intended for adding to Phobos?
> 
> Not that I am aware of. There was an attempt to PR an Option type way back when which never made it: https://github.com/dlang/phobos/pull/3915
> 
> There was a post here: https://forum.dlang.org/thread/hqtdekjtdgbhhbjgyvvo@forum.dlang.org

Perhaps there can be one implementation package of Nullable, Option,
and Result that is in the Dub repository that everyone can use even though they really ought to be part of Phobos.

-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk