Greetings everyone,

Question

What is your approach to integration testing in D? Do you use unittest blocks? Do you write stand-alone programs that interact with a running version of your program? Is there a library that makes certain kinds of testing easier?

For example, if I have a D project based on vibe.d, and I have custom converters to receive REST API request bodies in different formats based on the "Content-Type" HTTP header and other converter for the response based on the "Accepts" header, what is the best approach to test the entire program end-to-end?

Context

Having done considerable work in Java using Spring and Spring Boot, it's very common to have integration tests, which the Spring Boot framework makes quite easy to set up with annotations like @SpringBootTest.

In a nutshell, Spring Boot is built on top of dependency injection, with a "Context" acting as a container for all the objects (called Beans) that are created and injected into that container. @SpringBootTest lets the user pick and choose which objects will be created for the test, and then when the program is run, only those objects are loaded. This allows one to do things like, start a complete web server environment, test by sending a request to some endpoint, and then validate that the response is what you would expect. This is especially helpful to validate that you are using the Spring Framework correctly, e.g. that custom converters you created that allow messages to be received in binary protobuf format instead of JSON work correctly.

On 2/4/22 7:38 AM, Vijay Nayar wrote:

I recently moved mysql-native integration tests to a subproject. I don't think unittesting should do integration tests in the library itself. But doing them via a unittesting framework in an external program seems to work well.

Links:

• common integration test project: https://github.com/mysql-d/mysql-native/tree/master/integration-tests
• vibe tests: https://github.com/mysql-d/mysql-native/tree/master/integration-tests-vibe
• phobos tests: https://github.com/mysql-d/mysql-native/tree/master/integration-tests-phobos

-Steve

On Friday, 4 February 2022 at 12:38:08 UTC, Vijay Nayar wrote:

For server nodes, I've built a library just for this:
https://github.com/Geod24/localrest

It required careful planning on the application side (e.g. we're not calling runTask, we have some classes for dependency injection), but allowed us to write "integration tests" in unittests block: https://github.com/bosagora/agora/tree/75967aac4f41272c6e5e8487c4066174825290e0/source/agora/test#agora-test-folder

On Friday, 4 February 2022 at 17:39:00 UTC, H. S. Teoh wrote:

I am still in the process of experimenting, but the advice on this thread has all been very helpful. Right now I'm experimenting by creating a separate "integration" dub configuration which uses std.process : spawnProcess to run my vibe.d-based web server and try out requests on it.

However, in any kind of large organization using a build server, even when there are multiple services/builds running in the same environment, it can be very inconvenient use a fixed port on which to run this server. Another running service, another person's build, or a test that ran and died without closing its sockets can lead to conflicts during development.

In order to solve that problem of port conflicts during integration testing, I converted a utility class from the Java Spring Framework into D and wanted to share it here: https://gist.github.com/vnayar/04c6172d9f9991062974585bb3ccc8a4

The usage is very simple, and can be used by integration tests to pick random free ports on which to run their tests. Here is one of the available methods:

/**
 * Find an available TCP port randomly selected from the range
 * \[ [PORT_RANGE_MIN], [PORT_RANGE_MAX] \].
 * Returns: an available TCP port number
 * Throws: Exception if no available port could be found
 */
ushort findAvailableTcpPort() {
  return findAvailableTcpPort(PORT_RANGE_MIN);
}

unittest {
  foreach (ushort i; 0..10) {
    ushort port = findAvailableTcpPort();
    assert(port >= PORT_RANGE_MIN && port <= PORT_RANGE_MAX);
  }
}

On Sunday, 6 February 2022 at 17:36:05 UTC, Vijay Nayar wrote:

When working on a dub configuration needed to separately run integration tests that operate on the fully built program from the outside, treating it like a black-box, I had run into a number of problems because dub implicitly created configurations called application or library only if you have no configurations of your own defined. But as soon as I added my own configuration, those default configurations were no longer being created and I suddenly found myself seeing strange errors when trying to run unit-tests or builds.

See GitHub Dub Issue #1270.

However, I did finally navigate the tricky waters and make a configuration that does actually work for dub build, dub test and dub test --config=integration.

I thought I would share what I found here:

authors "Vijay Nayar"
copyright "Copyright © 2019, Vijay Nayar"
description "Receives data from sources, converts it to protobufs, and delivers it in batches to stem."
license "proprietary"
name "mouth"
targetPath "target"

# By default, 'dub' creates either an 'application' or 'library' config if no
# configuration exists.
# Because we need a configuration for integration tests, this forces us
# to create the 'application' config as well.
#
# The 'dub test' command searches for the first 'library' target type, and
# if none exists, the first 'executable'.

# Used by 'dub build' and 'dub test'.
configuration "application" {
  targetName "mouth"
  targetType "executable"
  dependency "funnel:common" path="../"
  dependency "funnel:proto" path="../"
  dependency "nanomsg-wrapper" version="~>0.5.3"
  dependency "poodinis" version="~>8.0.3"
  dependency "vibe-d" version="~>0.9.4"
  # This must be listed so that 'dub test' knows to ignore it.
  mainSourceFile "source/app.d"
}

# Used by 'dub test --config=integration'
configuration "integration" {
  targetName "mouth_integration"
  # This must not be 'library', or it will be used by 'dub test'.
  targetType "executable"
  # The "source/" directory is automatically included, it must be explicitly
  # excluded instead.
  excludedSourceFiles "source/*"
  # The integration tests' source is in './integration'.
  sourcePaths "integration"
  importPaths "integration"
  # Make sure the executable we are testing exists.
  preBuildCommands "echo IMPORT_PATHS=$$IMPORT_PATHS"
  preRunCommands "cd $PACKAGE_DIR ; dub build"
}