I'm currently in the process of annotating all of phobos with "export" and its quite cumbersome. To verify that I annoted all relevant functions and types with export I would like to run the unitests against the shared version of phobos. There is a problem with this though. The unittests are always compiled into the same binary the modules are in so the unittests would end up in the phobos.dll and would not test phobos across a shared library boundary. It would also be very usefull to have this verifycation for the future when DLL support is officially in and new features need to be tested if they are propperly annotated with "export". Once "export" is used for shared library symbol visibility on non-windows as well this will be an issue on all pattforms.

The best solution would be if you could simply compile the unittests and just the unittests into a executable. This would require some compiler modifications, but even when done would cause some problems:
- Unittests may use private functions / types of a module. Private functions and types are however not exported from an shared library so this would require making a lot of private functions / types public just to be able to run the unittests.

Instead of building this into the compiler you could use a tool which extracts all "unittest { }" blocks out of all source files and puts them into new .d files. Then compile these into an executable. Now the code would not even compile because private symbols would not be accessible.

Any suggestions how to fix this issue? I'm also open for implementation hints.

Kind Regards
Benjamin Thaut

On Tuesday, 3 March 2015 at 17:49:07 UTC, Benjamin Thaut wrote:
> Any suggestions how to fix this issue? I'm also open for implementation hints.
>
> Kind Regards
> Benjamin Thaut

Running unittests that access private symbols from the other side of the dll boundary sounds like a very hard problem to solve. On one hand you want to be able to access the symbols in order to run the unittests and on the other hand they should be private to clients of the api. The question is do we even want to run unittests that test private implementation on the other side of the dll boundery at all? These unittests test functionality the users of the dll are unable to access. As such they are not actually testing if symbols that should be exported are. However unittests that test the public interface of a module do. I think that any tool that is created should only care about the unittests which are testing the public interface of the module. Since this is the interface users of the dll will interact with.

Given the above, a tool will have to be able to identify if a unittest tests a public interface of a module and only extract the unittests that do. Since these unittests only test the public interface they should be able to be compiled if the programmer has remembered to export all the public interfaces.

You still have a problem if the unittest is actually testing the public interfaces but makes use of some private helper function in the process. Since this would not be identified by the tool as a public interface-testing unittest.
This problem would not matter if unittest that test the public interface are written in another testing module. In my opinion these kinds of unittests that test the public interface should always be written in another module to ensure that they don't unintentionally make use of private symbols.

Finding out if a unittest only accesses public symbols could be done by analyzing the ast of the method. Either inside the compiler of via one of the third party D parsers currently in use.

I hope that this reply was somewhat helpful to you.

On 2015-03-03 18:49, Benjamin Thaut wrote:
> I'm currently in the process of annotating all of phobos with "export"
> and its quite cumbersome. To verify that I annoted all relevant
> functions and types with export I would like to run the unitests against
> the shared version of phobos. There is a problem with this though. The
> unittests are always compiled into the same binary the modules are in so
> the unittests would end up in the phobos.dll and would not test phobos
> across a shared library boundary. It would also be very usefull to have
> this verifycation for the future when DLL support is officially in and
> new features need to be tested if they are propperly annotated with
> "export". Once "export" is used for shared library symbol visibility on
> non-windows as well this will be an issue on all pattforms.
>
> The best solution would be if you could simply compile the unittests and
> just the unittests into a executable. This would require some compiler
> modifications, but even when done would cause some problems:
> - Unittests may use private functions / types of a module. Private
> functions and types are however not exported from an shared library so
> this would require making a lot of private functions / types public just
> to be able to run the unittests.
>
> Instead of building this into the compiler you could use a tool which
> extracts all "unittest { }" blocks out of all source files and puts them
> into new .d files. Then compile these into an executable. Now the code
> would not even compile because private symbols would not be accessible.
>
> Any suggestions how to fix this issue? I'm also open for implementation
> hints.

Sounds like you want to test an application that uses Phobos as a DLL. To me that sounds like a more higher level test than a unit test, i.e. integration tests. I would put those tests in completely separate files, in a separate testing directory. Then it should be no problem to compile the parts separately.

As for private functions. That is more a question of if you really should write specific test for those or just test the public API. If you really want to, you can bypass the protection using pointers.

BTW, I think we need to have integration tests in general. Testing how different parts/modules interact with each other.

-- 
/Jacob Carlborg

On Thursday, 5 March 2015 at 07:28:12 UTC, Jacob Carlborg wrote:
> On 2015-03-03 18:49, Benjamin Thaut wrote:
>
> Sounds like you want to test an application that uses Phobos as a DLL. To me that sounds like a more higher level test than a unit test, i.e. integration tests. I would put those tests in completely separate files, in a separate testing directory. Then it should be no problem to compile the parts separately.
>
> As for private functions. That is more a question of if you really should write specific test for those or just test the public API. If you really want to, you can bypass the protection using pointers.
>
> BTW, I think we need to have integration tests in general. Testing how different parts/modules interact with each other.

I don't want to write new tests. I want to use the tests for phobos which are already there. And I want to use them in a way, that if new tests are added to phobos they are also tested against the dll version of phobos. Yes integration tests would be nice, but the PR for Dll support is already going to be huge without me adding integration tests.

On Thursday, 5 March 2015 at 03:24:05 UTC, TheFlyingFiddle wrote:

>
> Finding out if a unittest only accesses public symbols could be done by analyzing the ast of the method. Either inside the compiler of via one of the third party D parsers currently in use.
>

I thought about writing a tool which extracts all "unittest" blocks and puts each one into a template. Then I could use __pragma(compiles, ) on the template to check if it uses any private symbol and if not instanciate that template. That way unittests that use private functions would automatically be filtered out.

Kind Regards
Benjamin Thaut

On 2015-03-05 08:38, Benjamin Thaut wrote:

> I don't want to write new tests. I want to use the tests for phobos
> which are already there. And I want to use them in a way, that if new
> tests are added to phobos they are also tested against the dll version
> of phobos. Yes integration tests would be nice, but the PR for Dll
> support is already going to be huge without me adding integration tests.

Wouldn't it require more code to write a tool that extracts the unittest blocks?

-- 
/Jacob Carlborg

On Thursday, 5 March 2015 at 07:42:10 UTC, Jacob Carlborg wrote:
> On 2015-03-05 08:38, Benjamin Thaut wrote:
>
>> I don't want to write new tests. I want to use the tests for phobos
>> which are already there. And I want to use them in a way, that if new
>> tests are added to phobos they are also tested against the dll version
>> of phobos. Yes integration tests would be nice, but the PR for Dll
>> support is already going to be huge without me adding integration tests.
>
> Wouldn't it require more code to write a tool that extracts the unittest blocks?

I've written a simple D Tokenizer in a few lines of D code already. Building on that it would be pretty easy to extract unittest blocks.