I'm currently working on Windows DLL support which has stronger rules than linux shared objects for which symbols actually get exported from a shared library. But as we want to replicate the same behaviour on linux using symbol visibility (e.g. gcc 4 -fVisibility=hidden) this issue also applies to linux once implemented.

Currently export means two things:
- the symbol is publicy accessible (same as public)
- the symbol will be accisble across shared library boundaries


This has the following issue:

export void templateFunc(T)()
{
  someHelperFunc();
}

private void someHelperFunc()
{

}

And you don't even have to go into phobos to hit this problem. It is already in druntime see core.time.FracSec._enforceValid

This works with the current linux shared objects because they simply export all symbols. But once only symbols with export get exported this breaks.

The problem here is that you don't want to make someHelperFunc() export because that would mean users could call it directly, but you want it to be available for cross shared library calls. The cross shared library call happens if a template is instanced from a different shared library / executable than the module it was originally located in.

There are two solutions for this.

1) Given that export is transitive (that means if a struct or class is declared export every member that is _not_ private will be accessible across shared library boundaries. This behaviour is required to make the export protection level work on windows)

You can now do the following:

export struct SomeStruct
{
  static public void templateFunc(T)()
  {
    someHelperFunc();
  }

  static package void someHelperFunc()
  {

  }
}

Because of the transitivity someHelperFunc will be exported but still not be callable by the user directly. This can be used to fix the issue in core.time but may not be so well suited if you want the template to be on module level instead of nesting it into a struct.

2) Make export an attribute. If export is no longer an protection level but instead an attribute this issue can easily be solved by doing.

export public void templateFunc(T)()
{
  someHelperFunc();
}

export private void someHelperFunc()
{

}

But this would require grammar changes. Which are generally avoided if possible.

There would be a third option, which I rather avoid. Doing a "pramga(forceExport)" or something like that.

My implementation, which I ran into this issue with, currently usses approach 1. What do you think how this sould be solved?

Walter: What was the general idea behind export when you designed it, and how can it be used to solve this problem?

Kind Regards
Benjamin Thaut

I like the first version more as it fits better the natural way people design their interfaces and reduces attributed noise. Can't say if it will trigger some hidden issues.

First version also fits better some of ideas I had about automated API generation/verification (http://forum.dlang.org/post/otejdbgnhmyvbyaxatsk@forum.dlang.org)

Thanks for keeping to poke this issue - symbol visibility is currently a big undefined minefield in D ABI.

On Tuesday, 20 January 2015 at 12:23:32 UTC, Benjamin Thaut wrote:
> I'm currently working on Windows DLL support which has stronger rules than linux shared objects for which symbols actually get exported from a shared library. But as we want to replicate the same behaviour on linux using symbol visibility (e.g. gcc 4 -fVisibility=hidden) this issue also applies to linux once implemented.
>
> Currently export means two things:
> - the symbol is publicy accessible (same as public)
> - the symbol will be accisble across shared library boundaries
>
>
> This has the following issue:
>
> export void templateFunc(T)()
> {
>   someHelperFunc();
> }
>
> private void someHelperFunc()
> {
>
> }
>
> And you don't even have to go into phobos to hit this problem. It is already in druntime see core.time.FracSec._enforceValid
>
> This works with the current linux shared objects because they simply export all symbols. But once only symbols with export get exported this breaks.
>
> The problem here is that you don't want to make someHelperFunc() export because that would mean users could call it directly, but you want it to be available for cross shared library calls. The cross shared library call happens if a template is instanced from a different shared library / executable than the module it was originally located in.
>
> There are two solutions for this.
>
> 1) Given that export is transitive (that means if a struct or class is declared export every member that is _not_ private will be accessible across shared library boundaries. This behaviour is required to make the export protection level work on windows)
>
> You can now do the following:
>
> export struct SomeStruct
> {
>   static public void templateFunc(T)()
>   {
>     someHelperFunc();
>   }
>
>   static package void someHelperFunc()
>   {
>
>   }
> }
>
> Because of the transitivity someHelperFunc will be exported but still not be callable by the user directly. This can be used to fix the issue in core.time but may not be so well suited if you want the template to be on module level instead of nesting it into a struct.
>
> 2) Make export an attribute. If export is no longer an protection level but instead an attribute this issue can easily be solved by doing.
>
> export public void templateFunc(T)()
> {
>   someHelperFunc();
> }
>
> export private void someHelperFunc()
> {
>
> }
>
> But this would require grammar changes. Which are generally avoided if possible.
>
> There would be a third option, which I rather avoid. Doing a "pramga(forceExport)" or something like that.
>
> My implementation, which I ran into this issue with, currently usses approach 1. What do you think how this sould be solved?
>
> Walter: What was the general idea behind export when you designed it, and how can it be used to solve this problem?
>
> Kind Regards
> Benjamin Thaut

Just as heads up in case D ever comes to Aix, I don't know how it looks like nowadays, but Aix back in 2000 used to be have similar behavior to Windows.

The .def files in Windows were .exp (I think) files on Aix.

--
Paulo

> Thanks for keeping to poke this issue - symbol visibility is currently a big undefined minefield in D ABI.

Your welcome. At this point I'm so desperate for D Dll support that I stopped poking and started implementing it myself. I'm 3 unresolved symbol references away from actually building phobos into a dll (druntime already is).

By the way you could also export non accessible symbols like this:

static public void templateFunc(T)()
{
  Impl.someHelperFunc();
}

export struct Impl
{
  package:
  static void someHelperFunc()
  {

  }
}

There are uses in Phobos where workaround 1) would require some code changes:

private @property File trustedStdout() @trusted { return stdout; }

void write(T...)(T args) if (!is(T[0] : File))
{
    trustedStdout.write(args);
}

My workaround so far:

export struct _impl1
{
  package @property static File trustedStdout() @trusted { return stdout; }
}

alias trustedStdout = _impl1.trustedStdout;

On 1/20/2015 4:23 AM, Benjamin Thaut wrote:
> I'm currently working on Windows DLL support which has stronger rules than linux
> shared objects for which symbols actually get exported from a shared library.
> But as we want to replicate the same behaviour on linux using symbol visibility
> (e.g. gcc 4 -fVisibility=hidden) this issue also applies to linux once implemented.
>
> Currently export means two things:
> - the symbol is publicy accessible (same as public)
> - the symbol will be accisble across shared library boundaries
>
>
> This has the following issue:
>
> export void templateFunc(T)()
> {
>    someHelperFunc();
> }
>
> private void someHelperFunc()
> {
>
> }
>
> And you don't even have to go into phobos to hit this problem. It is already in
> druntime see core.time.FracSec._enforceValid
>
> This works with the current linux shared objects because they simply export all
> symbols. But once only symbols with export get exported this breaks.
>
> The problem here is that you don't want to make someHelperFunc() export because
> that would mean users could call it directly, but you want it to be available
> for cross shared library calls. The cross shared library call happens if a
> template is instanced from a different shared library / executable than the
> module it was originally located in.

exporting a template and then having the user instantiate outside of the library doesn't make a whole lot of sense, because the instantiation won't be there in the library. The library will have to instantiate every use case. If the compiler knows the library instantiated it, it won't re-instantiate it locally.


> There are two solutions for this.
>
> 1) Given that export is transitive (that means if a struct or class is declared
> export every member that is _not_ private will be accessible across shared
> library boundaries. This behaviour is required to make the export protection
> level work on windows)
>
> You can now do the following:
>
> export struct SomeStruct
> {
>    static public void templateFunc(T)()
>    {
>      someHelperFunc();
>    }
>
>    static package void someHelperFunc()
>    {
>
>    }
> }
>
> Because of the transitivity someHelperFunc will be exported but still not be
> callable by the user directly. This can be used to fix the issue in core.time
> but may not be so well suited if you want the template to be on module level
> instead of nesting it into a struct.
>
> 2) Make export an attribute. If export is no longer an protection level but
> instead an attribute this issue can easily be solved by doing.
>
> export public void templateFunc(T)()
> {
>    someHelperFunc();
> }
>
> export private void someHelperFunc()
> {
>
> }
>
> But this would require grammar changes. Which are generally avoided if possible.
>
> There would be a third option, which I rather avoid. Doing a
> "pramga(forceExport)" or something like that.
>
> My implementation, which I ran into this issue with, currently usses approach 1.
> What do you think how this sould be solved?

I'd be thinking that what a shared library exports is fixed, and expecting the user to make more instantiations makes for a fairly unresolvable issue. The solution is design the templates to be expanded only on one side of the dll/exe boundary, not straddle it.

Am 26.01.2015 um 23:24 schrieb Walter Bright:
>
> exporting a template and then having the user instantiate outside of the
> library doesn't make a whole lot of sense, because the instantiation
> won't be there in the library. The library will have to instantiate
> every use case. If the compiler knows the library instantiated it, it
> won't re-instantiate it locally.

Sorry, but wtf? So we just throw all of phobos away? Given this argument you can _never_ make phobos into a shared lirary becuse you can't possibliy pre-instaniate all possible template permutations in phobos. Your suggestion is completely un-pratical. Even C++ allows you to instanciate templates provided by a shared library.

>
> I'd be thinking that what a shared library exports is fixed, and
> expecting the user to make more instantiations makes for a fairly
> unresolvable issue. The solution is design the templates to be expanded
> only on one side of the dll/exe boundary, not straddle it.
>

Again this just makes it impossible to make phobos ever work as a shared library. A Language as template heavy as D should clearly allow users to instanciate templates across dll/exe boundaries.

Why am I getting the impression, everytime I read one of your comments regarding dlls on windows, that you don't want them to be useable? DO you have a special hate against Windows or their shared library solution?

Kind Regards
Benjamin Thaut

On 26.01.2015 23:24, Walter Bright wrote:
>> The problem here is that you don't want to make someHelperFunc()
>> export because that would mean users could call it directly, but
>> you want it to be available for cross shared library calls. The
>> cross shared library call happens if a template is instanced from a
>> different shared library / executable than the module it was
>> originally located in.
>
> exporting a template and then having the user instantiate outside of
> the library doesn't make a whole lot of sense, because the
> instantiation won't be there in the library. The library will have to
> instantiate every use case. If the compiler knows the library
> instantiated it, it won't re-instantiate it locally.

The problem is not about into which binary the template is generated to (this must be the binary where it is used), but how to access private non-templated methods called by the template.

From the core.time.FracSec example:

export struct FracSec
{
    ///...
    static FracSec from(string units)(long value)
        if(units == "msecs" ||
           units == "usecs" ||
           units == "hnsecs" ||
           units == "nsecs")
    {
        immutable hnsecs = cast(int)convert!(units, "hnsecs")(value);
        _enforceValid(hnsecs);
        return FracSec(hnsecs);
    }

    private static void _enforceValid(int hnsecs)
    {
        if(!_valid(hnsecs))
            throw new TimeException("FracSec must ...");
    }
    ///...
}

_enforceValid() could also be a free function. It is likely to be compiled into druntime.dll, but needs to be exported from the DLL to be callable by the instantiation of the template function in another DLL. The "private" forbids exporting, though.

On 20.01.2015 13:23, Benjamin Thaut wrote:
> I'm currently working on Windows DLL support which has stronger rules
> than linux shared objects for which symbols actually get exported from a
> shared library. But as we want to replicate the same behaviour on linux
> using symbol visibility (e.g. gcc 4 -fVisibility=hidden) this issue also
> applies to linux once implemented.
>
> Currently export means two things:
> - the symbol is publicy accessible (same as public)
> - the symbol will be accisble across shared library boundaries
>
>
> This has the following issue:
>
> export void templateFunc(T)()
> {
>    someHelperFunc();
> }
>
> private void someHelperFunc()
> {
>
> }
>
> And you don't even have to go into phobos to hit this problem. It is
> already in druntime see core.time.FracSec._enforceValid
>
> This works with the current linux shared objects because they simply
> export all symbols. But once only symbols with export get exported this
> breaks.

I would not mind if we export all symbols on Windows aswell. It doesn't seem to bother a lot of people for the linux version, even though it's unsafer and slower than on Windows (at least that was my experience more than 10 years ago). It might get us a first working version without adding "export" throughout the druntime/phobos source code.

>
> The problem here is that you don't want to make someHelperFunc() export
> because that would mean users could call it directly, but you want it to
> be available for cross shared library calls. The cross shared library
> call happens if a template is instanced from a different shared library
> / executable than the module it was originally located in.
>
> There are two solutions for this.
>
> 1) Given that export is transitive (that means if a struct or class is
> declared export every member that is _not_ private will be accessible
> across shared library boundaries. This behaviour is required to make the
> export protection level work on windows)
>
> You can now do the following:
>
> export struct SomeStruct
> {
>    static public void templateFunc(T)()
>    {
>      someHelperFunc();
>    }
>
>    static package void someHelperFunc()
>    {
>
>    }
> }
>
> Because of the transitivity someHelperFunc will be exported but still
> not be callable by the user directly. This can be used to fix the issue
> in core.time but may not be so well suited if you want the template to
> be on module level instead of nesting it into a struct.
>
> 2) Make export an attribute. If export is no longer an protection level
> but instead an attribute this issue can easily be solved by doing.
>
> export public void templateFunc(T)()
> {
>    someHelperFunc();
> }
>
> export private void someHelperFunc()
> {
>
> }
>
> But this would require grammar changes. Which are generally avoided if
> possible.

I don't have a clear favorite, but the second version makes it clearer that visibility and protection are separate issues.

A note on:
> export public void templateFunc(T)()

I don't think it is well defined what exporting a template is supposed to mean. My guess: whenever an instance of the template is created, its symbols are exported. This could make for a lot of duplicate symbols across multiple DLLs, though. Maybe there should be a method of explicitly exporting/importing a template instance from another DLL, e.g.

export alias symbol = templateFunc!int;

Please note, that the problem raised above by Benjamin applies just as well to non-exported templates.

On Tuesday, 27 January 2015 at 22:29:41 UTC, Rainer Schuetze wrote:
> I would not mind if we export all symbols on Windows aswell. It doesn't seem to bother a lot of people for the linux version, even though it's unsafer and slower than on Windows (at least that was my experience more than 10 years ago). It might get us a first working version without adding "export" throughout the druntime/phobos source code.

There are multiple reasons why I don't want to simply export every symbol:

1) Before I started this implementation I synchronized with Martin Nowak regrading hish plans for D shared libraries. It turns out that he wants to annotate all of druntime and phobos with export asap and use it to control symbol visibility on linux. He wants to get away from the "export everything" on linux because it hurts performance and prevents some optimizations.
2) Every data symbol that is considered for exporting adds a slight performance overhead through a additional indirection, even for static builds. That is because the compiler can't know if the symbol is imported or not, as export means both import and export at the same time. So if the compiler simply assumes that all symbols are exported this would add a lot of unnecessary overhead even in static builds. (but also in dynamic ones)
3) If we start with export everything on Windows now, it will be hared to go back to export only whats annotated.

>
> I don't have a clear favorite, but the second version makes it clearer that visibility and protection are separate issues.
>
> A note on:
> > export public void templateFunc(T)()
>
> I don't think it is well defined what exporting a template is supposed to mean. My guess: whenever an instance of the template is created, its symbols are exported. This could make for a lot of duplicate symbols across multiple DLLs, though.

Obviously its not yet well defined. But we can define it. And you are right, it means that all instances are exported. And we need that behavior because otherwise you have to spray in -allinst everywhere. Believe me I tried.

> Maybe there should be a method of explicitly exporting/importing a template instance from another DLL, e.g.
>
> export alias symbol = templateFunc!int;

I would rather not do that. You don't have to explicitly import template instances from static libraries either. We should try to keep the behavior of static and dynamic libraries as similar as possible. The ideal situation would be that you can simply compile something that was a static library into a dynamic one without doing any code changes (other then writing export: at the beginning of every file)