How do I call std_stdio_static_this() in std.stdio from a dynamically loaded shared library (osx) ?

I need to do this in my scenario:
1) main program is launched
2) dynamic library is created
3) dynamic library is loaded and a function from it is called
4) everything works fine until a writeln(0) is called from a function inside the dynamic lib, at which point it crashes. But by calling std_stdio_static_this2 (see below) BEFORE any call to writeln (and similar functions), everything works fine.

When I call std_stdio_static_this from another module, I'm pretty sure that code doesn't get called, because when I add a printf or assert(0) statement inside std_stdio_static_this, nothing happens.

My current (working!) workaround is shown below, but I was hoping for a solution that didn't involve modifying std.stdio.

----
void std_stdio_static_this2(T=void)()
{
    //copy contents of std_stdio_static_this() function here:
    //putting assert(0); here WILL crash, as expected; but it won't if the signature is std_stdio_static_this2().
    //Bind stdin, stdout, stderr
    __gshared File.Impl stdinImpl;
    stdinImpl.handle = core.stdc.stdio.stdin;
    .stdin.p = &stdinImpl;
    // stdout
    __gshared File.Impl stdoutImpl;
    stdoutImpl.handle = core.stdc.stdio.stdout;
    .stdout.p = &stdoutImpl;
    // stderr
    __gshared File.Impl stderrImpl;
    stderrImpl.handle = core.stdc.stdio.stderr;
    .stderr.p = &stderrImpl;
}
----

On 2012-10-03 02:55, timotheecour wrote:
> How do I call std_stdio_static_this() in std.stdio from a dynamically
> loaded shared library (osx) ?
>
> I need to do this in my scenario:
> 1) main program is launched
> 2) dynamic library is created
> 3) dynamic library is loaded and a function from it is called
> 4) everything works fine until a writeln(0) is called from a function
> inside the dynamic lib, at which point it crashes. But by calling
> std_stdio_static_this2 (see below) BEFORE any call to writeln (and
> similar functions), everything works fine.

Dynamic libraries are not properly working, see this thread:

http://forum.dlang.org/thread/k3vfm9$1tq$1@digitalmars.com

And this post:

http://forum.dlang.org/thread/k3vfm9$1tq$1@digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com

-- 
/Jacob Carlborg

Thanks for the links. Ya, I did see those threads and I understand there are limitations with dynamic load shared libraries, however, your answer is not very helpful. We shouldn't have to wait until they get 100% support to start using them (who knows when that happens); some applications just require them (eg writing matlab mex files).

In the meantime, the workaround I described above seems to adequately address the IO problem in shared libraries, it's just that it requires hacking inside std.stdio. So I'm just wondering whether I should:
* pull those changes back in phobos
* or if there's an alternative way of doing this

Thanks!

On 2012-10-03 08:56, timotheecour wrote:
> Thanks for the links. Ya, I did see those threads and I understand there
> are limitations with dynamic load shared libraries, however, your answer
> is not very helpful. We shouldn't have to wait until they get 100%
> support to start using them (who knows when that happens); some
> applications just require them (eg writing matlab mex files).

Yeah, it may not have been so helpful, sorry.

> In the meantime, the workaround I described above seems to adequately
> address the IO problem in shared libraries, it's just that it requires
> hacking inside std.stdio. So I'm just wondering whether I should:
> * pull those changes back in phobos
> * or if there's an alternative way of doing this

Module constructors if one of the things that don't work properly. I don't know exactly how this works in std.stdio I would need to study the code. If I recall correctly std.stdio uses some kind of hack to avoid circular dependencies that can otherwise occur if two modules, both having module constructors, import each other.

-- 
/Jacob Carlborg

> Module constructors if one of the things that don't work properly. I don't know exactly how this works in std.stdio I would need to study the code. If I recall correctly std.stdio uses some kind of hack to avoid circular dependencies that can otherwise occur if two modules, both having module constructors, import each other.

Yes, that hack consists in :
1) defining extern(C) void std_stdio_static_this() instead of static this()
2) defining an auxiliary module std.stdiobase who will call it at module construction:
module std.stdiobase;
extern(C) void std_stdio_static_this();
shared static this(){
    std_stdio_static_this();
}

I'm still puzzled as to why modifying the code inside std_stdio_static_this (say with assert(0)) doesn't have any effect (hence the need for the templated version I'm using).
Another workaround (besides adding std_stdio_static_this2) would be to make the field File.p non-private so that my std_stdio_static_this2 could be defined outside std.stdio. Any suggestion?

On 2012-10-03 20:23, timotheecour wrote:

> Yes, that hack consists in :
> 1) defining extern(C) void std_stdio_static_this() instead of static this()
> 2) defining an auxiliary module std.stdiobase who will call it at module
> construction:
> module std.stdiobase;
> extern(C) void std_stdio_static_this();
> shared static this(){
>      std_stdio_static_this();
> }
>
> I'm still puzzled as to why modifying the code inside
> std_stdio_static_this (say with assert(0)) doesn't have any effect
> (hence the need for the templated version I'm using).
> Another workaround (besides adding std_stdio_static_this2) would be to
> make the field File.p non-private so that my std_stdio_static_this2
> could be defined outside std.stdio. Any suggestion?

What happens if you just call "std_stdio_static_this" the first you do in your main function?

Usually these kinds of problems are solved by using lazy initialization. In this case that would mean converting "stdin", "stdout" and "stderr" to functions.

https://github.com/D-Programming-Language/phobos/blob/master/std/stdio.d#L2357

Something like this:

private __gshared
{
    File _stdin;
    File _stdout;
    File _stderr;

    File.Impl stdinImpl;
    File.Impl stdoutImpl;
    File.Impl stderrImpl;
}

@property File stdin ()
{
    if (!_stdin.p)
        _stdin.p = &stdinImpl;

    return _stdin;
}

The same for "stdout" and "stderr". But this may not be good for performance reasons. I don't know how good the compiler is at optimizing these kind of functions.

-- 
/Jacob Carlborg

> What happens if you just call "std_stdio_static_this" the first you do in your main function?

In my case I can't: I don't have control over the main function (it could be written in C for example).



> @property File stdin ()
> {
>     if (!_stdin.p)
>         _stdin.p = &stdinImpl;
>     return _stdin;
> }

I guess you also want to add inside the if(): "stdinImpl.handle = core.stdc.stdio.stdin;", etc.

> The same for "stdout" and "stderr". But this may not be good for performance reasons. I don't know how good the compiler is at optimizing these kind of functions.

I doubt the compiler can do much optimizing here, since the if won't always be true, but probably IO is slow enough compared to a branch so it might not matter.

That being said, templatizing std_stdio_static_this as above would avoid branching at every single call to writeXX functions. Would there be any disadvantage?

Also, I grepped for static_this, it appears this hack is only used in 2 places:
* std.stdio (=>stdiobase)
* std.process (=>processinit)
So the same template trick could be used in those 2 places.

I moved the discussion to http://forum.dlang.org/thread/iegurltmszdndxqnoyxw@forum.dlang.org#post-iegurltmszdndxqnoyxw:40forum.dlang.org
since the problem appears more general, ie how to redefine an extern(C) function in a dynamic load library

On 2012-10-03 23:51, timotheecour wrote:

> In my case I can't: I don't have control over the main function (it
> could be written in C for example).

If you're developing a library or similar you could have a requirement that the user need to call "Library.initialize()" before use. But of course, that's something anyone would like to avoid.

> I guess you also want to add inside the if(): "stdinImpl.handle =
> core.stdc.stdio.stdin;", etc.

Right, I missed that.

> I doubt the compiler can do much optimizing here, since the if won't
> always be true, but probably IO is slow enough compared to a branch so
> it might not matter.
>
> That being said, templatizing std_stdio_static_this as above would avoid
> branching at every single call to writeXX functions. Would there be any
> disadvantage?

I have no idea. I would guess as long as "std_stdio_static_this" or the corresponding code is run before std.stdio is used you should be fine. You have to be careful here about threads, you don't want to have a thread you don't know about sneaking up behind your back and starts using std.stdio before it's initialized.

> Also, I grepped for static_this, it appears this hack is only used in 2
> places:
> * std.stdio (=>stdiobase)
> * std.process (=>processinit)
> So the same template trick could be used in those 2 places.

You can write this instead:

void std_stdio_static_this2 () ()

Note the extra pair of empty parentheses. This will force the function to be a template. I think this look slightly better.

-- 
/Jacob Carlborg

> If you're developing a library or similar you could have a requirement that the user need to call "Library.initialize()" before use. But of course, that's something anyone would like to avoid.

I'm fine with that (and already doing this, to call Runtime.initialize), but that doesn't solve the issue of this thread.

> You can write this instead:
> void std_stdio_static_this2 () ()

Thanks! I didn't know that.