I am struggeling to get around the cycle detection kicking in when I have static init in modules that depend on eachother.

I have seen some threads on 'fixes' for this, e.g. adding a @standalone property to the module or similar. Has there been any progress on this?

If not would it be possible in e.g. main() to get a list of all compiled-in modules, and then iterate over them and call an init function where it exists? As long as there is a way to list the name of the modules at compile-time, this should be pretty easy..?

-Øivind

On Wed, 19 Sep 2012 22:25:46 +0200, Øivind <oivind.loe@gmail.com> wrote:

> I am struggeling to get around the cycle detection kicking in when I have static init in modules that depend on eachother.
>
> I have seen some threads on 'fixes' for this, e.g. adding a @standalone property to the module or similar. Has there been any progress on this?
>
> If not would it be possible in e.g. main() to get a list of all compiled-in modules, and then iterate over them and call an init function where it exists? As long as there is a way to list the name of the modules at compile-time, this should be pretty easy..?

There's no way to get that list at compile-time, because object files may
be added at link-time. However, D has a ModuleInfo object, which contains
information on all modules in the program:

import std.stdio;
void main( ) {
    foreach( m; ModuleInfo ) {
        writeln( m.name );
    }
}

For details on how this object works, have a look-see at
src/druntime/src/object_.d in your DMD installation folder.

I'm not sure what you're asking for is possible even given this object,
but it's probably the closest you'll (easily) get.

-- 
Simen

On Wednesday, 19 September 2012 at 20:56:17 UTC, Simen Kjaeraas wrote:
> On Wed, 19 Sep 2012 22:25:46 +0200, Øivind <oivind.loe@gmail.com> wrote:
>
>> I am struggeling to get around the cycle detection kicking in when I have static init in modules that depend on eachother.
>>
>> I have seen some threads on 'fixes' for this, e.g. adding a @standalone property to the module or similar. Has there been any progress on this?
>>
>> If not would it be possible in e.g. main() to get a list of all compiled-in modules, and then iterate over them and call an init function where it exists? As long as there is a way to list the name of the modules at compile-time, this should be pretty easy..?
>
> There's no way to get that list at compile-time, because object files may
> be added at link-time. However, D has a ModuleInfo object, which contains
> information on all modules in the program:
>
> import std.stdio;
> void main( ) {
>     foreach( m; ModuleInfo ) {
>         writeln( m.name );
>     }
> }
>
> For details on how this object works, have a look-see at
> src/druntime/src/object_.d in your DMD installation folder.
>
> I'm not sure what you're asking for is possible even given this object,
> but it's probably the closest you'll (easily) get.

If I define a function:

void getMembers() {

}

in one of my modules, all modules including this get the xgetMembers property of the ModuleInfo struct set. It seems like I should be able to do something with this, but i then get the following assertion failure:

./dboss(_d_assertm+0x26) [0x8aa1da]
./dboss() [0x7c8555]
./dboss(boss.core.proc.ProcBase!(boss.core.boss.Boss, "boss").ProcBase boss.core.proc.ProcBase!(boss.core.boss.Boss, "boss").ProcBase.__ctor()+0x31) [0x797805]
./dboss(boss.core.boss.Boss boss.core.boss.Boss.__ctor(immutable(char)[][])+0x21) [0x791099]
./dboss(_Dmain+0x33) [0x7488fb]
./dboss(extern (C) int rt.dmain2.main(int, char**).void runMain()+0x1c) [0x8aaa70]
./dboss(extern (C) int rt.dmain2.main(int, char**).void tryExec(scope void delegate())+0x2a) [0x8aa3ea]
./dboss(extern (C) int rt.dmain2.main(int, char**).void runAll()+0x3b) [0x8aaab7]
./dboss(extern (C) int rt.dmain2.main(int, char**).void tryExec(scope void delegate())+0x2a) [0x8aa3ea]
./dboss(main+0xd1) [0x8aa375]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f1d0edcc76d]

Removing the function from the module makes the assertion failure go away.. What is the cause of this? And what is the getMembers function used for?


Another way of approaching this would be if I could feed a list of modules into DMD during compile time. In C++, i would be able to do this by passing a define to g++ on the command line when invoking it. Is it possible to do something similar with DMD? E.g. create a list of modules before invoking DMD, pass it to DMD, and use this at compile time to import and call my init function?

> Another way of approaching this would be if I could feed a list of modules into DMD during compile time. In C++, i would be able to do this by passing a define to g++ on the command line when invoking it. Is it possible to do something similar with DMD? E.g. create a list of modules before invoking DMD, pass it to DMD, and use this at compile time to import and call my init function?

I can of course just generate a separate init.d file containing all of this info and then use that for static init of my modules. Think that should work :)

On Wednesday, September 19, 2012 22:25:46 Øivind wrote:
> I am struggeling to get around the cycle detection kicking in when I have static init in modules that depend on eachother.
> 
> I have seen some threads on 'fixes' for this, e.g. adding a @standalone property to the module or similar. Has there been any progress on this?

The solution is to do something like what std.stdio does. Instead of having a static constructor, it has an extern(C) function which does the same, and then it has std/stdiobase.d which has a static constructor which calls that extern(C) function in std.stdio. As long as you don't actually have any inter- dependencies, you can make it work.

Walter has refused suggestions which involve telling the compiler that there isn't actually dependency (via an attribute or whatnot) on the grounds that you're throwing away the compiler's checks, and it's too easy to screw up. In general, you can refactor things so that you don't have the circular dependency anymore, and if you can't, and the static constructors aren't actually forming a circular dependency, then you can pull the same trick that std.stdio does.

> If not would it be possible in e.g. main() to get a list of all compiled-in modules, and then iterate over them and call an init function where it exists? As long as there is a way to list the name of the modules at compile-time, this should be pretty easy..?

It will never be possible to determine the full list of modules at compile time due to C's linking model. For instance, you could link together modules which were built years apart. There's no way that they could know about each other. And since you could be using .di files, and the circular dependencies could actually be really indirect, the compiler can't possibly have all of the information that it needs to determine circular dependencies. A linker could do it but not as long as we use standard, C linkers. And that's not going to change.

So, the situation sucks, but there _are_ workarounds. The main one of course is to simply limit how much your modules import each other so that the few places that you need static constructors don't import each other even indirectly, but regardless, it can be worked around as long there isn't a true circular dependency.

- Jonathan M Davis

On Thursday, 20 September 2012 at 00:23:33 UTC, Jonathan M Davis wrote:
> On Wednesday, September 19, 2012 22:25:46 Øivind wrote:
>> I am struggeling to get around the cycle detection kicking in
>> when I have static init in modules that depend on eachother.
>> 
>> I have seen some threads on 'fixes' for this, e.g. adding a
>> @standalone property to the module or similar. Has there been any
>> progress on this?
>
> The solution is to do something like what std.stdio does. Instead of having a
> static constructor, it has an extern(C) function which does the same, and then
> it has std/stdiobase.d which has a static constructor which calls that
> extern(C) function in std.stdio. As long as you don't actually have any inter-
> dependencies, you can make it work.
>
> Walter has refused suggestions which involve telling the compiler that there
> isn't actually dependency (via an attribute or whatnot) on the grounds that
> you're throwing away the compiler's checks, and it's too easy to screw up. In
> general, you can refactor things so that you don't have the circular
> dependency anymore, and if you can't, and the static constructors aren't
> actually forming a circular dependency, then you can pull the same trick that
> std.stdio does.
>
>> If not would it be possible in e.g. main() to get a list of all
>> compiled-in modules, and then iterate over them and call an init
>> function where it exists? As long as there is a way to list the
>> name of the modules at compile-time, this should be pretty easy..?
>
> It will never be possible to determine the full list of modules at compile
> time due to C's linking model. For instance, you could link together modules
> which were built years apart. There's no way that they could know about each
> other. And since you could be using .di files, and the circular dependencies
> could actually be really indirect, the compiler can't possibly have all of the
> information that it needs to determine circular dependencies. A linker could
> do it but not as long as we use standard, C linkers. And that's not going to
> change.
>
> So, the situation sucks, but there _are_ workarounds. The main one of course
> is to simply limit how much your modules import each other so that the few
> places that you need static constructors don't import each other even
> indirectly, but regardless, it can be worked around as long there isn't a true
> circular dependency.
>
> - Jonathan M Davis

Thanks for the explination. The trick you talk about has worked for me before, but now I really need static init of modules that depend on eachother. Only solution I can think of is to list all modules in a script and generate d code from this to call init functions.. Then pass it to dmd along with the other code.

Sad that we can't get this to work like it really should. D is a very flexible language in many ways, and it seems strange that it should not allow the stuff I want to do here..

-Øivind

On Thursday, September 20, 2012 20:49:32 Øivind wrote:
> Thanks for the explination. The trick you talk about has worked for me before, but now I really need static init of modules that depend on eachother. Only solution I can think of is to list all modules in a script and generate d code from this to call init functions.. Then pass it to dmd along with the other code.
> 
> Sad that we can't get this to work like it really should. D is a very flexible language in many ways, and it seems strange that it should not allow the stuff I want to do here..

If you have a true circular dependency, then you're just plain screwed and need to redesign your code. That's fundamental and _can't_ be fixed.

Off the top of my head, the only reason that you wouldn't be able to use the trick that std.stdio uses but not have a true circular dependency is if your initializing const or immutable variables in a static constructor. But then the solution is to refactor your code so that you don't have the circular dependency, even if it's just moving those variables into a separate module, which is generally quite feasible, even if it's not necessarily quite what you wanted.

- Jonathan M Davis

On Thu, 20 Sep 2012, Jonathan M Davis wrote:

> On Thursday, September 20, 2012 20:49:32 ?ivind wrote:
> > Thanks for the explination. The trick you talk about has worked for me before, but now I really need static init of modules that depend on eachother. Only solution I can think of is to list all modules in a script and generate d code from this to call init functions.. Then pass it to dmd along with the other code.
> > 
> > Sad that we can't get this to work like it really should. D is a very flexible language in many ways, and it seems strange that it should not allow the stuff I want to do here..
> 
> If you have a true circular dependency, then you're just plain screwed and need to redesign your code. That's fundamental and _can't_ be fixed.
> 
> Off the top of my head, the only reason that you wouldn't be able to use the trick that std.stdio uses but not have a true circular dependency is if your initializing const or immutable variables in a static constructor. But then the solution is to refactor your code so that you don't have the circular dependency, even if it's just moving those variables into a separate module, which is generally quite feasible, even if it's not necessarily quite what you wanted.
> 
> - Jonathan M Davis
> 

I'm not picking on Jonathan's response, just needed an entry point to the thread...

Static initializers are overused in D, particularly phobos and druntime, imho.  They're a particular problem when it comes to use in library code. Mixing static and dynamic linkage along with initialization == things break in wierd ways.  I'd _much_ rather see them moved into explicit api calls or removed entirely where possible.

Every static initializtion is unavoidable (by users of the library) overhead.

My 2 cents,
Brad

On Wed, 19 Sep 2012 16:25:46 -0400, Øivind <oivind.loe@gmail.com> wrote:

> I am struggeling to get around the cycle detection kicking in when I have static init in modules that depend on eachother.
>
> I have seen some threads on 'fixes' for this, e.g. adding a @standalone property to the module or similar. Has there been any progress on this?

No.  If two initializations truly don't depend on each other, you can initialize in their own module.

The issue is that you have to move *all* the initializations into their own module.  The compiler doesn't keep track of complex dependencies, it's all on a module-level.

I redesigned the static cycle detection code a while back (it was broken (allowed certain cycles) and much less informative), and I can say it was a very difficult problem to solve.

> If not would it be possible in e.g. main() to get a list of all compiled-in modules, and then iterate over them and call an init function where it exists? As long as there is a way to list the name of the modules at compile-time, this should be pretty easy..?

No, you have no idea at compile-time what other modules will be linked in.

You need to do the cycle detection at link-time or run-time, and as long as we use standard C linkers, we cannot add that feature.

-Steve